Reforming education by committee rather than evidence

From AJC.com

by Maureen Downey

Statistician Howard Wainer doubts the salvation of public education will come from blue-ribbon commissions, a popular strategy in Georgia in which dense reports on how to fix schools stack higher than the Gold Dome. (As we discussed recently, the state is taking another swipe at funding reform, assembling its sixth commission to tackle the challenge.)

“If you try to change a very complicated system — and a school system is very complicated — the worst way is to appoint a blue-ribbon panel with a name like ‘Education 2030’ and ask them to come up with a plan to improve things,” Wainer said. “That is not going to work because we are not that smart.”

In an interview last week and in his new book “Uneducated Guesses” (Princeton, $24.95), Wainer maintains that education ought to look to manufacturing. Using paper-making as an example, Wainer said, “You might vary temperature a bit or you vary acidity by a little bit to see if it improves the quality of the paper. If it does, vary it some more in the same direction. If it makes things worse, retrace your steps and try something else. You should be in a constant state of experimentation all the time, seeing what makes things better. But you must make incremental changes so if it doesn’t work, it doesn’t kill you.”

Too often, schools blunder into change by mistaking anecdote for evidence, Wainer said. Tired of yelling at the TV when he saw news accounts of policy changes based on flawed evidence, Wainer uses his book to present evidence to help assess 11 such trends, including the entrance-exam-optional policies in many colleges and teacher evaluations based on student performance.

Wainer, who holds a doctorate in psychometrics from Princeton and lives near the university in New Jersey, was principal research scientist at the Educational Testing Service for 21 years and is now Distinguished Research Scientist at the National Board of Medical Examiners and an adjunct professor of statistics at the Wharton School of the University of Pennsylvania.

To test the growing assumption that entrance exams are not a quality predictor of student performance, Wainer reviewed the SAT scores of students who opted not to submit their scores when they applied to Bowdoin, a premier liberal arts college in Maine that has made SAT scores optional.

Wainer found that Bowdoin students who took the SAT but chose not to submit scores posted lower scores than their peers who did submit them. The mean score of students who submitted their scores was 1,323 out of 1,600, while nonsubmitters had a mean of 1,201.

Wainer went a step further to see how well these students fared in college. His finding: Students who didn’t submit SAT scores earned grades 0.2 points lower (on a four-point scale) in their first year than classmates who did submit their scores. Their poorer performance at Bowdoin was well-predicted by their SAT scores.

Wainer understands that most people won’t see grave concern, for instance, in a 3.0 grade-point average vs. a 2.8 (although Georgia students know that it’s a critical distinction for the HOPE scholarship).

But he isn’t arguing that an SAT score ought to trump a high school transcript. “What impresses me is that a two-and-a-half-hour test predicts performance in college about as well as four years of high school grades,” he said.

(Because college rankings incorporate the SAT scores of admitted students, Wainer points out that an optional SAT policy can enable a campus to climb in the rankings because lower-scoring students are less apt to submit their scores.)

While he has played a key role in the testing industry, Wainer said he’s not trying to bolster the College Board, which administers the SAT. In fact, he concludes in another chapter of his book that the national push to get more students enrolled in demanding AP math and science courses, also overseen by the College Board, is misbegotten.

“Someone asked me which side am I on,” he said. “I am on the side of data. What I hope people will do, when confronted with policy, is ask what’s the evidence.”

And Wainer said the evidence isn’t there yet on one of the most controversial new policies in education, basing teacher evaluations and pay on how much “value” they add to student learning as reflected in test scores.

Acknowledging that he goes “deep in the weeds” on the defects of value-added models, Wainer said, “It appears, at least at the moment, that the more you know about value-added models, the less faith you have in the value of inferences drawn from them.” He urges caution in adopting such models.

Wainer applies more than statistical evidence to education policy; he also brings common sense to bear. He dismisses attempts to compare U.S. schools with the idealized country du jour, saying, “You can take a Swedish model but anything works for Sweden because Sweden is full of Swedes. There are very few countries that have our diversity, our serious diversity, not skin color or hair curl, but the diversity of opinion, of background, that is in effect here.”

What Americans have to accept is that education, not to be confused with schooling, is neither cheap nor easy. “Schooling is six hours a day, 30 weeks a year. Education takes places in the home, in the church, in the community, all the time,” he said.

When Wainer served on the Princeton school board, parents asked him how they could help their children do better in school. He told them, “Turn off the television and read with them.”

–From Maureen Downey, for the AJC Get Schooled blog

http://blogs.ajc.com/get-schooled-blog/2011/09/05/uneducated-guesses-reforming-education-by-committee-rather-than-evidence/