Since standardized tests are so important, why aren't they fair, valid and reliable?

From the Diane Ravitch blog, by Diane

Many people have wondered how the New York State Education Department permitted the nonsensical story about the pineapple and the hare to get onto the state test.

This is not the first time a really bad reading passage got onto the test and it won’t be the last.

State Commissioner John King was quick to issue a defensive statement saying that people were reading the story “out of context,” as if the full story made sense (it didn’t). And he was quick to pin the blame on teachers, who supposedly had reviewed all the test items. It was the teachers’ fault, not his. In an era where Accountability is the hallmark of education policy, King was quick to refuse any accountability for what happened on his watch. These days, the ones at the top never accept accountability for what goes wrong, that’s for the “little people” like teachers and students, not for the bigwigs. No one holds them accountable, and they never accept any. None of them ever says, as President Harry S Truman did, “the buck stops here.”

So this is the reason that even a stupid, pointless story like the pineapple story–so thoroughly bowdlerized that it was disowned by Daniel Pinkwater, its original author–got past the review panel. I know about this process because I spent seven years as a member of the National Assessment Governing Board and served on a committee where we read every single question that would appear on a national test. When the review committee gets the items, with questions and answers, you are told that this particular item has been thoroughly field-tested. It has appeared in a children’s magazine; it has been used in a state assessment. Here are the results with all the accompanying statistics for this item. You are also told that the publisher’s own technical reviewers approved the item; so did the publisher’s bias and sensitivity reviewers.

By the time the item reaches the teachers or external panel, it has been vetted, you are told, by many others. There is tremendous implicit pressure to go along with the judgment of others whom you assume are very professional. They all agreed it was fine. Who are you to raise a question or complaint?

Since I am by nature a skeptic, I always read test passage and their questions and answers as if no one else had. And on more occasions than I can count, I said, “Stop. Wait. This doesn’t make sense. The question isn’t clear. None of the answers fits the question. There are two good answers,” or words to that effect.

But I understand the social pressure, the social consensus, that discourages questioning and criticism.

And that is how bad questions get onto standardized tests, and why the Pineapple question was not the first and will certainly not be the last to slip past the review panels.

The best remedy for this problem is to publish the questions and answers when the tests are finished. That way, everyone can see them. After all, as Mayor Bloomberg and Governor Cuomo and Secretary Duncan often remind us, when speaking of teacher evaluation ratings, “The public has a right to know.”

Since the tests are the linchpin of every national education policy today, the public has a right to know if the tests are fair, valid, reliable and reasonable ways to assess student learning.

http://dianeravitch.net/2012/04/24/how-stupid-items-get-onto-standardized-tests/