The FCAT is an ever moving target

From the Sun Sentinel

by Allison Ross

Florida's writing FCAT keeps undergoing changes. In the past seven years, the exam has had five major makeovers.

In 2006, state education officials stitched on a multiple-choice section, only to lop it off in 2009.

In 2010, they put it back to 3; in 2011, they boosted it to 4.

Despite those changes, the writing Florida Comprehensive Assessment Test will look different again this year.

And this year's changes will not be the last, with the state moving to adopt new national Common Core State Standards. By 2014-15, all students in Florida will be taking new tests under these core standards instead of the current math and English exams.

The saga of Florida's writing FCAT exemplifies the transient targets set by Florida's high-stakes standardized tests. The multitude of changes — made to nearly every major state assessment — has at times made valid year-to-year comparisons nearly impossible.

"It's hard to make informed decisions about how you're doing when you're changing the criteria for scoring," said Marc Baron, the Palm Beach County School District's chief of performance accountability.

For instance, in 2010 the state had a record number of A-rated and B-rated high schools. In Palm Beach County, 86 percent of high schools earned A's or B's, compared with 52 percent the previous year.

But later, an analysis by The Palm Beach Post found that the year-to-year jump was the result of state changes in how high schools were graded, not because the students at those schools did considerably better. The Post's analysis discovered that, had the state used the previous year's grading formula, most of that jump would not have happened.

"With an ever-changing test, just like ever-changing criteria for school grades, the year-to-year trends become less meaningful," said Robert Schaeffer, public education director for FairTest: The National Center for Fair and Open Testing in Massachusetts. "It becomes very hard to do trend analysis because the test has changed so much over time."

Florida's grading scale has changed most years since the test was put in place in the late 1990s, said Sherman Dorn, a professor of education at the University of South Florida. That's not a rare occurrence for high-stakes tests nationwide, he added.

"But are there problems with comparability?" he said. "Yeah. In some ways, it's the nature of the beast."

Dorn and others who study testing say that assessment systems usually change over time in an effort to make them more rigorous or comprehensive — or sometimes because of shifting political pressures, including budget concerns.

Before Florida's high school grading formula changed, Dorn said, "superintendents had been griping for years that their high schools couldn't get good grades." He suggested that the formula change was due at least in part to political pressure.

Kris Ellington, assistant deputy commissioner of the Florida Department of Education, said changes to the state's standardized tests and school-scoring system are intended to improve them and to help students become more college- and career-ready.

The state works hard to create ways to compare tests year to year despite changes, Ellington said.

"There are measurements, procedures for linking two tests," Ellington said. "While it's not as clean as having the same test year after year, there are ways to connect and continue to analyze our progress on student learning."

For instance, when the state switched some of its FCATs to a new, harder FCAT 2.0 last year, it reported the scores of the new tests on the old FCAT scale as FCAT-equivalent scores. The reason was to make the scores more comparable year over year.

But it wasn't a completely clean transition. The state warned at the time that comparisons between scores on the FCAT 2.0 and the previous year's FCATs "should be made with caution."

There were similar issues in 2010, when the state made changes to the writing FCAT because of budget cuts. Then-Education Commissioner Eric Smith called it a "stand-alone" year for the test because the changes had rendered comparisons to previous years difficult.

The Palm Beach County School District's accountability office has had many discussions about how to look at achievement and growth when the scoring or the test changes from year to year, Baron said. In the end, though, it's the final score that matters for schools and for students, he said.

Schools that receive an A or improve at least one letter grade from the previous year are eligible to earn an additional $70 per student, while schools that do poorly are subject to sanctions and state intervention.

"If you're keeping track of the number of A's, the number of B's, the number of C's, that's not comparable from year to year," Baron said. "But the rules are the rules, whether you footnote them or not.

http://www.sun-sentinel.com/news/palm-beach/pb-fcat-writing-changes-20110905,0,1141795.story