From the Miami Herald
By Laura Isensee and Sarah Butrymowicz
lisensee@MiamiHerald.com
At Coral Reef Senior High, calculus teacher Orlando Sarduy understands complicated formulas, and knows he will be graded on how his students perform on tests.
But despite his advanced knowledge of math, Sarduy cannot explain the statistics-packed formula behind the grade he’ll get.
It is so confusing that even a member of the state committee tasked with developing it abstained from a vote because she didn’t understand it.
The formula—in what is called a “value-added” model—tries to determine a teacher’s effect on a student’s FCAT performance by predicting what that student should score in a given year, and then rating the teacher on whether the student hits, misses or surpasses the mark.
But Sarduy, like thousands of other Florida teachers, doesn’t even teach a subject assessed by the FCAT. So his value-added score will not come from his math teaching or his particular students. Instead, it will be tied to the FCAT reading score of his entire school in South Dade—a notion that infuriates him, even though he appreciates the level of objectivity the new system brings, and the ways it strives to isolate a teacher’s impact on student learning.
Florida is among 25 states that have turned to student scores on standardized exams to help evaluate teachers and set their pay. By 2014, it will become mandatory to do so under a new state law. The model will initially use results on the FCAT, which has gotten tougher, and will expand to include other tests that are being developed in every subject at every grade level.
Florida’s revamped teacher-evaluation system is part of the education reform agenda pushed by the Obama Administration, which is giving states $4.3 billion in its Race to the Top grant program to come up with new ways to grade teachers and tie student performance to their paychecks.
In Florida, the stakes are high. Top-performing teachers can get permanent salary increases, while those with ratings near the bottom for two consecutive years can be let go.
“It’s interesting, but at the same time you have to realize, ‘That’s me in the line. I’m now part of a statewide experiment, and if the experiment doesn’t work out, am I excluded, am I excused, am I fired?’ That’s the concern,” said Sarduy, the calculus teacher.
In the past, teachers were evaluated by their principals alone. The result was that most were rated the same: proficient. And state-issued school grades come mainly from school FCAT scores—which don’t recognize individual teachers’ impact, said state Rep. Erik Fresen, the Miami Republican who helped pass the controversial law, known as SB 736.
“All we were looking at as a state and as districts was what the school did, not what the individual teacher did. This changes that paradigm completely,” he said.
In the new evaluations, half a teacher’s “grade” will be based on the new value-added formula, and half on the principal’s observations. Teachers who don’t instruct FCAT subjects will get grades based on the school’s FCAT reading performance.
The new system faces many challenges and much criticism.
• No research has shown that the value-added approach to teacher evaluations improves student learning, but there is research to suggest that some models yield unreliable results. For many teachers, adequate data points to plug into the formula aren’t available; about 60 percent of Florida teachers work in subjects not presently tested by the FCAT, which only covers reading, science and some math. And the test itself was designed to measure the performance of students, not teachers.
• The state teachers union is challenging the law in court, arguing that it takes away teachers’ right to bargain for their pay and working conditions.
• Miami-Dade’s new evaluation and merit pay system faces a separate challenge, that the district is exceeding the basic state requirements.
• Florida, facing an ever-shrinking education budget, has not figured out how to pay higher salaries to high-performing teachers.
“We can create this elaborate modeling system so we can gather performance data, but the big question is, ‘Where’s the money?’ ” said Lisa Maxwell, executive director of the Broward Principals and Assistant Principals Association. She served on the state committee that helped develop the model.
“Are these going to be $10 bonuses or $1,000 bonuses? All of that has yet to be determined,” she said.
Success will hinge on the data and the tests developed, said Enid Weisman, an assistant superintendent at Miami-Dade County Public Schools, the nation’s fourth-largest district.
“The better the data is, the more successful this will be,” she said. “It’s a challenging concept. We’re not selling widgets, and we’re not selling cars, and we’re not selling pharmaceuticals. We’re dealing with uniquely individual students.”
Even the biggest national supporters of value-added evaluations concede to caveats: Sufficient data exist for only about 20 percent of teachers nationwide to be given value-added scores. And questions abound about the accuracy and reliability of standardized tests like the FCAT.
“We don’t have evidence that this approach is going to improve teaching and learning,” said Douglas Harris, an expert on value-added modeling at the University of Wisconsin-Madison and author of the 2011 book Value-Added Measures in Education: What Every Educator Needs to Know.
Harris called Florida’s decision to use school-wide reading scores for individual teachers “backwards.”
CRAFTING THE FORMULA
Using nearly $4 million from its Race to the Top dollars, Florida contracted with the American Institutes for Research in Washington, D.C., to develop the complex formula.
“It’s not simply a matter about what they scored last year and did they improve,” said Juan Copa, director of research, evaluation and educator performance at the state’s Department of Education. “What you’re trying to do is isolate the impact of the teacher on the student’s learning.”
A committee of administrators, teachers, union leaders and parents from around the state met to decide which variables to include in the formula. Copa said they decided on a more complex model to gain more reliability and accuracy.
To try to isolate the teacher effect, the model weighs 10 factors, like student attendance and disability status.
Each factor has a different weight for each subject and grade level, explained Jon Cohen, executive vice president and program director for assessment at the American Institutes for Research.
The strongest factor in predicting a student’s future score—explaining about 60 percent of the variance in achievement—is that student’s prior test score. “If I know how you did on the math test last year and you did well, we’d expect you to do well this year,” Cohen said.
POVERTY LEFT OUT
The formula does not take into account a student’s race, gender or socioeconomic status—despite evidence that such characteristics are linked to student achievement.
Florida legislators opted to leave poverty out of the equation.
“They said every child should be able to learn, regardless of their poverty level,” said Gisela Feild, Miami-Dade’s administrative director for assessment, research and data analysis. Otherwise, it would have been good to include, Feild said. “We all know that poverty is a big indicator.”
In Miami-Dade, just over 60 percent of the 343,500 public-school students qualify for a free lunch in the National School Lunch program. In Broward, half of the 234,000 students are eligible.
Cohen said that if poverty influences a student’s score one year, it likely impacted the previous year’s score, too. “It’s pretty much embedded in the child’s scores when you control for prior achievement.”
Maxwell, who served on the committee, worries that with no weight given to schools in low-income neighborhoods, teachers may leave for charter or private schools.
“I don’t think you want to have a system that creates enclaves, where the best teachers leave a school based on that student population having poor performance,” Maxwell said.
RELIABLE AND COMPLEX
Karen Aronowitz, president of United Teachers of Dade, said the value-added formula is not a fair instrument and discourages collaboration among teachers. “It’s the wrong model and people know it.”
Parents also worry about how their children’s teachers will be evaluated. Jeanne Jusevic, a parent activist in Broward County, said, “We’re taking something that should be a diagnostic tool to let the school know where the kids are, and we’re making it the end-all, be-all.”
In one study, Harris and other researchers asked principals to rate specific teachers. They found that the results generally corresponded with those teachers’ value-added scores. But in several cases, a teacher deemed wonderful by a principal fared poorly when measured by student test-scores.
In another study, the error rate was calculated to be 25 percent, based on three years of data, according to a 2010 study by Mathematica Policy Research—meaning the model would incorrectly rate 1 in 4 teachers. And with one year of data, the error rate rose to 35 percent.
Harris points out that in Florida’s system—which will label teachers as “highly effective,” “effective,” “in need of improvement” or “unsatisfactory”—minor differences of just one or two points could separate the teachers classified as effective from their colleagues deemed in need of improvement.
William Sanders, the statistician considered to be the “grandfather” of the value-added model in education, said this approach can target that extra professional boost to teachers who are rated poorly early in their careers, but who have the potential to improve. A study by SAS Research, the software firm where Sanders now works, found that half of the teachers who scored poorly after two years improved after five more years.
In terms of the actual formula, Sanders maintains that complexity is a good thing. He said any formula must include multiple years of student data to be reliable.
“A lot of people are arguing the methodology used must be so transparent that a teacher can go to their dining room table with a $2 calculator and scratch pad and calculate her value-added,” Sanders said. “What I say to schools and districts is: To trade simplicity of calculation for reliability is a devil’s bargain.”
FORECASTING THE FUTURE
How will this new data-driven approach play out for teachers and their careers—and for students in the classrooms? Critics and proponents are sharply divided.
Officials with the state DOE say it will help teachers—and in turn, their students—improve.
“With value-added, we’re trying to give better information to teachers about their student performance so they can improve their instruction,” said Kathy Hebda, the state’s deputy chancellor for educator quality.
Rep. Fresen envisions the new system giving incentives to teachers to move from high-performing schools to struggling ones where their students can make bigger gains and they can earn more money.
Critics contend that the opposite will happen: Teachers will not want to teach at low-performing schools or teach students who are not poised to score well on standardized exams because the stakes are too high or the system is unreliable.
Many excellent teachers will get poor ratings, and many mediocre teachers (who are good at drilling) will get high scores,” Diane Ravitch, a former U.S. Assistant Secretary of Education and fierce critic of tying teacher pay to student test scores, wrote in an email to The Miami Herald.
Ravitch said that placing so much emphasis on tests will lead to teaching to the test, gaming the system and cheating, and the new system will be a “massive waste of money.”
Children will feel extra pressure, too, according to some experts and parents critical of standardized tests.
“Depending on the age and level of students, they may not realize that their test score has a connection to a teacher’s salary, but they are going to feel the effects indirectly in their schooling,” said Elisabeth Cramer, an education professor at Florida International University who has studied the effects of testing on students.
“Now that that there is going to be added emphasis from the teacher’s perspective of performing well on these exams, students are going to feel that extra pressure.”
This report is a collaboration of The Miami Herald and The Hechinger Report, a nonprofit news organization dedicated to in-depth education journalism. The Hechinger Report is an independently funded unit of Teachers College at Columbia University.
Miami Herald staff writer Laura Figueroa also contributed.
Read more here: http://www.miamiherald.com/2011/11/05/v-fullstory/2488961/complex-new-teacher-evaluations.html#storylink=cpy
No comments:
Post a Comment