If my email is any indication, teachers across New Jersey are just now being introduced to our new, top-down, state-mandated evaluation system, AchieveNJ. There's a lot of concern about this system - as well there should be. AchieveNJ has many serious flaws that I've documented over the last two weeks: see here, here, here, here, here, here, and here.

But none of these seem to concern Commissioner of Education Chris Cerf; if anything, he seems more confident than ever in AchieveNJ's infallibility:

Students in New Jersey are going to be placed in peer groups with students who scored similarly on the previous year’s test. They will get a score based on how much better or worse they do than others in their peer group.

“You are looking at the progress students make and that fully takes into account socio-economic status,” Cerf said. “By focusing on the starting point, it equalizes for things like special education and poverty and so on.” [emphasis mine]

According to Cerf, AchieveNJ doesn't just take socio-economic status into account: it takes SES "fully" into account. Notice the certainty in Cerf's tone; he is completely confident that the research backs up his position.

Alas - once again - the facts are not on Cerf's side:

There is substantial evidence, based on previous uses of test-based teacher evaluations, that teachers who educate students in poverty will pay a penalty when judged by AchieveNJ.

Everyone who cares about education in New Jersey - arguably, the highest-performing state in the nation when accounting for student characteristics - should be gravely concerned that Commissioner Cerf is selling his teacher evaluation plan under demonstrably false pretenses:

The Education Commissioner argues students who come from low-income areas will likely score similarly on the previous year's test, so he says they will likely end up in the same peer group where they are only compared to others like them. Same with students with disabilities, or students who come from high-income areas.

“The assumption he’s making is that the initial score embeds all of the background disadvantage of the student, therefore there’s no reason to use other measures for accounting for that,” said Bruce Baker, a professor st the Graduate School of Education at Rutgers University. 

Baker studies teacher evaluation models and argues the state’s system doesn’t accurately isolate the role a teacher has on learning. [emphasis mine]

Well, who's right? Is there a bias in a teacher's evaluation if that teacher has a disproportionate number of disadvantaged students? Has there been a previous use of SGPs that can be studied to determine if they are biased against students - and, therefore, their teachers - who are in economic distress?

Luckily for us - and apparently unknown to Commissioner Cerf - we know that there is.

AchieveNJ uses a method called Student Growth Percentiles (SGPs) to determine a teacher's "effectiveness" as evinced in state test scores. The method is fatally flawed to begin with: even it's "inventor" says it can't determine why a student has a particular level of "growth," which makes it inappropriate for use in teacher evaluation. But let's put this enormous flaw aside; have any other states used SGPs to evaluate teachers, and did they find a bias against students in need and, consequently, their teachers?

It turns out that just last year, New York collected data for millions of students on the results of state-level standardized tests. The state then calculated evaluations for thousands of teachers across the state. based on the SGPs of their students. The NY "growth model" was then analyzed by the American Institutes for Research and presented in a final Technical Report that was released last fall. Their conclusions (p.33)?

Table 11 indicates that there is a moderate correlation with the percent SWD [Students With Disabilities] and percent economically disadvantaged in a class. This correlation is negative, indicating that as the percent SWD or percent economically disadvantaged in a class increases, the MGP [median growth percentile] tends to decrease. However, teachers with high and low concentrations of SWD and economically disadvantagedstudents are still able to achieve MGPs across the distribution. [emphasis mine]

Translation: if a teacher has more children with disabilities or who are economically disadvantaged in his or her class, that class's average Student Growth Percentile will go down. This stands in direct contradiction to Chris Cerf's assertion that SGPs "fully take into account socio-economic status."

But hold on! Some of you might say: "Hey, Jazzman, it says there's a "moderate" correlation! What's the big deal?" Well, when you're dealing with forced decisions based on test scores, it is a big deal: again, some of the evaluation, all of the decision.

But aside from that, there's good reason to believe that AchieveNJ has a worse bias against economic disadvantage than the NY model. Because the New York growth model at least tries to account for the disadvantage that students in economic distress suffer on state tests (p. 14).

NYSED’s regulations permit three specific control variables at the individual student level for inclusion in the model designed to produce adjusted scores through 2011–2012, without additional classroom- or school-level variables. These variables, which are listed below, were selected after consultation with the Regents Task Force.. Additional variables may be included in a value-added model in future years, including classroom- or school-level variables that may reflect the context in which learning occurs. These are the student-level predictor variables: 

• ELL status: A Y/N variable was provided to indicate ELL status. 

• Students with disabilities (SWD) status: A Y/N variable was provided to indicate SWD status. 

• Poverty or economic disadvantage (ED): A Y/N variable was provided reflecting New York State’s rules related to family income levels and participation in economic support programs. A description is provided in Appendix E.

So the New York growth model at least attempts to account for the differences in test score outcomes, based on whether a child speaks English at home, whether they have a disability, and whether they are in poverty. Let's be very clear about this: the NY growth model fails - but at least they tried. It's reasonable to assume there is some mitigation of the effects of poverty in the teacher evaluation scores as reported by MSGPs.

But - and this is critical to understand - AchieveNJ doesn't even make the attempt to correct SGPs for poverty! 

In other words: the NY model attempts to account for students who are in economic distress, and tries to correct for the disadvantage they have on state tests, so that teachers who have these children in their classes are not unfairly penalized. However, even after this attempt, AIR found there is still a bias against these teachers. But AchieveNJ doesn't even try to account for students' socio-economic status.

It is, therefore, reasonable to assume that the bias against New Jersey's teachers who take on the arduous task of educating kids who are in poverty in will be even greater than the bias that has already been demonstrated in New York!

But hold on - it gets even worse! Because the New York model only disaggregates children in "economic disadvantage" by a "yes/no" variable. There's no accounting for different levels of poverty. Why does this matter?

As Bruce Baker has demonstrated, there is (at the school level at least) a substantial difference in the test score outcomes of children who are "less poor among the poor," and those who are in "deep" poverty. The New York model doesn't account for this; it treats all children below a certain threshold as equally disadvantaged. It is, therefore, reasonable to assume that the teachers of children who are in "deep" poverty will pay an even higher price on their evaluations for taking on this task.

So AIR's evaluation of New York State's growth model stands in direct contradiction to Commissioner Cerf's claim: SGPs are biased against teachers who educate our poorest children. And the penalty is most likely worse for New Jersey's teachers than it was for New York's. The evidence is clear and unambiguous.

And yet here stands Commissioner of Education Chris Cerf, absolutely certain in his righteousness, undeterred by the facts and past practices. He states, with the utmost confidence, that AchieveNJ "fully takes into account socio-economic status."

Teachers and parents of New Jersey: don't believe the hype. Your Commissioner of Education is flat out wrong:the teachers who educate poor children will be penalized under AchieveNJ.

Does everyone understand how incredibly irresponsible this is? Do Chris Cerf and the NJDOE even care?

Accountability begins at home.