A thought-experiment:

Ms. Wong is a fourth-grade teacher in Shanghai, and Ms. Stilton is a fourth-grade teacher in Ames, Iowa. Each has a class of 25 students, who on average score at the same level on international assessments of mathematics and proficiency in their native language. In both classrooms, the prescribed curriculum involves study of the history of the local community.

Combat at Guangzhou During the Second Opium War

At the end of the school year, both Ms. Wong’s students and Ms. Stilton’s students are administered a standardized test of their knowledge of Shanghai history, touching on the Qing Dynasty, the Opium Wars, World War II and the Japanese occupation, and the transition to Communism.

Ms. Wong’s students show a commendable grasp of the key historical events in Shanghai over the past two centuries. Ms. Stilton’s students get every question wrong.

The students started at the same point, on average, but by the end of the year Ms. Wong’s students dramatically outperform Ms. Stilton’s students on the standardized test. The results are entered into a value-added model, which determines that Ms. Wong is a highly effective teacher, while Ms. Stilton is ineffective.

Preposterous! How can we conclude that Ms. Wong is a more effective teacher than Ms. Stilton? They were teaching a different curriculum! Ms. Wong’s curriculum happened to be aligned with the end-of-year assessment, whereas Ms. Stilton’s was not. How can we conclude that one teacher is more effective than another if they are teaching different curricula?

Okay—admittedly, that’s an extreme example. Let’s try another one:

Ms. Collins is a fifth-grade teacher in a rural school district located in a state implementing the new Common Core standards. Mr. Brooks is a fifth-grade teacher in an urban district in the same state. Both have a class of 28 students whose average scores on the state’s English Language Arts and mathematics assessments the previous spring were identical. This year, the state is administering a new set of assessments that are aligned with the Common Core standards in reading and math.

In Mr. Brooks’ district, teachers have received extensive professional development and support on teaching to the new Common Core standards, and have been provided with new textbooks and curricular materials that are aligned with those standards. But in Ms. Collins’ district, teachers have not received any support on Common Core implementation, and new textbooks aligned with those standards will not be purchased for another three years.

In the spring, the students in both Ms. Collins’ class and Mr. Brooks’ class take the new state assessment aligned with the Common Core standards. Mr. Brooks’ students do very well on the assessment, while Ms. Collins’ students perform dismally. The results are entered into a value-added model, which determines that Mr. Brooks is a highly effective teacher, whereas Ms. Collins is ineffective.

Preposterous? Don’t bet on it. It’ll probably happen this year, in a state near you. As states begin aligning their assessments with the Common Core standards—which are, by all accounts, more challenging than the existing standards in much of the country—there is a high probability of uneven implementation of curriculum, professional development, and other supports within those states.

If some districts are using an older curriculum not aligned with the new standards and assessments, while others are using a newer curriculum that is aligned, then there’s a risk that differences in student performance on the new assessments will be improperly attributed to differences in the quality of the students’ teachers, rather than differences in the curriculum to which students were exposed. That’s the inference that would be drawn from a value-added model that doesn’t take into account variations in curriculum.

And value-added models rarely, if ever, do so. In some cases, it’s a reasonable assumption that curriculum is constant across classrooms in a district or a state; everyone knows the rules of the game, and has had ample time to develop curricula that are appropriate for the district’s or state’s expectations for what students should know and be able to do. But even in this scenario, schools and districts differ in the resources that they can devote to purchasing or developing curriculum, and curricular variation is ignored.

The claim that teacher quality is the most important school-based factor influencing student achievement rests on statistical models that fail to take into account variations across teachers in the curriculum they teach. And if we never look for curricular variation, we’ll never be able to assess how much it matters in relation to differences among teachers in their effects on student performance.

So the next time someone—a policy advocate, a district or state superintendent, a, I don’t know, secretary of education—says, “You know, research shows that teacher quality is the single most influential school-based factor affecting student achievement,” feel free to reply, “You know, curriculum is a school-based factor. And I’d bet the research you’re citing does not take curricular variations across schools and teachers into account.”

And then ask, “By the way, why did the British and the Chinese fight the Opium Wars?”