Research analysis at Shanker Blog is among the very best available online, notably the work of Matthew Di Carlo.

The posts there are predictably nuanced and careful, dispassionate—to a fault. As a critical educator, I am on edge when I read these careful explications of educational research because they tend to stand so far back from drawing critical conclusions that they leave a great deal of room for forgiving awful and baseless policy.

Teacher Evaluations And Turnover In Houston is an extremely important post as it unpacks new research on the current era of teacher evaluations spawned during the Obama years, notably the increased use of value-added methods (VAM) that link teacher quality to student test scores.

I highly recommend reading the post in full, but here I want to add a few annotations to address both my concerns the analysis tip-toes when it should stomp and to emphasize a few key takeaways well addressed by Di Carlo.

Let me share a few passages, and I will boldface what I want to address; first, the opening:

We are now entering a time period in which we might start to see a lot of studies released about the impact of new teacher evaluations. This incredibly rapid policy shift, perhaps the centerpiece of the Obama Administration’s education efforts, was sold based on illustrations of the importance of teacher quality.

The basic argument was that teacher effectiveness is perhaps the most important factor under schools’ control, and the best way to improve that effectiveness was to identify and remove ineffective teachers via new teacher evaluations. Without question, there was a logic to this approach, but dismissing or compelling the exits of low performing teachers does not occur in a vacuum. Even if a given policy causes more low performers to exit, the effects of this shift can be attenuated by turnover among higher performers, not to mention other important factors, such as the quality of applicants (Adnot et al. 2016).

To address incredibly flawed educational policy, I believe we must be much more careful about distinguishing between political/public claims and then how the research community poses the same issues.

As Adam Bessie has outlined, the “bad” teacher myth was never “sold” in the ways Di Carlo notes above. The film Waiting for Superman is a powerful example of how political and public discourse about “bad” teachers was primarily an argument that teacher quality was the singular or most important factor in student learning, period; politicians and the public almost never added the caveat “most important in-school factor.”

And we must acknowledge that the “bad” teacher movement driving new teacher evaluations including VAM was significantly grounded in anti-union sentiment and union-busting objectives—not about teacher quality or student learning.

The nuanced argument about teacher quality, in fact, was most often expressed among some researchers, while mostly absent from the mediaor political discourse, such as Di Carlo (from 2010):

But in the big picture, roughly 60 percent of achievement outcomes is explained by student and family background characteristics (most are unobserved, but likely pertain to income/poverty). Observable and unobservable schooling factors explain roughly 20 percent, most of this (10-15 percent) being teacher effects. The rest of the variation (about 20 percent) is unexplained (error). In other words, though precise estimates vary, the preponderance of evidence shows that achievement differences between students are overwhelmingly attributable to factors outside of schools and classrooms (see Hanushek et al. 1998; Rockoff 2003; Goldhaber et al. 1999; Rowan et al. 2002; Nye et al. 2004).

Next, further into the recent post:

Prior to ETI, there was a negative relationship between teacher effectiveness and exits – i.e., less effective teachers were more likely to exit than their more effective colleagues, with effectiveness here defined in terms of validated measures of teachers’ ability to raise students’ test scores (in part because the original value-added scores, unlike the other components of the system, are available both before and after the new evaluations were implemented).

A strong footnote for this important point—so-called weaker teachers were already leaving—is that the real teacher quality problems facing schools, notably high-poverty schools serving vulnerable populations of students, are a lack of equity in terms of teacher assignment (poor students, black/brown students, ELL students, and special needs students disproportionately are assigned year after year to new or inexperienced and un-/under-certified teachers; white and affluent students are gifted the most experienced and certified teachers) and the debilitating grind of high teacher attrition, turnover, in high-poverty and majority-minority schools.

Just as school choices increases educational problems such as segregation, VAM-based teacher evaluation does not address the real problems—equitable access to experienced and qualified teacher, and teacher turnover in high-poverty schools—while also increasing those problems.

And finally:

The big finding of Cullen et al. is that the relationship was stronger after the onset of the new evaluation system, with the estimated effects concentrated among low-performing teachers in schools serving low-performing students, who were more likely to exit the district than they were before ETI.

On the one hand, this suggests that the new evaluations worked as intended. Under a system in which principals were armed with better information about their teachers’ performance (full evaluation results instead of single year value-added scores), teachers who were less effective in raising test scores were more likely to exit the district (or be dismissed) post-ETI than they were prior to ETI, particularly in schools serving lower performing students. On the other hand, all exits increased under the new evaluations — including among teachers who were rated as average and high performers. The extent to which this spike is attributable to the new evaluation system per se is unclear, but it served to “dilute” the impact on student achievement of the increase in exits among low performers. There is also some indication that higher-rated teachers were more likely to switch out of schools with low-performing students after ETI (versus before the policy), which would also attenuate the impact of the policy.

The Big Caveat, of course, is that this evaluation process and concurrent analysis remain trapped in the efficient (read: lazy) use of test scores by students to determine teacher quality and effectiveness. That said, this study seems to show that VAM-type evaluations may actually push out so-called weak teachers—while also pushing out so-called effective and experienced teachers.

This preliminary evidence supports what many of us have been warning about during the Obama era of education reform: The “bad” teacher approach to education reform causes more harm than good because it misrepresents teacher quality and further de-professionalizes teaching, such as eroding the current teacher work force and discouraging the so-called “best and brightest” from choosing education as their career.

Di Carlo continues to offer incredibly important education research analysis, and I highly recommend anyone interested in education reform to return to this blog regularly. There you will find careful and crisp analysis—although I will continue to hope for the sort of analysis that will critically confront what lies beneath the political and public discourse about schools, education, teachers, and students.

The story inside the story of the research analyzed above is that beneath the “bad” teacher approach to education reform is a great deal of bad politics and bad media; and we must stop tip-toeing around those facts.