Economist Jesse Rothstein recently released a working paper about which I am compelled to write, as it speaks directly to so many of the issues that we have raised here over the past year or two. The purpose of Rothstein’s analysis is to move beyond the talking points about teaching quality in order to see if strategies that have been proposed for improving it might yield benefits. In particular, he examines two labor market-oriented policies: performance pay and dismissing teachers.

Both strategies are, at their cores, focused on selection (and deselection) – in other words, attracting and retaining higher-performing candidates and exiting, directly or indirectly, lower-performing incumbents. Both also take time to work and have yet to be experimented with systematically in most places; thus, there is relatively little evidence on the long-term effects of either.

Rothstein’s approach is to model this complex dynamic, specifically the labor market behavior of teachers under these policies (i.e., choosing, leaving and staying in teaching), which is often ignored or assumed away, despite the fact that it is so fundamental to the policies themselves. He then calculates what would happen under this model as a result of performance pay and dismissal policies – that is, how they would affect the teacher labor market and, ultimately, student performance.*

Of course, this is just a simulation, and must be (carefully) interpreted as such, but I think the approach and findings help shed light on three fundamental points about education reform in the U.S.

In short, the results (which, again, should not be overinterpreted) suggest that there is indeed a positive effect of both policies, but not a very large one. The impact of the performance bonuses is so small as to be only marginally meaningful, while deselection’s impact is larger, but still somewhat modest. Nevertheless, both policies are, according to Rothstein’s estimates, modestly more cost effective than money spent on class size reduction.

So, very quickly, here are my three takeaways.

Teachers must be paid more to assume more risk

As I have suggested in the past, as does Adam Ozimek in his post about this paper, it’s unlikely that we will attract “better candidates” to teaching by increasing their risks but not their rewards.

Districts will have trouble filling the vacancies from firing more teachers, at the same time the increased risk of dismissal, coupled with the uncertainty of salary based in part on performance bonuses, will, all else being equal, make teaching less attractive. Rothstein’s calculations support the commonsense notion that, in order to compensate for this, earnings of current and prospective teachers must be significantly higher.

Yet precisely the opposite is happening in many places. The relentless push for performance-based teacher pay, dismissals and other labor market-based reforms has largely occurred during a time in which many teachers’ compensation is not only not increasing (e.g., salary freezes), but, in some cases, actually decreasing (e.g., increasing benefits contributions).

Teachers don’t “teach for the money.” But, like all workers, they do consider compensation and job security when choosing their careers and whether to remain within them.

Increasing risk without increasing rewards is antithetical to market-based principles, and it may very well preclude the efficacy of even the best-designed selection-based policies.

Performance pay and deselection might, if done correctly, improve outcomes, but the realistic expectation is that the impact will be modest and gradual

Our education debate and policymaking are in no small part driven by the notion that simply removing the “bottom” teachers will generate drastic improvements over a relatively short period of time. These claims are usually based on other simulations, which are far simpler than Rothstein’s, and rely on assumptions that are considerably less plausible (though also see here).

For instance, Rothstein actually addresses the fact that the supply of new teachers – those who will replace the fired incumbents – is not necessarily unlimited, as is often assumed, at least if the dismissals occur on a large scale (e.g., not just in one district, which can siphon workers from elsewhere).

He also pays explicit attention to the precision (or lack thereof) in the measures used for these high-stakes decisions – in short, there will be mistakes.

These and other factors are likely to mitigate the benefits of any real world performance bonus or deselection policy. Such policies, done correctly, have potential, but the idea that they are guaranteed to work at all, to say nothing of generating large, almost immediate improvements, is probably unrealistic. It is also potentially destructive – for example, if we expect massive short-term improvements, we are more likely to end programs that might be working but haven’t been given enough time.

This brings me to my final takeaway.

Good measurement, and how teachers respond to the policies, aren’t just side issues

The “anything is better than the status quo” attitude that seems to permeate our debate reflects the (mostly fair) implication that evaluations and other policies weren’t working in many places, but, too often, it also seems to deemphasize the fact that how we measure teacher performance – and how teachers respond to the incentives embedded in the new systems – is exceedingly important, and requires careful attention and deliberate action.

For instance, Rothstein’s main analysis is a best case scenario. He explicitly assumes that value-added scores are perfect measures of teacher quality, and that their only limitation is random error due largely to small samples. When he relaxes these assumptions – and allows for the possibility of a “disconnect” between the performance measure (value-added) and outcomes, whether due to poor measurement or various forms of unwanted behavior (e.g., so-called “teaching to the test”) – the impacts are even smaller than they are under the “best case” assumptions.

This suggests, for one thing, that evaluations must be designed in a thorough manner, with field testing, continuous appraisal and attention to details such as accounting for error (which most states are largely ignoring). In addition, teachers’ perceptions of performance assessments – e.g., their confidence in the accuracy of the measurement, and their ability to affect it – almost certainly influences, at least to some degree, their decisions to pursue or remain in teaching. There is a serious need for research and attention to these and other issues, including how high-stakes applications affect performance measurement, which are too often casually dismissed in the rush to get new systems in place.

There’s a middle ground between “letting the perfect be the enemy of the good” and doing something just for the sake of getting it done. Failure to occupy that ground may in some places be threatening the potential of policies – whether deselection, performance bonuses, or any other – designed to improve teacher quality.

- Matt Di Carlo

*****

* Here’s an extremely simplified version of Rothstein’s approach: He compares a baseline scenario, in which there is no performance pay or deselection regime, with a scenario in which these policies are introduced. In both the baseline and intervention scenarios, every year, individuals’ decision to choose teaching – and whether to remain in the classroom – is modeled as a function of their projections of how much they will earn – and how much job security they’ll have – in the future. In the baseline scenario, the decision to teach, stay or leave is easier, since there is no firing and salaries are fixed in advanced (there are no bonuses). When the bonuses and deselection are introduced, however, things get more complicated. Under these alternative scenarios, current and prospective teachers have to make predictions (about future earnings and security) based on imperfect information. If teachers are confident in their ability – and the capacity of their employer to measure it correctly – they’ll choose teaching or stick around, because they think they’ll get the bonuses and face minimal risk of being fired. If they’re not, they’ll either be dismissed or, in anticipation of that outcome (or a desire to avoid the uncertainty), they’ll seek out other opportunities in the labor market. The longer they remain in teaching, the better idea they’ll have of how they’ll fare in the future.