The idea of closing “low performing schools” has undeniable appeal, at least in theory. The basic notion is that some schools are so dysfunctional that they cannot be saved and may be doing irreparable harm to their students every day they are open. Thus, it is argued, closing such schools and sending their students elsewhere is the best option – even if students end up in “average” schools, proponents argue, they will be better off.

Such closures are very controversial, however, and for good reason. For one thing, given adequate time and resources, schools may improve – i.e., there are less drastic interventions that might be equally (or more) effective as a way to help students. Moreover, closing a school represents a disruption in students’ lives (and often, by the way, to the larger community). In this sense, any closure must offer cumulative positive effects sufficient to offset an initial negative effect. Much depends on how and why schools are identified for closure, and the quality of the schools that displaced students attend. In practice, then, closure is a fairly risky policy, both educationally and (perhaps especially) politically. This disconnect between the appeal of theoretical school closures and the actual risks, in practice, may help explain why U.S. educational policy has been designed such that many schools operate at some risk of closure, but relatively few ever end up shutting their doors.

Despite the always contentious debates about the risks and merits of closing “low performing schools,” there has not been a tremendous amount of strong evidence about effects (in part because such closures have been somewhat rare). A new report by the Center for Research on Education Outcomes (CREDO) helps fill the gap, using a very large dataset to examine the test-based impact of school closures (among other things). The results speak directly to the closure debate, in both specific and general terms, but interpreting them is complicated by the fact that this analysis evaluates what is at best a policy done poorly.

The study, which includes schools in 26 states between 2006-07 and 2013-14, provides a useful overview of the characteristics of closed schools and their students, but the primary focus is on the outcomes of students whose schools closed (note that some of these schools may have closed for reasons other than performance per se – e.g., capacity, finances, etc.). The CREDO team looked at students in “low performing” schools – i.e., those with math and reading proficiency rates in the bottom 20 percent of their respective states for two consecutive years.

When a school closed, the data allowed the students to be “followed” to see how they did in subsequent years. Put simply, the testing growth of students who attended “low performing” schools that were closed was compared to the growth of similar peers who attended “low performing” schools that did not close (bear in mind that the comparison group is constrained to include only students in "low performing," non-closed schools).

The primary finding of the report is that students from closed schools ended up making less testing progress than similar students in “low performing” schools that didn’t close. The difference was statistically discernible but very small (about 0.01-0.02 standard deviations). In other words, if you (cautiously) take these results at face value, closing schools didn’t help students, on average.

There is a problem with interpreting these results, however, and it speaks directly to U.S. education policy broadly: The indicators most often used to identify schools for closure (e.g., absolute proficiency rates or other status measures) are poor measures of school performance. As we’ve discussed many times here, absolute performance measures such as proficiency rates tell you very little about the performance of a school. Some schools serve students who enter performing at very low levels, but those students make fantastic progress while attending; these are high performing schools, but could easily be classified as low performing based solely on proficiency rates or other status measures. Although federal and state policies are gradually moving toward incorporating growth measures in assessing school performance, many of the schools that were closed during the years in this study may actually have been average or even above average in terms of the progress made by their students. They were closed for no reason other than the students they happen to serve.

So, what do we make of the performance of students displaced from closed schools when the schools were closed based mostly or even entirely on measures that don’t actually measure performance? This is a serious tension in this CREDO study (one which is not the fault of the researchers). That is, the study defines performance properly (in terms of growth over time), but it is evaluating policies that define it improperly (in terms of how highly students score in a given year).

In this sense, the study is less an evaluation of school closures than it is an evaluation of a school closure policy done poorly. It is plausible, for example, that a policy identifying “low performing” schools based on the progress their students make would have resulted in larger gains (though any closure policy must contend with the disruption effect and the availability of nearby alternatives).

Note also that the CREDO study found that the schools that were closed served a disproportionate number of Black and Latino children. This is not so surprising, given that, in the U.S., students of color are also disproportionately likely to be from lower income households versus their white peers, and lower income students, on average, tend to score less well than their higher income peers. That is, Identifying schools for closure based on absolute performance levels is not only bad policy in terms of testing results, but it may also have a disparate impact by race and ethnicity.

Overall, then, this study illustrates the fact, which is obvious but still important to emphasize, that closing schools is very risky and not even close to the guarantee that a few diehard advocates sometimes imply. Certainly, there are at least some schools that, despite adequate time and resources to try and improve, remain dysfunctional enough that they should be closed for performance-related reasons. It would be absurd to argue otherwise (though there may be far fewer than some think). The problem is identifying them in a fair and rigorous manner. And we have done a very poor job of that so far.