Author’s Note: This post was drafted before the election, but held until after the anticipated defeats of Donald Trump and Massachusetts’s State Question 2. The assumption was that the timing would be right for a balanced discussion of the actual record of choice-driven reforms. Now, both Trump and true believers in scaling up charters plan to ramp up their reality-free agendas. Regardless, even in the age of Trumpism, educators need evidence—based analyses of the actual outcomes of market-driven reforms.

If test-driven, competition-driven school reform has taught us anything, it is this: When schools post gains in state test scores, those numbers may mean something or they may mean nothing. Some of those bubble-in increases might be evidence of meaningful learning – just as they might mean that teach-to-the test has taught students destructive habits that will undermine their futures in college and post-school life.

By now, scholars should forsake the use of the terms “student achievement” and “student performance” when talking about test score “outputs.” Especially in schools and systems that focus on bubble-in accountability, the jury is in and sharp increases on graphs displaying test score boosts do not show that students are getting better educations. Researchers should accept the burden of proving that test score growth is evidence of increased learning or school improvement before saying that they are evidence of policies that are beneficial to students.

The latest report by the Education Research Association on New Orleans school reform, “Extreme Measures: When & How School Closures & Charter Takeovers Benefit Students,“ is consistent with the solid research that informs its previous scholarship.  And it includes the important qualifying statement that, “prior evidence suggests that schools that are effective in generating high test scores are not consistently more effective with other student outcomes.” When the ERA released its landmark “What Effect Did the Post-Katrina Reforms Have on Student Outcomes?,“ however, a diverse variety of scholars participated in the discussion of its findings. Fewer dissenting opinions were expressed when the new paper was released.

As with the case with the ERA’s “What Effect Did the Post-Katrina Reforms Have on Student Outcomes?”, by Doug Harris and Matthew Larsen, a graph that illustrates the NOLA trend line in test score growth presents the central argument. Harris and Larsen showed that New Orleans test score growth was increasing incrementally from 2002 to 2005. That was the time when NCLB prompted unbelievably huge increases in state test scores across the nation, as the reliable NAEP test score gains slowed.

Between 2007 and 2010, NOLA scores shot up by more than .2 standard deviations. This was the time, however, when its reformers had great freedom in terms of suspending and pushing out students who interfered with their mission to dramatically raise test scores. They also had thousands of additional dollars, per student, that could be used for constructive policies such as helping students graduate from high school, as well as for test-obsessed pedagogies. Growth then slowed and in the next two years’ tests gain were almost the same as the two years preceding the hurricane, about .1 standard deviation.

Harris argued that the gains from 2007 to 2010 were not reversed, and that could contribute to the case for seeing NOLA gains as meaningful. But, it would take future research to determine whether the NOLA approach produced real improvements. I would argue that the new “When & How School Closures & Charter Takeovers Benefit Students” fails to provide such evidence. This raises the question of whether the ERA’s methodology can provide real-world evidence in favor of NOLA’s extreme measures.

The ERA’s single most important finding was illustrated by a graph showing elementary schools’ test score gains before, during, and after closures. It charted the gains of schools that were closed, as well as gains by a matched group of students. Yes, during the two years after the interventions, tests scores in the schools that were taken over caught up and surpassed the matched group. So, if those gains reflect real learning, the ERA makes a case for taking over elementary schools. Moreover, it can then argue, “All future cohorts (of students) benefit, which means the net effect is clearly positive.”

Left unexplained is the time before and after the closure or takeover was announced. Neerav Kingsland, the former CEO of New Schools for New Orleans, would guess that “closing schools slightly harms existing students but is much better for future students.” But, test score gains were just as dramatic during that transition year, before interventions began! And, if gains before and after the announcement were not real, then it’s not possible to conclude that future gains that grow from that policy are positive. The gains during that year were comparable to the difference between the matched group and the intervention group, and no plausible explanation was given as to how that transition could have conceivably produced amazing test score gains that reflect actual learning increases, as opposed to what targeted schools are likely to do – juke the stats. Moreover, these gains represent a third of the total increases from the time of the intervention announcement.

In theory, I guess, the improvements during the transition year could have been bogus but the test score increases afterwards reflect real learning. More likely, however, they set the stage for the same approach to future learning – or lack of learning – for subsequent students. They may have been the first phase in creating schools that are so preoccupied with test score growth that who knows what else is ignored.

Moreover, ERA panelist Greg Richmond, of the National Charter School Association, acknowledges the evidence that high school closures and takeovers may be more difficult to accomplish, and the harm done by the process is likely to be greater. For instance, high school students may drop out during the transition. The ERA also warns against the harm done by disrupting schools, and notes that charters prefer to have a clean start with a new school. Consequently, charters have often been loath to takeover schools in ways that smooth the transitions. Moreover, the ERA notes both the increase in high school graduation rates and the lack of effects on college entry. Most importantly, the paper also reminds us of the oft-ignored principle which is especially important when experimenting with children’s live, “the principle of ‘do no harm.'” But, it still asserts that between 25 and 40% of the NOLA gains to takeover/closures.

High school graduation rates also are metrics that are easily fabricated, and they might or might not reflect increased learning. I don’t challenge those outcomes, however, because their downsides are so much smaller. They are likely the result of win-win policies, such as hiring more counselors and mentors who help students reach their goals. My concerns are a) the inherent harm of such a focus on test scores, and b) the games that tend to be played with numbers in order to pretend that bubble-in accountability actually increases learning.

And that raises the question of why the ERA did not include a timely qualitative component to its research. Surely they knew that transition years would occur. The likely scenario during those years would be that adults and students would be learning about fear and loathing. At least for educators with actual experience in the inner city, the last thing to be expected in such environments would be a dramatic increase in meaningful academic learning. As it stands, the gains of the two post-intervention years are joined at the hip with the seemingly impossible gains between the closure announcement and the actual closure. Until the ERA can provide a plausible explanation that the metrics in the transition years isn’t bogus, it still has the burden of proving that the post-takeover gains are real.

What do you think? Can you think of a rational explanation for real gains before interventions began? Why didn’t ERA researchers ask this question? Do you think President Trump will ask questions about such gains, or just promise that they will be “huge”?