This report, published by the National Bureau of Economic Research and lead authored by Hoover Institution Senior Fellow Caroline Hoxby, initially escaped our attention. But the Washington Post and many charter advocates trumpeted the findings so loudly that we really had no choice but to seek a review. We were glad we did. In a revolutionary reversal of research procedures, Hoxby and her colleagues announced her results to the media and policymakers while withholding much of the actual information on which her results were based. Hers was a breakthrough in scientific methodology and a tribute to the non-accountability of pro-accountability researchers.

As for the report itself, Hoxby claimed that charter schools in New York City worked better than public schools, pointing out that she had taken advantage of a natural experimental design as students were assigned by lottery. However, the expert third party review notes several likely sources of bias which probably resulted in Hoxby’s inflated finding. Among these, the study relies on statistical models that include 3rd grade test scores, measured after the admission lotteries had taken place. Because of that timing, those scores could be affected by whether students attend a charter school, meaning that Hoxby’s chosen statistical models destroy the benefits of the very randomization that she and her supporters rely on as the main strength of the study’s design. Of course, the reviewer couldn’t quantify the extent of the overestimate since Hoxby had left out the information that would be needed for readers to engage in such an independent review and analysis. Trust, and don’t verify appears to be the operational accountability philosophy.

Nevertheless, accompanied by a formidable cloud of statistical formuli and dressed up in elaborate theoretical assumptions and explanations, this work was widely heralded as proving the policy wisdom of the charter school reform. As our reviewer pointed out, New York City’s charter schools might genuinely be improving student outcomes; however, this study—because of the information it withheld and its methodological shortcomings—does not and cannot resolve the issue.