The contemporary school reform movement was rooted in the Michael Deaver, Lee Atwater, Karl Rove, and Dick Morris era of public relations spin.  Being able to hire the best possible public relations expertise, reformers borrowed from Deaver’s success in using photos of Ronald Reagan getting off his helicopter, with his hand cupped on his ear, not being able to hear or answer the questions that reporters shouted at him.  He and his successors understood the truism that, “the eye always predominates over the ear when there is a fundamental clash between the two.”

Accountability-driven reformers also drew upon another legacy of Reagan era public relations – the proliferation of “think tanks” publishing “papers” that are touted as research.  Deep-pocketed donors fund an alphabet soup of advocacy organizations to publish “pseudo-studies” that typically conclude that test-driven, choice-driven policy experiments “can” increase student performance.  Since most commentators will read no more than the studies’ abstracts or the sponsors’ press releases, the papers provide an endless supply of soundbites and power points. The press continually gets eye-fulls of graphics indicating that accountability and charter schools can increase student performance. Rarely are these studies peer reviewed and almost none ask the questions that policy researchers should investigate. Few ask what will be the most likely results of reforms.  These papers shout out the supposed benefits of favored policies while ignoring their inherent costs.

Learning from the Federal Market-based Reforms: Lessons for the Every Student Succeeds Act (ESSA), edited by William Mathis and Tina Trujillo, is a wonderful corrective for the unfortunate trend of twisting evidence to reach predetermined edu-political agendas.  Learning from the Federal Market-based Reforms is a part of the National Education Policy Center series of publications, and it comes from a perspective that is very different than that of test-driven, competition-driven reformers. In contrast to so much of the research generated by reform advocates – where the best papers are often science-informed but shun the scientific method – the NEPC’s work measures up to the highest standards of  academic excellence.  As with most social science, the contributors to this 697 page anthology come with a perspective(s), but they are objective and intellectually honest.

“The Impacts of School Choice Reforms on Student Achievement,” by Gary Miron and Jessica Urschel, is one of the anthology’s best examples of an analysis which is essential for the discussion of charter schools and other forms of school choice.  It evaluates of the quality and the social scientific rigor of studies on the effectiveness (in terms of raising test scores) of vouchers, home schools, magnet schools, online learning and charter schools.  It then calculates how much each of those approaches increase or decrease (or fail to make much of a difference to) student performance.  In other words, Miron and Urschel provide a thoroughly researched but readable guide into the benefits these policies can contribute. Moreover, their graphics convey both the number and the quality of the studies, and whether they produce slightly or very positive or negative or mixed results.  These scattergrams thus provide a Rorschach test.  Readers are free to decide whether the various forms of choice are likely to produce gains that are big enough to justify policies that also have costs.

Starting with the last issue, Miron and Urschel only found three meaningful studies in terms of student performance results for virtual schools.  All three focused on virtual charters. Like the pro-reform CREDO, “The Impacts of School Choice Reforms on Student Achievement” concludes that students in the virtual schools made far smaller gains in comparison to demographically similar students in brick and mortar schools.  Miron and Urschel also conclude that the 19 studies of home schools were mostly of lower quality, while the 15 studies of vouchers were of higher quality.  The Weighted Mean Gains of home schools were the highest of all approaches (1.33 on a +2 to a -2 scale), while the voucher gains were the second-highest (.62). I doubt many people, regardless of their opinions on home schooling and vouchers will be determined by the test score increases that they produce.  Home school decisions will include a great deal of consideration of the downside of not sending children to a public school with their peers. Vouchers debates will be driven by questions in terms of the First Amendment, educational values, and the costs of pulling students out of public schools by subsidizing private schools.

I suspect that the more important analyses, in terms of policy-making, should be informed by the scattergrams regarding studies of magnet and charter schools.  Both have similar costs and advantages, although the magnets include inter-district magnets which could become very beneficial if used for socio-economic integration and racial desegregation in a new generation of school improvement.  I would argue that, in our reform environment where distant charter management organizations (CMOs) have so much power, today’s charters have far greater downsides than magnets.  Since charters have so much more freedom to “cream” and/or “push out” students who make it more difficult to raise test scores, their effects on student performance should be very similar to magnets. To read more detailed analyses of the downsides of charters, a reader must turn to other essays in Learning from the Federal Market-Based Reforms. But, Miron and Urschel are particularly impressive in describing the increased segregation caused by charters. In contrast to such an important social cost of charters, their positive Weight Mean Impact is only about 1/4th as great as the impact of magnets!

So, here’s the Rorschach test – the charts of the student performance effects of magnets and charter. The quality of the 26 studies on magnet schools and the 83 charter studies are very similar.  But, 14 of the magnet studies show slightly or very positive effects, while only 5 show slightly or very negative effects, with 7 showing mixed results. By contrast, 30 charters showed positive results, with 30 showing comparable negative results, and with 23 showing mixed results.  Miron and Urschel thus conclude,  “the magnitude of change evidence from charter school studies is not substantial and those looking for the achievement gap to be closed will be disappointed.”

Those seeking evidence to discredit Miron and Urschel for bias are likely to be disappointed also.  As testimony to their objectivity, Miron et. al’s 2007 study was rated somewhat lower in terms of methodological quality than the 2009 study by Zimmer, Gill, Booker, Laveru, Sass, and Witte, which has several authors generally linked with pro-reform analyses.

Miron’s and Urschel’s methodology required them to sort through and exclude studies that lack technical reports. The “piles of these pseudo-studies/evaluations” are testimony to the legacy of Michael Deaver’s spin that has infused corporate school reform.  Miron’s and Urschel’s scholarship shows that “the research and evaluation literature has not yet produced clear and unambiguous factual statements about achievement.” They further conclude, “It appears that policy decisions are being shaped by research that does not merit the emphasis it is receiving.”

By the way, they address the CREDO study which is “the most prominent study of charter schools, and it “has been repeatedly cited by pro-charter advocacy organizations.” They cite CREDO’s “carefully crafted sentence” which states, “While much ground remains to be covered, charter schools in the 27 states are outperforming their TPS peer schools.” Then Miron and Urschel add that reviewers noted: “. . . the study overall shows that less than one hundredth of one percent of the variation in test performance is explainable by charter school enrollment. With a very large sample size, nearly any effect will be statistically significant, but in practical terms these effects are so small as to be regarded, without hyperbole, as trivial.”

What do you think? Will policy-makers pay proper respect to Learning from the Federal Market-Based Reforms? Will they start to pay more attention to both the minimal gains produced by the proliferation of choice and the substantial harm it has produced?