There has been a great deal written about the recent national study of charter schools by the Center for Research on Education Outcomes, housed at Stanford University, but here’s the bottom line. This was written by Kevin Welner, director of the National Education Policy Center, located at the University of Colorado Boulder School of Education.  He is the author of the 2008 book, “NeoVouchers: The Emergence of Tuition Tax Credits for Private Schooling.”

By Kevin Welner

Charter school policy is important. It’s worth arguing about. But those arguments can get a bit off-track and even ridiculous. Recently, the Center for Research on Education Outcomes (CREDO) released a national study of charter schools, the results of which suggest that less than one hundredth of one percent (<0.01 percent) of the variation in test performance in reading is explainable by charter school enrollment. Yet based on this, CREDO issued a press release stating that “charter school students now have greater learning gains in reading than their peers in traditional public schools.” This conclusion was repeated in newspapers across the nation.

I hope that, upon reading such claims, readers immediately ask two questions: Is that difference of any practical significance whatsoever? and Is this study really able to detect such infinitesimal differences? For both questions, the answer is ‘no’. But like a Kardashian, the import of the CREDO studies goes way beyond the justifiable.

CREDO’s charter school research burst onto the scene in 2009, with the release of a national study backed by a strong promotional effort. The 2009 study is probably best known through “Waiting for Superman“, a film backed by an even greater promotional effort. The ‘Superman’ narrator tells the audience that “one in five” charter schools is excellent. The actual finding from the study is that of the charters researched, 17% (which is really one in six) had better results than the comparison student results attributed to conventional public schools, while 37% did worse.

The publicity effort soon combined with the appeal of the study to some charter critics, who used those numbers to argue that conventional publics performed more than twice as well as charters. The buzz around the study was enormous; at one point, it seemed like “the CREDO study” was synonymous with “charter school research.” And the strengths, weaknesses, methods, and even the findings of the study were not well understood.

In truth, sometimes the best and most responsible understanding of a controversial report is neither black nor white. Sometimes we need to be able to embrace some nuance—in a specific study and in the overall evidence. That’s certainly the case here.

The National Education Policy Center (NEPC), which I direct, has reviewed three CREDO charter school studies: the original 2009 national study that is described briefly above, as well as the 2013 follow-up national study and the 2013 study of Michigan charters. These reviews are part of the NEPC’s “Think Twice”think tank review project, which provides the public, policy makers, and the press with timely, academically sound reviews of selected publications.

Because the CREDO studies have taken on such outsized importance, it is worthwhile to take a step back and consider those studies—with the help of the three NEPC reviews—and also to consider how they fit within the larger body of charter school research about test-score outcomes. The truth has always been that the CREDO studies are valuable and sound, but limited, contributions to an overall body of charter school research. This was true of the 2009 report/study, and it’s true of the more recent CREDO studies. The scope of the CREDO studies makes them among the most important in the field—they are based on datasets that capture most charters in the nation—but they have never deserved the ‘single, definitive study’ status sometimes accorded them. And there are good reasons for exercising caution when using the studies’ findings.

NEPC was not among those who argued that the original CREDO findings proved charters were doing meaningfully worse than publics. Instead, the review and the press release placed the CREDO study within the larger body of research and summarized the findings as follows: “The primary findings of the CREDO report show that charter school students’ test performance is basically the same as the performance of students enrolled in traditional public schools.” Notwithstanding the 17 percent/37 percent figures, the study if read properly did not show much separation between the sectors.

But for each of the CREDO studies, two questions need to be asked: What are the findings? and How strong are the data and analyses? The 2009 findings were somewhat favorable to conventional public schools—although not as favorable as some skeptics argued. The 2013 findings appear to show some improvement for charters, with great variation between schools and between states—and with an overall national estimate of students in charter schools scoring approximately 0.01 standard deviations higher on reading tests and 0.005 standard deviations lower on math tests than their peers in conventional public schools (the former being statistically significant; the latter not). That is, the ‘findings’ question can be answered as follows: small differences shown in 2009 are even smaller in 2013. The CREDO findings are highly consistent with an overall body of research concluding that the test-score outcomes of the sectors are almost identical.

How strong are the data and analyses? The CREDO data have always been impressive, reflecting the compilation of an unprecedented national dataset. The analyses, however, have been based on an atypical “virtual twin” approach that raised some concerns for the teams of NEPC reviewers in 2009 as well as in 2013. (Note that the 2009 reviewers’ main expertise was in charter school research, while the 2013 reviewers were experts on research methods.) Among the concerns that have been raised are the following two:
1. The study fails to use methods that could have directly modeled both individual student growth and school-level effects, such as hierarchical linear modeling, which would have been better matched to the goals of making generalizable statements about both students and schools. That is, the regression models used by CREDO fail to address independence of observations and the absence of measurement error, which are two key assumptions required in such analyses.

2. The so-called virtual twins in traditional public schools may not adequately control for differences between families who select a charter school and those who do not, which could bias the results.

Here’s the final sentence of the NEPC’s 2009 press release, “Because of the potential value of the CREDO work, the reviewers urge the authors to answer those questions [the methods concerns raised by the reviewers] in technical follow-up papers to the report and in later work with their data base.” Unfortunately, the CREDO researchers did not exercise the caution suggested or otherwise alter their approach.

Oddly, the 2013 CREDO studies have been used to argue that charters are outperforming conventional publics. In response, we have stressed the two points outlined above: (1) taking the CREDO results as gospel (setting aside methodological concerns) the results consistently show essentially no difference between sectors; and (2) we shouldn’t forget that the CREDO approach does have some weaknesses, those weaknesses have not been addressed even though they were first identified four years ago, and those weaknesses appear likely to result in stronger numbers for charters.

NEPC’s press releases for the 2013 reviews also included push back about inappropriate claims. In particular, Michigan’s pro-charter Mackinac Center held up the CREDO study of that state as showing charters to be a “smashing success;” Louisiana’s Gov. Jindal held up the CREDO study of that state as showing charters to be “proof of the success of charter schools in Louisiana.” These claims were nonsense, and we didn’t hesitate to say so.

Stepping back from the CREDO study specifically, there exists a very large body of evidence about charter schools (see “Exploring the School Choice Universe.” Among this research, studies like CREDO’s that look at test-score outcomes are the most common. Because this social science research takes place outside the controlled conditions of a laboratory, in the real (messy) world, each study comes with its own strengths and weaknesses. A broad use of the research base is accordingly much preferable to reliance on even the best study. Yet as noted, it matters not in this case: the overall body of research tells us the same thing as the CREDO research: the sectors are the same in terms of test-score outcomes.

Charter school critics and charter school advocates should be able to agree that there are excellent and awful charter schools, just as there are excellent and awful conventional public schools. What CREDO and others have shown is that, on average, the two sectors are very similar in terms of test-score outcomes. For those of us interested in improving overall educational opportunities it’s time to set aside the “which sector is better” test-score argument and instead invest across the board in the sorts of practices and supports that the best schools in each sector have provided for their students.