A recent Economist article on charter schools, though slightly more nuanced than most mainstream media treatments of the charter evidence, contains a very common, somewhat misleading argument that I’d like to address quickly. It’s about the findings of the so-called “CREDO study,” the important (albeit over-cited) 2009 national comparison of student achievement in charter and regular public schools in 16 states.

Specifically, the article asserts that the CREDO analysis, which finds a statistically discernible but very small negative impact of charters overall (with wide underlying variation), also finds a significant positive effect among low-income students. This leads the Economist to conclude that the entire CREDO study “has been misinterpreted,” because it’s real value is in showing that “the children who most need charters have been served well.”

Whether or not an intervention affects outcomes among subgroups of students is obviously important (though one has hardly “misinterpreted” a study by focusing on its overall results). And CREDO does indeed find a statistically significant, positive test-based impact of charters on low-income students, vis-à-vis their counterparts in regular public schools. However, as discussed here (and in countless textbooks and methods courses), statistical significance only means we can be confident that the difference is non-zero (it cannot be chalked up to random fluctuation). Significant differences are often not large enough to be practically meaningful.

And this is certainly the case with CREDO and low-income students.

The actual size of the difference (about 0.01 standard deviations in both math and reading) is very small. For all intents and purposes, there is little meaningful discrepancy between the CREDO-estimated performance of low-income students in charters versus those enrolled in regular public schools.*

Now, this does not mean that there are no important differences anywhere (e.g., within individual states), or that CREDO should supercede the fairly large body of evidence on charters. For instance, Mathematica’s 2010 experimental study of 36 charter middle schools found a large positive impact of charters on low-income students’ test growth, while a recent analysis of Massachusetts charters found that they generally outperformed regular public schools in urban districts.

It’s difficult to draw any sweeping conclusions at this point (many studies do not make the comparison at all), but, in many places, low-income students do tend to fare better in charters, though the differences are sometimes moderate to small (as in CREDO).

Of course, as with all charter school research, the most useful question is not whether some low-income students do better, but rather why. This is a tough question, which I have raised before in the context of charters in general, and which I hope  to address more fully in a subsequent post.

- Matt Di Carlo

*****

* It’s worth noting that this is a single-year impact, and that, as always, there is variation by state, school age and other characteristics.  Also, as discussed here, keep in mind that this is a relative, not absolute, estimated effect – it is a comparison with regular public schools.