An important article in the Journal of Policy Analysis and Management presents results from one of the published analyses to look at the long term impact of attending charter schools.

The authors, Kevin Booker, Tim Sass, Brian Gill, and Ron Zimmer, replicate part of their earlier analysis of charter schools in Florida and Chicago (Booker et al. 2011), which found that students attending charter high schools had a substantially higher chance of graduation and college enrollment (relative to students that attended charter middle schools but regular public high schools). For this more recent paper, they extend the previous analysis, including the addition of two very important, longer term outcomes – college persistence and labor market earnings.

The limitations of test scores, the current coin of the realm, are well known; similarly, outcomes such as graduation may fail to capture meaningful skills. This paper is among the first to extend the charter school effects literature, which has long relied almost exclusively on test scores, into the longer term postsecondary and even adulthood realms, representing a huge step forward for this body of evidence. It is a development that is likely to become more and more common, as longitudinal data hopefully become available from other locations. And this particular paper, in addition to its obvious importance for the charter school literature, also carries some implications regarding the use of test-based outcomes in education policy evaluation.

Booker et al. analyze the data for cohorts of students in Florida and Chicago public schools starting in the late 1990s. Only one cohort of students from Chicago could be followed through four (potential) years of college, while the Florida dataset permits the researchers to follow students up until 2011, including their entry into the labor market. A summary of their findings is as follows:

• Consistent with the results of Booker et al. (2011), enrollment in charter high schools is associated with a higher chance of graduation within five years, by 7-10 percentage points, vis-à-vis the comparison group (students who attended charter middle schools and regular public high schools);
• Students who attended charter high schools were roughly 10 percentage points more likely to attend college;
• Charter school attendees in Florida were 13 percentage points more likely to persist in college for at least two years, and seven percentage points more likely in Chicago, though the latter estimate is imprecise and not statistically discernible at any conventional level;
• When they reached age 23-25, former charter students (Florida only) earned about $2,300 more annually, on average, which represents roughly 13 percent of average annual earnings among this group.

(Note that the choice of comparison group – students who attended charter middle schools but not charter high schools – is designed to strengthen internal validity at the possible expense of external validity. It may be the case, in other words, that students who attended charter middle schools "respond differently" to the treatment than do their peers who attended regular public middle schools. Note also that this study focuses exclusively on high schools.)

These estimated effects are clearly substantial in magnitude, and they suggest that charter high schools in these two locations may be having a meaningful positive impact on student outcomes beyond the typical standardized test scores that dominate our education policy debate. As additional studies from other locations start to emerge, it will be very interesting and important to see whether these results are found elsewhere. In the meantime, there are a couple of broader issues here that bear mention.

First, as we have discussed here countless times, the more important question is not whether these outcomes are better among charter versus regular public school students, but rather why. What specific charter school policies and practices might be responsible for these estimated impacts? These types of questions, while extremely difficult to address empirically, will hopefully be part of the evidence going forward.

On a second, possibly related note, one of the most striking aspects underlying the positive graduation, college attendance/persistence, and earnings results in Florida is that empirical evaluations of charter schools in the state have consistently shown no positive charter impact on test scores. It seems that attending Florida charter high schools is associated with better attainment and higher earnings after graduation, but not higher test scores before graduation. This seems consistent with other research, for example that showing the lack of strong relationship between teachers’ estimated effects on test scores versus other types of student outcomes (e.g., Jackson 2012).

One might speculate on the reason for this apparent disconnect by suggesting that, for example, charter high schools in Florida manage to imbue students with skills that are useful in college and on the labor market but do not help with standardized test performance. Or, perhaps, Florida’s charter high schools are particularly effective in counseling their students into college, and maybe beyond.

In any case, unless we are to conclude that standardized testing results tell us nothing at all (which is absurd), we have two student outcomes that are telling us different things about Florida charters. And the plausible, albeit painfully non-specific interpretation is that the concrete policies and practices of charter high schools in Florida may, on average, improve longer terms outcomes but not test scores. Understanding these factors is the key for studies like this one to help all schools improve, whether regular public or public charter.

It’s also clear that research on the longer term effects of charter school attendance (and other policy interventions as well) are important not only for their value in assessing the impact of these policies, but also for the potential they have to shed light on the strengths and weaknesses of common outcomes that we use for the purposes of such evaluation.