Here’s a quick run-down on the 2010-11 New Jersey School Report Card data on charter schools. No-one else is putting out decent summaries of this stuff, so I feel obligated to revisit these data periodically. They don’t change much over time. But those older blog posts get buried over time. So, here we go.

Let’s take a specific look at Newark because that’s where most of our attention has been paid regarding high flying charter performance.

Data sources:

1. NJDOE Report Card

2. NJDOE Enrollment File

3. NJDOE Directory File (for City location)

Percent Free Lunch

Percent ELL

Percent Female

Regression Model of Charter Performance

More explanation is provided below. But this regression model (raw output on link below) is simply intended to compare the average proficiency rates across all tests and grades of charter schools to other schools in the same city and with similar characteristics. The bottom line is that as in previous similar regressions, there remains a small statistically non-significant margin of difference in average overall proficiency. But, the graphs that follow are perhaps more fun/interesting to explore.

CharterRegression

Now, for the following figures, the overall charter effect variable is removed, so that we can see how individual charter schools lie with respect to expected proficiency levels. The following figures compare schools to their predicted performance given each of the characteristics in the regression model. On the vertical axis is the standardized residual or the standard deviations above or below predicted performance. Along the horizontal axis is the percent free lunch of the schools, just so that we can see how they sort out by poverty concentration. Note that poverty concentration is already controlled for in the models. I begin with a few figures for select tests in Newark, and then present some statewide figures.

Newark Schools over and under predicted performance

Statewide schools over and under predicted performance

On average, this statewide picture is actually pretty ugly. It would certainly be very hard to argue that charter school expansion across New Jersey has led to any substantive overall improvement of educational opportunities. Numerous charter schools are substantial underperformers. And overall, as the regression model indicates, the net performance is bread even.

Take home points

This analysis merely compares the average proficiency rates of schools with similar characteristics in the same city. It does not measure whether charters “add value” per se.  This isn’t really ideal from a research perspective, because it doesn’t attempt to sort out whether these schools are actually doing something that leads to higher performance.

To address this question we might try either of two strategies – estimating achievement gains across matched schools – or hypothetically matched schools/children, or by a lottery based analysis comparing kids lotteried in to those lotteried out and staying in neighborhood schools.

But, I would argue that we still might not learn much of policy relevance for Newark from either of these approaches. Why?

Well, let’s consider the first approach – a matched school analysis (or virtual match based on individual students).  Let’s say we specifically wanted to determine the effectiveness of schools like North Star, Robert Treat or Gray charter.  The problem is that there really aren’t any “matched” schools or match-able kids – except perhaps those in magnet schools.  Note on matching-based-analyses… given that nearly all kids in a city like Newark qualify for Free OR REDUCED lunch, matching would have to be done on the basis of Free Lunch! If not, substantial precision/accuracy is lost and the comparisons invalid.

We might look outside of Newark for matched schools or students, but then other contextual factors might compromise the analysis quite substantially, and this might cut either for or against the charters.

Further, it appears that gender balance matters – not just a little – but a lot. Gordon McInnis tipped me off to this.  I hadn’t realized how big a deal it was in these schools.

Note that I’ve also left out attrition here, so that even if the schools were matched by poverty rates, gender and ELL concentration, there might be substantive differences in which students leave over time, altering the peer group composition over time (as weaker students leave).  Again, it may be most relevant to compare Newark Charters to Newark Magnets and/or children who attend them, which are most similar to these Newark Charters.

We could try to construct hypothetical or virtual matches based on similar individual children (to those in the charters) across the district who may or may not actually attend school together. But therein lies the problem, most other similar kids left in district schools would be attending school in substantively different peer groups than those in charters like North Star, Gray or Treat.

AND if we did find an “effect” on student achievement growth what the heck would it mean? And how would it inform our policy decisions?

Well, if we did, we would still have significant difficulty sorting out as to whether that effect has anything to do with school quality, or with student peer group  – quite possibly the largest in school factor affecting achievement.

Alternatively, one could attempt a lottery based analysis in which we look at the gains of kids lotteried in and lotteried out of the charters – left in their neighborhood schools. But in this case we would certainly have kids whose peer groups differ dramatically.  Again, we could try to “correct” for that uneven distribution, but the fact is that we simply can’t fully correct for the substantial contextual differences across these schools.  Too many Newark charters (and those in Jersey City and Hoboken) simply don’t even come close to resembling the student composition of traditional public schools in the same area.

So who cares? Well, it matters a great deal for policy implications whether the effect is created by concentrating less poor, English speaking females in a given school or by actually providing substantively better curriculum/instruction.  The latter might be scalable but the FORMER IS NOT! There just aren’t enough non-poor girls in Newark to create (or expand) a whole bunch of these schools!