The folks at the American Enterprise Institute have released a study comparing charter schools to public schools. "Difference on Balance: National Comparisons of Charter and Traditional Public Schools" announces in its title that AEI continues to support the modern charter school industry (charter schools are not public schools in any meaningful way except their willingness to accept public tax dollars).

That said, a quick scan of the end notes shows a broad assortment of articles and even some studies not from the usual suspects. So this could be interesting. We should never pretend that AEI is anything but a huge fan of charter schools as a tool in the invisible hand of free market economics, but AEI is also, on the whole, one of the more intellectually rigorous thinky tanks in the biz. They may be working to sell reformster PR, but at least its not flimsy thin baloney PR, and after you've read a hundred or so of these things, you start to appreciate people who at least do a little homework and don't just base their "research" on five "reports" from their own organization.

At any rate, let's take a look at this thirty-two page paper. Once again (I believe this is the closest thing I have to a bloggy slogan) I've read it so that you don't have to. Here we go.

Our Author 

The paper is by Nat Malkus, a senior research analyst with AEI. From 2009 to 2015 he worked with American Institutes for Research, the outfit that sounds like a research organization but is in the test manufacturing business. He graduated in 1997 from Covenant College, a liberal arts Christian college in Tennessee, with a degree in historical studies and later earned a Ph.D. in educational policy and leadership from the University of Maryland. His AEI bio lists four years as a teacher.

In a US News piece about the release of his report, Malkus lays out the root of the issue-- there's a huge debate about how charters compare to public schools, and rather than suggest that one side is honest and decent and the other side is composed of delude liars, he suggests there's a different problem. His theory-- people are looking at different chunks of evidence.

Many studies have carefully matched charters to comparable traditional public schools in limited areas, but even the broadest of these don't generalize across the nation. Other evidence attains a larger scope by matching all charters with all traditional public schools, but doesn't account for the fact that the latter serve students everywhere, while charters exist in select locations.

Methodology

So we're going to start in the back of the report, looking at the methodology, because that's the whole show here. Again, from his US News summation:

I attempt to bridge the tradeoffs in scope and rigor by comparing student populations in charters across the nation to the public schools that neighbor them. Specifically, I match all brick and mortar charter schools to the nearest 5 traditional public schools of the same grade level in the same jurisdiction.

The report uses 2011-2012 data from the National Center for Educational Statistics. Malkus looked for public schools that could be compared to the charters based on shared enrollment. Where the charters were authorized by the local school district, only schools within that district were compared on the theory that only those students were eligible to attend the charter. Where charters could draw from outside any district boundaries, Malkus made his selections based on distance and grade level. Public schools were compared to other public schools on the same basis. Malkus also defined "local" as "within 30 miles."

Malkus started with a pool of 5,700 charter schools from 2011-2012. 890 were eliminated because they were special-purpose schools or just too tiny.

Malkus does make some questionable choices. For instance, 580 charters were found to have either missing or 0% Free and Reduced Lunch enrollment. Malkus takes this as a sign that these schools do not participate in the program, and tosses the 580 schools out of the study. However, given that one of the criticisms of charters is that some enroll few-or-none FRPL students, that seems like a lot of data to toss out.

Oh, and he threw out all the cyber schools. Would that we all could do the same. 

Ultimately there were 3,670 charters that had fully matching sets of data and five public schools that fell locally. I didn't see anything about what he did with any of those charters that had folded since 2012.

And here's an important part to understand-- he compares comparisons. In other words, he looks at how Gotrox Charter High compares to the public schools in the area, but he also looks Regular Public High compares to other public schools, which lets him compare GCH to RPH. 

So Let's Get To It

Malkus opens by noting that there's a huge debate involving two different pictures of who charters serve and that both sides are over-simplifying. Of course, it's only nice to make an equivalency if it's true. I'd suggest (and I'm sure some charter supporters would agree) that it's equally possible that one side is over-simplifying and the other side is right.

But let's see what Malkus found. Let's see this data-based description and comparison of charter and public school student populations.

Black Student Enrollment

Malkus finds that fewer than half of the charters have a black student enrollment that matches their local public schools; public schools match other public schools more closely. About a third of charters had more black students; about a fifth had fewer.

60% of the public schools had similar black enrollment which means-- well, I'm not sure. Only 10% of public schools had substantially more or fewer black students than the surrounding public schools, which would seem to suggest that segregation by housing is not a huge thing, and that would seem to be... counter-intuitive? Wrong?

Hispanic Enrollment

Wow. 42% of the charters enroll fewer, or substantially fewer Hispanic students.

White Enrollment

Once again, charters do not match their surrounding schools. Only about a third of all charters have the same white enrollment as surrounding public schools. The rest are evenly split between way more and way fewer white students.

Special Ed Enrollment

Here the public schools show the greatest similarity-- a full 71% have the same enrollment level, while fewer than half of charters have a similar enrollment level. It will come as no surprise at all that 35% of charters enroll fewer students with special needs. And Malkus does not get into the distinctions between levels of special needs; there's a big difference between a charter enrolling a student with a mild learning disability and one with profound (and expensive) issues.

Limited English Proficiency

The same pattern. 46% of charters match their local public schools. Again, Malkus's model does not allow for different levels of proficiency or the lack thereof.

Distribution of Poverty

Hmm. Only 16% of charter schools have similar levels of poverty to comparable public schools, with the rest evenly distributed between more and fewer students of poverty. But-- for public schools, only 31% have a similar level of poverty enrollment compared to schools around them (again, half and half for the differences).

It's pretty stark-- the numbers here suggest that it is economics more than anything else that leads to the sorting of our students into different schools, whether those schools are public or charter.

However-- and this is important-- we need to remember that FRPL numbers hide three different levels of poverty, and that the differences between those levels are not insignificant. Check out this piece from Jersey Jazzman for further explaining. Bottom line: the data in this section are incomplete.

Student Proficiency

Only 22% of charters have student proficiency levels similar to the local public schools. 44% of the charters have higher proficiency rates than public schools, suggesting that creaming and skimming are actually things. And because "student proficiency" actually means "student test scores," test prep may have a roll here as well. As always, I'd be happy to talk about actual student achievement, but as long as we're just talking about test scores, it's a waste of everyone's time.

Suspension Rates

The charter numbers here are not super-surprising-- slightly over half of the charters have similar suspension rates to public schools, with 17% having higher rates. Malkus notes that a full 29% of charters show lower suspension rates, but given that charters have more made-to-order student bodies, a lower suspension rate ought to be more common than it is.

The surprising number for me was in the public school. 79% of public schools have suspension rates similar to schools around them. It's important to remember that suspension rates can be pretty wobbly data; a low suspension rate can mean well-behaved students, a great positive atmosphere, or a spineless administration. But given how many people know the story of The Bad School with all of the Bad Children, it's surprising to me that public schools resemble each other when it comes to suspensions.

Other Data Crunching Games

For even more data fun, Malkus then goes back and crunches each pod of six schools (charter plus five public comparison schools) to see how charters rank out of the six for each trait. The crunching doesn't yield any big insights. We are going to skip it.

Discussion

Malkus then moves on to talk about the implications of his results. And by "implications" I mean "implications for continued attempts to market charter schools."

First, Malkus repeats a thread that he's been injecting throughout the report, which can be roughly summarized as "Look at how different charters are!" He means this observation to counter the dueling cartoon depiction of charters that he blames on a loud conversation between charter cheerleaders and charter critics. People have been oversimplifying, and charters are all really different.

Keep the point about differences in mind as Malkus moves on to his next point, which is that the data both confirms and debunks charter myths. Charters really do serve fewer students with special needs, but they really don't suspend students at higher rates. Except for the ones that don't and the that do, because charters are different.

To Malkus's credit, he follows the implications of the low enrollment numbers for students with special needs and language issues. There's no real hard data to prove that charters cream and skim, he says, but you don't have to squint at all to see that this data totally suggests that such skimming and creaming is in fact going on, and it would behoove charters to go take a look at this issue.

The Next Wave of Charterdon

This report, like the recent charter call for spanking cyber-charters, can be seen as the newest wave, the next logical step in the charter movement. Call it the Steady Maturing of the Charter Sector or call it Charterdon's Night of the Long Knives, but at some point, charters were going to have to turn on their own weaker brethren. When everyone was just trying to get things up and running, it was fine to let anything go. If you're just trying to get your foot in the door, it doesn't matter whether your shoes are nicely polished or not.

But at some inevitable point, the market was going to start paying attention to the product and some charters were going to start embarrassing all the other charters. As everyone from the movie industry to comic book publishers has learned at some point, if you don't police your own ranks, Uncle Sugar will show up to do it for you. And as Chi-Chi's learned, if just one part of your operation screws up, everyone in the business suffers.

So it's in the charter market interest to start observing that some charters actually do kind of suck. This report is particularly adept at making the point that charters are many and varied and just because you've heard about one terrible one, that doesn't mean that the charter one town over isn't awesomely better than your public school. 

Bottom Line 

This is four year old data, which in the world of modern charters is a lot of years. But it is certainly worth noting that a charter-loving think tank has now published research that says, "Yes, some of the criticisms of charter schools are valid."

AEI has always taken a patient long-term view of the charter biz, and this report concedes some ground in order to consolidate a more defensible position. Do not imagine for a minute that the next line after "Some criticisms of charters are valid" will be "and so we withdraw our support of them." The data, though it may have some problems, is worth a look. But don't expect it to change any charter fan's mind. Their premise remains the same-- charters must exist and must go on. Data like this will not change their goals; it will just help them fine tune their pursuit of those goals.