As someone who spends a good bit of his time debunking many of the claims of the education reformsters, one continuing frustration is how many of them don't seem to learn their lessons. Certainly, we can have good faith debates about education policy, and reasonable people can disagree on many things...

But when you've been called out in public for making a big mistake, and you don't at least attempt to correct yourself... well, it's hard to take you seriously -- even if other, less discriminating minds do.

Case in point:

We parents all have heard the claim that something wasn’t fair. “Suzie got a bigger piece of cake than I did!” “Tommy got to go fishing while I had to clean the garage!” “Malachi had a lot more money spent on his education because you sent him to a traditional public school and me to a public charter school!” Well, maybe we haven’t actually heard that last one very often but it would be a more legitimate gripe than the other ones. 

Students in public charter schools receive $5,721 or 29% less in average per-pupil revenue than students in traditional public schools (TPS) in 14 major metropolitan areas across the U. S in Fiscal Year 2014. That is the main conclusion of a study that my research team released yesterday.

This is from the crew at the University of Arkansas's "Department of Education Reform" -- yes, there is such a thing, I swear -- led by the author here, Patrick Wolf. The study Wolf's team produced purports to show that charters are getting screwed out of the revenues they deserve, which are instead flowing to public district schools.



(Side note: if charters "do more with less," why do they need the same money as public district schools? Isn't that part of their "awesomeness"?)



But here's the thing: the methods this study uses are similar to a study they produced back in 2014 -- a study that was thoroughly debunked a month later. In a brief published by the National Center for Education Policy, Dr. Bruce Baker* notes that even if we put aside many problems the U-Ark study has with documenting its data sources and explaining its methodologies, one enormous flaw renders the entire report useless:

As mentioned earlier, the major issue that critically undercuts all findings and conclusions of the study, and any subsequent “return on investment” comparisons, is the report’s misunderstanding of intergovernmental fiscal relationships. Again, as the authors note, they studied “all revenues” (not expenditures), because studying expenditures, while “fascinating” would be “extremely difficult” (Technical appendix, p. 385). 

Any “revenue per pupil” figure includes two parts that may significantly affect the figure. What goes into the total revenue measure? And how are pupils counted? If one’s goal is to compare “revenues per pupil” of one entity to another, one must be able to appropriately align the correct revenue measure with the correct pupil measure for each entity. That is, for the district, one must identify the revenues intended to provide services to the district’s pupils and revenues intended to provide services to the charter school’s pupils. If numbers are missed or—worse yet—wrongly attributed, the comparison becomes invalid and misleading. [emphasis mine]

Baker cites several examples of how U-Ark gets this basic idea wrong time and again -- including, in his first example, U-Ark's analysis of Newark, NJ:

One can get closer to the $28,000 figure by dividing total revenue for that year by the district enrollment, excluding sent pupils (charter school, out of district special education, etc.). But this would be particularly wrong and the result substantially inflated because the numerator would include all revenues for both district and sent charter students, but the denominator would include only district students. 

Again, Baker pointed this out in 2014. But guess what? By all appearances, U-Ark made the same mistake once again in 2017. Let me see if I can explain this with a few pictures.

Unlike the U-Ark report, I'm going to tell you exactly where I'm getting my data for all these slides. The school year is 2013-14, just like the U-Ark report. The fiscal data comes from the User-Friendly Budget Guide** published by the New Jersey Department of Education; U-Ark says in its data comes from the NJDOE, so the figures should be the same. I get my charter enrollment numbers from the Enrollment data of the NJDOE, using the 2013-14 files.

There were, according to these sources, 17,273 students in Camden's total enrollment for 2013-14. These include contracted pre-school and out-of-district placements, which we will set aside for now (even though that is a deeply flawed thing to do -- more later). If we take the total full-time enrollment -- 15,546 -- and subtract 4,251 charter students, we get 11,295 Camden City Public School students.

Total revenues for the district in that year were $369,770,349. This included $54,902,533 in transfers to the Camden charter schools. Understand that this was not necessarily the only source of revenue for the charters, who might also collect funds directly from the federal government or from private sources. It's also worth pointing out here that all 4,251 Camden charter students may not come from Camden (although it's safe to assume the vast majority are city residents). But, as we'll see, that doesn't matter anyway.

Using these figures, U-Ark steps in to make its per pupil calculations. In the numerator is the revenue  collected by the district or the charter schools; in the denominator are the students enrolled in each sector.



See the problem?

If we use all of the $370 million in the district's per pupil figure, but we only count the students in CCPS and not the charters, we wind up double-counting about $55 million. Because that money is in both the district per pupil figure and in the charter figure.

Even U-Ark admits they should not do this:

That $370 million figure -- a figure, by the way, that is deeply flawed (more in Part II), should not be the figure that U-Ark uses to calculate CCPS's per pupil figure. I'm not saying this: U-Ark is.



So did they?

My calculation using these figures comes out to $32,738 -- which is very close to U-Ark's figure of $32,569.

Like I said, there are several reasons the figures don't match exactly: precise charter enrollment figures, including various students in out-of-district placements, minor adjustments to the revenue, etc. 



But it's clear that Wolf and his U-Ark team used the wrong revenue figure when making their calculation of Camden's per pupil spending; worse, they made the same mistake they made in 2014, even after they had been publicly corrected!

 

(Side note: we know they read Bruce Baker's review of their earlier report, because they cite it multiple times.)



Now, as I’ll explain in the next post, fixing this problem still makes for a deeply flawed analysis. But let's suppose, just for illustration purposes, they had corrected it. What would the figure be?

Here, we subtract the charter school transfer (find it on page 5 of the User-Friendly Summary). Which, according to U-Ark themselves, is the correct way to approach the calculation. What's the result?

Again, this is a deeply  flawed comparison. But it's a much smaller gap than using U-Ark's methods.

Let me end this part by addressing Professor Wolf and his team directly:

Gentlemen, I have shown in this post exactly where my data came from. Maybe you have different, equally credible sources. None of us would know, however, because your sole citation for state data is: "New Jersey Department of Education, School Finance." (p. 33) If you'd care to share your sources, your data, and how you arrived at your calculations (in appropriate detail to allow for replication, a common standard in our field), then please do; I'll happily publish them here. You can reach me at the email address on the left side of the blog.

But as it stands right now, there is more than enough evidence, in my opinion, to entirely dismiss your report and its conclusions.

Part II in a bit...

Reformy since 2005!

ADDING: Previous atrocities have been documented. 

* As always: Bruce is my advisor in the PhD program at Rutgers GSE.



** I use the 2015-16 guide because it gives the latest "actual" figures for 2013-14 available from NJDOE.