Earlier this week, two Kennesaw State University economists wrote about their study that suggests the private school tax credit program saves Georgia money. Today, another researcher questions their conclusions and their analysis, saying they relied on flawed assumptions to buttress their case.

Kevin G. Welner is a professor of education at the University of Colorado Boulder School of Education where he chairs the Educational Foundations Policy and Practice program area. Welner co-founded and is director of the National Education Policy Center. He has written extensively on vouchers and tax credits, including the book, “NeoVouchers: The Emergence of Tuition Tax Credits for Private Schooling.”

By Kevin G. Welner

In a guest column published here Monday by Heidi Holmes Erickson and Benjamin Scafidi, the two Kennesaw State University researchers contend that Georgia’s “Qualified Education Expense (QEE) Tax Credit Scholarship Program” generates economic benefits to the state. The QEE program uses tax-credited donations to generate vouchers that fund private school tuition.

The claims made by Erickson and Scafidi are eye-catching. But, upon closer examination, their study clearly lacks sufficient data and rigor for them to reach valid conclusions. Below, I explain just three fundamental problems with the study. In doing so, I do not mean to suggest that additional problems are not also present.

Let’s begin by accepting the underlying figures from 2018 used by Erickson and Scafidi: 13,895 students received a “scholarship” (voucher). The average taxpayer cost per voucher was $3,713 per student. The state average cost per pupil in public school was $5,717.

At this point, the Erickson and Scafidi study turns to speculation: “Using a cautious approach—that 90% of these students would have enrolled in public schools if these scholarships had not been available, the analysis found it would have cost almost $105 million to educate these students in public schools” (emphasis added). This is not a cautious approach. In fact, the Southern Education Foundation in 2011 published an analysis of the Georgia program and concluded, “it is extremely difficult to identify any factual basis upon which to assume or conclude that most recipients of SSO scholarships in Georgia have transferred from public to private schools since 2008.”

The pro-voucher CATO Institute, looking at the early days of the similar Arizona policy, interviewed participating private schools about their voucher recipients and concluded, “On the basis of our data collection and conversations with representatives of scholarship organizations, we estimate that between 15 and 30 percent of scholarships were dedicated to students who would have otherwise attended public school.” That’s a far cry from a “cautious” 90%.

This is not a trivial matter; the calculations presented by Erickson and Scafidi hinge on this key number. To understand its importance, imagine that four students received vouchers: Ann, Barbara, Curt, and David. Using the figures above, if all four were prompted by the voucher program to enroll in private schools instead of public schools, the state savings would be four times the difference between what the state would pay for public school attendance and what the voucher costs the state: 4 x ($5,717 - $3,713) = $8,016. But if Ann and Barbara would have attended private school anyway, and the state is now subsidizing their private school tuition instead of paying nothing, then state still saves $4,008 total on Curt and David but suffers a loss of two times the voucher cost (2 x $3,713) due to the Ann/Barbara subsidy, with a net calculated as $4,008 – $7,426 = -$3,418. That is, if only 50% of the students are “switchers”, the state suffers a loss of $3,418.

Plugging in the numbers from Erickson and Scafidi, and using an approach outlined by researchers at Georgia State University, we can see from the below table that as the switcher number drops from 90 to 50 percent, the state “savings” drop from plus $19.9 million to minus $11.9 million. The break-even point is at almost exactly 65% switchers, as this chart shows.

The savings at the local level involve more complicated calculations. Erickson and Scafidi derived a figure of $8,381 for the marginal cost of educating one additional student in a Georgia public school. They then subtracted the $5,717 contribution from the state ($8,381 - $5,717 = $2,664), and determined an overall savings (again using the 90% switcher figure) of $33,3 million in 2019. But this number, too, appears to be highly inflated. The Georgia State University researchers mentioned earlier came up with a bottom-line marginal-cost figure in 2014 of only $1,272. Using that figure, the overall break-even point – accounting for the state and local fiscal effects – is slightly over 53% switchers.

So why do I (and policymakers, and Erickson and Scafidi) have to speculate about that key switcher rate? The answer takes us back to the QEE law itself, which appears to have been designed to cloak the fiscal effects and the academic effects of the voucher program. Regarding the latter, we do know from studies of more transparent voucher programs – in Louisiana, Indiana and Ohio – that students tend to suffer an academic disadvantage, particularly in math, when they receive the vouchers. But Georgia has not been able to accurately determine academic outcomes for its voucher-receiving children. And the fiscal effects of tax-credit voucher programs – in Georgia and elsewhere – are always designed to be hidden.

At the outset, I mentioned three fundamental problems with the Erickson and Scafidi study. The second one is the linchpin of their analysis of fiscal benefits tied to graduation rates and college matriculation. The researchers were given some data from GOAL, the largest among the private organizations that receive the tax-credited donations and then hand out the vouchers. Those data are, I suppose, better than nothing – which is what the law generally offers in terms of transparency. But the one control variable provided and used is eligibility for “free or reduced price lunch” (FRL).

I should note here that the GOAL dataset apparently also included the race/ethnicity of the voucher recipients. But while the report discloses that GOAL voucher recipients are, as compared to the public-school student population, substantially more likely to be white (and to not be eligible for FRL), no reported analyses control for race/ethnicity.

More importantly, the dataset apparently did not disaggregate among the FRL students to show how many are at the lower income level, qualifying for free (rather than reduced-price) lunch. A family of four qualifies for reduced-price lunch if its income is up to $47,638; it qualifies for free lunch if its income is from zero to $33,475. Since over 55% of all Georgia children are eligible for one or the other, merely controlling for overall FRL eligibility does little to account for the predictive value of family wealth in driving graduation rates and college matriculation. Given that the analyses also did not use the race/ethnicity variable, let alone non-available key predictors such as parental education, the causal inferences made by Erickson and Scafidi are simply not justified. Some of the data limitations are acknowledged by Erickson and Scafidi at the end of their full report.

The final issue with the study concerns something called “attrition bias.” Looking at the full report, we see the following: “Of the 1,191 GOAL students, 407 students transferred out of the GOAL program to other high schools leaving a cohort adjusted sample of 784 students.” The report – which is therefore based on an analysis of those remaining 784 students – includes no analysis attempting to determine if this attrition was, as might be expected, non-random. That is, might the students who left the program, and presumably their private schools, have had worse academic experiences than those who stayed? If so, then the results of the program would be worse than as reported.

In short, we all need far better reporting and data if we are to seriously analyze the fiscal or the academic impacts of the Georgia voucher program. Erickson and Scafidi shouldn’t be faulted for trying to come up with an estimate, but they should be faulted for overblown claims about their analyses and about the program itself.