BOULDER, CO (June 15, 2021) – A recent brief from the School Choice Demonstration Project at the University of Arkansas is being used by voucher advocates to argue that increasing school choice can spur broad test score improvements.
However, T. Jameson Brewer of the University of North Georgia and Joel Malin of Miami University reviewed Education Freedom and Student Achievement: Is More School Choice Associated with Higher State-Level Performance on the NAEP?, and found significant methodological weaknesses and flaws that render the report useless for guiding policy.
The report ranks states based on their expansion of market-oriented school policies such as vouchers, charters, homeschooling, and inter-district choice. It then constructs a regression model using this “education freedom” ranking along with per-pupil spending and student/teacher ratio, and using each state’s combined math plus reading NAEP scores as the dependent variable.
This creative approach yields an unexpected negative relationship between higher spending and the combined NAEP levels. The student/teacher ratio variable does show the expected inverse relationship to outcomes (the more students per teacher, the lower NAEP scores). Oddly, while the two variables—spending and student/teacher ratio—would be expected to be highly correlated, the study does not explore or address this concern by, for example, exploring alternative modeling choices.
The report’s main finding is that, after controlling for spending, student/teacher ratio, household income, and percent White students, the model shows a positive correlation between “freedom” and these scores (as well as NAEP score gains since 2003). While repeatedly stating that their data and methods “cannot establish conclusively whether education freedom caused those changes,” the authors also repeatedly trumpet the association teased out by their model and urge policymakers to embrace school choice policies. Readers are ultimately informed of the “reality” that “[s]chool choice has its best chance to influence NAEP scores and gains across an entire state by delivering competitive pressure to district-run public schools.”
Professors Brewer and Malin point out, however, that the report’s data and methods can, at best, suggest a relationship that should then be examined using a stronger research design. They also explain that the report, by ignoring relevant peer-reviewed research that has found negative consequences of school choice reforms, does not engage meaningfully with the larger body of research. Indeed, the reviewers identify significant methodological flaws that cast doubt on the report’s findings. Major faults include issues related to independent variable construction, the use of an unusually combined dependent variable, and the inclusion of a student group that is untested via the NAEP.
Moreover, the methodology fails to scrutinize dubious findings emerging from their models—particularly with regard to spending on education. Instead, the report uses such findings to buttress its concluding claim that a package of school choice reforms is desirable and beneficial.
These shortcomings undermine the report’s conclusions and render the study, as currently presented, useless for purposes of guiding policymaking.
Find the review, by T. Jameson Brewer and Joel Malin, at:
http://nepc.colorado.edu/thinktank/education-freedom
Find Education Freedom and Student Achievement: Is More School Choice Associated with Higher State-Level Performance on the NAEP?, written by Patrick J. Wolf, Jay P. Greene, Matthew Ladner, and James D. Paul and published by the School Choice Demonstration Project at the University of Arkansas, at:
https://cpb-us-e1.wpmucdn.com/wordpressua.uark.edu/dist/9/544/files/2018/10/education-freedom-and-naep-scores.pdf