Review presents new criticism of report that claimed a positive competition effect of Florida voucher policy
Contact: John T. Yun, (805) 893-2342; (email) jyun@education.ucsb.edu
Kevin Welner, (303) 492-8370; (email) kevin.welner@gmail.com
TEMPE, Ariz and BOULDER, Colo. (May 22, 2008 ) - A recent Manhattan Institute report claims Florida's program of publicly funded, private-school vouchers for special education students fosters competition that helps special education students who remain in public schools. A new review of the report, however, concludes that vague and flawed statistical analyses provide little of value to decision-makers.
"The Effect of Special Education Vouchers on Public School Achievement: Evidence from Florida's McKay Scholarship Program," was written by Jay P. Greene and Marcus Winters and published by the Manhattan Institute for Policy Research. It was reviewed for the Think Tank Review Project by Professor John T. Yun of the University of California at Santa Barbara.
Florida's McKay Scholarship Program provides vouchers for special education students to attend private schools. The scholarship program is open to any Florida student classified as having a learning disability; as of the 2006-2007 school year, about 4.5% of Florida's special education students received these vouchers. The Manhattan Institute analysis appears to cover an earlier period, from school years 2000-2001 through 2004-2005, during which time the program enrolled fewer students.
The Manhattan report is based on statistical analyses that, its authors conclude, shows that the McKay program spurred public schools to improve achievement for those special education students who remained in public schools during that time. The report presents relatively small effect sizes (a small competition benefit), but asserts that these results are probably understated.
Yun's review finds the statistical evidence weak. Among his criticisms are the following:
* The conclusions rest on multiple unsubstantiated assumptions about the direction of possible "selection bias." This bias occurs because the students using the vouchers are not randomly selected, meaning that important differences likely exist between "average" special education students and those who use vouchers. In each case, the report presumes that the selection bias in Florida results in an understating of the competition effects. But Yun points out serious problems with those presumptions. The most important one is that the report completely ignores other plausible sources of possible section bias that might lead to overestimating the effect of the voucher program.
* The model specification for the report's analysis of achievement measures includes several questionable and difficult-to-justify choices that are unexplained and unaddressed in the report.
* The Manhattan report defines competition by the number of private schools willing to accept vouchers within a 5- or 10-mile radius of the public school. Yet, Yun argues, if the loss of students is the critical signal to public schools, the report should have considered the number of available spots relative to the number of students who could fill them. This would more accurately capture the magnitude of the competition pressure felt by public schools.
* The report's approach conflates exposure to vouchers with urban, rural or suburban location. Because the analysis did not directly include urbanicity, and because urban schools generally perform worse than their suburban and many of their rural counterparts (and thus may have an easier time improving their scores), the results could reflect factors unrelated to voucher exposure and be specific to changes occurring in urban schools.
* The report does not explain how public school officials would know how many private schools in the local area were accepting vouchers or the level of capacity in these private schools to enroll additional students with disabilities, nor how very small percentages of students leaving public schools for private schools under the program is likely to encourage widespread change in public schools. Nor does it explain how the mere presence of schools would trigger immediate changes in public school behavior that would be quickly reflected in student test scores.
In reality, Yun points out, "the number of voucher recipients was quite modest until nearly halfway through the sampled time period. In order for the hypothesized competitive effects to have caused improvements in nearby public schools, those schools would have had to almost immediately receive the signal that special education students were leaving their schools and then adjust their practices accordingly, with the effects of these changes then very quickly having an impact on test scores. Such a series of events seems unlikely."
Yun concludes that the report offers policymakers little guidance: "Any attempt to use this report for decision-making or policy evaluation, prior to validation using different methods and more robust approaches, should be viewed with extreme skepticism."
CONTACT:
John T. Yun, Professor of Education
University of California at Santa Barbara
(805) 893-2342
jyun@education.ucsb.edu
Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@gmail.com
About the Think Tank Review Project
The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.
Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."
**********
###
The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder partners with the Education Policy Research Unit (EPRU) at Arizona State University to produce policy briefs and think tank reviews. These centers provide a variety of audiences, both academic and public, with information, analysis, and insight to further democratic deliberation regarding educational policies.
Visit their website at http://educationanalysis.org
###
**********