Key Takeaway: Report’s utility is severely limited by methods of analysis
Contact:
William J. Mathis, (802) 383-0058, wmathis@sover.net
Joseph P. Robinson-Cimpian, (217) 333-8527, jpr@illinois.edu
URL for this press release: http://tinyurl.com/prhse73
BOULDER, CO (December 3, 2015) – A recent report from the National Bureau of Economic Research examined the short- and long-term outcomes of Florida’s retention policy aimed at low-scoring third graders. Several shortcomings severely limit the report’s usefulness, however, according to a new review.
Professor Joseph P. Robinson-Cimpian, of the Department of Educational Psychology at the University of Illinois at Urbana-Champaign, reviewed The Effects of Test-Based Retention on Student Outcomes Over Time: Regression Discontinuity Evidence from Florida, for the Think Twice think tank review project at the National Education Policy Center, housed at the University of Colorado Boulder’s School of Education.
A Florida policy flags students for retention, to repeat third grade, based on a state-specified cut-score on the Florida Comprehensive Achievement Test. Students just below the threshold (one-third of whom were retained) performed better than those just above the threshold (5% of whom were retained) on the next year’s tests. Based on these differences, a new report concludes that third-grade retention has immediate positive effects on the following year’s test results—although these effects fade over the next six years, with no effect on graduation.
Professor Robinson-Cimpian notes that the Regression Discontinuity methods used to estimate the effects, comparing students immediately above and below the law’s cut-score, lend themselves to making causal claims. However, he finds serious shortcomings. The law requires that students below the cut-score receive intensive extra services intended to raise their subsequent achievement, and this applies to the one-third of students who were retained and the two-thirds promoted. These promises of extra supports do not apply to those above the threshold, whether or not they are retained. Thus the researchers cannot know if positive outcomes for those below the cut-score were due to the greater likelihood of retention or to the assurance of additional services. Professor Robinson-Cimpian concludes that the report problematically attributes its finding of positive effects solely to retention, ignoring other required services such as summer reading camp and intensive reading interventions.
A second problem is that the researchers exacerbate the outcome differences between those below and above the threshold by using an Instrumental Variable approach, which attributes the entire difference to just the one-third of students who are retained, effectively making the outcome difference appear more than three times as large. Importantly, the very use of the Instrumental Variable approach is inappropriate because the method assumes that failing to attain the threshold has no effect on outcomes other than through increasing the likelihood of retention. The Florida policy, however, introduces another change in services—specifically, intensive interventions for promoted students just below the threshold but not above. Thus, services for promoted students change at the policy threshold, in addition to the change in the likelihood of retention. As Professor Robinson-Cimpian explains, “for the report’s claim about the causal effect of third-grade retention to be valid, there can be no other services that change at the policy-threshold.”
Further, Professor Robinson-Cimpian points out, the methods used have extremely limited generalizability, which is restricted to students at or very near the threshold and directly affected by the policy. Moreover, a number of additional factors—ranging from a changing sample to confounding third-grade retention effects with later-grade retention effects—further call into question any causal claims of the longer-term effects. Even setting aside the problems generated by confounding retention effects with the effects of other interventions and supports, the findings are not easily generalizable to lower- or higher-achieving students, to other grades, or to other states with similar test-based retention policies.
Find Joseph P. Robinson-Cimpian’s review on the NEPC website at:
http://nepc.colorado.edu/thinktank/review-NBER-retention
Find The Effects of Test-Based Retention on Student Outcomes over Time: Regression Discontinuity Evidence from Florida, by Guido Schwerdt and Martin R. West, published by the National Bureau of Economic Research, at: http://www.hks.harvard.edu/pepg/PDF/Papers/PEPG12-09_West.pdf
The Think Twice think tank review project (http://thinktankreview.org) of the National Education Policy Center (NEPC) provides the public, policymakers, and the press with timely, academically sound reviews of selected publications. NEPC is housed at the University of Colorado Boulder School of Education. The Think Twice think tank review project is made possible in part by support provided by the Great Lakes Center for Education Research and Practice.
The mission of the National Education Policy Center is to produce and disseminate high-quality, peer-reviewed research to inform education policy discussions. We are guided by the belief that the democratic governance of public education is strengthened when policies are based on sound evidence. For more information on the NEPC, please visit http://nepc.colorado.edu/.
This review is also found on the GLC website at http://www.greatlakescenter.org/.