Review faults inappropriate causal inferences
Contact: Gregory Camilli, (732) 932-7496; (email) camilli@rutgers.edu
Kevin Welner, (303) 492-8370; (email) kevin.welner@gmail.com
TEMPE, Ariz and BOULDER, Colo. (July 29, 2008) -- A recent report suggests that high-achieving students are losing out under the No Child Left Behind Act and recommends incentives for schools to better serve such students. An independent review of the report praises the scrutiny given to high achievers, especially those from resource-poor schools, but concludes that the report's presentations of findings and policy implications inappropriately overreach.
The report, High-Achieving Students in the Era of NCLB, by Tom Loveless (Part 1) and Steve Farkas and Ann Duffett (Part 2), was published by the Thomas B. Fordham Institute. It was reviewed for the Think Tank Review Project by Professor Gregory Camilli of Rutgers University. The report comes as Congress considers a redesign of NCLB, and Camilli notes it is framed to influence that discussion.
The two-part Fordham Institute report consists of 1) an analysis of mathematics and reading achievement among higher- and lower-achieving students as measured by a national standardized test, and 2) the results of a survey of teachers about how schools serve high achievers while meeting NCLB's requirements.
Analyzing results of the National Assessment of Educational Progress (NAEP) in reading and mathematics for students at the 10th (lower-achieving) and 90th (higher-achieving) percentiles on those exams, the report concludes that the gap between the two groups has narrowed since NCLB took effect. This result was due to achievement gains by low rather than higher achievers. The report then suggests that lower-income and minority high achievers may be at risk for losing out on opportunities under NCLB.
The second part of the report highlights findings concerning teachers' perceptions and beliefs regarding the education of higher achievers. For instance, 86% of the teachers responding indicated that it is more important to focus on achievement among all students, regardless of their advantages, rather than to focus primarily on struggling, disadvantaged students.
Camilli applauds the spotlight placed by the report on Black, Hispanic, and poor high achievers, and he concludes that it provides a "meaningful statistical description of these student populations." But he also offers several criticisms of the report:
• While the report includes acknowledgements that its correlational analyses cannot be used to draw causal inferences, such inferences nonetheless are used pervasively to bolster policy recommendations. Camilli is particularly critical of such overreaching in the report's foreword, written by Fordham's Chester Finn and Michael Petrelli, as well as in the executive summary.
• There are in fact two NAEP data sets from which gap statistics can be calculated. The report's analysis of the gap between high and low achievers uses only one. Camilli shows that with the other data set, much less convincing results are obtained, thus limiting the generalizability of the gap trends presented in the report.
• The report suggests policies that would increase stratification of educational opportunities, contradicting "a large body of research" that questions the efficacy of tracking students by perceived ability. This research is not discussed in the report.
• Response bias to the survey of teachers may be a problem. It received only a 15% return rate and did not provide appropriate breakdowns by such characteristics as grade level and urbanicity that might have better informed readers. The wording of certain questions may have inadvertently led to response bias.
Camilli concludes that the report's two studies inappropriately "attempt to inform broad policies on the strength of two fairly narrow analyses."
CONTACT:
Gregory Camilli, Professor
Rutgers University
(732) 932-7496
camilli@rutgers.edu
Kevin Welner, Professor and Director
Education and the Public Interest Center
University of Colorado at Boulder
(303) 492-8370
kevin.welner@gmail.com
About the Think Tank Review Project
The Think Tank Review Project (http://thinktankreview.org), a collaborative project of the ASU Education Policy Research Unit (EPRU) and CU-Boulder's Education and the Public Interest Center (EPIC), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.
Kevin Welner, the project co-director, explains that the project is needed because, "despite their garnering of media attention and their influence with many policy makers, reports released by private think tanks vary tremendously in their quality. Many think tank reports are little more than ideological argumentation dressed up as research. Many others include flaws that would likely have been identified and addressed through the peer review process. We believe that the media, policy makers, and the public will greatly benefit from having qualified social scientists provide reviews of these documents in a timely fashion." He adds, "we don't consider our reviews to be the final word, nor is our goal to stop think tanks' contributions to a public dialogue. That dialogue is, in fact, what we value the most. The best ideas come about through rigorous critique and debate."
**********
###
The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder partners with the Education Policy Research Unit (EPRU) at Arizona State University to produce policy briefs and think tank reviews. These centers provide a variety of audiences, both academic and public, with information, analysis, and insight to further democratic deliberation regarding educational policies.
Visit their website at http://educationanalysis.org
###
**********