Review finds that CEP publication inappropriately infers test progress of English Language Learners

Contact: Jeff MacSwan, (480) 965-4967; macswan@asu.edu

BOULDER, Colo. and TEMPE, Ariz. (May 19, 2010) - A recent Center on Education Policy (CEP) report asks, Has Progress Been Made in Raising Achievement for English Language Learners? and finds that in some states the number of English Language Learners (ELLs) meeting proficiency standards under No Child Left Behind (NCLB) has increased, while in others it has fallen. But a review released today finds the report suffers from a fatal flaw: using data that measure the performance at the level of states and school districts to draw conclusions about the progress of individual students.

The CEP report, by Naomi Chudowsky and Victor Chudowsky, was reviewed for the Think Tank Review Project by Jeff MacSwan, who is Professor and Director of the Applied Linguistics Program at Arizona State University and specializes in ELL research.

The CEP report relies on an analysis of trends in the achievement-test performance of ELLs from 2006 through 2008. Based on those data, it concludes that states have generally made progress in raising ELLs' achievement.

MacSwan, in his review, finds two principal flaws with the study. The first is that the report fails to adequately account for the fact that the language in which a person is taught can be a serious source of measurement error for ELL achievement test scores.

The more sweeping problem, however, is in the source of data that the CEP report uses to draw its conclusions. The report relies on the percentages of ELLs in each year's cohort for a given grade level who meet NCLB-related adequate yearly progress (AYP) standards. Because the makeup of those cohorts changes each year, "drawing growth-related conclusions from [these data] is inappropriate," MacSwan writes.

Moreover, while the report's title, "Has progress been made in raising achievement for English language learners?" asks readers to infer a direct causal relationship between changes in reported percentages of ELLs meeting AYP and improvements in academic achievement for ELLs, no such inference can legitimately be drawn from these data. And the report all but ignores a wide range of plausible, competing explanations for changes in scores over time other than actual improvement of achievement levels.

"A causal inference would require minimally that competing explanations be eliminated," MacSwan writes. "While the report notes these potential complications in the data, they are apparently not taken seriously enough to prevent the authors from drawing very specific conclusions about the achievement trajectories of ELLs."

The result is a report that "is not a useful guide for policy or practice related to ELLs," MacSwan concludes. "The data analyses presented do not provide useful insight into ELL progress, and no judgments can be made regarding the relative success of current federal policy based on the report."

The Think Tank Review Project (http://thinktankreview.org), a collaborative project of University of Colorado at Boulder's Education and the Public Interest Center (EPIC) and the ASU Education Policy Research Unit (EPRU), provides the public, policy makers, and the press with timely, academically sound reviews of selected think tank publications. The project is made possible by funding from the Great Lakes Center for Education Research and Practice.

Find Jeff MacSwan's review on the web at:
http://epicpolicy.org/thinktank/review-progress-ELL

Find the Center on Education Policy report, Has Progress Been Made in Raising Achievement for English Language Learners?, on the web at:
http://www.cep-dc.org/index.cfm?fuseaction=document_ext.showDocumentByID&nodeID=1&DocumentID=305

The Education and the Public Interest Center (EPIC) at the University of Colorado at Boulder and the Education Policy Research Unit (EPRU) at Arizona State University collaborate to produce policy briefs and think tank reviews. Our goal is to promote well-informed democratic deliberation about education policy by providing academic as well as non-academic audiences with useful information and high quality analyses.

Think Tank Research Quality: Lessons for Policy Makers, the Media, and the Public, our new book based on the work of the Think Tank Review Project, is now available from Information Age Publishing at http://www.infoagepub.com/products/Think-Tank-Research-Quality, or from Barnes & Noble at http://tinyurl.com/TTRQ-B-N.

Visit EPIC and EPRU at http://www.educationanalysis.org/

EPIC and EPRU are members of the Education Policy Alliance
(http://educationpolicyalliance.org).

###
**********