A Review of the Chetty Study - the Value of Teachers?

Kenneth J. Bernstein

Researchers Raj Chetty, John Friedman and Jonah Rockoff made headlines when The New York Times wrote about their study on teacher value-added in early January.  They concluded that teachers whose students show high gains on standardized test scores are less likely to become pregnant as teenagers, more likely to enroll in college, and more likely to earn more money as adults.

Their working paper: The Long-term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood, was published by National Bureau of Economic Research.

The study was a preliminary scientific paper on value-added, but that didn't stop The Timesand other media outlets from overvaluing the findings and sensationalizing the headlines - Big Study Links Good Teachers to Lasting Gain.  Typically such papers are used to solicit response before being peer reviewed.

Looking only at test scores, previous studies had shown, the effect of a good teacher mostly fades after three or four years. But the broader view showed that the students still benefit for years to come.

It didn't take long for reformers to pick up on the talking points by the authors.  "The message is to fire people sooner rather than later," (John) Friedman said then.

A review released today finds that the study didn't take into account plausible alternative explanations—for example, that parents who did the most to promote their offspring’s long-term success also endeavored to secure high value-added teachers for their children.  The review cautions policy makers about their reaction.

The original study was reviewed for the Think Twice think tank review project by Dale Ballou, an Associate Professor of Public Policy and Education at Vanderbilt University.

The review was produced by the National Education Policy Center (NEPC) with funding from the Great Lakes Center for Education Research and Practice (GLC).

Ballou’s review acknowledges that the report is impressive in many ways, but that important tests for bias were omitted and it failed to account for external factors which could impact test scores.

While the media overestimated the reach of the research to create flashy headlines, the data presented offer important tools to be further studied.  Ballou cautions that good teachers do more than just raise test scores.

The report includes several persuasive tests that substantiate the claim that teachers with high measured value-added are in fact raising students’ test scores: it is not merely an observed association, but a causal impact. However, the report’s findings with regard to students’ long-term success do not include adequate tests of this sort.

So while the original paper is useful, Ballou's review offers important cautions that need to be worked out before making judgements or firing teachers.

                                                            --

 

Ballou's review offers insight into how the study can be improved and used to analyze the results.  Unfortunately, The Times took the report placed it on page one and misused the data to sell newspapers.

Nicholas Kristoff, also of The Times, furthered the media blitz with: The Value of Teachers. His column took some wild leaps with the data.

Having a good fourth-grade teacher makes a student 1.25 percent more likely to go to college, the research suggests, and 1.25 percent less likely to get pregnant as a teenager. Each of the students will go on as an adult to earn, on average, $25,000 more over a lifetime — or about $700,000 in gains for an average size class — all attributable to that ace teacher back in the fourth grade. That’s right: A great teacher is worth hundreds of thousands of dollars to each year’s students, just in the extra income they will earn.

In his piece, Kristoff acknowledged that others warned him against the type of policy conclusions he was making, yet he made them anyway.

Kristoff concluded:

Suppose that the bottom 5 percent of teachers could be replaced by teachers of average quality. The three economists found that each student in the classroom would have extra cumulative lifetime earnings of more than $52,000. That’s more than $1.4 million in gains for the classroom.

Ballou specifically warns that using the data in the way Kristoff did is inaccurate, because there are too many variables which are not yet accounted for in the original study.

Inevitably, concern arises that the model has not controlled for enough other factors: that something omitted becomes confounded with the effect of the teacher. The mere fact that the model does not control for every influence on student achievement is not automatically a source of bias. Such influences need to vary systematically by teacher—that is, some teachers are normally assigned students who, for reasons not observed, score higher than expected, while other teachers are normally assigned students who do worse.

So while the report offers "one of the most dense, important and interesting analyses on this topic in a very long time" (Matthew DiCarlo, the Shanker Blog), Ballou's review should raise caution about overemphasizing the findings.

Thus, the report’s key findings linking teacher value-added scores to outcomes such as later earnings are not sufficiently validated; more evidence on that point needs to be presented before policy and practice are shaped in response to this report.

So much has been written about the Chetty study that blogger Larry Ferlazo has created The Best Posts On The NY Times-Featured Teacher Effectiveness Study.

Matthew Di Carlo at the Shanker Blog tackled the study with The Persistence Of Both Teacher Effects And Misinterpretations Of Research About Them.

Soon after the media blitz, Rutgers Bruce Baker wrote on his School Finance 101 blog: Fire first, ask questions later? Comments on Recent Teacher Effectiveness Studies.

Much has been written about the Chetty report on teacher value-added, but nothing as detailed as Ballou's review.  I encourage you to read Ballou's review and dig deeper.  Critical analysis of research, peer review, and policy implications need to be taken into consideration before we start jumping to conclusions.

ORIGINALLY POSTED TO ITEACHQ ON THU FEB 16, 2012 AT 04:20 AM PST.

ALSO REPUBLISHED BY COMMUNITY SPOTLIGHT.

This blog post has been shared by permission from the author.
Readers wishing to comment on the content are encouraged to do so via the link to the original post.
Find the original post here:

Daily Kos

The views expressed by the blogger are not necessarily those of NEPC.

Kenneth J. Bernstein

Kenneth J. Bernstein is now proudly 64 years young, teacher in DC metro area, Quaker liberal - and still passionate about learning with his students (http://teach...
Author Profile
Top