A “recent” (2012) economics-based study has recently come to VAMmers’ attention, about effective teachers’ effects on students’ non-cognitive skills, with evidence supporting that teachers’ effects on non-cognitive skills matter more than test scores. Given the econometric approach of the author, I invited my colleague — ASU Assistant Professor of Education Economics, Margarita Pivovarova — to give us a review of the study. She writes:

“Going back to the debate about what constitutes a “good” or “effective” teacher and whether value-added based on test scores alone could potentially capture all about which we care in a “good” or “effective” teacher when we find one, a working paper released back in 2012 in the National Bureau of Education Researh (NBER) series on economics of education (a paper, though, recently reviewed here in the Better Living through Mathematics blog and recently reviewed here in Diane Ravitch’s blog) takes a big leap forward in that discussion and provides empirical evidence that not all value-added are created equal. The paper, titled “Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina” is available here.

C. Kirabo Jackson, an Associate Professor at Northwestern University’s Institute for Policy Research, proposes and empirically estimates (i.e., using actual data from North Carolina) a model that assumes that teachers may affect not only test scores, but also non-cognitive abilities of their students, and that these two effects may not overlap within the same teacher. In other words, a teacher who is very good at boosting test scores may not be the one who has high value-added in the behavioral and social skills domain. The author focuses on establishing the multidimensionality of a teacher’s effect – something many parents and students believe in and appreciate.

Statistically speaking, the strategy in the paper decomposes the value-added for the same teacher into two separate effects – one that affects cognitive skills (i.e., traditionally measured by test scores) and the other one that is widely overlooked by the adepts of value-added models, but nevertheless impacts important non-cognitive traits (i.e., behavioral and social characteristics of students). In fact, non-cognitive skills have been shown to be better predictors of adult outcomes than test scores when it comes right down to it.

But complicated statistical manipulations aside (although I should mention that the author does establish a credible causal link between teachers’ effect and students’ outcomes), let’s look at the findings. Here is where things become interesting.

First, the paper documents two interesting patterns: test scores are only weakly correlated with other measurable outcomes such as absences, suspension, and on-time progression, all of which can be called behavioral outcomes. At the same time, non-test-score outcomes are correlated with long-run outcomes – dropout rates, graduation rates, and SAT taking behaviors, independent of test scores. Moreover, behavioral outcomes are almost twice as effective as are test scores in improving long-run outcomes!

The next step would be to establish whether the teachers who are good at raising test scores are also as good at influencing students’ behavioral characteristics. It turns out, according to the results in the paper, that teachers “value-added” for test scores and non-test-scores outcomes are only weakly correlated, which implies that teachers have different skills and skills are not perfectly balanced within teachers. So, when we label a teacher as good, bad, or average, as based on the test scores included in value-added models alone, we are at the same time overlooking those who could have changed the life of their students for better by simply encouraging them to go to school more often and by helping them to progress through school (i.e., this flies in the face of retention policies based on test scores).

That being said, the main take away from this paper is that when we evaluate teachers based on their value-added as based on their test scores, we fail to measure the impact of good teachers on non-cognitive outcomes. This is highly unfortunate, and unfortunately non-trivial, as teachers who “add value” to non-cognitive factors and improve these in addition to just test scores, impact the future outcomes of their students to a much greater extent that teachers with the high value-added for test scores alone.”

Thanks to Dr. Pivovarova for her summary and post!! And thanks to C. Kirabo Jackson for such a unique article of so much “added value!”

*I should also mention, though, and in all fairness particularly given my prior posts and criticisms of the NBER and their premature releases of non-peer-reviewed studies (see, for example, here and here), this too has not yet made it to publication in a peer-reviewed journal outlet, although it is currently “under review” which is certainly farther along than many of the other NBER publications that never make it to print.