Recall that the peer-reviewed journal Educational Researcher (ER) – recently published a “Special Issue” including nine articles examining value-added measures (VAMs). I have reviewed the next of nine articles (#3 of 9) here, titled “Exploring the Potential of Value-Added Performance Measures to Affect the Quality of the Teacher Workforce” as authored by Dan Goldhaber – Professor at the University of Washington Bothell, Director of the National Center for Analysis of Longitudinal Data in Education Research (CALDER), and a Vice-President at the American Institutes of Research (AIR). AIR is one of our largest VAM consulting/contract firms, and Goldabher is, accordingly, perhaps one of the field’s most vocal proponents of VAMs, also self-described as an “advocate of using value-added measurements carefully to inform some high-stakes decisions” (see original reference here). Hence, it makes sense he writes about VAMs’ potentials herein.

Here’s what he has to add to the conversation, specifically about “the various mechanisms through which the use of value added might affect teacher quality and…what we know empirically about the potential of each mechanism” (p. 87).

Most importantly in this piece, and in my opinion, Goldhaber discusses the “[s]everal [which turns out to be two] studies that simulate the effects of using value-added estimates for high-stakes purposes [and] suggest there may be significant student achievement benefits” (p. 88). Here are the two sections in support of these benefits as defined and claimed:

• “There is evidence that high value-added teachers are perceived to engage in better teaching practices, and they are valued by principals as reflected in formal evaluations (Harris, Ingle, & Rutledge, 2014)” (p. 88). Contrary to this claim as interpreted herein by Goldhaber, however, is that these authors actually found “that some principals give high value-added teachers low ratings” which implies that the opposite is also true (i.e., inconsistencies in ratings), and “that teacher value-added measures and informal principal evaluations are positively, but weakly [emphasis added], correlated.” This puts a different spin on both of the actual results derived via this study, as per Goldhaber’s interpretation (see the cited study here).
• “Perhaps most importantly, value added is also associated with long-term schooling (e.g., high school graduation and college-going), labor market (e.g., earnings), and nonacademic outcomes (e.g., teen pregnancy) (Chetty, Friedman, & Rockoff, 2014)” (p. 88). As you all likely recall, this study is of much controversy (see prior posts on this study here, here, here, and here.)

Otherwise, Goldhaber explores “the various mechanisms through which the use of value added might [emphasis in the original] affect teacher quality and describe[s] what we know empirically about the potential of [emphasis added] each mechanism.” The word “might” (with or without emphases added) is notably used throughout this manuscript, as is the word “assumption” albeit less often, which leaves us with not much more than a clear impression that most of what is offered in this piece, is still conjecture.

I write this even though some of the other research cited in this piece is peripherally related, for example, given what we know from labor economics. “We know” that “teachers who believe they will be effective and rewarded for their effectiveness are more likely to see teaching as a desirable profession” (p. 89). But do we really know this? Little mention is made of our reality here, however, given the real and deleterious effects we witness, for example, as current teacher educators when we work with potential/future teachers who almost daily express serious concerns about joining a profession now with very little autonomy, not much respect, and a stark increase in draconian accountability measures that will be used to hold them accountable for that which they do, or do not do well. This also makes no mention of the prospective teachers who have now chosen not to enter teacher education, pre-profession, either, and for similar reasons. “On the other hand, the use of value-added performance measures might lead to positive changes in the perception of teachers, making teaching a more prestigious profession and hence leading more people to pursue a teaching career” (p. 89). Hmm…

Nonetheless, these conjectures are categorized into sections about how VAMs might help us to (1) change the supply of people who opt into pursuing a teaching career and who are selected into the labor market, (2) change the effectiveness of those currently teaching, and (3) change which teachers elect to, or are permitted to, stay in teaching. Unfortunately again, however, there’s not much else in terms of research-based statements (other than the two articles briefly mentioned in this manuscript, bulleted above) that Goldhaber “adds” in terms of “value” regarding the “Potential of Value-Added Performance Measures.”

I write this with some regret in that it would be fine with me if this thing actually worked, and more importantly, helped any of the three above desired outcomes come to fruition, or helped teachers improve their professional practice, professional selves, and the like. Indeed, in theory, this should work, but it doesn’t….yet. I write “yet” here with serious reservations about whether VAMs will ever satisfy that for which they have been tasked, largely via educational policies.

Related, and on this point we agree, “teacher pay incentives is one area that we know a good deal about, based on analysis of actual policy variation, and the results are not terribly promising…experiments generally show performance bonuses, a particular form of pay for performance, have no significant student achievement effects, whether the bonus is rewarded at the individual teacher level” (p. 89). We disagree, though, again on Goldhaber’s conjectures for the future, that “there are several reasons why it is premature to write off pay for performance entirely…” (p. 89; see also a prior post on this here as related to a study Goldhaber (overly) cites in support some of his latter claims).

In the end (which is actually near the beginning of the manuscript), Goldhaber notes that VAMs are “distinct” as compared to classroom observations, because they offer “an objective measure that does not rely on human interpretation of teacher practices, and by design, [they offer] a system in which teachers are evaluated relative to one another rather than relative to an absolute standard (i.e., it creates a distribution in which teachers can be ranked). It is also a more novel measure” (p. 88). As just stated, in theory, this should work, but it just doesn’t….yet.

In the actual end (actually in terms of Goldhaber’s conclusions) he suggests we “Take a Leap of Faith?” (p. 90). I, for one, am not jumping.

*****

If interested, see the Review of Article #1 – the introduction to the special issue here and the Review of Article #2 – on VAMs’ measurement errors, issues with retroactive revisions, and (more) problems with using standardized tests in VAMs here.

Article #3 Reference: Goldhaber, D. (2015). Exploring the potential of value-added performance measures to affect the quality of the teacher workforce. Educational Researcher, 44(2), 87-95. doi:10.3102/0013189X15574905