There is currently an ongoing rhetorical war of sorts over the gender wage gap. One “side” makes the common argument that women earn around 75 cents on the male dollar (see here, for example).

Others assert that the gender gap is a myth, or that it is so small as to be unimportant.

Often, these types of exchanges are enough to exasperate the casual observer, and inspire claims such as “statistics can be made to say anything.” In truth, however, the controversy over the gender gap is a good example of how descriptive statistics, by themselves, say nothing. What matters is how they’re interpreted.

Moreover, the manner in which one must interpret various statistics on the gender gap applies almost perfectly to the achievement gaps that are so often mentioned in education debates.

Let’s start with the gender gap, and take the common “75 cents on the dollar” side first. If you calculate the average hourly wage of all the women who work full-time and compare it with the wage of all the men who work, women earn about 75-80 cents on the male dollar (estimates vary over time, sample selection and by how one collects wage and earnings data).

In that sense, the frequently-cited statistic is “correct.” But what does it mean? If you’re simply trying to describewhat women earn, on average, vis-à-vis men, then you’re all set. And the fact that women earn less is a meaningful statistic, one with wide-ranging societal implications.

But that’s not quite how people usually interpret the gender gap. Typically, they’re making a statement related to discrimination – that the discrepancy between male and female wages reflects “unequal pay for equal work” by equally-qualified workers. This is a crucial distinction. It means you have to make sure you’re comparing the proverbial apples to other apples.

Let’s say we’re only comparing one man and one woman. The man makes $20 an hour while the woman earns around $16. But the $4 gap may very well have little to do with gender. Perhaps the man is much older, and would naturally have worked himself up to higher earnings. Perhaps the man has more education, or they have equal education but the man chose a higher-paying occupation.

Put simply, to the degree that there are such differences, on average, between men and women in terms of their characteristics and choices, the average gap in their wages may be due to those factors. For example, women are more likely to work part-time (some estimates include part-time workers), they select into different occupations (e.g.,teaching and nursing), and many of them leave and reenter the labor force for extended periods of time due to family considerations (labor force attachment).

To be sure, one might argue that some of these factors reflect an institutionalized form of discrimination – e.g., insufficient maternity and other family-related policies – but it’s misleading to imply that women make 75 percent of what men make solely because of “unequal pay for equal work,” when men and women are different, on average, in terms of their characteristics, work histories and choices.

Overall, when you control for all these factors, at least to the degree possible, a portion of the gender gap disappears (see here for an excellent review of this literature).

The discrepancy that remains (the “residual gap”) – usually around 5-15 cents on the male dollar, depending on data and methods – might be regarded as a more accurate estimate of the extent of discrimination.*

In short, there are different ways to measure the gender gap, and their “accuracy” is not about the statistics as much as how they’re interpreted. The gap is 75-80 cents on the male dollar if you’re making no claims that the difference is attributable solely to discrimination. When you account for the underlying factors – and you must do so to interpret the data in this manner – you get a somewhat different picture of the extent of the problem (problem though it still is).

Now, think about how easily this all applies to test data in education. We are inundated every day with average scores and rates – for schools, districts, states, subgroups of students, etc. These data are frequently compared between groups and institutions in much the same way as wages are compared between men and women.

One excellent example for our purposes is the race-based achievement gap. You’ll frequently hear that black students score much lower than their white counterparts. And, like the raw wages of men versus women, this is true and meaningful. It’s a powerful measure of unequal educational outcomes, one which has repercussions in terms of social, political, and economic inequality and other important dynamics.

If, however, this large discrepancy is used to argue (as is usually the case) that black students are receiving a commensurately inferior education – i.e., that the raw achievement gap entirely reflects the extent of inequality of educational opportunity – it is being misinterpreted.

Like women’s earnings, the test scores of black students reflect a host of other demographic and background-related differences, many of which have little to do with the quality of schooling. For instance, on average, black public school students are from families that are more impoverished and have parents who have less education than the national average. These and other interrelated non-school factors, such as access to health care, have a considerable influence on students’ test performance.

Researchers have long known that characteristics such as income and parental education levels “explain away” some portion of the black/white achievement gap. It is, however, important to note that the degree to which controlling for measurable background factors “shrinks” the size of the gap varies by study, grade and other factors, and there’s strong evidence that the gap grows as students progress through school (suggesting, in part, disparities in school quality). But, in all cases, confounding factors do explain much of the variation in performance.

(Side note: A recent working paper by economists Jesse Rothstein and Nathan Wozny, which is worth reading, finds that standard income variables might actually underestimate the importance of parental resources.**)

All this means that the unadjusted gap – the raw difference between black and white students’ scores – cannot by itself be interpreted as a measure of school quality, or be used to compare such quality between states and districts (as inter-group differences and thus residual gaps vary by location). Without question, the raw gap is still hugely disturbing and very meaningful. It is clear evidence of inequality of educational performance, but not entirely inequality of educational opportunity.

So, raw achievement gaps, like raw gender gaps, are what they are. It’s how they’re interpreted that determines whether we end up with productive policy debates or a political free-for-all in which statistics are blamed or disputed when the real problem is our own failure to draw proper conclusions from them.

- Matt Di Carlo

*****

* It is of course possible that there are other unmeasured factors responsible for part of the residual gap, and benefits may also change these calculations.

** Specifically, whereas most studies use parents’ current income, which is subject to year-to-year fluctuations (including unemployment), Rothstein and Wozny use “permanent income” – put simply, how much parents earn over multiple years. They find that permanent income explains twice as much of the gap as current income, which suggests that the adjusted black/white achievement differential may be somewhat smaller than is generally thought.