Key Takeaway: "Gold-Standard Research" is (April) Fools' Gold 

Find Documents:
Press Release: http://nepc.info/node/7914

Contact:
Dr. Fess Au: (303) 492-8370

BOULDER, CO (April 1, 2016) - A startling discovery made last week in a Chicago laboratory is poised to turn the educational research world on its head. Randomized, controlled trials (RCTs), which were long thought to be the gold standard for research, were found to consist almost entirely of iron pyrite, popularly known as “fool’s gold.”

In the realm of the hard sciences, most notably in pharmaceutical testing, experimental designs such as double-blind studies using a placebo are indeed golden. Educational researchers naturally assumed that experimental designs were equally ideal for their work. 

Yet when the Chicago researchers put RCTs under the microscope they discerned few of the benefits of true experimental studies. “The subjects ordinarily weren’t blind to whether they were in the treatment or control group, there was almost always no placebo, the subjects usually were not representative of the general population, and neither the treatment nor control subjects would stay in their proper place,” explained Dr. Fess Au, the principal investigator of the group of scientists who made the discovery.

Au notes that the research team did find some RCT specimens, such as those investigating a well-defined curricular intervention, that merited more attention. But the majority of RCTs clearly had little gold content.

To illustrate, Au pointed to studies of charter schools. Because some charter schools are popular and then use a lottery to determine who will be offered admission, researchers can take advantage of the random assignment of applicants to each of the two groups: those offered admission and those denied admission. Au pointed to five problems:

1. Those offered admission sometimes don’t accept.
2. Those denied admission sometimes find a comparable treatment (a similar school), so the idea of a “control” group becomes murky.
3. Those who start at the charter school sometimes leave.
4. Both groups of subjects know their status (treatment or control), potentially generating a Hawthorne Effect.
5. Even assuming all the other problems are addressed, the results cannot be generalized beyond the specific context and types of subjects.
    

“It’s this last point that really made us realize that we were looking at fool’s gold,” said Au. “How much does it help us to know that a particular, over-enrolled charter school has X effect on Y parents?” He explained that the results cannot be generalized to other charter schools, and certainly not to less popular charter schools. The results also cannot be generalized to the broad population of “non-chooser” families that did not apply to the school (or to any charter school).

“The usual result,” the scientists concluded, “is good internal validity but weak external validity. So why all the fuss?”

Seeking to answer this question, we tracked down three researchers known for their RCT work. All three spoke only on condition of anonymity, and all three acknowledged the scarcity of true gold in their work. “The real gold in this research comes from the IES,” said one, with a knowing chuckle.