External Validity in U.S Education Research

As methods for internal validity improve, methodological concerns have shifted toward assessing how well the research community can extrapolate from individual studies. Under recent federal granting initiatives, over $1 billion has been awarded to education programs that have been validated by a single randomized or natural experiment. If these experiments have weak external validity, scientific advancement is delayed and federal education funding might be squandered.
Sean Tanner conducted a meta-analysis of over 300 education-related interventions conducted in the United States in the past two decades. He found that study samples in U.S. trials are less white, more socioeconomically disadvantaged, and less proficient in English than districts as a whole. He also found that that effect sizes tend to decay in the second and third trials of interventions. These results point to serious challenges in generalizing causal knowledge from rigorous trials of education interventions in the U.S.