Achieving Statistical Significance with Covariates

The use of covariates in statistical models is an important, yet understudied area of researcher discretion. The size and statistical significance of study estimates can be affected by researchers’ choices of which covariates to include in a statistical model.
How often does the statistical significance of published findings depend on such discretionary choices? Gabriel Lenz and Alexander Sahn use newly available replication data to answer this question, focusing primarily on observational studies. In nearly 40% of articles published in the American Journal of Political Science (AJPS) between 2012 and 2015, they find that researchers likely achieved targeted statistical significance levels through covariate adjustment without disclosure or justification. Covariate adjustments lowered p-values to statistically significant levels by increasing the absolute value of researchers’ key effect estimates rather than increasing estimate precision. While these findings do not necessarily demonstrate that researchers exploit this discretion, they show that because of a lack of transparency requirements, there are ample opportunities to do so.