Assessing Bias from the (Mis)Use of Covariates: A Meta-Analysis

Lenz and Sahn examine how often research findings depend on suppression effects, or covariate-induced increases in effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that researchers have a strong explanation for effect size increases, especially when the statistical significance of the key finding depends on them.

They find that 30-40% of observational articles in a leading journal depend on suppression effects for statistical significance. Although suppression effects are of course potentially justifiable — to address suppressor variables — none of the articles justifies or discloses them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.

Publications associated with this project:

• Lenz, Gabriel S., and Alexander Sahn. “Achieving Statistical Significance with Control Variables and Without Transparency.” Political Analysis, (November 2020) ed, 1–14. https://doi.org/10.1017/pan.2020.31.