The Social Science Prediction Platform

What will be the effect of raising the minimum wage by $2 an hour? How will cash transfers impact local prices in rural Kenya? Will a basic income support economic recovery from the coronavirus pandemic? When faced with such questions, policymakers and practitioners rely on research and expert perspectives to make decisions. Making good policy decisions often comes down to being able to reasonably predict the effects of policy alternatives, and choosing those that maximize societal benefits. Too often, however, hindsight bias leads us to believe we knew how a policy, intervention, or an evaluation would play out even if our priors don’t actually match the outcomes.

Collecting and recording predictions systematically can help us understand how results relate to our prior beliefs, as well as how we can improve their accuracy. They can also reveal how surprising results really are, potentially protecting against publication bias or the mistaken discounting of results as “uninteresting” after the fact. Finally, tracking prediction accuracy over time makes it possible to identify super-forecasters—individuals who make consistently accurate predictions—who can help prioritize research questions, as well as design strategies and policy options in the absence of rigorous evidence. All of these benefits become even more important in times of emergency, such as the COVID-19 pandemic which has disrupted the collection of field data for many research projects.

The Social Science Prediction Platform (SSPP), led by Stefano DellaVigna and Eva Vivalt, allows researchers to standardize how they source and record predictions, similar to how study registries have systematized hypothesis and design pre-registration. Perhaps most importantly, the SSPP streamlines the development and distribution of forecast collection surveys. With the SSPP, researchers can:

• Use our survey template or develop their own to collect predictions for various outcomes such as effect sizes, standard errors, and confidence intervals. And because Qualtrics is integrated within the SSPP itself, users can upload .qsf files directly to the platform.
• Distribute the survey directly from the platform to a sample of their choosing, including topic or method experts such as senior academics and professionals, disciplinary experts such as pre-screened Ph.D. students, or members of the general public who sign up on the platform. They can also send auto-generated anonymous links directly to individuals via email. Importantly, the SSPP enables more efficient coordination of survey distribution, reducing the risk of overburdening a small group of popular forecasters and mitigating “forecaster fatigue” (a common problem for journal reviewers) and low response rates.
• Access survey results on a pre-specified date and easily download them in .csv files.

To learn more about key issues to consider when designing surveys for collecting forecasts, as well as how to use the platform, see our Forecasting Survey Guide. The SSPP team is also offering consulting support for those who need it—contact Nick Otis at support@socialscienceprediction.org for help.