Motivation

The last few decades have seen explosive growth in the number of impact evaluations—randomized evaluations in particular—that seek to guide policy decision-making in low- and middle-income countries. However, there are important gaps in translating this ever-expanding wealth of knowledge into actionable and credible policy insights. When moving from evidence generation to aggregation, translation, and adaptation, a key challenge is the lack of standardization and comparability of the main results produced by each study.

Activities

CEGA and the World Bank collaborated as part of a global consortium of institutions to fill this gap by constructing an Impact Data and Evidence Aggregation Library (IDEAL). Hosted by the World Bank, the IDEAL data will be fully open-access and free for the world to use. Users can search the data to find relative effect sizes on a range of outcomes across countries and contexts. We expect IDEAL to facilitate quantitative meta-analyses as well as more qualitative systematic reviews, whether such efforts are centered around an intervention, policy, population, or outcome.

Importantly, the consortium designed the IDEAL schema to help guide the standardization of key information from randomized evaluations, enabling more meaningful comparisons across studies while unlocking the potential to answer a wide range of methodological questions.

The IDEAL approach piloted extracting information with graduate student coders, generating lessons learned about which information is most difficult to extract due to ambiguity in research design. CEGA is excited to continue this work exploring how Large Language Models (LLMs) can facilitate the extraction of information at scale from thousands of published papers.

If you are interested in learning more about CEGA’s contributions to IDEAL, and how to collaborate with us, please contact Research Transparency Program Manager Jo Weech.