Researchers evaluating highly visible interventions are often subjected to external pressures from government and donors to demonstrate positive results. In the absence of research transparency, investigators can conform to pressures and decide to report biased, imprecise or dishonest results. Alternatively, they run the risk of facing politically charged criticism from outside stakeholders if the results are unflattering. Furthermore, if researchers report null findings from an evaluation, the ambiguity associated with interpreting research findings opens the door for donors to interpret (or purposely misinterpret) the results as they find most convenient.
To combat these issues, investigators are developing tools to increase the transparency and integrity of research practices. One such tool, pre-specification plans, increases transparency by encouraging researchers to determine and document their methodology before their results are found. This precludes investigators from choosing their methodology based on what is most likely to produce positive results, increases confidence in research findings, and renders it more difficult for outside parties to discredit the results.
To both prevent a return to violence and to stimulate economic development, the Government of Sierra Leone implemented a number of reforms that give communities, and vulnerable groups within them, a greater voice in local decision-making. Alongside a national decentralization program that re-established district-level councils, the World Bank and major donors heavily committed to community driven development (CDD) projects helped the government pilot a project that went one step further by providing small grants to be administered by village development committees.
The World Bank and other major donors have allocated billions of dollars towards CDD projects on the basis that they foster democracy, the inclusion of marginalized groups, and enhance the capacity of communities to engage in collective action. Despite their high profile nature, there is little evidence confirming the social value of CDD programs. In such highly visible impact evaluations, pre-analysis plans limit pressure from government official and donors to show successful program outcomes by ”cherry picking” specific aspects of the results. The impact evaluation in question collected 316 indicators. On average 16 outcomes would be statistically significant due purely to random chance. If a research decided to ignore all other findings and only present those 16 results the program couldbe shown to have generated positive results.
Two hundred thirty-six villages from two ethnically and politically distinct districts were randomly assigned into a treatment group or a control group. Treatment villages were regularly visited by a facilitator, who helped community members create or revamp Village Development Committees (VDCs), set up bank accounts for the VDCs, establish transparent budgeting practices, and create village development plans that included specifics on how GoBifo grants would be used. The participation and inclusion of marginalized groups was central to this process – for example, each social group (women, youth, and adult men) came up with their own development plan, and these plans were then combined into a single unified vision.
Household surveys, which covered participation in local decision-making, attitudes to minorities, and engagement in collective action, as well as demographic and socioeconomic information, were collected in late 2005 and again in mid-2009, along with village-level focus group discussions. In addition, three structured community activities (SCAs) were conducted in late 2009, shortly after the program activities had ended, to capture any persistent impacts on collective action, participation of minorities, and elite capture.
Pre-analysis Plan Implementation
In 2005, before data was collected, the research team established nine hypotheses which would be tested to find the possible areas of program impacts. As the project came to a close in 2009, the researchers fleshed out a document with the exact outcome measures and econometric specifications that were used to test each hypothesis, and archived this ex ante analysis plan before the follow-up analysis. A mean effects approach averaging the results from each group of indicators was used to produce an index for the validity of each hypothesis.
Results and Policy Implications
There is no evidence that the program led to fundamental changes in local institutions or decision-making. Despite the fact that many women in treatment villages participated in GoBifo decisions, they were no more likely to voice an opinion in community meetings after the project ended or to play a leadership role in other areas. Similarly, the establishment of a democratically elected village development committee that carried out multiple projects did not lead treatment villages to be any more successful at raising funds in response to a later matching grant opportunity. Lastly, there were no program impacts on elite capture, although levels of capture were low in the research communities (at least as measured by the third SCA).
The use of a pre-analysis plan ensured that the researchers did not alter their results for political reasons and helped protect the team from potential criticism from donors and/or CDD advocates. Despite the results, CDD programs continue to be widely implemented for the purpose of affecting political economy.
2005 - 2009