The surge in experimental evaluations observed in the last decades is closely related with the pursuit of an efficient allocation of resources in social policy. By measuring how effective different programs are, it is possible to assess whether such policies should be continued and to provide policymakers with a groundwork to an informed decision when comparing different intervention possibilities. Issues such as external validity and failures to replicate results on a larger scale, however, are a reminder of the difficulties in interpreting these experimental results. Nevertheless, even these asymmetries can be analyzed, providing us with a better understanding of the barriers to scaled implementation and replication.
As stated by Eva Vivalt in her 2020 article (“How much can we generalize from impact evaluations?”), “generalizability is not binary but something that we can measure.” By exploiting a data set of impact evaluation results, the author highlights some of the systematic differences in experimental results: small-scale interventions consistently report higher effects, while government-implemented programs usually have smaller effect sizes than academic or non-governmental organization.
These results put the implementation of these policies into perspective, showing that there is scope to understand the drivers of these gaps, allowing for more realistic estimates of large-scale interventions or finding measures to keep its effectiveness at scale. The article presented by professor Ashish Shenoy at the last LEAP Development Coffee is another contribution to this discussion.
You may read the full article by Erick Baumgartner on Knowledge Bocconi