Development#
A fundamental aspect of inferential thinking and statistical measurement is how to design experiments in order to be measuring causation instead of correlation. An intervention can be thought of as some sort of change that is thought to bring about some desired outcome. Correlation between two measurements can be due to many underlying factors besides the intervention being studied. A properly designed experiment can isolate the effect of the intervention being studied.
The main idea is to take an overall population, create two subsets and randomly assign which of the subsets of the population gets an intervention (called treatment) and another subset who does not get an intervention ( called control). The population can be studied before and after an intervention is rolled out. The application of this methodology in Development Economics has caused a revolution in how the field measures the impacts of different development interventions.
Historically many of the early statistical models came from agriculture, where a scientist might be looking to increase the yield of a crop being planted. The experiment might be to plant two fields side by side with the same seeds, but one field gets two times the fertilizer of the second field. The amount that the yield increases due to fertilizer can be measured by the yield on the treated field (more fertilizer) minus the yield on the control field with baseline fertilizer. (And an economist would then balance the cost of the extra fertilizer against the revenue from the additional crop yield)
More recently, most people are familiar with drug trials, where new medicines are evaluated using an experimental design. In 2020 the whole world waited for the results of randomized controlled trials of the COVID-19 vaccines to be carried out before the vaccines could be used. In these trials, 30,000 - 40,000 volunteers were recruited from a diverse set of ages and ethnicities and randomized into two groups, treatment and control. In the treatment groups the volunteers received the vaccine, in the control groups the volunteers received an injection of a neutral saline solution. The study continued until a certain number of people in the control group had gotten COVID-19, and the number of people in the two study arms were compared.
In the current construction of the internet as we use it, A/B testing is a very common tool for UX design, and a very common work for data scientists working in internet companies. Every time you surf the web or visit an internet site you may be unknowingly part of a test of the size or color of a button, flow of the page, or other details. The internet companies constantly divide site visitors into treatment and control groups and measure site engagement metrics to calibrate changes to their site. A manager of any site, no matter how small, has tools such as Google Analytics that can allow them to measure changes in site design on user interaction metrics.
In the world of Development Economics, this has taken the form of evaluating interventions that are usually aimed at reducing poverty, or improving health and education outcomes. This field is generally called Randomized Controlled Trials (RCTs), or also sometimes Impact Evaluation. In many cases these are pilot programs for new interventions that can be measured well in a pilot case, and if measured well and a strong effect is found, can be used as evidence to make the case for a broader rollout. In many cases the pilot study may only have resources to provide the intervention to a subset of the overall population and this could lead naturally to a control subset and treatment subset, where the control subset will eventually get the treatment in future time periods.
Some examples of treatments that were tested out in pilot RCT are:
Malaria bednets to reduce childhood malaria
Household water treatment to reduce child diarrhea
Clean cooking stoves to reduce child respiratory illness
Conditional payments to incentivize health clinic visits
Payments for completion of school to incentivize girls schooling
In each of these we can think of reasons that randomization can be needed to screen out confounding factors. For example if malaria bednets were just sold at the store, it might be households with more wealth, more connected with social capital, or more education who might be the ones to purchase and use them. These confounding factors would make it hard for researchers to know what the effect of bednets were alone.
One seminal RCT was a study of deworming - giving out an anti-helminthic pill to kill intestinal parasites - to primary school children in Rural Kenya. This study was carried out to help a local non-profit that was working on increasing the rate of primary school attendance where there was high absenteeism due to childhood illness. The economics professors who carried this study out (and actually continue to carry it out) are Edward Miguel and Michael Kremer. Edward Miguel is now a Professor at UC Berkeley who teaches the Economics of Development Course (ECON 172) and Michael Kremer, Miguel’s PhD thesis advisor at Harvard, won the Nobel Prize for Economics in 2019. The randomization of populations was more elaborate than just two groups and the comparison of students who got the deworming pill and who did not showed positive health and school attendance effects not just for the students who got it, but also for students at the same school who didn’t get it and students who just happened to live in nearby villages to where the deworming was carried out. The researchers were also able to carefully document these effects using the correct study design and thus argue for the expansion of the program, which has since been carried out nationally in Kenya and in as can be seen on the Deworm the World project page “with over 1 billion treatments delivered since 2014 and 280 million treatments in 2019”
The researchers involve in this seminal study later moved on to study other interventions that could reduce the burden of childhood diseases by focusing on water-borne diseases such as diarrhea. Water-borne diarrhea is a leading cause of mortality in sub-Saharan Africa but is fatal primarily in infants under 5 years old. So the target of these studies was measuring health outcomes in children under 5. These interventions included the protection of water sources, Spring Cleaning, and the as can be seen in the Social Engineering: Evidence from a Suite of Take-up Experiments in Kenya, sold in Kenya as Water Guard to purify drinking water. A subset of this study is the focus of this week’s lab.
The pedagogical purpose of this week’s lab is also for students to think about the process that goes on behind the scenes of an economics journal article. The researchers work with the local NGO to map a set of villages where Spring Protection or Water Guard Promotion are going to be carried out. The populations are divided into a control arm, which will get the intervention in a future time period, and a set of comparable populations who get different treatment interventions, or even combinations of interventions. The NGO employs surveyors, who visit the households at baseline, before the intervention begins, and at various intervals later on after the intervention has been deployed. This household survey data is used to create a data set that is used to measure the effects of the intervention.