Experimental Evaluation Methods: How to Randomize?

Submitted by jyuan@worldbank.org on Tue, 04/12/2016 - 18:16

Lecture Overview

  • Unit and method of randomization
  • Real-world constraints
  • Revisiting unit and method
  • Variations on simple treatment-control


Experimental Evaluation Methods from clearsateam

Download File
Pdf as plain
Experimental Evaluation Methods
1. How to Randomize
2. Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How to randomize? 5. Sampling and sample size 6. Threats and Analysis 7. Scaling Up 8. Project from Start to Finish
3. Lecture Overview  Unit and method of randomization  Real-world constraints  Revisiting unit and method  Variations on simple treatment-control
5. Unit of Randomization: Options 1. Randomizing at the individual level 2. Randomizing at the group level- “Cluster Randomized Trial” Which level to randomize?
6. Unit of Randomization: Considerations  What unit does the program target for treatment? • Groups in microfinance. • Political constituencies for governance projects. • Schools for education projects.  What is the “unit” of analysis? • What are the outcomes we care about? • At what level are we able to measure them? • Examples-  Test scores for school children.  Health outcomes for individuals.  Vote shares for politicians.
7. Unit of Randomization: Individual?
8. Unit of Randomization: Individual?
9. Unit of Randomization: Clusters? “Groups of individuals”: Cluster Randomized Trial
10. Unit of Randomization: Class?
11. Unit of Randomization: Class?
12. Unit of Randomization: School?
13. Unit of Randomization: School?
14. How to Choose the Level  Nature of the Treatment • How is the intervention administered? • What is the catchment area of each “unit of intervention”? • How wide is the potential impact?  Aggregation level of available data  Power requirements  Generally, best to randomize at the level at which the treatment is administered.
16. Constraints: Political  Not as severe as often claimed  Lotteries are simple, common and transparent  Randomly chosen from applicant pool  Participants know the “winners” and “losers”  Simple lottery is useful when there is no a priori reason to discriminate  Perceived as fair and transparent
17. Constraints: Resources  Most programs have limited resources • Vouchers, Farmer Training Programs  Results in more eligible recipients than resources will allow services for.  More often than not, a lot of resources are spent on programs that are never evaluated.  Randomized experiments are usually no more costly than other methods.
18. Constraints: Contamination Spillovers/Crossovers  Remember the counterfactual!  If control group is different from the counterfactual, our results can be biased  Can occur due to- • Spillovers • Crossovers
19. Constraints: Logistics  Need to recognize logistical constraints in research designs.  For example- Individual de-worming treatment by health workers • Many responsibilities. Not just de-worming. • Serve members from both T/C groups • Different procedures for different groups?
20. Constraints: Fairness Is the randomization “fair”?  Randomizing at the child-level within classes  Randomizing at the class-level within schools  Randomizing at the community-level
21. Constraints: Sample Size  The program has limited scale  Primarily an issue of statistical power  Will be addressed tomorrow
22. What would you do? An intervention proposes to study the impact of regular washing of hands by children on their attendance in school. What would be your unit of randomization? A. Child level B. Household level C. Classroom level D. School level E. Village level F. Don’t know 0% 0% 0% 0% 0% 0% Child level Household level Classroom level Village level School level Don’t know
24. Phase-in: Takes Advantage of Expansion  Extremely useful for experiments where everyone will / should get the treatment  Natural approach when expanding program faces resource constraints  What determines which schools, branches, etc. will be covered in which year?
25. Phase-in Design Round 3 Treatment: 3/3 Control: 0 1 1 3 3 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Round 1 Treatment: 1/3 Control: 2/3 Round 2 Treatment: 2/3 Control: 1/3 Randomized evaluation ends
26. Phase-in Designs  Advantages • Everyone gets the treatment eventually • Provides incentives to maintain contact  Concerns • Can complicate estimating long-run effects • Care required with phase-in windows • Do expectations change actions today?
27. Rotation Design  Groups get treatment in turns: • Group A benefits from the program in period 1 but not in in period 2 • Group B does not have the program in period 1 but receives it in period 2
28. Rotation Design Round 1 Treatment: 1/2 Control: 1/2 Round 2 Treatment from Round 1  Control —————————————————————————— Control from Round 1  Treatment
29. Rotation Design  Advantages • Perceived as fairer • Easier to get accepted (at least initially).  Concerns • Control group anticipates future eligibility • AND Treatment group anticipates the loss of eligibility • Impossible to estimate a long term impact.
30. Encouragement Design: What to do When you Can’t Randomize Access  Sometimes it’s practically or ethically impossible to randomize program access  But most programs have less than 100% take-up  Randomize encouragement to receive treatment
31. What is “Encouragement”?  Something that makes some folks more likely to use program than others  Not itself a “treatment”  For whom are we estimating the treatment effect?  Think about who responds to encouragement
32. Which two groups would you compare in an encouragement design? A. Encouraged vs. Not encouraged B. Participants vs. Non-participants C. Compliers vs. Non-compliers D. Don’t know 0% 0% 0% 0% Participants vs. Non-part... Encouraged vs. Not enco... Compliers vs. Non-compliers Don’t know
33. Encouragement Design Encourage Do not encourage Participated Did not participate Complying Not complying Compare encouraged to not encouraged These must be correlated Do not compare participants to non-participants Adjust for non-compliance in analysis phase
34. Randomization « In the Bubble »  Sometimes there are as many eligible as treated individuals • Suppose there are 2000 applicants • Screening of applications produces 500 « worthy » candidates • There are 500 slots: you can’t do a simple lottery.  Consider the screening rules • Selection procedures may only exist to reduce eligible candidates in order to meet a capacity constraint • It may be interesting to test the relevance of the selection procedure and evaluate the program at the same time.  Randomization in the bubble: • Among individuals who just below the threshold
35. Randomization in “The Bubble” Within the bubble, compare treatment to control Participants (scores > 700) Non-participants (scores < 500) Treatment Control
36. To Summarize: Possible designs  Simple lottery  Randomized phase-in  Rotation  Encouragement design • Note: These are not mutually exclusive.  Randomization in the “bubble” • Partial lottery with screening
37. Methods of Randomization: Recap Design Most useful when… Advantages Disadvantages Basic Lottery  Program over subscribed  Familiar  Easy to understand  Easy to implement  Can be implemented in public  Control group may not cooperate  Differential attrition
38. Methods of Randomization: Recap Design Most useful when… Advantages Disadvantages Phase-In  Expanding over time  Everyone must receive treatment eventually  Easy to understand  Constraint is easy to explain  Control group complies because they expect to benefit later  Anticipation of treatment may impact short-run behavior  Difficult to measure long-term impact
39. Methods of Randomization: Recap Design Most useful when… Advantages Disadvantages Rotation  Everyone must receive something at some point  Not enough resources per given time period for all  More data points than phase-in  Difficult to measure long-term impact
40. Methods of randomization: Recap Design Most useful when… Advantages Disadvantages Encouragement  Program has to be open to all comers  When take-up is low, but can be easily improved with an incentive  Can randomize at individual level even when the program is not administered at that level  Measures impact of those who respond to the incentive  Need large enough inducement to improve take-up  Encouragement itself may have direct effect
41. What randomization method would you choose if your partner requires that everyone receives treatment at some point in time? A. Basic lottery B. Phase-in C. Rotation D. Randomization in the bubble E. Encouragement F. Don’t know 0% 0% 0% 0% 0% 0% Basic lottery Randomization in the b... Phase-in design Rotation design Don’t know Encouragement
43. Multiple Treatments  Sometimes core question is deciding among different possible interventions  You can randomize these programs  Does this teach us about the benefit of any one intervention?  Do you have a control group?
44. Multiple Treatments Control Treatment 1 Treatment 2
45. Cross-cutting Treatments  Test different components of treatment in different combinations  Test whether components serve as substitutes or complements  What is most cost-effective combination?  Advantage: win-win for operations, can help answer questions for them, beyond simple “impact”!  An example?
46. Varying Levels of Treatment  Some villages are assigned full treatment • All households get double fortified salt  Some villages are assigned partial treatment • 50% of households get double fortified salt  Testing subsidies, prices and cost-effectiveness!
47. Stratification  Objective: balancing your sample when you have a small sample  What is it? • Dividing the sample into different subgroups • Assign treatment and control within each subgroup
48. When to Stratify  Stratify on variables that could have important impact on outcome variable  Stratify on subgroups that you are particularly interested in (where may think impact of program may be different)  Stratification more important when small data set  Can get complex to stratify on too many variables  Makes the draw less transparent the more you stratify
49. Mechanics of Randomization  Need sample frame  Pull out of a hat/bucket  Use random number generator in spreadsheet program to order observations randomly  Stata program code  What if no existing list?
Regional Center Tag
Resource Type Tag
Language Type Tag
Image Thumb