Threats and Analysis

Submitted by jyuan@worldbank.org on Wed, 04/13/2016 - 12:37
Image
threat

Lecture Overview

  • Attrition
  • Spillovers
  • Partial Compliance and Sample Selection Bias
  • Intention to Treat & Treatment on Treated
  • Choice of Outcomes
  • External Validity

Threats and Analysis from clearsateam

Download File
Pdf as plain
Threats and Analysis
1. Threats and Analysis
2. Course Overview 1. What is evaluation? 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How to randomize? 5. Sampling and sample size 6. Threats and Analysis 7. Scaling Up 8. Project from Start to Finish
3. Lecture Overview  Attrition  Spillovers  Partial Compliance and Sample Selection Bias  Intention to Treat & Treatment on Treated  Choice of Outcomes  External Validity
4. ATTRITION
5. Attrition Bias: An Example  The problem you want to address: • Some children don’t come to school because they are too weak (undernourished)  You start a feeding program where free food vouchers are delivered at home in certain villages and want to do an evaluation • You have a treatment and a control group  Weak, stunted children start going to school more if they live in a treatment village  First impact of your program: increased enrollment.  In addition, you want to measure the impact on child’s growth • Second outcome of interest: Weight of children  You go to all the schools (in treatment and control districts) and measure everyone who is in school on a given day  Will the treatment-control difference in weight be over-stated or understated?
6. Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 Ave. Difference Difference
7. Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30 Ave. 25 25 27 25 Difference 0 Difference 2
8. Attrition  Is it a problem if some of the people in the experiment vanish before you collect your data? • It is a problem if the type of people who disappear is correlated with the treatment.  Why is it a problem?  When should we expect this to happen?
9. What if only children > 21 Kg come to school? A. Will you underestimate the impact? B. Will you overestimate the impact? C. Neither D. Ambiguous E. Don’t know 20% 20% 20% 20% 20% Will you overestimate th... Will you underestimate t.. Neither Ambiguous Don’t know Before Treatment After Treament T C T C 20 20 22 20 25 25 27 25 30 30 32 30
10. What if only children > 21 Kg come to school? Before Treatment After Treament T C T C [absent] [absent] 22 [absent] 25 25 27 25 30 30 32 30 Ave. 27.5 27.5 27 27.5 Difference 0 Difference -0.5
11. When is attrition not a problem? A. When it is less than 25% of the original sample B. When it happens in the same proportion in both groups C. When it is correlated with treatment assignment D. All of the above E. None of the above 20% 20% 20% 20% 20% When it is less than 25% .. When it is correlated wit... When it happens in the ... All of the above None of the above
12. Attrition Bias  Devote resources to tracking participants after they leave the program  If there is still attrition, check that it is not different in treatment and control. Is that enough?  Also check that it is not correlated with observables.  Try to bound the extent of the bias • Suppose everyone who dropped out from the treatment got the lowest score that anyone got; suppose everyone who dropped out of control got the highest score that anyone got… • Why does this help?
13. SPILLOVERS
14. What Else Could Go Wrong? Target Populatio n Not in evaluation Evaluation Sample Total Population Random Assignment Treatment Group Control Group
15. Spillovers, Contamination Target Populatio n Not in evaluation Evaluation Sample Total Population Random Assignmen t Treatment Group Control Group Treatment 
16. Spillovers, Contamination Target Populatio n Not in evaluation Evaluation Sample Total Population Random Assignmen t Treatment Group Control Group Treatment 
17. Example: Vaccination for Chicken Pox  Suppose you randomize chicken pox vaccinations within schools • Suppose that prevents the transmission of disease, what problems does this create for evaluation? • Suppose spillovers are local? How can we measure total impact?
18. Spillovers Within School Without Spillovers School A Treated? Outcome Pupil 1 Yes no chicken pox Total in Treatment with chicken pox Pupil 2 No chicken pox Total in Control with chicken pox Pupil 3 Yes no chicken pox Pupil 4 No chicken pox Treament Effect Pupil 5 Yes no chicken pox Pupil 6 No chicken pox With Spillovers Suppose, because prevalence is lower, some children are not re-infected with chicken pox School A Treated? Outcome Pupil 1 Yes no chicken pox Total in Treatment with chicken pox Pupil 2 No no chicken pox Total in Control with chicken pox Pupil 3 Yes no chicken pox Pupil 4 No chicken pox Treatment Effect Pupil 5 Yes no chicken pox Pupil 6 No chicken pox
19. 0% 100% -100% 0% 67% - 6767 % Spillovers Within School Without Spillovers School A Treated? Outcome Pupil 1 Yes no chicken pox Total in Treatment with chicken pox Pupil 2 No chicken pox Total in Control with chicken pox Pupil 3 Yes no chicken pox Pupil 4 No chicken pox Treament Effect Pupil 5 Yes no chicken pox Pupil 6 No chicken pox With Spillovers Suppose, because prevalence is lower, some children are not re-infected with chicken pox School A Treated? Outcome Pupil 1 Yes no chicken pox Total in Treatment with chicken pox Pupil 2 No no chicken pox Total in Control with chicken pox Pupil 3 Yes no chicken pox Pupil 4 No chicken pox Treatment Effect Pupil 5 Yes no chicken pox Pupil 6 No chicken pox - 67%
20. How to Measure Program Impact in the Presence of Spillovers?  Design the unit of randomization so that it encompasses the spillovers  If we expect spillovers that are all within school: • Randomization at the level of the school allows for estimation of the overall effect
21. PARTIAL COMPLIANCE AND SAMPLE SELECTION BIAS
22. Non Compliers Target Populatio n Not in evaluation Evaluation Sample What can you do? Can you switch them? Treatment group Participants No-Shows Control group Non- Participants Cross-overs Random Assignment No!
23. Non Compliers Target Populatio n Not in evaluation Evaluation Sample What can you do? Can you drop them? Treatment group Participants No-Shows Control group Non- Participants Cross-overs Random Assignment No!
24. Non Compliers Target Populatio n Not in evaluation Evaluation Sample Treatment group Participants No-Shows Control group Non- Participants Cross-overs Random Assignment You can compare the original groups
25. Sample Selection Bias  Sample selection bias could arise if factors other than random assignment influence program allocation • Even if intended allocation of program was random, the actual allocation may not be
26. Sample Selection Bias  Individuals assigned to comparison group could attempt to move into treatment group • School feeding program: parents could attempt to move their children from comparison school to treatment school  Alternatively, individuals allocated to treatment group may not receive treatment • School feeding program: some students assigned to treatment schools bring and eat their own lunch anyway, or choose not to eat at all.
27. INTENTION TO TREAT & TREATMENT ON TREATED
28. ITT and ToT  Vaccination campaign in villages  Some people in treatment villages not treated • 78% of people assigned to receive treatment received some treatment  What do you do? • Compare the beneficiaries and non-beneficiaries? • Why not?
29. Intention to Treat (ITT)  What does “intention to treat” measure? “What happened to the average child who is in a treated school in this population?”  Is this difference the causal effect of the intervention?
30. Intention School 1 to Treat ? Treated? Pupil 1 yes yes 4 Pupil 2 yes yes 4 Pupil 3 yes yes 4 Pupil 4 yes no 0 Pupil 5 yes yes 4 Pupil 6 yes no 2 Pupil 7 yes no 0 Pupil 8 yes yes 6 School 1: Pupil 9 yes yes 6 Avg. Change among Treated (A) Pupil 10 yes no 0 School 2: Avg. Change among Treated A= Avg. Change among not-treated (B) School 2 A-B Pupil 1 no no 2 Pupil 2 no no 1 Pupil 3 no yes 3 Pupil 4 no no 0 Pupil 5 no no 0 Pupil 6 no yes 3 Pupil 7 no no 0 Pupil 8 no no 0 Pupil 9 no no 0 Pupil 10 no no 0 Avg. Change among Not-Treated B= Observed Change in weight
31. 3 3 0.9 2.1 0.9 Intention School 1 to Treat ? Treated? Pupil 1 yes yes 4 Pupil 2 yes yes 4 Pupil 3 yes yes 4 Pupil 4 yes no 0 Pupil 5 yes yes 4 Pupil 6 yes no 2 Pupil 7 yes no 0 Pupil 8 yes yes 6 School 1: Pupil 9 yes yes 6 Avg. Change among Treated (A) Pupil 10 yes no 0 School 2: Avg. Change among Treated A= Avg. Change among not-treated (B) School 2 A-B Pupil 1 no no 2 Pupil 2 no no 1 Pupil 3 no yes 3 Pupil 4 no no 0 Pupil 5 no no 0 Pupil 6 no yes 3 Pupil 7 no no 0 Pupil 8 no no 0 Pupil 9 no no 0 Pupil 10 no no 0 Avg. Change among Not-Treated B= Observed Change in weight
32. From ITT to Effect of Treatment on the Treated (TOT)  The point is that if there is leakage across the groups, the comparison between those originally assigned to treatment and those originally assigned to control is smaller  But the difference in the probability of getting treated is also smaller  Formally, we obtain the probability of being induced into getting treatment by “instrumenting” the probability of treatment by the original assignment
33. Estimating TOT  What values do we need?  Y(T)  Y(C)  Prob[treated|T]  Prob[treated|C]
34. Treatment on the Treated (TOT) 퐵 = 퐸 푌푖 푧푖 = 1 − 퐸 푌푖 푧푖 = 0 퐸 푠푖 푧푖 = 1 − 퐸 푠푖 푧푖 = 0 푌 푇 − 푌 퐶 푃푟표푏 푡푟푒푎푡푒푑 푇 − 푃푟표푏[푡푟푒푎푡푒푑|퐶]
35. TOT Estimator Intention School 1 to Treat ? Treated? Observed Change in weight Pupil 1 yes yes 4 Pupil 2 yes yes 4 Pupil 3 yes yes 4 A = Gain if Treated Pupil 4 yes no 0 B = Gain if not Treated Pupil 5 yes yes 4 Pupil 6 yes no 2 Pupil 7 yes no 0 ToT Estimator: A-B Pupil 8 yes yes 6 Pupil 9 yes yes 6 Pupil 10 yes no 0 A-B = Y(T)-Y(C) Avg. Change Y(T)= Prob(Treated|T)-Prob(Treated|C) School 2 Pupil 1 no no 2 Y(T) Pupil 2 no no 1 Y(C) Pupil 3 no yes 3 Prob(Treated|T) Pupil 4 no no 0 Prob(Treated|C) Pupil 5 no no 0 Pupil 6 no yes 3 Pupil 7 no no 0 Y(T)-Y(C) Pupil 8 no no 0 Prob(Treated|T)-Prob(Treated|C) Pupil 9 no no 0 Pupil 10 no no 0 Avg. Change Y(C) = A-B
36. TOT Estimator Observed Change in weight 3 3 0.9 60% 20% 2.1 40% 0.9 5.25 Intention School 1 to Treat ? Treated? Pupil 1 yes yes 4 Pupil 2 yes yes 4 Pupil 3 yes yes 4 A = Gain if Treated Pupil 4 yes no 0 B = Gain if not Treated Pupil 5 yes yes 4 Pupil 6 yes no 2 Pupil 7 yes no 0 ToT Estimator: A-B Pupil 8 yes yes 6 Pupil 9 yes yes 6 Pupil 10 yes no 0 A-B = Y(T)-Y(C) Avg. Change Y(T)= Prob(Treated|T)-Prob(Treated|C) School 2 Pupil 1 no no 2 Y(T) Pupil 2 no no 1 Y(C) Pupil 3 no yes 3 Prob(Treated|T) Pupil 4 no no 0 Prob(Treated|C) Pupil 5 no no 0 Pupil 6 no yes 3 Pupil 7 no no 0 Y(T)-Y(C) Pupil 8 no no 0 Prob(Treated|T)-Prob(Treated|C) Pupil 9 no no 0 Pupil 10 no no 0 Avg. Change Y(C) = A-B
37. ITT vs TOT  If obtaining estimate is easy, why not always use TOT?  TOT estimates Local Average Treatment Effect (LATE)  ITT may be policy relevant parameter of interest • For example, we may not be interested in the medical effect of deworming treatment, but what would happen under an actual deworming program. • If students often miss school and therefore don't get the deworming medicine, the intention to treat estimate may actually be most relevant.
38. CHOICE OF OUTCOMES
39. Multiple outcomes  How do we decide on which outcomes to focus on? • Only outcomes that are statistically significantly different?  The more outcomes you look at, the higher the chance you find at least one significantly affected by the program  How do you account for this? • Pre-specify outcomes of interest (JPAL-AEA web registry) • Report results on all measured outcomes, even null results • Correct statistical tests (Bonferroni)
40. EXTERNAL VALIDITY
41. Threat to External Validity:  Behavioral responses to evaluations  Generalizability of results
42. Threat to external validity: Behavioral responses to evaluations  One limitation of evaluations is that the evaluation itself may cause the treatment or comparison group to change its behavior • Treatment group behavior changes: Hawthorne effect • Comparison group behavior changes: John Henry effect  Minimize salience of evaluation as much as possible  Consider including controls who are measured at end-line only
43. Generalizability of Results  Depend on three factors: • Program Implementation: can it be replicated at a large (national) scale? • Study Sample: is it representative? • Sensitivity of results: would a similar, but slightly different program, have same impact?
44. Further Resources  Using Randomization in Development Economics Research: A Toolkit (Duflo, Glennerster, Kremer)  Mostly Harmless Econometrics (Angrist and Pischke)  Identification and Estimation of Local Average Treatment Effects (Imbens and Angrist, Econometrica, 1994).
Date
Regional Center Tag
Resource Type Tag
Language Type Tag
Image Thumb
3