Framing Impact Evaluation through Theory of Change

Submitted by jyuan@worldbank.org on Tue, 04/12/2016 - 17:18
Image
whatisevaluation

Course Overview

1.Framing Impact Evaluation through Theory of Change

2.Measuring impacts (outcomes, indicators)

3.Why randomize?

4.How to randomize?

5.Sampling and sample size

6.Threats and Analysis

7.Project from Start to Finish

 

What is Evaluation from clearsateam

 

Download File
Pdf as plain
What is Evaluation
1. Framing Impact Evaluation through Theory of Change
2. Course Overview 1. Framing Impact Evaluation through Theory of Change 2. Measuring impacts (outcomes, indicators) 3. Why randomize? 4. How to randomize? 5. Sampling and sample size 6. Threats and Analysis 7. Project from Start to Finish
3. What is Evaluation? Evaluation Program Evaluation Impact Evaluation
4. Evaluation Program Evaluation Impact Evaluation Program Evaluation
5. Monitoring and Evaluation Evaluation Program Evaluation Impact Evaluation Monitoring
6. Monitoring Evaluation Program Evaluation Impact Evaluation Program Evaluation
7. Components of Program Evaluation  Needs Assessment  Theory of change  Process Evaluation  Impact Evaluation  Cost Effectiveness  What is the problem?  How, in theory, does the program fix the problem?  Does the program work as planned?  Were its goals achieved? The magnitude?  Given magnitude and cost, how does it compare to alternatives?
8. Evaluation should usually be conducted: Externally and independe.. 0% 0% 0% 0% Externally and closely in... Internally Don’t know A. Externally and independent from the implementers of the program being evaluated B. Externally and closely integrated with program implementers C. Internally D. Don’t know
9. Who is this Evaluation For?  Academics  Donors • Their Constituents  Politicians / policymakers  Technocrats  Implementers  Proponents, Skeptics  Beneficiaries
10. DOES AID WORK?
11. Aid Optimists “I have identified the specific investments that are needed [to end poverty]; found ways to plan and implement them; [and] shown that they can be affordable.” Jeffrey Sachs End of Poverty
12. Aid Pessimists “After $2.3 trillion over 5 decades, why are the desperate needs of the world's poor still so tragically unmet? Isn't it finally time for an end to the impunity of foreign aid?” Bill Easterly The White Man’s Burden
13. How can Impact Evaluation Help Us?  Surprisingly little hard evidence on what works  Can do more with given budget with better evidence  If people knew money was going to programs that worked, could help increase pot for anti-poverty programs  Instead of asking “do aid/development programs work?” should be asking: • Which work best, why and when? • How can we scale up what works?
14. What do you think is the most cost-effective way to increase immunization rates? A. Community mobilization campaign B. Improve healthcare worker attendance C. Develop new vaccines, such as pneumococcal D. Hold special ‘immunization camps’ E. Incentivize parents to immunize their children 0% 0% 0% 0% 0% Community mobilization... Improve healthcare work.. Develop new vaccines, su.. Hold special ‘immunizati.. Incentivize parents to ...
15. Components of Program Evaluation  Needs Assessment  Program Theory Assessment  Process Evaluation  Impact Evaluation  Cost Effectiveness  What is the problem?  How, in theory, does the program fix the problem?  Does the program work as planned?  Were its goals achieved? The magnitude?  Given magnitude and cost, how does it compare to alternatives?
16. Identifying the problem NEEDS ASSESSMENT
17. The Need  Every year, between 2 and 3 million people die from vaccine-preventable diseases  Only 54% of 1-2 year olds in India receive the basic package of immunizations  In rural Rajasthan, this rate falls to 22%
18. The Goal  To increase the full immunization rate among children in rural Rajasthan
19. The Problem  In India, immunizations are offered for free… but the immunization rate remains low  Average household is within 2 kilometers of the nearest clinic  High absenteeism at government health facilities – 45% of Auxiliary Nurse Midwives are absent on any given workday
20. The Solution(s)
21. Really the Problem?  Cultural resistance, distrust in public health institutions– memories of Emergency India  People don’t value immunizations: short-term cost for long-term (and invisible) benefits  Limited income: parents can’t afford to take a day off
22. Alternative Solution(s)?
23. Devising a Solution  What is the theory behind your solution?  How does that map to your theory of the problem?
24. Blueprint for Change THEORY OF CHANGE
25. Theory of Change  Program Theory Assessment  Logical Framework (Log Frame)  Results Framework  Outcome Mapping
26. What is Theory of Change? “A theory of change is a road map of where we are going (results) and how we are getting there (process)”
27. Causal Hypothesis Q: How do I expect results to be achieved? A: If [inputs] and [activities] produce [outputs] this should lead to [outcomes] which will ultimately contribute to [goal]
28. Theory of Change Establish regular Incentives for full course Parents bring children to regular camp Increased immunization rates camps Parents believe camps are regular Supply-side limits on immunization Parents value incentive Parents do not value immunization Incentives regularly paid Camps provide immunizations
29. Assumptions Assumption: A necessary and positive external condition that should be in place for the chain of cause and effect (in an intervention) to go forward
30. Theory of Change Establish regular Incentives for full course Parents bring children to regular camp Increased immunization rates camps Parents believe camps are regular Supply-side limits on immunization Parents value incentive Parents do not value immunization Incentives regularly paid Camps provide immunizations
31. Results Levels Inputs Activities Outputs Outcomes Goal Resource s Actions Products and services KASBs Dev. status
32. Log Frame Objectives Hierarchy Indicators Sources of Verification Assumptions / Threats Impact (Goal/ Overall objective) Increased immunization Immunization rates Household survey Adequate vaccine supply, parents do not have second thoughts Outcome (Project Objective) Parents attend the immunization camps repeatedly Follow-up attendance Household survey; Immunization card Parents have the time to come Outputs Immunization camps are reliably open; Incentives are delivered Number of kg bags delivered; Camp schedules Random audits; Camp administrative data Nurses/assistants will show up to camp and give out incentives properly Inputs (Activities) Camps + incentives are established Camps are built, functional Random audits of camps Sufficient materials, funding, manpower Needs assessment Impact evaluation Process evaluation
33. Theory of Change: Product vs Process “Theory of change thinking is a habit not a product.”
34. Making the program work PROCESS EVALUATION
35. Components of Program Evaluation  Needs Assessment  Program Theory Assessment  Process Evaluation  Impact Evaluation  Cost Effectiveness  What is the problem?  How, in theory, does the program fix the problem?  Does the program work as planned?
36. Solving the Black Box Problem Low immunization rates Needs Assessment Intervention Intervention design/Inputs Final Black Box No increase in full immunization outcome
37. Identifying Theory Failure vs. Implementation Failure Successful intervention Inputs Activities Outputs Outcomes Goal Implementation failure Inputs Activities Outputs Outcomes Goal Theory failure Inputs Activities Outputs Outcomes Goal
38. Why is Theory of Change Important For evaluators, reminds us What is it? Example Components Assumptions Conclusion to consider process For programmers, it helps us be results oriented
39. Process Evaluation  Supply Side • Logistics • Management  Demand Side • Assumptions of response • Behavior Change?
40. Process Evaluation: Logistics  Establish camp • Hiring nurses and administrators • Installing temporary camp site • Procuring vaccines and other medical supplies  Organize incentive scheme • Identify viable incentive • Purchase kilos and dinner plate sets
41. Process Evaluation: Supply Logistics
42. Process Evaluation: Demand-side  Do parents visit the camps?  Do they come back?
43. Process was okay, so….  What happened to immunization rates?
44. Measuring how well it worked IMPACT EVALUATION
45. Did we Achieve our Goals?  Primary outcome (impact): did camps (or camps + incentives) raise the full immunization rates?  Also distributional questions: what was the impact for households who had come once vs. households who had never come?
46. Intervention Time Primary outcome Impact Counterfactual What is Impact?
47. How to Measure Impact?  What would have happened in the absence of the program?  Take the difference between what happened (with the program) …and - what would have happened (without the program) = IMPACT of the program
48. Non-random Treatment and Comparison Groups
49. Non-Random Treatment and Comparison Groups HQ
50. Constructing the Counterfactual  Counterfactual is often constructed by selecting a group not affected by the program  Randomized: • Use random assignment of the program to create a control group which mimics the counterfactual.  Non-randomized: • Argue that a certain excluded group mimics the counterfactual.
51. How Impact Differs from Process?  When we answer a process question, we need to describe what happened.  When we answer an impact question, we need to compare what happened to what would have happened without the program
52. The “gold standard” for Impact Evaluation RANDOMIZED EVALUATION
53. Random Sampling and Random Assignment Randomly sample from area of interest
54. Random Sampling and Random Assignment Randomly sample from area of interest Randomly assign to treatment and control Randomly sample from both treatment and control
55. Immunization Example Target Populatio n (134) Not in evaluation (0) Evaluation Sample (134) Total Population (700+ villages) Random Assignment Camps (30) Camps + Incentives (30) Control (74)
56. Impact 6% 17% 38% 40% 35% 30% 25% 20% 15% 10% 5% 0% Control Camps Camps + Lentils
57. Making Policy from Evidence  National scale-up? • How representative is rural Rajasthan? (Recall: 22% vs. 44% nationally) • Same barriers to immunization?
58. Evidence-Based Policymaking COST-EFFECTIVENESS ANALYSIS
59. Costs per fully Immunized Child Rs. 2202 Cost of Incentive Cost of Camp Rs. 372 Rs 730 + Immunization Camps Camps + Incentives
60. Cost-Effectiveness Diagram
61. When is a good time to do a Randomized Evaluation? A. After the program has begun and you are not expanding it elsewhere B. When a positive impact has been proven using rigorous methodology C. When you are rolling out a program with the intention of taking it to scale D. When a program is on a very small scale e.g one village with treatment and one 0% 0% 0% 0% When a positive impact ... After the program has b... When you are rolling out.. When a program is on a ...
62. When to do a Randomized Evaluation?  When there is an important question you want/need to know the answer to  Timing--not too early and not too late  Program is representative not gold plated • Or tests an basic concept you need tested  Time, expertise, and money to do it right  Develop an evaluation plan to prioritize
63. When NOT to do an RE  When the program is premature and still requires considerable “tinkering” to work well  When the project is on too small a scale to randomize into two “representative groups”  If a positive impact has been proven using rigorous methodology and resources are sufficient to cover everyone  After the program has already begun and you are not expanding elsewhere
64. Developing an Evaluation Strategy  Start with a question  Verify the question hasn’t been answered  State a hypothesis  Design the evaluation  Determine whether the value of the answer is worth the cost of the evaluation  With key questions answered from impact evaluations, process evaluation can give your overall impact  If you ask the right question, you’re more likely to care  A few high quality impact studies are worth more than many poor quality ones
65. Components of Program Evaluation  Needs Assessment  Theory of Change  Process Evaluation  Impact Evaluation  Cost Effectiveness  What is the problem?  How, in theory, does the program fix the problem?  Does the program work as planned?  Were its goals achieved? The magnitude?  Given magnitude and cost, how does it compare to alternatives?
Date
Regional Center Tag
Resource Type Tag
Language Type Tag
Image Thumb
S