Introduction to Cost-effectiveness Analysis

Submitted by on Tue, 03/21/2017 - 13:41

Suppose the Department of Health and Family Welfare has conducted a rigorous evaluation to determine the impact of a maternal and child health program on a range of health outcomes. The results of the evaluation show that the program has had a significant, positive and high magnitude effect on a number of important health outcomes. The Health and Family Welfare Department is pleased to know that the program is effective, but wants to know what policy action to take given this positive impact. Should the program be continued, stopped, modified, or scaled up?

Consider another example. A community’s water supply has been contaminated with effluents leading to a large incidence of diarrhea within the population. An international aid organization has come up with two very different strategies to tackle this problem. One project manager in the organization is advocating for investments in modern water and sanitation infrastructure, including sewage and a piped-water supply, while another manager has proposed a distribution system where households are given free chlorine tablets to treat their own water at home. Through a randomized impact evaluation, these two methods were shown to be equally effective – each reducing diarrheal incidence by 80 percent. Which intervention should the organization implement?

Before we can answer both of these questions, we need to know additional information related to the costs of the program. Perhaps the maternal and child health program, while effective, is actually so costly that government budgets would not be able to sustainably afford a statewide scale up. In the second example, it is highly likely that modern infrastructure investments in an otherwise remote village would be prohibitively expensive. In this case, distributing chlorine tablets may well be the better program to implement.

Any program or policy we introduce has opportunity costs. In other words, there are alternative ways to spend money and time. It is not always enough to know that a policy or program has a positive impact on the lives of the poor; it is helpful to know whether the program is the best use of limited resources. Users of evidence, be it government, NGOs, donors, or other organizations that make evidence-informed policy, would like to consider not only whether or not a program had a positive impact, but also whether the program, when compared to its costs, is of sufficiently good value. The term “sufficiently good value” can be an abstract concept when comparing one program in isolation, but if we have information on the impact and costs of other related programs, we can begin to compare across programs to determine which yields the greatest value for money. One way of doing this is to conduct a Cost-Effectiveness Analysis, which is a method of summarizing complex programs in terms of a simple ratio of impacts to costs.

This manual introduces the concept of a Cost-Effectiveness Analysis (CEA) and provides practical steps to conducting this method of program evaluation.

Download File
Pdf as plain
introduction to
cost-effectiveness analysis
the abdul latif jameel poverty action lab (j-pal)
clear south asia
front cover: j-pal/ipa
Suggested Citation: Jetha, Qayam. 2017. "Introduction to Cost-Effectiveness Analysis.” J-PAL South Asia and
CLEAR South Asia.
Design: Amanda Kohn
I. Introduction to Cost-Effectiveness Analysis 4
The Importance of Program Cost Data 5
What is Cost-Effectiveness Analysis and Comparative 5
Cost-Effectiveness Analysis?
Example: Diarrheal Disease Cost-Effectiveness Analysis 6
When to Conduct Cost-Effectiveness Analysis 7
Cost Benefit Versus Cost-Effectiveness Analysis 7
II. Conducting A Cost-Effectiveness Analysis 9
Step 1: Quantifying Impact 10
Step 2: Quantifying Cost 11
Defining the Program 11
Identifying Ingredients 12
Gathering Unit Cost Information 14
Standardizing Costs Across Programs 15
Putting Together Costs and Benefits 16
III. Example: Immunization Incentives Program 17
Additional Resources 20
Table of Figures
Figure 1. Diarrheal Disease CEA, Incidents Averted per US$1,000 Spent 6
Figure 2. Cost-Effectiveness Analysis vs Cost-Benefit Analysis 7
Figure 3. Percentage of Children Aged 1-3 Years Fully Immunized by 18
Treatment Status
Figure 4. Disaggregation of Camps and Camps plus Incentives Program 19
Cost Data
Figure 5. Costs per Fully Immunized Child 19
table of contents
aude guerrucci
introduction to
cost-effectiveness analysis
aude guerrucci
pover t yac t 5
the importance of program
cost data
Suppose the Department of Health and Family Welfare has
conducted a rigorous evaluation to determine the impact of a
maternal and child health program on a range of health outcomes.
The results of the evaluation show that the program has had a
significant, positive and high magnitude effect on a number of
important health outcomes. The Health and Family Welfare
Department is pleased to know that the program is effective,
but wants to know what policy action to take given this positive
impact. Should the program be continued, stopped, modified,
or scaled up?
Consider another example. A community’s water supply has been
contaminated with effluents leading to a large incidence of diarrhea
within the population. An international aid organization has come
up with two very different strategies to tackle this problem. One
project manager in the organization is advocating for investments
in modern water and sanitation infrastructure, including sewage
and a piped-water supply, while another manager has proposed a
distribution system where households are given free chlorine
tablets to treat their own water at home. Through a randomized
impact evaluation, these two methods were shown to be equally
effective – each reducing diarrheal incidence by 80 percent. Which
intervention should the organization implement?
Before we can answer both of these questions, we need to know
additional information related to the costs of the program. Perhaps
the maternal and child health program, while effective, is actually
so costly that government budgets would not be able to sustainably
afford a statewide scale up. In the second example, it is highly likely
that modern infrastructure investments in an otherwise remote
village would be prohibitively expensive. In this case, distributing
chlorine tablets may well be the better program to implement.
Any program or policy we introduce has opportunity costs. In
other words, there are alternative ways to spend money and time.
It is not always enough to know that a policy or program has a
positive impact on the lives of the poor; it is helpful to know whether
the program is the best use of limited resources. Users of evidence, be
it government, NGOs, donors, or other organizations that make
evidence-informed policy, would like to consider not only whether or
not a program had a positive impact, but also whether the program,
when compared to its costs, is of sufficiently good value. The term
“sufficiently good value” can be an abstract concept when comparing
one program in isolation, but if we have information on the impact
and costs of other related programs, we can begin to compare
across programs to determine which yields the greatest value for
money. One way of doing this is to conduct a Cost-Effectiveness
Analysis, which is a method of summarizing complex programs in
terms of a simple ratio of impacts to costs.
This manual introduces the concept of a Cost-Effectiveness Analysis
(CEA) and provides practical steps to conducting this method of
program evaluation.
what is cost-effectiveness
and comparative costeffectiveness
Cost-Effectiveness Analysis (CEA) – A CEA shows the
impact of a program on one outcome measure for a given
cost incurred. To calculate this, take the impact of a program
on a particular outcome (e.g. percent reduction in the
incidence of diarrhea) and divide by the total cost of the
program. Formally, a cost-effectiveness analysis is calculated
using the following equation:
CE Ratio=
(Total Impact of the Program on a Specified Outcome)
(Total Cost of Implementing the Program)
The resulting cost-effectiveness ratio is a statistic that
describes the number of cases of diarrhea prevented per
rupee spent. Or, if the ratio is flipped, the amount it costs
to reduce the incidence of diarrhea by one case. Both
measures give an indication of the value for money of a
particular program.
Comparative Cost-Effectiveness Analysis – Takes
multiple programs (possibly from different contexts) and
compares them using the same ratio of costs to impact. This
ratio, calculated across a range of alternative programs that
address the same policy goal, conveys the relative impacts
and costs of these programs in an easily understandable
and intuitive way. In this manner, policymakers can ask:
per US dollar spent, how much do each of these programs
reduce diarrhea? Comparing CEA calculations across
multiple programs in this manner provides an indication of
the program that gives the most “bang for the buck,” which
can help government or other organizations make the most
out of limited budgets.
1 Dhaliwal, Iqbal, Esther Duflo, Rachel Glennerster, and Caitlin Tulloch. 2013.
“Comparative Cost-Effectiveness Analysis to Inform Policy in Developing
Countries: A General Framework with Applications for Education.” Education
Policy in Developing Countries. 285-338 | Available online.
pover t yac t 6
It is important to note a few defining features of a CEA:
1. Before a CEA can be conducted, some measure of the
program’s impact must be known. Therefore, a CEA usually
follows from a rigorous impact evaluation.
2. It is important to emphasize that the relevant cost measure is
not the cost of the evaluation; it is strictly the total aggregate
cost of all components of implementing the program itself.
3. A comparative CEA provides an estimate of the relative
effectiveness of a variety of programs on one outcome and
does not rely on any judgements about the monetary value of
the outcomes.
4. A CEA of a single program can only provide an estimate
of how much effect that program generates per currency
unit spent. While this can be a useful starting point, rarely
does it provide adequate information to base investment
decisions. On the other hand, comparative CEA can help
inform investment decisions by allowing users to compare the
value of a particular intervention with other policy alternatives.
5. Any comparative CEA relies on comprehensive and consistent
calculation of costs and effects across the studies included.
The ability to draw comparisons across different projects requires
comparability between the ways program costs are measured. To
achieve comparability, a number of assumptions must be
made on how to measure costs and standardize impacts.
6. CEA is just one input into the policymaking process.
Comparative CEA provides policymakers with a ranking
of how cost-effective various programs have been in the
specific context in which they were evaluated. CEA should
not be interpreted as a promise for exactly how costeffective
a program model will be in every context.
example: diarrheal disease cost-effectiveness analysis
The CEA depicted below shows an example of a comparative CEA that the Abdul Latif Jameel Poverty Action Lab (J-PAL) undertook to
compare the reduction in diarrheal disease incidents per US$1000 spent across five different programs implemented across various contexts1.
These numbers and assumptions are based on a report from 2015, which may be out of date. Note that each of the five interventions is compared
across a single outcome measure: reduction in diarrheal incidents. While a program may actually show impact on a number of different
outcomes, a CEA ensures comparability by only focusing on one key outcome.
Changing Behavior Source Improvements
figure 1. sensitivity to population density
free chlorine
dispensers at
water sources
free home delivery
of chlorine
encasing water
sources in
free home delivery
of chlorine
promotion with
free soap
Chlorine Treatment
pover t yac t 7
Three of the five interventions involve dispensing chlorine treatment,
one program provides improvements to the water source, and one
intervention focuses on changing health behavior through the
promotion of hand-washing with soap. The analysis shows that
dispensing chlorine for free at community water sources was the most
cost-effective way of those tested to prevent diarrheal disease, leading
to 494 fewer diarrhea incidents for each US$1,000 spent on the
program. In comparison, free home delivery of chlorine treatment
prevented anywhere from 115 to 333 diarrheal incidents per US$1,000
spent. A hand-washing promotion intervention in Pakistan had the
lowest impact per US$1,000 spent, reducing diarrheal disease
by 71 incidents. This analysis provides useful information for a
government department on what is the best way to maximize
reductions in diarrheal incidence for a fixed pool of resources.
There are a number of caveats and assumptions behind these
figures, which we will discuss in detail throughout the rest of
this manual. But even when we change these assumptions, some
programs consistently generate a much greater reduction in
diarrhea per US$1,000 spent than other programs.
when to conduct a costeffectiveness
Cost-effectiveness calculations are most useful when:
1. You have a specific outcome measure you want to
affect and there are many possible interventions that
address this goal.
Many solutions to the problem of low student attendance have been
proposed. For example, J-PAL affiliates have evaluated a number
of programs that aim to increase student attendance such as
conditional cash transfers, merit scholarships, free primary school
uniforms, menstrual cups for teenage girls, information sessions
on the returns to education, midday meals, iron supplementation
programs, and many others. In this case, a comparative CEA
could answer the question: out of these programs, which one
increases attendance the most per a given cost.
2. You want to demonstrate that a non-obvious program
is a good idea.
One non-obvious program that has an impact on school
attendance is providing children with deworming pills. Due to
the low cost of these pills, deworming has been found to be
incredibly cost-effective, increasing student participation by 12.5
years per US$100 spent on the program. This finding provided
the important information that lead to the scale-up of mass
school-based deworming programs throughout the world.
3. You want to understand how the cost-effectiveness
of a program could vary with contextual and
implementation factors.
Say an NGO focused on education for disadvantaged children
learned about the cost-effectiveness of school based deworming
in Kenya, and as a result wants to implement a similar intervention
across the Indian state of Tamil Nadu. However, the NGO is not
sure whether procuring the deworming pill would be more expensive
in Tamil Nadu and thus negate the cost-effectiveness of the program.
To check whether this is the case, the NGO can use the Kenyan
deworming cost-effectiveness data to conduct what is known as a
sensitivity analysis.
A sensitivity analysis involves repeating the initial analysis but
substituting alternate decisions or a range of values to understand
how results would differ according to contextual factors. In
this case, the NGO could calculate the same CE ratio as the
Kenyan study but to account for the ambiguity on pill price
conduct three separate analyses. The first analysis, the worstcase
scenario, could use a very high pill price, while the second
and third analyses could use a middle and low price respectively.
Comparing the CE ratios of these three sensitivity analyses with
CE ratios for other education programs that the NGO could
implement, will provide valuable information as to whether
the deworming program remains comparably cost-effective at
different price levels.
cost benefit versus costeffectiveness
Cost-Benefit Analysis (CBA) is another method to estimate
the value of a program. The key difference is that a CBA compare
the monetary value of all program benefits to the costs,
whereas a CEA shows the impact of a program on a single
outcome relative to cost. While a CEA gives us an estimate of
the effectiveness of the program versus its cost, CBA monetizes
the benefit due to a program and compares this with the cost.
figure 2. cost-effectiveness analysis vs
cost-benefit analysis
Cost-Benefit Analysis
Monetized dollar value of
all benefits of the program
Total dollar cost of
the program
Cost-Effectiveness Analysis
Non-monetized impact of
the program on one outcome
Total dollar cost of
the program
pover t yac t 8
Because CBA compares the monetary value of benefits, one
advantage of using this method is that it makes it easier to assess
a program with multiple outcomes. For example, suppose the
corporate social responsibility arm of a large private organization
wants to know something about the return of investing in an
unconditional cash transfer program targeted at female primary
school students. An impact evaluation of the cash transfer shows
that the program has an impact on students’ test scores as well as
a smaller impact on student’s health. A CBA will make a series of
assumptions that determine a monetary value for the increase in
test scores and the improvements in health, add the two values,
and compare this total benefit to the total cost of the program.
Another advantage of CBA is that by putting both costs and
benefits on a currency scale, we can identify not only a relative
but an absolute judgement of whether the program is worth the
investment. A CBA ratio of a single program that is less than one
implies that the program may not be worth investing in, as the
costs exceed the total benefits. This means that unlike CEA, CBA
does not necessarily have to compare across different programs
to give some idea of whether a program is a good investment.
The downside of CBA is the large number of detailed assumptions
needed to estimate a monetary value for the different benefits of
the program. In our example, how can we estimate the monetary
value of increasing students’ test scores? Similarly, for a program
that reduces child mortality, a CBA will require that a monetary
value be placed on saving a life. Different organizations or individuals
may have very different views on assigning a monetary value to
certain benefits. For the sake of simplicity and to remove the
value judgement about the relative weight given to certain benefits,
J-PAL conducts CEAs.
aude guerrucci
conducting a
cost-effectiveness analysis
arvind eyunni | pratham
pover t yac t 10
As outlined in the sections above, conducting a CEA requires
quantifying two pieces of information: the program impact
and the program cost. Again, the key concern when conducting
a comparative CEA is ensuring that the way program costs and
program impacts are measured is consistent across the various
programs included in the analysis. This chapter describes how to
conduct a CEA, by going through the steps required to quantify
the impact (the numerator of the CE Ratio) and the cost (denominator
of the CE Ratio) of a program. As we will see, conducting a CEA
involves making many decisions, assumptions, and judgement calls
about how to quantify costs and impact. What follows are the steps
and practices employed by J-PAL.
step 1: quantifying impact
Before a CEA can be conducted, a program’s impact must be
estimated through the use of a rigorous impact evaluation. The
fact that the impact evaluation serves as the precursor to CEA
underscores the importance of carrying out impact evaluations
to generate precise estimates of the effect of various programs.
Without a large body of rigorous evidence generated through
impact evaluations it would be impossible to compare the relative
cost-effectiveness of programs, thereby constraining our ability
to make evidence informed policy decisions.
For every program included in a comparative CEA, it is not
strictly necessary that an organization or agency itself conduct
or commission the impact evaluation. Instead, an organization
can draw from previous impact evaluations of programs inside
and outside its operating location. The organization must decide
on a set of standards and requirements for determining which
impact evaluations warrant inclusion in their CEA. The key is to be
transparent in these inclusion criteria. Some example criteria are
the following:
• Studies that are of sufficiently large scale/sample. E.g. only
include evaluations that are drawn from samples that are
representative of at least a district.
• Studies which are sufficiently rigorous in terms of method.
E.g. only include randomized evaluations into the CEA.
• Studies in a particular geography. E.g. only include
evaluations of Indian programs.
• Studies that have been externally replicated, i.e. where the
same program has been examined with a different dataset
drawn from a different context or population. E.g. only
include evaluations of programs that have been externally
replicated at least twice.
For example, an organization might want to specify that their CEA
will compare all impact evaluations of agricultural interventions
in Southeast Asia whose main outcome of interest is crop yields
and which use either a randomized evaluation or a regression
discontinuity design.
For the purpose of this chapter, we will assume that an impact
evaluation has been completed on a particular program, and we
have a positive and statistically significant impact estimate on an
outcome of interest. At J-PAL, if a program has a statistically
insignificant impact on an outcome, that program remains part
of the CEA but no cost-effectiveness ratio is calculated. Once an
impact evaluation has been completed and we have an estimate
of program impact, we are part way to having the denominator
value of our CE Ratio. What remains to be done is to aggregate
total impact, address spillover effects, and identify programs
that achieve multiple impacts.
1. Aggregating Total Impact.
Often, the impact estimate of a program that is found through
an impact evaluation is calculated in a manner that is difficult
to interpret or is in terms of a single unit or a single beneficiary.
For example, suppose a government ministry commissions an
evaluation of a Midday Meal Scheme. Suppose that the evaluating
agency finds that the program increases child attendance by 11
percentage points per year, from 60 percent to 71 percent. To
get a simple and easily interpretable measure of program impact,
we will need to convert this percentage point increase into a
figure such as the number of additional school days or years for
all children who benefited under the program.
If there are 180 days in the school year, then for one child the program
increases the number of school days attended by approximately
twenty (from 108 days without the Midday Meal Program (180*0.6)
to 128 days with the Midday Meal Program (180*0.71)). Now,
to get the total impact, the individual student impact must be
extrapolated to the number of children that participated in the
evaluation. Say that the program affected 10,000 children across
100 schools. In this case the total impact of the program would
be 200,000 additional school days attended. If the program ran
for two years, then the impact would have to be multiplied by
two (400,000 additional school days).
In its simplest form, calculating the total impact of a program
follows this formula:
Total Impact = Impact (per unit) × Sample Size
× Program Duration
In the case of the midday meal example:
Total Impact = 20 days × 10,000 children × 2 years
When calculating the program costs associated with the Midday Meal
Program, make sure that the costs are associated with the same sample
size and same duration as used to calculate the total impact. In our
example, we will need the cost of implementing the Midday Meal
Scheme for those 10,000 children in 100 schools over two years.
2 For more information on spillover effects: see Duflo, Esther, Glennerster, Rachel,
Kremer, Michael. "Using Randomization in Development Economics Research: A
Toolkit." Handbook of Development Economics.
pover t yac t 11
questions to consider
• What is the specific intervention or program the
impact evaluation assesses?
• Is the program an extension or a modification of an
existing intervention?
• What is the comparator case?
• What time period did the evaluation cover?
2. Spillover Effects.
In some cases, the effects of a particular program may spill over
onto nonrecipient populations. For example, suppose private school
students come to public schools to receive the midday meal. Or
consider an anti-malarial program that provides free insecticidetreated
bed nets to households within a certain district. Since
an individual’s risk of contracting malaria depends on the overall
prevalence of the disease where they live, decreasing malaria rates
within one district may have positive health spillovers on a
neighbouring district2.
When calculating a CEA for a program that contains a spillover
effect, one must decide whether or not to include an estimate of
this spillover in the total impact of a program. While there are no
right or wrong answers, a useful thought experiment to consider
when making this decision is to ask whether the spill-over benefit
would be present in a scaled up version of the program. For example,
if the anti-malarial bednet program was scaled up across all
households in Tamil Nadu, then there would no longer be a district
that does not receive the program and thus no additional spillover
benefit. In this case, it makes sense to forgo the spillover effect in the
calculation of program impact.
3. Programs Achieving Multiple Impacts.
As discussed above, by definition a CEA compares the costs and
impacts of a set of programs on one outcome. However, a program
may have multiple impacts on a variety of important outcomes.
A conditional cash transfer might increase student attendance,
but also have impacts on health outcomes and an individual's
empowerment or confidence. A midday meal program may increase
both student attendance and nutrition outcomes. Unfortunately, a
CEA does not have a way to include multiple outcomes. At J-PAL,
in order to call attention to this, we make a note in the CEA if a
program has impacts on a number of different outcomes. While
this is far from an ideal solution, it does highlight that the program’s
other impacts should also be taken into account.
step 2: quantifying cost
When quantifying costs, the goal is to include cost information on
all the “ingredients” or components of a program to get a sense of
how much it costs to implement (or how much it would
cost to replicate the program). Quantifying these costs can
appear deceptively simple, particularly when aggregate cost data
(such as the entire budgetary total or total personnel costs) are
reported. However, without an adequate explanation of what the
budget includes and over what time period, using this cost data
can lead to an erroneous estimate of the total cost of the program.
To illustrate, suppose a program evaluation consulting firm
is calculating a CEA of a computer-assisted learning program
implemented by the School Education Department that provides
underperforming schools in a particular district with computers
and teacher training on how to effectively integrate the computers
into the classroom. The School Education Department gives the
consultant an aggregate cost associated with implementation of
the program, but fails to give a detailed breakdown of what is
encapsulated within this budget. The consultant has no way of
knowing whether the cost data accurately reflects the total cost of
the program as it was implemented. Perhaps the aggregate figure
does not include the cost of computers as computers were leftover
from a previous government program or schools already had
computers. Maybe the program budget failed to incorporate the
opportunity cost to teachers for their time spent in computer training
or perhaps the aggregate figure did not include the wages paid to
the lead trainer for his time training the group of trainers since
the lead trainer was already a government employee. Alternatively,
maybe the budget included compensation for the lead trainer, but
not the costs of the facilities and materials for the training.
As we will see, the costs of goods and services provided for free, user
costs of beneficiaries giving their time, and costs associated with all
aspects of staff training, are all important to keep in mind. Failure to
include the costs of staff training for one program while doing so for
another will lead to an inaccurate comparison and could lead
policymakers to misallocate resources. To ensure consistency in cost
calculations across programs, it is necessary to obtain detailed,
granular cost data and be systematic in the way costs are collected.
A helpful way to do this is to use the ingredients approach
to cost collection. This method is a way to ensure that all relevant
costs have been included in an analysis. It requires a complete
and accurate description of the program. Then, we can generate
a complete listing of all the necessary resources or ingredients
required (both items and amounts) for the program to achieve its
impact. After a systematic specification of a program’s ingredients,
unit costs are gathered. Finally, following identification and
valuation, the ingredients can be added together to produce a total
cost figure. We outline the different components of the ingredients
approach in more details below.
The ingredients approach involves four steps.
defining the program
pover t yac t 12
Before adding up the costs of ingredients, one must have a clear
understanding of what is meant by “the program”. Defining the
program can be a tricky endeavor because many evaluations
examine different variations of an already existing program or
simply add on to existing government or NGO infrastructure.
In this case, a useful way of thinking about the constituents of
the program is to imagine what the intervention would look like
and what costs would be involved if it were being replicated in a
new context. This thought exercise is another way of getting at the
notion of a comparator case, which is the starting situation
against which the program is being compared. Underlying all
cost-effectiveness calculations is an implied basic level of costs
and benefits that would exist even in the absence of the program.
Recall that in the terminology of impact evaluation, the level
of benefits that would exist without the program is called the
counterfactual3. Likewise, the level of costs that would exist
had the program not been implemented is the comparator case.
CE Ratio=
[outcome with program] – [outcome that would have existed
without program (counterfactual)]
(costs with program] – [costs that would have existed without
program (comparator case)]
An understanding of the comparator case allows one to parse out
the incremental costs of the program itself. In some cases, a
program starts from a comparator case of zero. For example,
a program implemented by a Department of Energy to hire an
organization to distribute solar power LED-lights to low income
households can be considered to start from zero, if there is no
preexisting government infrastructure that the program is adding
to, modifying, or extending. Supposing the costs in the absence
of the solar energy program were zero, then all the costs of the
program itself should be included in the CEA.
A program that provides merit scholarships to public school
students based on their standardized test grades is an example of
a program that does not start from a comparator case of zero. In
this case, the verification of test scores and selection of winners
would be done by school administrators, whose salaries would be
paid even if the program did not exist. Since administrators would
likely be present in most contexts in which the program is replicated,
it is reasonable to assume that the costs of school administrators
would be borne in the comparator case and thus not be included
as an incremental cost of the merit-scholarship program itself.
As a final example, let’s consider the comparator case of the
computer assisted learning program mentioned earlier. Suppose
that all of the underperforming schools in the district already had
computers, but they were underutilized. Technically, the comparator
case in this example would include computers that were already
present in the schools, meaning that the cost of the computers
would not show up in the program’s cost. This analysis answers the
question: “what is the cost-effectiveness of the computer-assisted
learning program in contexts where schools already have computers?”
However, the existence of the computers prior to the program
should be clearly stated in the assumptions, as one may still
consider including the cost of the existing computers in the CEA
if conducting a sensitivity analysis for another context. If the
Education Department were looking to scale up the program to a
new district where schools did not have computers, the relevant
question would be: “what is the cost-effectiveness of the computerassisted
learning program if the school district needed to buy
computers?” It is possible to do the cost calculation either way –
with or without the cost of computers – but the general rule is to
include the marginal costs of the program as it was implemented
and clearly state assumptions and details about the context.
identifying ingredients
Some important cost categories are as follows:
1. Program Administration Costs
2. Targeting Costs
3. Staff Training Costs
4. User Training Costs
5. Implementation Costs
6. User Costs
7. Averted Costs
8. Monitoring Costs
A few notes before describing each category in more detail:
• Programs differ substantially in terms of resources required
and as such, some cost categories may not be necessarily relevant
to the program in question or conversely additional cost categories
may need to be added. The listing of these eight categories is
only intended as a framework for thinking methodically about
the possible costs of a program.
• Sometimes the numerical cost figure will not exist for a certain
line item under a cost category and an estimate will have to
questions to consider
• What are the necessary ingredients of the program?
• Is the program or intervention saving any cost
formerly incurred?
• Are there any donated goods and services provided
for free?
• Does the program impose on beneficiaries any implicit or
explicit cost?
3 For a fulsome overview of the counterfactual and the intuition of impact evaluation,
see the "Intro to Evaluations" section of the J-PAL website: https://www.
pover t yac t 13
be made. In these cases, it is important to make explicit the
assumptions used and if possible also include a minimum and
maximum value that the cost figure may take.
Program Administration Costs: This major cost item
represented in this category is staff costs. We want to include
all costs associated with hiring the staff involved in the
implementation (not the evaluation!) of the program. Be sure
to include cost values for full-time salaried staff as well as nonsalaried
full-time staff. Also included in this category are any
capital costs incurred for the purchase of facilities, utilities, and
other materials needed to support program implementation.
Targeting Costs: Were there any costs associated with identifying
the location of the program catchment area or the location of the
beneficiaries? Did the implementer have to raise popular awareness
of the existence of the program? Was any money spent on identifying
beneficiaries themselves? If the answer is "yes" to either of these
questions, the program will incur targeting costs. Examples of
costs in this category could include the costs of doing a census, a
participatory rural appraisal, a proxy means test survey, a door-todoor
informational campaign, putting up flyers or other marketing
materials, etc.
Staff Training Costs: Was there any cost associated with training
the staff responsible for implementing the program? For example,
were workers at a government department trained in using a new
technology that they would then teach program beneficiaries to
use? Staff training costs may comprise remuneration for external
trainers (not full-time staff) including their fees, labor, lodging, and
transport; costs for facilities, materials, and food; and for non-full
time staff the wages that trainees would earn while participating
at the training (costs for full time staff would already be captured
under program administrative costs).
User Training Costs: Are program beneficiaries required to
receive any training? If so, relevant costs to include are: capital
costs of the event (facilities, materials, food, etc.), wages,
transportation, and lodging for non-full time staff (again, fulltime
staff are covered under program administrative costs).
Compensating Users’ Time: J-PAL includes in the User
Training Cost category an estimate of a beneficiary’s opportunity
cost for attending a required training (including significant
amounts of travel time). While this is not a direct accounting cost
for a government or implementing organization, the time spent
in training does represent a real cost to the user and is therefore
included. To estimate the value of forgone income, we either use
an average measure of household income that is collected in the
evaluation itself, or when this data is not available, we use the local
wage rate.
Including a compensating value for the time beneficiaries spend
contributing to the program is another example of a discretionary
choice to be made by the researcher. Like all judgement decisions
in a CEA, either choice is acceptable so long as the researcher can
explain their reasoning and the decision is consistent across all
of the different programs within the same CEA.
Implementation Costs: For many projects, implementation
will be the largest and most important category. Captured here
would be all costs directly associated with the implementation
of the program, such as asset or in-kind transfers, vouchers,
incentive or award payments, holding meetings, developing and
printing material, etc.
Goods and Services Provided for Free: Another question is
whether or not to include the cost of goods and services that are
provided for free. The answer depends on how the CEA will be
used. As an example, say an NGO implemented a program to
provide households with fully subsidized improved cook stoves.
As part of the program, the NGO had planned on contracting
an organization to go door-to-door to maintain and repair cook
stoves, but instead members of the village decided to mobilize
and volunteer their time and labor to fix damaged and nonfunctional
cook stoves within their community. If the objective
is to examine the costs to society as a whole, then one should
include an estimate of the market cost of the services provided
by the volunteers. Furthermore, another consideration is to ask
whether the inputs provided for free in a particular context would
be made available at no cost if the program were to be scaled up
in a different context. If one can rationalize that even in a different
context a volunteer cadre would form to maintain the cook stoves,
then it may make sense to not include this cost item.
Transfer Payments: Monetary transfer payments, such as a cash
stipend provided in conditional and unconditional transfer programs,
represents a redistribution of wealth from one party to another
and not a change in the size of the total resource pie. If the CEA
is being conducted from a broader social planner standpoint,
then we should not include transfers as a cost. However, if we
are concerned with what the program costs to implement then
transfers should be included under the implementation cost
ingredient category.
User Costs: This includes the cost that a particular program
imposes on beneficiaries, including the cost of their time (apart
from beneficiaries’ opportunity costs for attending a training
– as this is collected under User Training Costs). If a program
partially subsidizes a good or service, the remainder cost borne
by the beneficiary is included under this category. These costs
can be divided into new costs and averted user costs. Averted
user costs would occur if program beneficiaries worked fewer
hours as a result of the program or if beneficiaries used fewer
capital goods. As the costs to labor and capital are no longer
incurred due to the program, they must be reported as negative
costs or program savings.
Averted Costs: Were there any existing programs or nonuser
costs that were discontinued or reduced in size as a result
of the program? For example, say a government’s agricultural
department had a program to send qualified trainers to farms to
answer farmers’ questions and provide extension services. This
year, however, a new intervention was introduced by the
Agricultural Department that provided the same service over
cell phones, thus rendering the trainer program obsolete. In this
pover t yac t 14
case, the labor cost savings of the now obsolete trainers would
enter the CEA as a positive saving. Note that if a new program
completely supplants an existing one, it is likely that averted costs
will arise across many different cost categories such as program
administration, implementation costs, staff training, etc. The
monetary value of these averted ingredients should be included
under this category.
Monitoring Costs: Was there any expenditure related to
overseeing, monitoring, or measuring the progress of beneficiaries
or staff? The answer to this question would be "yes", if for example,
administrators must monitor whether beneficiaries meet the
conditionality imposed in a cash transfer program, or if an education
program employs a group of monitors to conduct periodic spot checks
to ensure that teachers are implementing the correct teaching
materials. If a program does have monitoring outlays, be sure to
include the cost of monitoring materials (e.g. cameras, questionnaires,
etc.), the cost of aggregating and analyzing the monitoring data
(again, this is separate from analyzing data from the evaluation!),
and the labor of part-time staff involved in monitoring activities,
as well as their conveyance, and accommodation (costs for full time
staff are included in the program administration category).
gathering unit cost information
Once we know the program’s ingredients and the specific cost
items that are needed under each ingredient category, we need
to collect the relevant data. The cost collection process will be
quite different depending on whether an organization is calculating
the cost-effectiveness of one of their own evaluations or if the
organization is including a previous study done by other
researchers. If it is the latter, and if the study does not include a
cost-effectiveness calculation, the organization will have to define
the program, identify its ingredients, and identify costs on behalf
of the researchers. This can be a difficult task, especially if the
program is conducted in a foreign context. In such a case, most,
if not all, cost figures will have to be gathered through secondary
data sources and be supplemented with assumptions and
estimations, which will decrease the reliability of estimates and
potentially jeopardize the validity of CE ratios. It is a good
practice to try to contact the researchers for more clarification
before attempting to take this on.
questions to consider
• Where will you get your data on each of the cost items?
• At what stage of the program should you collect this data?
• Is the CEA a prospective or retrospective analysis?
• Who should you work with to collect this information?
Collecting accurate cost data will be much easier for those evaluations
that an organization itself is conducting or commissioning.
However, it is crucial that cost collection is seen as an integral
component of the evaluation and not as an afterthought. Cost
collection should be explicitly included in any terms of reference
that is created and it should be built in as a general expectation
that the evaluator will be collecting this program data for the
duration of the evaluation.
The evaluator should work with the implementing organization
or government department to collect these costs. The implementing
agency may have many of the main cost estimates readily available
in the form of program budgets and the main work required will
be ensuring that the data can be shared. Cost data from program
budgets can be supplemented with secondary data sources (e.g.,
data on district-wide wage rate) as well as primary data collected
by the evaluating agency during general data collection (e.g., data
on beneficiaries’ indirect costs due to the program).
When to Collect Cost Data: Generally, a CEA can take place
at two distinct stages of program implementation:
• A Prospective Analysis takes place prior to the start of a
program or pilot and before an actual impact evaluation has
been undertaken. A prospective analysis uses projected or
budgeted costs and impact estimates from impact evaluations
of comparable programs in other contexts. This analysis cannot
yield a precise prediction of the actual CEA of the program; it
is only an indication of the program’s potential CEA if all of the
assumptions play out as anticipated. This type of preliminary or
scoping analysis can help to answer the following questions:
• “Roughly how cost-effective could this proposed program be?”
• “How big an impact must this program achieve to meet our
minimum cost-effectiveness requirement to make the
program worthwhile?”
For example, say the Labor and Employment Department of a
particular government wants to test the feasibility of a new job
training program. The department is unsure whether this would
be a good investment to make and so, before they implement the
program they decide to examine whether the training program
will reduce the youth unemployment rate in a cost-effective
manner. In such a case, the department would be conducting a
prospective analysis.
• A Retrospective Analysis – This takes place after the program
is implemented and following an impact evaluation. Although
a retrospective CEA is conducted following an evaluation,
ideally cost data will be collected during the program
evaluation and not after the evaluation has concluded. Cost
collection that is built into the evaluation itself and occurs
in conjunction with program implementation provides the
most accurate data for CEAs. A retrospective CEA can answer:
• “Exactly how cost-effective was the program in the context
it was evaluated?”
pover t yac t 15
• “Roughly how cost-effective might we expect this program
to be if it were rolled out on a larger scale?”
standardizing costs across programs
questions to consider
• What is the base year? What is the year of analysis?
• Do costs across programs need to be converted to the
same currency?
• Do costs over multiple years need to be discounted into
the base year?
• Do inflation rates need to be taken into account?
Base Year - The year that the program begins (usually
coincides with the year that the program costs were incurred).
This is the year in which the present value is taken.
Year of Analysis – The year in which the actual costeffectiveness
calculation is undertaken.
Emphasized throughout this chapter has been the one-word rule
for conducting a comparative CEA - consistency. Consistency is
essential because it ensures that the differences in cost-effectiveness
across programs are not due to discrepancies in calculation, but
instead due to actual differences in the relative value for money
of programs. Consistency is the perquisite for comparability. We
established that consistency matters at all stages of the CEA, from
forming a uniform selection process for determining evaluations
to include in the CEA, to consistency in calculating the impacts
and costs of programs, to detailing assumptions.
The standard order of operations across all CEAs is the following:
The last step in calculating costs, standardizing cost data across
programs, is directly related to this idea of consistency. Included
in comparative CEAs are impacts and costs of programs that have
been calculated at different points in time and across different
contexts. Costs inherently vary due to time and location. US$100
spent on a program in India in 2006 does not equal US$100 spent
on a program in Spain in 2015. Even within the same program,
a dollar spent in year 1 of implementation is not the same as
a dollar spent in year 2. Due to variations in time and context,
we must ensure that consistency across and within programs is
upheld by standardizing costs across time and location.
Following collection of cost data for each respective program
included in a comparative CEA, aggregate program costs need
to be standardized by accounting for three additional factors:
time discounting, inflation, and currency exchange rates. In the
process of standardizing costs, it is important to distinguish
between two temporal concepts, base year and year of analysis.
step# operation unit of currency for a 3-year
program beginning in 2004 (base year)
1. Gather cost data for programs using the ingredients approach Year 1: 2004 (USD)
Year 2: 2005 (USD)
Year 3: 2006 (USD)
Exchange into the CEA’s common currency using year-specific
exchange rates
2. Year 1: 2004 (INR)
Year 2: 2005 (INR)
Year 3: 2006 (INR)
Deflate nominal costs back to the real value in a particular base
year prices. Use average annual inflation rates over time between
base year selected and incursion of costs
3. 2004 (INR) ( incurred in 2004, 2005, and 2006)
Time discounting - take present value (PV) of cost stream for
programmes that incur costs over multiple years
4. PV of cost stream in 2004 (in 2004 INR)
Inflate costs forward from the common base year to the year of
analysis. Use average annual inflation rates (for the currency of
analysis) over time between base year selected and year of analysis
5. PV of cost stream in 2015 (in 2015 INR)
pover t yac t 16
1. Exchanging into a common currency: Due to the
varying purchasing powers of different currencies, if programs report
costs in multiple currencies, it is necessary to exchange them
into a common unit (e.g., US dollar or Indian rupee). Of course,
converting into a common currency is not needed when comparing
in a comparative CEA two or more programs that all run in the
same country. The decision whether to use a purchasing power
parity (PPP) exchange rate, which is the rate of currency conversion
that equalizes the buying power of different currencies, versus a
standard exchange rate is at the researcher’s discretion. Standard
and PPP exchange rates both entail certain advantages and drawbacks.
Generally using a PPP versus a standard exchange rate does not
change the relative cost-effectiveness of programs, as long as this
decision does not change for different programs within the same
comparative CEA.
2. Deflate nominal costs to the real value in the base
year: A real value is a value that has adjusted to remove the
effects of changes in the price level (inflation or deflation),
whereas a nominal figure is unadjusted to the price level.
Program costs are usually entered as nominal costs. Therefore,
when nominal costs are compared across time, any difference
may be due to two things: (1) underlying differences in costs, and
(2) changes in the price level that have occurred between the
two time periods. It is crucial to convert nominal figures into
real figures so that we can only looks at the underlying differences
in costs that are not associated with changes in prices.
To do this, it is important to identify the appropriate base year
for each program and deflate all cost data to that base year. In
the table above, the costs for the example program incurred in 2005
(year 2) and 2006 (year 3) were converted to 2004 Indian rupee
amounts. This is done using the average inflation rate for the
common currency (measured by the consumer price index [CPI])
and making the following calculation:
Real value in base year=
(nominal value in year cost incurred )
(average inflation rate between base year and year cost incurred)
3. Time Discounting: Time discounting or taking the present
value, is a necessary step when evaluations report the impacts
and costs for more than one year. Disregard this step if all of the
evaluations of programs in your CEA are one year long (i.e., do
not have costs and impacts distributed over different years).
The rationale for discounting is given by the time value of
money, which suggests that a dollar is worth more today than
tomorrow because a dollar today can earn interest. The discounting
of costs is representative of the choice a funder faces between
incurring costs this year, or deferring expenditures to invest
for a year and then incurring costs the next year. Therefore, with
multiple year programs we must discount costs after the
second year back into the first year using what is known as a
discount rate, usually calculated as the Social Opportunity
Cost of Capital, which is the forgone rate of return on capital
markets. Generally, J-PAL uses a social opportunity cost of capital
of 10 percent, which is approximately the median rate according
to the Asian Development Bank. It is important to realize that the
discount rate choice will have implications on long-run programs
that have very different benefit-cost streams. A sensitivity analysis can
be conducted to test the impact of using different discount rates.
Say a program has the following stream of costs: a large upfront
fixed cost of US$1,000 in year one and subsequent costs in year
2 of US$300 and US$400 in year 3. The program’s impact was
measured as a three-year impact so we need the corresponding
three-year cost of this program. To calculate this cost we must
calculate the following
Present Value of Costs=
I0 +
Where r is equal to the social opportunity cost of capital, I0
represents the initial outlay at time zero, and I1, I2, ..., In are the
costs associated in time period 1, 2, all the way to the last time
period, n.
4. Inflate Costs to the Year of Analysis: The second inflation
related adjustment is to inflate the common currency, present
valued costs for each program from the base year to the year
of analysis. This is calculated similarly to step 2 but using the
average inflation rate for the common currency that occurred
between the base year and the year of analysis.
put ting together costs and benefits
Following completion of this four-step cost standardization
process, the calculated costs of a program should reflect the
actual costs of the program and across programs, costs should be
standardized. We are now ready to integrate program cost data
with program impact data and calculate the final CE Ratio. Recall
that this requires dividing the aggregated cost of the program by
the impact of the program on a specified outcome.
Cost-Effectiveness Ratio=
Total Impact of the Program on a Specified Outcome
Total Cost of Implementing a Program
Following calculation of the cost-effectiveness ratio for each
program, the final step is to present the CE Ratios for each
program in a visually appealing way that allows the reader to
easily discern the relative cost-effectiveness of programs.
(1+r)2 + + ... +
aude guerrucci
e x ample – immunization
incentives progr am
pover t yac t 18
The next section presents a simplified example of a CEA of an
immunization program that was run in 134 villages in rural Udaipur,
Rajasthan from 2004-2007. The immunization program was
implemented by the NGO Seva Mandir and evaluated by J-PAL
affiliated researchers (Abhijit Banerjee, Esther Duflo, and Rachel
Glennerster) using a randomized evaluation4.
Background on the Program: Although immunization is a
very cheap and effective way of improving child survival, only
22 percent of children in rural Rajasthan have received the basic
package of immunizations. This rate is even lower for those living
in tribal areas. The public health facilities serving rural and tribal
areas are characterized by high absenteeism: 45 percent of Auxiliary
Nurse Midwives (ANM) who carry out immunizations are absent
from their village-level health center on any given day. Given that
a full immunization course requires at least five visits to a public
health facility, the unreliability of the ANMs increases the opportunity
cost of a visit to the sub-center and may deter families from taking
their children to complete their full immunization schedule.
The Program: Two different programs were implemented and
evaluated. Thirty villages received program 1, thirty received
program 2, and in thirty villages no new programs were
implemented; this control group and received the status quo –
access to the standard public health facilities).
1. Program 1: To increase the supply of infrastructure for
immunization, Seva Mandir hired a mobile immunization
team to conduct monthly immunization camps in villages.
The camps were held on the same time each month and the
presence of a nurse was verified. A Seva Mandir worker
informed villages about the immunization camps and
educated parents on the benefits of immunization.
2. Program 2: In addition to the immunization camps of
program 1, an incentive scheme was instituted to
simultaneously increase the demand for immunizations.
The incentive scheme offered parents a 1 kg bag of lentils per
immunization administered, and parents also received a set of
plates once a child completed the full immunization course.
Impact Evaluation Results: Relative to the control group,
reliable camps but no incentives (Program 1) increased the number
of fully immunized children by 12 percentage points, from 6 percent
in the comparison group to 18 percent in villages with reliable
camps. The combined camps plus incentive (Program 2) caused
fewer children to drop out after the first two or three immunizations.
Camps plus incentives increased full immunization rates by
33 percentage points over the comparison group, and by 21
percentage points over Program 1.
4 See J-PAL Policy Briefcase, “Incentives for Immunization”, available at: http:// Academic
paper: Banerjee, Abhijit V., Duflo, Esther, Glennerster, Rachel, and Dhruva
Kothari. 2010. “Improving Immunisation Coverage in Rural India: Clustered
Randomised Controlled Evaluation of Immunisation Campaigns with and without
Incentives.” British Medical Journal.
Which of the two programs was most cost-effective?
To answer this, J-PAL conducted a CEA comparing the two
programs in terms of their respective cost per fully immunized
child. First, the percentage increase in children fully immunized,
which was calculated in the impact evaluation, was converted
to an easier impact figure to interpret the number of fully
immunized children. Next, costs were gathered for each of the
two programs using the ingredients approach. In this case, costs
were disaggregated using slightly different cost categories than
those outlined above.
figure 3. percentage of children aged 1-3 years
fully immunized by treatment status
camps +
pover t yac t 19
And finally, the CE ratio was calculated for each program by
dividing the costs (1,950,465 and 1,206,486 rupees) by the
number of fully immunized children that occurred as a result of
the program to get the costs per fully immunized child.
figure 5. costs per fully immunized child
These results show that providing incentives, in addition to
improving the supply of services through immunization camps,
actually halved the cost of fully immunizing an additional child.
It turned out that the camps with incentives were busier than
those without incentives, meaning that nurses’ time was used
more efficiently. Since more than twice as many children were
fully vaccinated in camps with incentives, each nurse vaccinated
more children, thus reducing the cost per shot. This is an incredibly
valuable result for informing policy decisions. Since budgets are
limited, a policymaker may be tempted to implement the less costly
immunization camp program, forgoing the incentives. However,
the CEA demonstrates that the relative value of the camps plus
incentives was actually roughly twice that of just the immunization
camps alone.
figure 4. disaggregation of camps and camps plus incentives program cost data
Rs. 372
+ Rs.730
camps +
Cost of Incentives Cost of Camp
Rs. 2,202
cost components details camps with
% of
camps without
% of
Team of 4 GNMs and 4 GNM Assistants
+ Coordinators Salary
558,500 29% 558,500 46%
Travel Staff and Incentive transport to camps 171,460 9% 63,460 5%
Honorarium USD0.26 per child under 2 yrs per shot,
given to village workers
119,580 6% 62,370 5%
Daily allowance USD1.10 for attending bi-monthly
meetings, given to village workers
19,500 1% 19,500 2%
Consultancy fees Paid for training of nurses and assistants 2,200 0% 2,200 0%
Lodging + boarding Expenses incurred during trainings 7,333 0% 7,333 1%
Travel For village worker's transport to trainings 4,645 0% 4,645 0%
Training material Office supplies disbursed during trainings 1,500 0% 1,500 0%
Medicines Includes paraceptemol, syringes and
needles, needle cutters, blood pressure
instruments, and stethoscopes
43,925 2% 15,320 1%
Refrigerators For vaccine storage 25,178 1% 25,178 2%
Cost of Monitoring Includes cameras, film, and manpower
required for monitoring camps, entering,
and analyzing data
446,480 23% 446,480 37%
Incentive Utensils and lentils (includes storage boxes) 550,164 28% - 0%
Total 1,950,465 100% 1,206,486 100%
additional resources
pover t yac t 21
for more information on conducting
ceas consult:
1. Comparative Cost-Effectiveness Analysis to Inform Policy in
Developing Countries: A General Framework with
Applications for Education: by Iqbal Dhaliwal, Esther Duflo,
Rachel Glennerster, and Caitlin Tulloch
2. J-PAL Costing Template: a template to help users generate
an estimate of total program costs. It provides users with a
comprehensive list of the different cost categories that may
be included in a program and prompts the user to input
various details about cost data for their respective program.
3. J-PAL Costing Guidelines: a useful document accompanying
the J-PAL costing template which instructs users on how to
approach collecting cost information.
J-PAL Latin America
& Caribbean
Pontificia Universidad
Católica de Chile
J-PAL Africa
University of Cape Town
J-PAL Europe
Paris School of Economics
J-PAL Southeast Asia
University of Indonesia
J-PAL South Asia
IFMR, India
J-PAL Global
J-PAL North America
The Abdul Latif Jameel Poverty
Action Lab (J-PAL) is a network
of more than 140 affiliated
professors from over 40
universities. Our mission is to
reduce poverty by ensuring that
policy is informed by scientific
evidence. We engage with
hundreds of partners around
the world to conduct rigorous
research, build capacity, share
policy lessons, and scale up
effective programs.
For more information, visit
Regional Center Tag
Resource Type Tag
Language Type Tag
Image Thumb