QUICK LINKS

Model Evaluation Frameworks

Health Financing Models

Health Delivery Models

 

Evaluating Organizations CHMI's Reported Results Initiative Background Methodology GiveWell.org: Charity Review and Ranking Process Background Methodology Selected Resources Evaluation Approaches Evaluation for Models and Adaptive Initiatives Evaluating Social Innovation Business Model Evaluation Business Model Evaluation Scorecard Evaluating Your Business Model Additional Web Resources Impact Evaluation Frameworks

 

 

Evaluating Organizations

---

 

CHMI's Reported Results Initiative

 

Background

 

The CHMI database currently profiles more than 1000 programs from 105 countries. While much rich descriptive information has been captured, an important informational gap remains – which programs are actually “working”, or achieving the kind of health and financial protection results that are important to national and global health policymakers, donors, investors, and other program implementers looking to emulate proven models.

 

The ultimate goal of the system is to enable greater transparency and standardization of how programs’ performance is tracked and shared with the global health community.

 

Methodology

Reported results are measures of program performance across a number of key categories, including the following examples [Click here to review the full set of results categories and definitions.]:

 

• Quality:          Showing the program reduced the percentage of incorrect medication dispensed
• Affordability:  Giving evidence that services are priced below market price
• Availability:    Demonstrating that program increased the number of services or products available to an under-served community

 

CHMI collects Reported Results through a standardized template. All results statements are self-reported. Where available, Reported Results generated by third-party evaluations are included. Statements may be edited for consistency and layout restrictions.

 

 

 

 

GiveWell.org: Charity Review and Ranking Process

 

Background

The mission of GiveWell is to review charities and publish detail analysis to help donors identify charities for donation.

 

Methodology

 

General Approach (quoted)

• Understand as much as we can about a cause by reading as much of the background literature as we can and talking to individuals with subject-matter expertise;
• Cast a wide net to identify as many charities that could potentially receive a top rating from us;
• Formulate heuristics to prioritize some of the many charities we have identified as more or less promising;
• Looking briefly at all of the charities and considering them with regard to our heuristics and categorizing some as "promising and worthy of further investigation" and others as "not promising and unlikely to receive a recommendation;"
• Investigating the most promising ones in more depth through phone calls with charity representatives; reviewing internal documents including monitoring and evaluation reports, budgets, and plans for using additional funding; reviewing independent literature and evidence of effectiveness of the charities' programs.
• Identifying the top contenders for our recommendations and doing very in-depth investigations, including visits to their programs in the field.

 

Measuring Cost-effectiveness

http://www.givewell.org/international/technical/criteria/cost-effectiveness#Howcosteffectiveiscosteffective

 

Considerations:

• Charities frequently cite misleading and overly optimistic figures for cost-effectiveness.
• Our cost-effectiveness estimates include all costs (including administrative costs) and generally look at the cost per life or life-year changed (death averted, year of blindness averted, etc.) However, there are many ways in which they do not account for all possible costs and benefits of a program.
• Because of the many limitations of cost-effectiveness estimates,we give estimated cost-effectiveness only limited weight in recommending charities. Confidence in the organization's track record generally carries heavier weight when differences in estimated cost-effectiveness are not extremely large.
• The impact we are most often able to estimate is the cost per life saved. There are many kinds of impact besides saving lives; we try to do our best to quantify the different ways in which a program may be improving lives, and to help donors decide (sometimes with the aid of formal frameworks such as the disability-adjusted life-year) how to compare these.

 

Limitations of Cost-effectiveness methods

The estimates we use do not capture all considerations for cost-effectiveness. In particular:

• We generally draw effectiveness estimates from studies, and we would guess that studies often involve particularly well-executed programs in particularly suitable locations. While we make efforts to assess how representative studies are of average conditions, our ability to do so is often limited.
• Estimates consider only direct impact, and do not attempt to incorporate the many ways in which a program may affect people beyond (or as a result of) its immediate impact on health/income. This issue is discussed specifically as it relates to GiveWell in a paper by Leif Wenar.4
  
• Estimates are generally based on extremely limited information and are therefore extremely rough.

 

Survey Questions:

 

     Example organization: Partners in Health

     Questions:

• What type of conditions are treated? How serious are they?
• What are the qualifications of the staff?
• Were staff members employed before you hired them? If so, what were their jobs?
• Who are the patients? What is their standard of living?
• What other primary care services are there in the area? What services were available before you started working in the area?
• Do you monitor the quality of care?
• Has the number of patient visits grown over time?
• Do you survey clients to determine their satisfaction with the care they receive?
• Are patients required to pay a fee for care? What proportion of the cost of care is covered by patient fees?
• Are you replacing government services? How has government spending on primary care changed since you began working in the area?
• How much has been spent on this program? How many people have been served?
• How would your activities likely change if you had more revenue than expected? Less? Would additional revenue translate directly into more patient treatments for moderate to serious conditions, and up to what point?

 

 

 

Selected Resources

---

 

Evaluation Approaches

 

 

Evaluation for Models and Adaptive Initiatives Heather Britt and Julia Coffman Center for Evaluation Innovation November, 2012 Britt and Coffman.pdf	Heather Britt and Julia Coffman outline a framework for selecting evaluation approaches for two main types of grant-making programs: models and adaptive initiatives. Key Points: Although there has been a growing emphasis on use of experimental designs in evaluation, there is also increasing agreement that evaluation designs should be situation specific. The nature of the program is one of the key factors to consider in evaluation design. Two types of programs: Models: provide replicable or semi-standardized solutions; evaluation of models requires understanding the stage of development of the model program, with summative evaluation done only when the model is fully developed Adaptive initiatives,: are flexible programming strategies used to address problems that require unique, context-based solutions – require different evaluation designs; require consideration of both the timing and scale of the initiative in determining the appropriate evaluation design
Evaluating Social Innovation Hallie Preskill and Tanya Beer Center for Evaluation Innovation August, 2012 EvaluatingSocialInnovation.pdf	Hallie Preskill and Tanya Beer explore how grantmakers must re-envision evaluation so that social innovations have a better chance of success. Key Points: Traditional evaluation approaches (formative & summative evaluations) fail to meet the fast-paced information needs of "philanthropic decision makers" and innovators in the midst of complex social change efforts; they restrict implementers to pre-set plans that lose their relevance as the initiative unfolds. TEA are based on strategic philanthropy (eg. articulated goals, theory of change, well-aligned partners and grantees, performance metrics, evaluation to measure progress against desired outcomes. But these principles can work against social innovation because innovators have to conform to the plans and metrics that don't evolve in response to the dynamic context.  Social innovation is a fundamentally different approach to change than implementing program models with a known set of elements or “ingredients.” While the long-term goals of a social innovation might be well defined, the path to achieving them is less clear—little is known about what will work, where, under what conditions, how, and with whom. Instead, decision makers need to explore what activities will trigger change; and activities that successfully trigger a desired change may never work again. Developmental evaluation: The DE evaluator works collaboratively with social innovators to conceptualize, design, and test new approaches in a long-term, on-going process of adaptation, intentional change, and development. DE’s focusis on social innovations where there is no accepted model (and might never be) for solving the problem. Types of questions answered by DE: What is developing or emerging as the innovation takes shape? What variations in effects are we seeing? What do the initial results reveal about expected progress? What seems to be working and not working? What elements merit more attention or changes? How is the larger system or environment responding to the innovation? How should the innovation be adapted in response to changing circumstances? How can the project adapt to the context in ways that are within the project’s control?     Choosing evaluation approach:

 

 

Business Model Evaluation

 

Business Model Evaluation Scorecard Mark Sniukas - Sniukas.com	Provides a scorecard template for evaluating a business model Additional resources: The Innovation Map: A framework for defining innovation outcomes Defining Innovation Types. from Marc Sniukas
Evaluating Your Business Model FastTrac, Kauffman Foundation
Additional Web Resources	BizFilings: Business Owner's Toolkit | www.bizfilings.com

 

 

Impact Evaluation Frameworks

 

Nirali M Shah, Wenjuan Wang, and David M Bishai. (2011) Comparing private sector family planning services to government and NGO services in Ethiopia and Pakistan: how do social franchises compare across quality, equity and cost? Health Policy and Planning 2011;26:i63–i71. doi:10.1093/heapol/czr027  PDF

 

ABSTRACT: Policy makers in developing countries need to assess how public health programmes function across both public and private sectors. We propose an evaluation framework to assist in simultaneously tracking performance on efficiency, quality and access by the poor in family planning services. We apply this framework to field data from family planning programmes in Ethiopia and Pakistan, comparing (1) independent private sector providers; (2) social franchises of private providers; (3) non-government organization (NGO) providers; and (4) government providers on these three factors. Franchised private clinics have higher quality than non-franchised private clinics in both countries. In Pakistan, the costs per client and the proportion of poorest clients showed no differences between franchised and non-franchised private clinics, whereas in Ethiopia, franchised clinics had higher costs and fewer clients from the poorest quintile. Our results highlight that there are trade-offs between access, cost and quality of care that must be balanced as competing priorities. The relative programme performance of various service arrangements on each metric will be context specific.