It can identify: The evaluation may confirm the theory of change or it may suggest refinements based on the analysis of evidence. Methodologies to evaluate the impact of large scale nutrition programs(Habicht et al. Many cover unexpected or negative change as well. KEQ2 To what extent were the programme objectives met? Review: An assessment of the performance of an intervention, periodically or on an ad hoc basis. International Initiative for Impact Evaluation Working Paper , Rethinking Impact Evaluation for Development - IDS Special Issue, InterAction Impact Evaluation Guidance Notes and Webinar Series, Realist impact evaluation: An introduction, Evaluability assessment for impact evaluation, When and how to develop an impact-oriented monitoring and evaluation system, Additional guidance documents can be found here. Yet there is a need to put the theory into practice in a hands-on fashion for practitioners. Handbook on Impact Evaluation : Quantitative Methods and Practices Impact evaluation is an assessment of how the intervention being evaluated affects outcomes, whether these effects are intended or unintended. It also includes new material on research ethics and partnerships to conduct impact evaluation. Principles forEvaluation of Development Assistance. Impact Evaluation in UN Agency Evaluation Systems: Guidance on Selection, Planning and Management. BonbrightD(2012). PerrinB (2012). Realist impact evaluation is an approach to impact evaluation that emphasises the importance of context for programme outcomes. ; World Bank, 2006). and ', Michael Lechner - Universitt St Gallen, Switzerland, The authors are to be congratulated for their comprehensive and delightfully modern treatmentof causal inference based on observational data. Kalamazoo: Western Michigan University Checklist Project. And, some are used for particular types of development interventions such humanitarian assistance such as: coverage, coordination, protection, coherence. Overview: Data Collection and Analysis Methods in Impact Evaluation, Some Reflections on Current Debates in Impact Evaluation. The specific evaluative rubrics should be used to interpret the evidence and determine which considerations are critically important or urgent. Impact Evaluation M. Frlich, S. Sperlich Published 28 February 2019 Economics, Political Science In recent years, interest in rigorous impact evaluation has grown tremendously in policy-making, economics, public health, social sciences and international relations. This book spans the most promising approaches towards impact evaluation, and I am particularly impressed by their inclusion of Pearls (2000) work alongside many other notable contributions. Barbeito, Ins An impact evaluation assesses the extent to which a program has caused desired changes in the intended audience. In other words, an impact evaluation is able to attribute impacts to a program and to that program alone. It specifies designs for causal attribution, including whether and how comparison groups will be constructed, and methods for data collection and analysis. PDF Die Europische Kommission First published in 2011, it has been used widely across the development and academic communities. It is a particularly important one due to its ability to isolate the interventions impact. 500 L'Enfant Plaza SW Washington, DC 20024 This report is in the public domain. Some interventions cannot be fully planned in advance, however for example, programmes in settings where implementation has to respond to emerging barriers and opportunities such as to support the development of legislation in a volatile political environment. Elsewhere, its fundamental basis may revolve around adaptive learning, in which case the theory of change should focus on articulating how the various actors gather and use information together to make ongoing improvements and adaptations. If this is the first time you use this feature, you will be asked to authorise Cambridge Core to connect with your account. [assessing effectiveness, impact, equity, gender equality], KEQ4: What unintended results positive and negative did the intervention produce? Guidance Note No. The study highlighted diversity in the approaches taken to impact evaluation and informed a chapter on impact evaluation in the 2013-2014 Annual Evaluation Report. This approach to evaluation focuses on the practicali-ties of defining successful outcomes and success cases (Brinkerhoff, 2003) and uses some of the processes from theory-driven evaluation to determine the linkages, which may take the form of a logic model, an impact model, or a results map. PDF impact evaluation methods - Abdul Latif Jameel Poverty Action Lab How did these occur? PDF Quantitative Methods for Impact Evaluation - Harvard T.H. Chan School The handbook is divided into four sections: Part One discusses what to evaluate and why; Part Two presents the main impact evaluation methods; Part Three addresses how to manage impact evaluations; Part Four reviews impact evaluation sampling and data collection. It is concerned with the net impact of an intervention on households and institutions, attributable only and exclusively to that intervention. Ease of access to subjects . They tend to emphasize operational aspects. But how do we know were all talking about the same thing? Case studies illustrate different applications of impact evaluations. Impact evaluation might be appropriate when there is a need to understand the impacts that have been produced. Reviews are usually less comprehensive and/or in-depth than evaluations. A guide to water and sanitation sector impact evaluations(Poulos et al. Evaluative reasoning is the process ofsynthesizingthe answers to lower- and mid-level questions into defensible judgements that directly answer the high-level questions. The Stata code and data are provided as a courtesy only. 1. It can be undertaken of a programme or a policy, or upstream work - such as capacity building, policy advocacy and support for an enabling environment. 2020. This is what makes evaluation so much more useful and relevant than the mere measurement of indicators or summaries of observations and stories. Skip to content Features Data Collection Data Management Results Framework & Indicators Dashboards Ozabaci, Deniz PDF Development Impact Evaluation (DIME) - World Bank The book incorporates real-world examples to present practical guidelines for designing and implementing impact . This book also details challenges and goals in other realms of evaluation, including monitoring and evaluation (M&E . Technol. and Both parametric and non-parametricestimation are treated and numerous examples illustrate the presentation. PDF Planning and Managing Impact Evaluations - USAID Learning Lab "Are public housing projects good for kids?." 5. For example, if decisions have already been made on the basis of existing credible evidence, or if decisions need to be made before it is possible to undertake a credible impact evaluation. Wollni, Meike For answeringcausalKEQs, there are essentially three broad approaches to causal attribution analysis: (1) counterfactual approaches; (2) consistency of evidence with causal relationship; and (3) ruling out alternatives (see above). In what circumstances? We use cookies to distinguish you from other users and to provide you with a better experience on our websites. This work is increasingly important as the global development community works to reduce poverty and achieve the 2030 Sustainable Development Goals. PDF Impact Evaluation - Intrac Washington DC:InterAction. Effective/Applicability Date. Is it to build ownership of a donor-funded programme? The following links offer an overview of the main methodologies: To implement your impact evaluation you may require technical assistance from evaluators and data specialists. (PDF) Impact Evaluation in Practice - ResearchGate This blog post by Simon Hearn(ODI) wasoriginally posted by Action to Research. Impact Evaluation in Practice - Second Edition - World Bank Group Evaluative criteria should be thought of as concepts that must be addressed in the evaluation. The book links to complementary instructional material available online, including an applied case as well as questions and answers. This paper, written by Sandra Nutley, Alison Powell and HuwDavies for the Alliance for Useful Evidence, discusses the risks ofusing a hierarchy of evidence and suggests an alternative in whichmore complex matrix approaches for identifying evidencequ. techniques in impact evaluation and a set of reasoning skills intended to help them become both a consumer and producer of applied empirical research. PDF Developing Research Questions for Impact Evaluations Guha, Sharmistha Liu, Piaomu See:http://www.oecd.org/dac/evaluation/50584880.pdf, OEDC-DAC(2010). ; IDB,2013). and Designing an impact evaluation work plan: a step-by-step guide - TolaData A simple guide to help you design an impact evaluation work plan more accurately & effectively. One way of doing so is to use a specific rubric that defines different levels of performance (or standards) for each evaluative criterion, deciding what evidence will be gathered and how it will besynthesizedto reach defensible conclusions about the worth of the intervention. Should it be considered a success if the average household income increases but the income of the poorest households is reduced? PDF Overview of Impact Evaluation - unicef-irc.org This will also ensure that data from other M&E activities such as performance monitoring and process evaluation can be used, as needed. 3. ; World Bank, 2007). If evaluative rubrics are relatively small in size, these should be included in the main body of the report. It is also important to consider the timing of an impact evaluation. The emphasis on nonparametric identification and semiparametric and nonparametric estimation makes this text standout. Los criterios de evaluacin especifican los valores que se emplearn en esta. This data will be updated every 24 hours. Accessibility, Access to 4. You must have JavaScript enabled to use this form. Building in an evaluation component for active labor market programs: a practitioners guide(Card et al. Identifying the information to be collected and methods of doing this - planning stage. Full text views reflects the number of PDF downloads, PDFs sent to Google Drive, Dropbox and Kindle and HTML full text views for chapters in this book. 2023. Impact evaluations need to go beyond assessing the size of the effects (i.e., the average impact) to identify for whom and in what ways a programme or policy has been successful. 2023. Randomized Controlled Trials, UNICEF Brief 8. If you continue to navigate this website beyond this page, cookies will be placed on your browser. Case studies illustrate different applications of impact evaluations. Impact Evaluation Fund (SIEF). [assessing effectiveness, efficiency], KEQ3: Did the intervention produce the intended results in the short, medium and long term? Usage data cannot currently be displayed. The updated version covers the newest techniques for evaluating programs and includes state-of-the-art implementation advice, as well as an expanded set of examples and case studies that draw on recent development challenges. For example, should an economic development programme be considered a success if it produces increases in household income but also produces hazardous environment impacts? Regardless of the type of evaluation, it is important to think through who should be involved, why and how they will be involved in each step of the evaluation process to develop an appropriate and context-specific participatory approach. JPAL offers data and code for many of their impact evaluations. Once the data is analyzed, the results are then reported through presentations, evaluation reports, papers and policy brief. All evaluation questions should be linked explicitly to the evaluative criteria to ensure that the criteria are covered in full. 2022. 2022. Students will learn to critically analyze evaluation research and to gauge how convincing the research is in identifying a causal impact. FunnellS and Rogers P (2012). Washington DC:InterAction. Methodological Briefs - Impact Evaluation No. This definition does not require that changes are produced solely or wholly by the programme or policy under investigation (UNEG2013). There are three design options that address causal attribution: Some individuals andorganisationsuse a narrower definition ofimpact evaluation, and only include evaluations containing acounterfactual of some kind. Good data management includes developing effective processes for: consistently collecting and recording data, storing data securely, cleaning data, transferring data (e.g., between different types of software used for analysis), effectively presenting data and making data accessible for verification and use by others. and PDF Guidance for the Evaluation of Immobile Chemicals for the Impact to Once the data required to conduct an impact analysis are available, the data will be analyzed using statistical software and appropriate estimation strategies. 2022. Evidence-based policy-making has become a recurring theme in public policy, alongside greater demands for accountability in public policies and public spending, and requests for independent and rigorous impact evaluations for policy evidence. For example, the findings of an impact evaluation can be used to improve implementation of a programme for the next intake of participants by identifying critical elements to monitor and tightly manage. Organisation for Economic Co-operation and Development Development Assistance Committee (OECD-DAC). The updated version covers the newest techniques for evaluating programs and includes state-of-the-art implementation advice, as well as an expanded set of examples and case studies that draw on recent development challenges. In other words, the analytical framework the methodology for analysing the meaning of the data by looking for patterns in a systematic and transparent manner should be specified during the evaluation planning stage. There are three broad strategies for causal attribution in impact evaluations: Using a combination of these strategies can usually help to increase the strength of the conclusions that are drawn. 2021. Ifan impact evaluation fails to systematically undertake causal attribution, there is a greater risk that the evaluation will produce incorrect findings and lead to incorrect decisions.