Why M & E?

IFC/GEF Efficient Lighting Initiative (ELI) Monitoring and Evaluation Overview

The IFC/GEF has incorporated a rigorous program monitoring and evaluation (M&E) component into the development of the Efficient Lighting Initiative (ELI). The purpose of the M&E activity is to measure the extent to which the various components of the ELI program as implemented in the participating countries have achieved their objective of moving the market toward a sustainable increase in the availability and adoption of high efficiency lighting equipment.

A secondary aim of the project is to impose a state-of-the-art yet "standardizable" monitoring and evaluation protocol for market transformation programs, such as ELI, that are aimed at achieving reductions in greenhouse gas emissions (GHG). Indeed, if the international lending community is to engage in end-user strategies for achieving GHG mitigation to complement more traditional energy supply strategies then it must have a way to be certain that such investments have their intended effect on both the market for energy and, ultimately, the environment.

The ELI monitoring and evaluation effort will address such broad questions as:

• "Can a coordinated set of best-practice approaches to market intervention be effectively implemented in a range of countries and conditions?"
• "Can such a program be successful at altering energy technology supply and purchasing behaviors in a significant portion of the targeted populations?"
• "What impacts can these changes to market conditions and behaviors be expected to have on greenhouse gas emissions reductions in the near term and into the future?"
• "If successful, can this type of worldwide program activity be replicated for other technologies, other markets, and other greenhouse gas mitigation strategies?"

These types of questions will guide the various components of this M&E activity. Since the overall aim of the ELI is market transformation, the M&E components of the research will provide the basis for assessing the extent to which market transformation has occurred in each country, and globally over the entire ELI effort.

More specifically, the M&E activity will also examine several questions at the country level, covering all seven ELI countries: Peru, Argentina, South Africa, Philippines, Latvia, Czech Republic and Hungary. Three major M&E activities will be carried out in each country, in parallel to the implementation of the ELI program itself:

• Process Evaluation - This type of M&E activity examines the design, delivery and follow-up processes that are conducted on behalf of a program. It includes review of both implementation and delivery organizations and procedures, as well as market segments that are targeted by a program, their awareness, satisfaction levels, and behaviors associated with a program and its services. While much of the methodology in process evaluation is qualitative, it also includes a significant market research component, aimed at quantifying those elements that can be statistically measured.

• Impact Evaluation - This M&E activity specifically attempts to quantify the energy and environmental impacts that are attributable to a program. To that aim, it must try to control for exogenous influences on outcomes - such as changes in economic conditions - so that conclusions can be drawn about the specific effects of program-related activities. It includes measurement of both direct and indirect economic and energy consumption and usage impacts, with specific attention to program participants. Then, once energy impacts are determined, the Impact Evaluation will estimate the relative impact that these outcomes will have on the environment, with specific focus on greenhouse gas avoidance from reductions in the need for electricity produced by each country's electric power plants.

• Market Transformation Analysis - This M&E activity focuses on the broader effect of the program on the overall market, not just participants. Here, evaluators will examine changes in producer, supplier and end-user knowledge, behaviors and attitudes that have resulted from the program's existence. Of key importance is to determine the extent to which changes can be expected to be sustained over a long term, since the primary aim of the ELI program is to permanently accelerate the adoption of higher efficiency equipment. This part of the evaluation will examine non-participants both in each country and within the regions, so that any spillover effects can be captured. Market transformation analysis deals mostly with economic issues and elements of market behavior over a long term.

The M&E activity also includes an important objective of building an understanding of the value of careful monitoring of market intervention strategies. Only by incorporating M&E conventions into such programs can stakeholders be able to determine whether they worked as intended, information that is invaluable in obtaining support for continuing such efforts in the future. The main training activity will be an evaluation conference to be held at the conclusion of the ELI program in 2005 that will present papers on various aspects of the seven-country program with a focus on data collection and monitoring challenges. It is anticipated that this conference will to build a broader foundation for the development of stronger M&E culture and community within each country and region. In addition, the opportunity to conclude both the M&E project and the ELI program with a major information sharing event will itself help to further the aims of the program in creating a broad sustainable knowledge base for further market transformation activities.

The structure of the M&E research team consists of (1) a core group of North American energy and environmental program evaluation experts leading the technical design and analysis, (2) an international market research firm with affiliates in seven countries to be responsible for data collection, and (3) set of seven energy efficiency/environmental experts within the seven countries to serve as in-country liaisons for the M&E project, providing in-country oversight and interpretation. The M&E team has secured the participation of in-country experts in various aspects of consumer energy markets - market research entities, engineering firms, economists and environmental analysts - to serve as liaisons for the M&E team. It is the hope of the project that through their involvement with the ELI M&E project, these individual s and firms will serve as catalysts for the further development of an evaluation culture within the regions.

The M&E team members, including the in-country experts, must have no involvement in the implementation of ELI, nor benefit in any way from the outcomes of the program. It is essential that the integrity of the M&E effort be preserved so that any findings are invulnerable to conflict of interest criticisms. Ethical guidelines of the Qualitative Research Consultants Association (QRCA) will be followed in the conduct of the data collection and analysis, maintaining confidentiality of information among the M&E research team throughout the project. Our aim is not to evaluate the performance of individuals, but rather to assess the effectiveness of a strategy at changing a market. That being the case, we encourage and seek candor in all contact activities throughout the project so that an accurate picture of how the program has been implemented can be gleaned. It is only with full knowledge and understanding of conditions, barriers, actions and outcomes that accurate assessments can be obtained, and conclusions reached about the ELI program's effectiveness.

As part of this project it is the IFC's intention to strive for and test a standard methodology, approach and strategy for conducting international, multi-country evaluations of market transformation programs. To that aim, the M&E team will strive for the design and implementation of easily transferable data collection techniques and replicable analysis methods. To the maximum extent possible, the M&E team will incorporate standard M&E protocols and baseline data from previous IFC/GEF and other reputable research efforts dealing with global GHG emissions mitigation efforts.

The ELI Monitoring and Evaluation Work Plan is accessible from the ELI Website, www.efficienctlighting.net. Updates on evaluation activities will be posted on this site throughout the project (2000 - 2005). The M&E Project Manager is Ms. Luisa Freeman - Favrdin of the Applied Energy Group, (lufreeman@aol.com or 703-242-3870 ). Carol Mulholland of the Academy for Educational Development (cmulholl@aed.org, 202-884-7034) has M&E oversight responsibilities on behalf of IFC. The IFC ELI Program Manager is Russell Sturm (rsturm@ifc.org).