Impact Evaluation Research
Usage Impacts: Assessment of the impact of energy efficiency services and bill payment programs on the energy usage of participating households.
Payment Impacts: Assessment of the impact of payment programs, arrearage forgiveness, and energy efficiency services on the payments made by participating households.
Collections Cost Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the collections costs associated with participating customers.
Termination Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the incidence of account termination due to nonpayment for participating customers.
Impact Evaluation Activities
Data Attrition Analysis: Data attrition is often a serious problem in program evaluation research. Without careful attention to data attrition, measured results may be biased and inaccurately portray the impact of the program. APPRISE always conducts a careful analysis of the extent and causes of data attrition, presents results using various levels of data cleaning, and cautiously draws conclusions based on available data.
Weather Normalization: Variations in weather from one year to the next can often obscure the impacts of usage reduction programs. APPRISE uses sophisticated weather normalization procedures to control for the impact of the weather on energy usage when conducting usage impact analysis.
Comparison Group Utilization: In addition to weather, there are many other variables that can impact a household’s energy usage and bill payment behavior. To control for these other factors, we make use of comparison groups. We compare the change in usage or payment behavior for the treatment group to the change for the comparison group. The comparison group may be comprised of future program participants or another group of similar utility customers.
Impact Evaluation Outputs
- Measurement of program impacts
- Analysis of program achievements as compared to program goals
- Recommendations for program modification
APPRISE Impact Evaluations
Below we provide a few examples of some of our recent projects that included impact evaluation research. Click the tabs below to learn more about these projects, or see here for a full list of projects that included impact evaluation research.
South Jersey Gas has been operating energy efficiency programs in coordination with the New Jersey Clean Energy Program (NJCEP) since 2009. The programs provide additional rebates and financing to customers who participate in the NJCEP residential and commercial and industrial programs. APPRISE conducted an evaluation of these programs to assess program design, implementation, and impacts. The research included interviews with program administrators, contractors, and participants; surveys with participants of three different programs; analysis of the impacts of the programs on natural gas usage; and assessment of the non-energy impacts of the programs.
PECO’s Low-Income Usage Reduction Program (LIURP) provides energy efficiency services and energy education to PECO’s low-income customers to reduce their energy usage and increase the affordability of their energy bills. The LIURP evaluations estimate the impact of LIURP on participants’ energy usage, energy bills, and payment compliance; and survey customers to analyze the impact of energy education and program satisfaction. Evaluations also include observations of audits on a sample of jobs. This research provides information on potential changes to program services and implementation that may lead to increased program savings.
APPRISE was the lead contractor for the evaluation of the National Low-Income Weatherization Assistance Program (WAP). APPRISE had three key partners for this effort – Blasnik and Associates, Dahloff Associates, and the Energy Center of Wisconsin, and was also supported by technical experts from across the country. The evaluation consisted of data collection from state WAP offices, local agencies, and participating and nonparticipating clients. Utility data was collected and analyzed for a sample of clients from every state and submetering studies were conducted to assess impacts on clients who use bulk fuels and to conduct special technical studies. The evaluation assessed program operations and delivery, energy impacts, and non-energy impacts, and made recommendations for modifications to program design and delivery.
APPRISE conducted an evaluation of Peoples Natural Gas Universal Service Programs. Peoples’ Universal Service Programs provide reduced natural gas bills, arrearage forgiveness, emergency bill payment assistance, energy efficiency services, and case management services to low-income households. The evaluation assessed the efficiency and effectiveness of these programs through background research, needs assessment research, program administrator interviews, customer surveys, program database analysis, and analysis of the impacts of the program on affordability, bill payment, and collections actions and costs.
The Public Service Commission of Maryland (PSC) authorized BGE to develop a one-year Prepaid Pilot Program with electric-only and dual-service customers to test the feasibility and customer acceptance of a utility prepay service program. This program provides customers with the opportunity to pay for their energy in advance of use and make additional energy payments as needed. APPRISE conducted a comprehensive evaluation of the program to assess how the pilot impacted energy usage, energy costs, payment compliance, collections actions and costs, and disconnections. The research found that the pilot had several benefits for participants. Participants reported that it improved their control over energy expenses, helped them to pay off arrearages, avoid or reduce the length of their disconnections, and reduce their energy usage. A weather-normalized, comparison group adjusted billing analysis found that participants reduced their energy usage and improved their energy payments. However, it was difficult to draw conclusions about collections and disconnections because the pilot was during the COVID-19 Pandemic when disconnections were suspended.