Impact Evaluation Research
Usage Impacts: Assessment of the impact of energy efficiency services and bill payment programs on the energy usage of participating households.
Payment Impacts: Assessment of the impact of payment programs, arrearage forgiveness, and energy efficiency services on the payments made by participating households.
Collections Cost Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the collections costs associated with participating customers.
Termination Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the incidence of account termination due to nonpayment for participating customers.
Impact Evaluation Activities
Data Attrition Analysis: Data attrition is often a serious problem in program evaluation research. Without careful attention to data attrition, measured results may be biased and inaccurately portray the impact of the program. APPRISE always conducts a careful analysis of the extent and causes of data attrition, presents results using various levels of data cleaning, and cautiously draws conclusions based on available data.
Weather Normalization: Variations in weather from one year to the next can often obscure the impacts of usage reduction programs. APPRISE uses sophisticated weather normalization procedures to control for the impact of the weather on energy usage when conducting usage impact analysis.
Comparison Group Utilization: In addition to weather, there are many other variables that can impact a household’s energy usage and bill payment behavior. To control for these other factors, we make use of comparison groups. We compare the change in usage or payment behavior for the treatment group to the change for the comparison group. The comparison group may be comprised of future program participants or another group of similar utility customers.
Impact Evaluation Outputs
- Measurement of program impacts
- Analysis of program achievements as compared to program goals
- Recommendations for program modification
APPRISE Impact Evaluations
Below we provide a few examples of some of our recent projects that included impact evaluation research. Click the tabs below to learn more about these projects, or see here for a full list of projects that included impact evaluation research.
South Jersey Gas has been operating energy efficiency programs in coordination with the New Jersey Clean Energy Program (NJCEP) since 2009. The programs provide additional rebates and financing to customers who participate in the NJCEP residential and commercial and industrial programs. APPRISE conducted an evaluation of these programs to assess program design, implementation, and impacts. The research included interviews with program administrators, contractors, and participants; surveys with participants of three different programs; analysis of the impacts of the programs on natural gas usage; and assessment of the non-energy impacts of the programs.
PECO’s Low-Income Usage Reduction Program (LIURP) provides energy efficiency services and energy education to PECO’s low-income customers to reduce their energy usage and increase the affordability of their energy bills. The evaluations estimated the impact of LIURP on participants’ energy usage, energy bills, and payment compliance and surveyed customers to analyze the impact of energy education and program satisfaction. In the 2016 program evaluation, APPRISE also conducted observations of audits on a sample of jobs. This research provided information on potential changes to program services and implementation that may lead to increased program savings. The evaluation of 2017 participants is assessing the program process, examining the impacts of program treatments on energy usage and affordability, and surveying customers to assess program satisfaction.
APPRISE was the lead contractor for the evaluation of the National Low-Income Weatherization Assistance Program (WAP). APPRISE had three key partners for this effort – Blasnik and Associates, Dahloff Associates, and the Energy Center of Wisconsin, and was also supported by technical experts from across the country. The evaluation consisted of data collection from state WAP offices, local agencies, and participating and nonparticipating clients. Utility data was collected and analyzed for a sample of clients from every state and submetering studies were conducted to assess impacts on clients who use bulk fuels and to conduct special technical studies. The evaluation assessed program operations and delivery, energy impacts, and non-energy impacts, and made recommendations for modifications to program design and delivery.
APPRISE conducted an evaluation of Peoples Natural Gas Universal Service Programs. Peoples’ Universal Service Programs provide reduced natural gas bills, arrearage forgiveness, emergency bill payment assistance, energy efficiency services, and case management services to low-income households. The evaluation assessed the efficiency and effectiveness of these programs through background research, needs assessment research, program administrator interviews, customer surveys, program database analysis, and analysis of the impacts of the program on affordability, bill payment, and collections actions and costs.
Ohio’s Percentage of Income Payment Program (PIPP) was revised as the PIPP Plus in November 2010. The program helps low-income households maintain utility service by providing a reduced bill based on a percentage of the household’s annual income. APPRISE conducted an evaluation of this program to assess the effectiveness of the changes to PIPP that aimed to provide incentives for regular, on-time payments and consequences for failure to adhere to program rules. The research included administrative interviews, participant surveys, and analysis of program and utility data.