RESEARCH ACTIVITIES – Impact Evaluation Research 2017-02-27T15:11:24+00:00

Impact Evaluation Research

  • Usage Impacts: Assessment of the impact of energy efficiency services and bill payment programs on the energy usage of participating households.
  • Payment Impacts: Assessment of the impact of payment programs, arrearage forgiveness, and energy efficiency services on the payments made by participating households.
  • Collections Cost Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the collections costs associated with participating customers.
  • Termination Impacts: Assessment of the impact of usage reduction programs and bill payment programs on the incidence of account termination due to nonpayment for participating customers.

Impact Evaluation Activities

  • Data Attrition Analysis: Data attrition is often a serious problem in program evaluation research. Without careful attention to data attrition, measured results may be biased and inaccurately portray the impact of the program. APPRISE always conducts a careful analysis of the extent and causes of data attrition, presents results using various levels of data cleaning, and cautiously draws conclusions based on available data.
  • Weather Normalization: Variations in weather from one year to the next can often obscure the impacts of usage reduction programs. APPRISE uses sophisticated weather normalization procedures to control for the impact of the weather on energy usage when conducting usage impact analysis.
  • Comparison Group Utilization: In addition to weather, there are many other variables that can impact a household’s energy usage and bill payment behavior. To control for these other factors, we make use of comparison groups. We compare the change in usage or payment behavior for the treatment group to the change for the comparison group. The comparison group may be comprised of future program participants or another group of similar utility customers.

Impact Evaluation Outputs

  • Measurement of program impacts
  • Analysis of program achievements as compared to program goals
  • Recommendations for program modification

APPRISE Impact Evaluations – Payment Programs

APPRISE conducted a comprehensive evaluation of Allegheny Power’s Universal Service Programs. The evaluation research included interviews with Allegheny and agency program managers and staff, surveys with participating and nonparticipating customers, analysis of program databases, and analysis of the impacts of the payment assistance program on energy affordability, payment compliance, arrearages, and collections costs.
Ameren introduced a pilot of a low-income payment assistance program that provides electric bill payment assistance to low-income customers. Participants receive monthly bill credits toward their electric bill and additional credits toward their arrearages. APPRISE provided an evaluation of this program in 2012 that included interviews with local agencies that conduct intake for the program, a survey of participating customers, analysis of program data, and analysis of program impacts on affordability, payment coverage, and terminations. Ameren made program modifications in response to findings from the initial evaluation. APPRISE conducted a follow-up Process and Impact Evaluation of the program in 2014 that documented the impacts of the program changes. APPRISE is currently conducting a third evaluation of this program that includes interviews with defaulted customers, analysis of program data, and analysis of the impacts of the program on affordability and bill payment. Findings from this research will be used to further refine the program.
Baltimore Gas and Electric (BGE) designed and implemented pilot programs to determine the effect of energy bill discounts and payment counseling on the payment timeliness of limited-income customers. The Graduated Rate Discount (GRAD) Program provided a range of usage-based discounts. Some of the customers also received a home energy audit or payment counseling phone calls. The Customer Assistance Maintenance Program (CAMP) pilot tested changes to BGE’s CAMP including increased bill credits, payment counseling, or both. APPRISE’s process and impact evaluation assessed the effectiveness of these pilots in increasing customers’ on-time bill payment and energy conservation behavior.
Duquesne Light implemented Universal Service Programs to improve affordability for low-income customers. The programs include the Customer Assistance Program (CAP), which provides reduced payments and arrearage forgiveness; Smart Comfort, which provides energy efficiency and energy education services; CARES, which provides outreach and referral services; and the Hardship Fund, which provides emergency assistance. APPRISE conducted an evaluation of the performance of these programs and the integration among the different programs. The research included interviews with program and agency staff, a participant survey, and analysis of the impacts of the program on affordability and payment compliance.
FirstEnergy implemented Universal Service Programs to help low-income customers maintain electric service and protect customers’ health and safety. The programs include the CAP Program, which provides reduced payments and arrearage forgiveness; LIURP/WARM, which provides energy efficiency and energy education services; CARES, which provides outreach and referral services; the Hardship Fund, which provides emergency assistance; and Gatekeeper, where field personnel recognize and report customers who may be in distress. APPRISE conducted an evaluation of the performance of these programs, their ability to help low-income customers maintain affordable service, and the integration among the different programs. This evaluation included analysis of program databases, interviews with FirstEnergy managers and staff, interviews with staff at organizations that implement the programs; and analysis of the impact of CAP on bills, customer payments, and affordability.
The Illinois Department of Healthcare and Family Services (HFS) has designed the Percent of Income Payment Plan (PIPP) pilot to help low-income households maintain energy service throughout the year and reduce the health and safety risks associated with energy service disruption. The PIPP program offers LIHEAP recipients arrearage forgiveness and a fixed payment plan in the place of a one-time LIHEAP grant. It also tests the effectiveness of offering weatherization and energy conservation, as well as counseling and education services to program participants. It is important for HFS to have information on the appropriateness of the program development, the effectiveness of the program implementation, and the program impact on participating clients and the participating utility. APPRISE conducted a process and impact evaluation of this program that assessed whether the program was implemented effectively and whether the program achieved its intended effect on clients.
Louisville Gas and Electric Company and Kentucky Utilities Company offer a Home Energy Assistance (HEA) Program to assist the poorest households in their service territories least able to afford to pay their energy bills. Funded by ratepayers and shareholders, the programs (structured differently by company) provide fixed monthly credits to LIHEAP participants. APPRISE conducted a comprehensive HEA assessment to analyze the program management, availability, and customer participation; assess impacts on payments, arrearages, collections actions, and weatherization participation; and make recommendations for program improvement.
The New Jersey Statewide Heating Assistance and Referral for Energy Services (NJ SHARES) is a nonprofit corporation organized to provide assistance to individuals and families living in New Jersey who are in need of temporary help paying their energy bills. APPRISE conducts an annual impact evaluation of the New Jersey SHARES program. The evaluation includes analysis of administrative and utility data to assess the program’s impact on maintenance of electric and gas service and energy bill payment. In 2012, APPRISE also conducted a survey of 2011 grant recipients to assess whether the crises had been resolved, what additional assistance was needed, and the interest and potential for energy efficiency program participation. APPRISE also serves as an advisor to the NJ SHARES Board of Directors.
The New Jersey Universal Service Fund (USF) program provides fixed credits to eligible low-income families to limit electric and gas bills to six percent of household income. APPRISE conducted a process and impact evaluation of this program. This evaluation provided information on the benefits and costs of the USF program and alternatives for program design and operations. Program evaluation research consisted of administrative interviews, client surveys, program and utility data analysis, and financial data analysis.
Ohio’s Percentage of Income Payment Program (PIPP) was revised as the PIPP Plus in November 2010. The program helps low-income households maintain utility service by providing a reduced bill based on a percentage of the household’s annual income. APPRISE conducted an evaluation of this program to assess the effectiveness of the changes to PIPP that aimed to provide incentives for regular, on-time payments and consequences for failure to adhere to program rules. The research included administrative interviews, participant surveys, and analysis of program and utility data.
PECO implemented Universal Service Programs to help low-income customers maintain energy services and protect customers’ health and safety. The programs include CAP which provides reduced payments and arrearage forgiveness, LIURP which provides energy efficiency and energy education services, CARES which provides outreach and referral services, and MEAP which provides emergency financial assistance for overdue energy bills. APPRISE conducted the 2006 and the 2012 evaluations of PECO’s Universal Services Programs. The research included review of program procedures and implementation; customer needs assessment; interviews with program managers and implementation partners; customer surveys; and an analysis of the impact of the program on energy affordability, bill payment, and collections. The work also included preparation of the Universal Services Programs Evaluation Report, submitted to the Pennsylvania Public Utilities Commission.
APPRISE is conducting an evaluation of Peoples Natural Gas Universal Service Programs.  Peoples’ Universal Service Programs provide reduced natural gas bills, arrearage forgiveness, emergency bill payment assistance, energy efficiency services, and case management services to low-income households.  The evaluation will assess the efficiency and effectiveness of these programs through background research, needs assessment research, program administrator interviews, customer surveys, program database analysis, and analysis of the impacts of the program on affordability, bill payment, and collections actions and costs.
PG Energy implemented their Customer Assistance Program (CAP), called the PG Energy Partners Program (PGEPP), to assist eligible low-income residential customers to pay their gas bills and reduce arrearages. Participating households pay a monthly bill that equals a percentage of their income, and receive monthly arrearage forgiveness when bills are paid on time and in full. APPRISE evaluated this program to determine how to increase efficiency of administration and to measure the impacts of the program on affordability, bill payment, collections activity, and gas usage.
APPRISE conducted a Process and Impact Evaluation of PGW’s Customer Responsibility Program in 2006 and 2012. This program provides low-income PGW customers with a reduced bill to increase energy affordability. The evaluations assessed the characteristics of program participants, payment compliance by program participants, and arrearage forgiveness received by program participants. The research estimated the impact of the program on affordability, payments, and collections actions and costs. Evaluation activities included interviews with program managers and staff, a survey of participating and non-participating customers, and analysis of customer billing and payment data.
APPRISE conducted a Process and Impact Evaluation of PPL Electric Utilities’ (PPL) Universal Services Programs in 2008 and in 2014. The programs provide energy efficiency services and bill payment assistance to PPL’s low-income customers. The 2014 evaluation included a needs assessment, interviews with program managers and staff, interviews with service delivery agencies, surveys with participating and non-participating customers, and analysis of the impact of the programs on energy affordability and bill payment practices.
T.W. Phillips established a customer assistance program called the Energy Help Fund (EHF) as part of its Universal Services Programs. The EHF provides low-income customers with rate discounts and arrearage forgiveness. APPRISE conducted a Process and Impact Evaluation of the program. The evaluation included administrative interviews, customer surveys, and analysis of payment and program data.
UGI has a Customer Assistance Program (CAP) to improve energy affordability for low-income customers. The program provides a reduced natural gas bill based on the customer’s poverty level and energy usage. APPRISE conducted a Process and Impact Evaluation of this program. The research included an assessment of the program design; review of the program costs and implementation; participant and non-participant feedback; and analysis of the impacts of the program on bill payment, arrearages, and collections costs.

APPRISE Impact Evaluations – Usage Reduction Programs

AmerenUE partnered with the Missouri Department of Natural Resources Energy Center and other utilities to provide weatherization services to low-income households through the Low Income Weatherization Program (LIWP). The goal of the program is to provide energy efficient improvements to low-income customers to reduce their utility bills and help them prepare for the heating and cooling seasons. APPRISE conducted a process and impact evaluation of the weatherization program. The goals of this research were to assess the effectiveness of the program and to identify opportunities for program improvement.
The 2004 Colorado REACh grant developed a comprehensive cooling program to assist Colorado’s low-income households. Local agency partners implemented cooling program services, including equipment installation and client education, and the state LIHEAP office furnished community-wide education and notification services. APPRISE conducted a comprehensive Process and Impact Evaluation of this program.
FirstEnergy offers a Low-Income Usage Reduction Program (LIURP) and Act 129 Low-Income Programs to assist low-income customers. The programs provide energy efficiency and energy education services, help low-income customers maintain electric service, and protect customers’ health and safety. APPRISE conducted a study to assess recent program changes, and to provide guidance to FirstEnergy on future tracking, reporting, and process improvements. The research included an assessment of what changes should be made to program procedures, whether additional measures should be added, and how projected energy savings for additional energy efficiency measures should be calculated.
First State CAA is implementing a heating replacement and energy conservation program for the Delaware Department of Health and Social Services. The Repair Replace Heaters and Conserving Energy (RRHACE) program is designed to assess the needs of low-income households and to deliver a customized set of services, including heating equipment assessment and services, Healthy Homes assessment and services, and energy education. APPRISE is advising First State on the Data Tracking System and conducting Process and Impact Evaluations. The Process Evaluation will assess challenges and opportunities, and will develop performance measures to be assessed in the Impact Evaluation.
APPRISE was the lead contractor for the evaluation of the National Low-Income Weatherization Assistance Program (WAP). APPRISE had three key partners for this effort – Blasnik and Associates, Dahloff Associates, and the Energy Center of Wisconsin, and was also supported by technical experts from across the country. The evaluation consisted of data collection from state WAP offices, local agencies, and participating and nonparticipating clients. Utility data was collected and analyzed for a sample of clients from every state and submetering studies were conducted to assess impacts on clients who use bulk fuels and to conduct special technical studies. The evaluation assessed program operations and delivery, energy impacts, and non-energy impacts, and made recommendations for modifications to program design and delivery.
The New Hampshire REACh Furnace Cleaning Program aimed to decrease energy usage and reduce home energy insecurity for low-income households in northern New Hampshire by cleaning furnaces and identifying furnaces in need of repair and replacement. Extensive furnace repairs and furnace replacements were also provided in those homes where the need was identified. APPRISE’s evaluation included interviews with program managers, community agency staff, and furnace cleaners; and analysis of the impact of the program on energy usage, energy insecurity, and furnace safety and efficiency indicators.
The Comfort Partners Program is a statewide collaborative utility residential low-income program, providing joint delivery of services through electric and gas utilities. Participating customers receive direct installation of energy efficiency measures, energy education, and payment plans aimed to reduce arrearages. APPRISE conducted a comprehensive evaluation of this program, including a Process Evaluation and analysis of the program’s data tracking system, the comprehensiveness of service delivery, and the impacts on energy usage and bill payment.
The NJ Comfort Partners Program provides no-cost energy efficiency services to low-income utility customers throughout the state. APPRISE conducted a comprehensive evaluation of this program. The Data Tracking Research assessed the comprehensiveness, sufficiency, and accuracy of the program database. The Process Evaluation included review and assessment of materials and procedures; interviews with utility managers; interviews and on-site observations at service delivery contractors; on-site observation of audits, installations, and third party inspections; and post-completion inspections of program installations. The Affordability Evaluation included a telephone survey with program participants and analysis of the program’s impact on affordability of energy bills and participation in the NJ Universal Service Fund program. The Impact Evaluation included analysis of program data, engineering impact analysis, usage impact analysis, and savings realization analysis. The Energy Savings Protocols Research included review and assessment of protocols currently in use to estimate program savings.
The NJ Comfort Partners Program provides no-cost energy efficiency services to low-income utility customers throughout the state. APPRISE last conducted a comprehensive evaluation of this program in 2013.  The 2017 evaluation will review program changes that have been made since the last evaluation, estimate energy savings with a billing analysis, review and make recommendations for changes to the Energy Saving Protocols, and conduct targeted participant telephone interviews and on-site inspections of completed jobs.
The New Jersey Comfort Partners Seniors Pilot extends Comfort Partners weatherization services to seniors in Monroe Township, New Jersey, with incomes above the guidelines for the Comfort Partners Program. APPRISE conducted an Impact Evaluation of this program. The evaluation measured the electric usage reduction that resulted from the program.
The purpose of this research was to furnish the New Jersey Board of Public Utilities (BPU) and the New Jersey Clean Energy Council with detailed documentation of the existing weatherization programs in New Jersey and estimates of the usage reduction impacts of the NJ WAP. The analysis addressed the differences and similarities between the program models and the rationale for each program’s design. The evaluation also provided impact results for two agencies that provide WAP services in New Jersey. This information was designed to help the BPU and the Clean Energy Council assess which model might be most appropriate for a ratepayer-funded low-income usage reduction program.
APPRISE conducted national evaluations of the Department of Energy’s Weatherization Assistance Program (WAP) for program years 2008 and 2010.   In this study of New York’s WAP for the New York State Division of Housing and Community Renewal (DHCR), APPRISE is using data collected from New York State to furnish detailed information on the performance of the New York WAP program.  This analysis will provide statewide statistics, results for selected subgroups, and program savings and cost-effectiveness statistics.
Niagara Mohawk’s Low-Income Customer Assistance Program (LICAP) was designed to address the problems of low-income customers who are unable to pay their full energy bills. Payment-troubled customers may receive an affordable payment arrangement, energy use management education, and a variety of energy efficiency services including weatherization and energy efficient appliance replacement. APPRISE conducted Process and Impact Evaluations of this program.
Ohio’s Department of Development, Office of Community Services, was awarded a REACh grant to research how to provide additional resources to low-income weatherization services in a way that addresses the more comprehensive needs of low-income households. These funds were used to assess the in-home environmental health risks for vulnerable individuals, prioritize usage reduction measures to meet the energy and health needs of vulnerable individuals, and provide additional measures that enhance the health and safety of the home. The program also established a partnership with vulnerable households to take actions to mitigate the environmental health risks in the home and refer clients to other services that can improve the health of the home and the clients who live there. APPRISE conducted a process and impact evaluation of this project.
PECO’s Low-Income Usage Reduction Program (LIURP) provides energy efficiency services and energy education to PECO’s low-income customers to reduce their energy usage and increase the affordability of their energy bills. The evaluations estimated the impact of LIURP on participants’ energy usage, energy bills, and payment compliance and surveyed customers to analyze the impact of energy education and client behavioral changes. In the 2012 program evaluation, APPRISE also conducted observations of service delivery and inspections on a sample of completed jobs. This research provided information on potential changes to program services and implementation that may lead to increased program savings. The current evaluation of 2015 participants will assess the program process, examine the impacts of program treatments on energy usage and affordability, and survey customers to assess program satisfaction.
PGW’s ELIRP provides no-cost comprehensive weatherization treatments to selected, high-usage PGW bill payment assistance program participants. APPRISE conducted an impact evaluation and a market study for the program. The impact evaluation estimated energy savings, assessed savings achieved through individual major measures, calculated savings obtained by each contractor, and analyzed client targeting to make recommendations for how greater savings may be achieved. The market study included interviews with program managers, interviews with implementation contractors, and review of inspection reports to identify how PGW can improve the program’s efficiency and effectiveness.
PPL Electric Utilities (PPL) implemented the Winter Relief Assistance Program (WRAP) in 1984 to help reduce electric bills and improve home comfort for low-income customers. The objectives of the WRAP are to reduce energy usage and bills of low-income customers and to increase low-income customers’ ability to pay their electric bills, resulting in reduced arrearages. The program also aims to increase health, safety, and comfort for low-income occupants; create and maintain partnerships with community based organizations and contractors; and make referrals to other low-income assistance programs. APPRISE conducted a comprehensive Process and Impact evaluation of PPL’s WRAP. The research consisted of administrative interviews, review of program procedures, on-site observation of service delivery, customer interviews, service delivery provider interviews, and usage and payment impact analyses.
Sacramento Municipal Utility District’s (SMUD) Low Income Weatherization Program provides no-cost energy services to qualified low-income residential customers. The goals of the program are to reduce electric bills, improve safety and comfort, provide energy education to reduce usage and lower bills, and partner with contractors and community organizations in providing these services. APPRISE’s comprehensive evaluation included needs assessment, interviews with program administrators and contractors, a participant survey, on-site inspections of completed jobs, and billing analysis to measure the impacts on energy usage.
UGI’s Low-Income Usage Reduction Program (LIURP) provides no-cost energy efficiency services to low-income customers in its service territory. APPRISE conducted a Process and Impact Evaluation of this program. The research included review and assessment of program materials and procedures, on-site interviews with UGI managers and staff, in-depth telephone interviews with a sample of program participants, and analysis of the program’s impact on the energy usage of participating customers.

APPRISE Impact Evaluations – Market Transformation Programs

New Jersey Natural Gas operates energy efficiency programs that are complementary to the New Jersey Clean Energy Programs (NJCEP) as the SAVEGREEN project. The residential program provides grants or on-bill repayment plans to assist with the purchase and installation of furnaces/boilers and water heaters. The commercial program provides NJCEP Direct Install participants with a no-interest on-bill repayment plan for the value of the project not covered by the NJCEP incentive. APPRISE conducted an evaluation of the SAVEGREEN project to assess program management, analyze the incremental impact on energy efficiency activity, determine the extent to which the program influences residential customer implementation of whole house improvements, and measure the program’s energy savings.
PGW’s CIER provides prescriptive rebates to PGW customers who purchase large commercial and industrial natural gas boilers and commercial food service equipment to encourage the purchase of the most energy-efficient units available. APPRISE conducted an impact evaluation and a market study for the program. The impact evaluation estimated the change in energy usage for program participants due to the program, evaluated the application rejection rate, frequency of measure installation, and measure-specific savings. The market study interviewed program participants and nonparticipants, installers, and distributers to understand how participants learned about the program, incremental costs for high-efficiency equipment, availability of high-efficiency equipment, sufficiency of current rebate levels, and how the program could be modified to encourage increased participation.
PGW’s CIRI provides incentives to multi-family residential, commercial, and industrial customers to install comprehensive energy efficiency upgrades. APPRISE conducted a market study and impact evaluation for the program. The market study included interviews with program managers, technical advisors, and program participants and nonparticipants to identify how PGW could improve marketing, participation, measure selection, and financing availability. The impact evaluation assessed installed measures, estimated energy savings, and made recommendations for how greater savings may be achieved.
PGW’s HECI provides incentives to home builders and commercial developers who build properties that save a specified percentage of natural gas. PGW provides offerings for single-family residential properties and for commercial, industrial, and multi-family properties. APPRISE conducted a market study of the program to understand the marketing efforts that have been most successful, the barriers to current program participation, the adequacy of current incentive levels, and opportunities to encourage increased program participation. The market study included in-depth interviews with program administrators and technical advisors, participants and nonparticipants who expressed interest, and architects and design build firms. APPRISE also conducted an impact evaluation to estimate the impact of the program on natural gas usage.
PGW’s Home Rebate Program provides incentives to residential customers and participating Conservation Service Providers (CSPs) who complete comprehensive natural gas energy efficiency retrofits.  Financial incentives are designed to encourage customers and CSPs to pursue as many cost-effective energy efficiency measures as possible and achieve at least twenty percent energy savings on average for the program.  The market study and impact evaluation included interviews with program administrators and CSPs, in-depth interviews with partial and full program participants, analysis of the measures installed, assessment of the impact of the program on natural gas usage, and estimation of the program’s realization rate and cost-effectiveness.
PGW’s RHER provides rebates to customers who purchase residential-sized natural gas heating equipment to encourage purchase of the most efficient boilers and furnaces available. The program provides prescriptive cash rebates to offset most of the incremental cost of the higher efficiency equipment and installation. APPRISE previously conducted an impact evaluation and a market study for the program soon after the program was implemented. The impact evaluation assessed the impacts of the program on natural gas usage by measure type, efficiency, and pre-treatment usage, and compared estimated savings to deemed savings. The market study included interviews with the program manager, HVAC contractors, and program participants to assess the impact of the program on the market for high efficiency equipment, obstacles faced in program participation, and the level of financial incentives needed to encourage adoption of this equipment. APPRISE also conducted a second impact evaluation to assess the energy savings achieved by the program in the 2014 program year.
South Jersey Gas has been operating energy efficiency programs in coordination with the New Jersey Clean Energy Program (NJCEP) since 2009. The programs provide additional rebates and financing to customers who participate in the NJCEP residential and commercial and industrial programs. APPRISE conducted an evaluation of these programs to assess program design, implementation, and impacts. The research included interviews with program administrators, contractors, and participants; surveys with participants of three different programs; analysis of the impacts of the programs on natural gas usage; and assessment of the non-energy impacts of the programs.