RESEARCH ACTIVITIES – Performance Measurement Research 2018-06-14T16:55:43+00:00

Performance Measurement Research

Performance measurement involves the development of metrics to assess program success and the use of program data to measure those metrics. Some examples of metrics used include the number of households served by a program, the program penetration rate, a targeting index for vulnerable households, and the percentage reduction in energy usage or energy burden.

Performance Measurement Outputs

  • A Logic Model that describes the program assumptions, inputs, immediate outcomes, intermediate outcomes, and final program impacts.
  • A set of performance measures that allows program managers and policymakers to assess the success of a program in meeting defined goals.
  • A Data and Indicator Model that maps the specific outcomes to performance indicators and the data that are needed to measure those performance indicators.
  • Periodic calculations of performance measures to determine how a program performs over time and with changes in program design and implementation.

APPRISE Performance Measurement Research

APPRISE conducted a comprehensive evaluation of Allegheny Power’s Universal Service Programs. The evaluation research included interviews with Allegheny and agency program managers and staff, surveys with participating and nonparticipating customers, analysis of program databases, and analysis of the impacts of the payment assistance program on energy affordability, payment compliance, arrearages, and collections costs.
Ameren introduced a pilot of a low-income payment assistance program that provides electric bill payment assistance to low-income customers. Participants receive monthly bill credits toward their electric bill and additional credits toward their arrearages. APPRISE provided an evaluation of this program in 2012 that included interviews with local agencies that conduct intake for the program, a survey of participating customers, analysis of program data, and analysis of program impacts on affordability, payment coverage, and terminations. Ameren made program modifications in response to findings from the initial evaluation. APPRISE conducted a follow-up Process and Impact Evaluation of the program in 2014 that documented the impacts of the program changes. APPRISE conducted a third evaluation of this program in 2016 that included interviews with defaulted customers, analysis of program data, and analysis of the impacts of the program on affordability and bill payment. Findings from this research were used to further refine the program.
The purpose of this study was to assist DDOE’s review of its LIHEAP design and to make recommendations for program modifications. This study assessed DDOE’s LIHEAP procedures, analyzed the population of households eligible for energy assistance, and calculated the impacts of the assistance on participants’ energy burden. The study furnished DDOE with information needed to modify its program benefits design matrix in a way that meets the statutory guidance furnished to DDOE by the federal LIHEAP program.
The purpose of this study was to assist DDOE’s efforts to mitigate the rising energy costs on those with the highest home energy needs, greatest energy burdens, and least amount of available resources. The study focused on households that pay utility gas and/or electric bills, and are income-eligible and program-eligible for rate discounts. The study characterized the population of households that are eligible for rate discounts, estimated the rate discount program penetration rates, and analyzed how the rate discounts complement the LIHEAP program to reduce energy burdens for participating households.
Duquesne Light implemented Universal Service Programs to improve affordability for low-income customers. The programs include the Customer Assistance Program (CAP), which provides reduced payments and arrearage forgiveness; Smart Comfort, which provides energy efficiency and energy education services; CARES, which provides outreach and referral services; and the Hardship Fund, which provides emergency assistance. APPRISE conducted an evaluation of the performance of these programs and the integration among the different programs. The research included interviews with program and agency staff, a participant survey, and analysis of the impacts of the program on affordability and payment compliance.
FirstEnergy offers a Low-Income Usage Reduction Program (LIURP) and Act 129 Low-Income Programs to assist low-income customers. The programs provide energy efficiency and energy education services, help low-income customers maintain electric service, and protect customers’ health and safety. APPRISE conducted a study to assess recent program changes, and to provide guidance to FirstEnergy on future tracking, reporting, and process improvements. The research included an assessment of what changes should be made to program procedures, whether additional measures should be added, and how projected energy savings for additional energy efficiency measures should be calculated.
The Illinois REACh project was an innovative pilot program aimed to help low-income households maintain energy service throughout the year and reduce the health and safety risks associated with energy service disruption. The comprehensive program worked with clients to increase resources available for paying energy bills, reduce the amount of energy used through effective energy-saving actions and participation in usage reduction programs, and develop the skills that are needed to maintain energy service through proactive management of their monthly budgets and energy bills. APPRISE conducted an evaluation of this program to assist program managers in assessing whether the program design is appropriate, whether there are any barriers to implementing the program as designed, and whether the program has achieved the intended impacts.
Louisville Gas and Electric Company and Kentucky Utilities Company offer a Home Energy Assistance (HEA) Program to assist the poorest households in their service territories least able to afford to pay their energy bills. Funded by ratepayers and shareholders, the programs (structured differently by company) provide fixed monthly credits to LIHEAP participants. APPRISE conducted a comprehensive HEA assessment to analyze the program management, availability, and customer participation; assess impacts on payments, arrearages, collections actions, and weatherization participation; and make recommendations for program improvement.
APPRISE previously worked with the federal LIHEAP office to develop a performance measurement plan for targeting heating assistance to households with vulnerable members and a logic model to support that performance measurement plan. APPRISE also developed a separate model to examine the targeting of elderly households. APPRISE then extended the logic models to address all LIHEAP program components including cooling assistance, crisis assistance, home energy-related equipment repair and replacement, weatherization, and Assurance 16 activities. APPRISE also developed a logic model for targeting LIHEAP income-eligible households with at least one young child. The final report recommended an integrated strategy for implementing all of the logic models.

The Minnesota Conservation Improvement Programs (CIP) portfolios include low-income components to assist income-eligible households with energy conservation. APPRISE conducted research through the Conservation Applied Research & Development (CARD) Grant Program to provide a comprehensive review of the low-income efficiency programs, evaluate the program processes, and assess program policies and guidelines. The goal of the research was to facilitate more effective program operations and policymaking.

The Community Service Block Grant Program provides funding to alleviate the causes and conditions of poverty in communities through services that address employment,  income management, education, housing, nutrition, and health.  APPRISE is supporting the National Association of State Community Service Programs to develop a Data Warehouse for the federal Community Services Block Grant (CSBG) program.  We will develop tools for reporting, validation, and analysis of performance measure data at the local, state, territory, and national level.  Additionally, APPRISE is supporting state efforts to collect and report new performance metrics established by the Department of Health and Human Services, Office of Community Services.  The project will improve data quality, enhance understanding of CSBG outcomes, and inform program decision-making. 
APPRISE was the lead contractor for the evaluation of the National Low-Income Weatherization Assistance Program (WAP). APPRISE had three key partners for this effort – Blasnik and Associates, Dahloff Associates, and the Energy Center of Wisconsin, and was also supported by technical experts from across the country. The evaluation consisted of data collection from state WAP offices, local agencies, and participating and nonparticipating clients. Utility data was collected and analyzed for a sample of clients from every state and submetering studies were conducted to assess impacts on clients who use bulk fuels and to conduct special technical studies. The evaluation assessed program operations and delivery, energy impacts, and non-energy impacts, and made recommendations for modifications to program design and delivery.
The New Hampshire REACh Furnace Cleaning Program aimed to decrease energy usage and reduce home energy insecurity for low-income households in northern New Hampshire by cleaning furnaces and identifying furnaces in need of repair and replacement. Extensive furnace repairs and furnace replacements were also provided in those homes where the need was identified. APPRISE’s evaluation included interviews with program managers, community agency staff, and furnace cleaners; and analysis of the impact of the program on energy usage, energy insecurity, and furnace safety and efficiency indicators.
The NJ Comfort Partners Program provides no-cost energy efficiency services to low-income utility customers throughout the state. APPRISE conducted a comprehensive evaluation of this program. The Data Tracking Research assessed the comprehensiveness, sufficiency, and accuracy of the program database. The Process Evaluation included review and assessment of materials and procedures; interviews with utility managers; interviews and on-site observations at service delivery contractors; on-site observation of audits, installations, and third party inspections; and post-completion inspections of program installations. The Affordability Evaluation included a telephone survey with program participants and analysis of the program’s impact on affordability of energy bills and participation in the NJ Universal Service Fund program. The Impact Evaluation included analysis of program data, engineering impact analysis, usage impact analysis, and savings realization analysis. The Energy Savings Protocols Research included review and assessment of protocols currently in use to estimate program savings.
The NJ Comfort Partners Program provides no-cost energy efficiency services to low-income utility customers throughout the state. The 2017 evaluation reviewed program changes that were made since the 2013 evaluation, estimated energy savings with a billing analysis, reviewed and made recommendations for changes to the Energy Saving Protocols, and conducted targeted participant telephone interviews and on-site inspections of completed jobs.  The evaluation found improved energy savings and recommended additional program refinements to further increase savings achieved through the program.
The New Jersey Statewide Heating Assistance and Referral for Energy Services (NJ SHARES) is a nonprofit corporation organized to provide assistance to individuals and families living in New Jersey who are in need of temporary help paying their energy bills. APPRISE conducts an annual impact evaluation of the New Jersey SHARES program. The evaluation includes analysis of administrative and utility data to assess the program’s impact on maintenance of electric and gas service and on energy bill payment.  Some of the previous evaluations also included surveys and in-depth participant interviews to assess whether the crises had been resolved, what additional assistance was needed, and the interest and potential for energy efficiency program participation.   APPRISE also serves as an advisor to the NJ SHARES Board of Directors.
APPRISE conducted national evaluations of the Department of Energy’s Weatherization Assistance Program (WAP) for program years 2008 and 2010.   In this study of New York’s WAP for the New York State Division of Housing and Community Renewal (DHCR), APPRISE used data collected from New York State to furnish detailed information on the performance of the New York WAP program.  This analysis provided statewide statistics, results for selected subgroups, and program savings and cost-effectiveness statistics.
The Ohio Department of Development, Office of Energy Efficiency, has implemented programs funded by the Universal Service Fund. Participating customers receive cost-effective energy efficiency services and education targeted at reducing electric usage. APPRISE conducted a Process Evaluation of this program. The work included review of all program documents; interviews with program managers, staff, and contractors; observation of service delivery; customer surveys; contractor surveys; and economic analysis of the impacts of the program on Ohio’s economy.
Ohio’s Percentage of Income Payment Program (PIPP) was revised as the PIPP Plus in November 2010. The program helps low-income households maintain utility service by providing a reduced bill based on a percentage of the household’s annual income. APPRISE conducted an evaluation of this program to assess the effectiveness of the changes to PIPP that aimed to provide incentives for regular, on-time payments and consequences for failure to adhere to program rules. The research included administrative interviews, participant surveys, and analysis of program and utility data.
Ohio’s Department of Development, Office of Community Services, was awarded a REACh grant to research how to provide additional resources to low-income weatherization services in a way that addresses the more comprehensive needs of low-income households. These funds were used to assess the in-home environmental health risks for vulnerable individuals, prioritize usage reduction measures to meet the energy and health needs of vulnerable individuals, and provide additional measures that enhance the health and safety of the home. The program also established a partnership with vulnerable households to take actions to mitigate the environmental health risks in the home and refer clients to other services that can improve the health of the home and the clients who live there. APPRISE conducted a process and impact evaluation of this project.
PECO implemented Universal Service Programs to help low-income customers maintain energy services and protect customers’ health and safety. The programs include CAP which provides reduced payments and arrearage forgiveness, LIURP which provides energy efficiency and energy education services, CARES which provides outreach and referral services, and MEAP which provides emergency financial assistance for overdue energy bills. The 2019 evaluation will include review of program procedures and implementation; customer needs assessment; interviews with program managers and implementation partners; customer surveys; and an analysis of the impact of the program on energy affordability, bill payment, collections, and energy usage.
APPRISE conducted an evaluation of Peoples Natural Gas Universal Service Programs.  Peoples’ Universal Service Programs provide reduced natural gas bills, arrearage forgiveness, emergency bill payment assistance, energy efficiency services, and case management services to low-income households.  The evaluation assessed the efficiency and effectiveness of these programs through background research, needs assessment research, program administrator interviews, customer surveys, program database analysis, and analysis of the impacts of the program on affordability, bill payment, and collections actions and costs.
PGW’s ELIRP provides no-cost comprehensive weatherization treatments to selected high-usage PGW bill payment assistance program participants. APPRISE is conducting an impact evaluation for the 2016 program participants.  The impact evaluation will estimate energy savings, assess savings achieved by individual contractors, and assess bill savings.  Previous evaluations studied energy saving impacts achieved through individual major measures and analyzed client targeting to make recommendations for how greater savings may be achieved. A previous market study also included interviews with program managers, interviews with implementation contractors, and review of inspection reports to identify how PGW can improve the program’s efficiency and effectiveness.
PGW’s Customer Responsibility Program provides low-income PGW customers with a reduced bill to increase energy affordability. Their other Universal Service Programs provide energy efficiency services, case management services, and emergency bill payment assistance.  The 2019 evaluation will assess the characteristics of program participants, payment compliance by program participants, and arrearage forgiveness received by program participants. The research will estimate the impact of the program on affordability, payments, and collections actions and costs. Evaluation activities will include interviews with program managers and staff, needs assessment, in-depth interviews with participating customers, and analysis of customer billing, payment, and collections data.
APPRISE conducted a Process and Impact Evaluation of PPL Electric Utilities’ (PPL) Universal Services Programs in 2008 and in 2014. The programs provide energy efficiency services and bill payment assistance to PPL’s low-income customers. The 2014 evaluation included a needs assessment, interviews with program managers and staff, interviews with service delivery agencies, surveys with participating and non-participating customers, and analysis of the impact of the programs on energy affordability and bill payment practices.
Sacramento Municipal Utility District’s (SMUD) Low Income Weatherization Program provides no-cost energy services to qualified low-income residential customers. The goals of the program are to reduce electric bills, improve safety and comfort, provide energy education to reduce usage and lower bills, and partner with contractors and community organizations in providing these services. APPRISE’s comprehensive evaluation included needs assessment, interviews with program administrators and contractors, a participant survey, on-site inspections of completed jobs, and billing analysis to measure the impacts on energy usage.
UGI’s Low-Income Usage Reduction Program (LIURP) provides no-cost energy efficiency services to low-income customers in its service territory. APPRISE conducted a Process and Impact Evaluation of this program. The research included review and assessment of program materials and procedures, on-site interviews with UGI managers and staff, in-depth telephone interviews with a sample of program participants, and analysis of the program’s impact on the energy usage of participating customers.
UGI has a Customer Assistance Program (CAP) to improve energy affordability for low-income customers. The program provides a reduced natural gas bill based on the customer’s poverty level and energy usage. APPRISE conducted a Process and Impact Evaluation of this program. The research included an assessment of the program design; review of the program costs and implementation; participant and non-participant feedback; and analysis of the impacts of the program on bill payment, arrearages, and collections costs.
The Vermont REACh program created a private, non-profit energy consumers’ cooperative called Consumerco. This cooperative aims to achieve the Vermont REACh goal by providing comprehensive energy and efficiency services to its members. Consumerco aims to assist its low-income consumers in attaining energy self-sufficiency by instructing them how to minimize energy usage, choose the most cost-effective fuel, obtain the least cost supply of fuel, receive fuel assistance, and finance weatherization. In this project, we conducted background research, provided a process evaluation of the Consumerco, provided a process evaluation of the case management, and conducted an impact evaluation. We furnished concrete performance data and program documentation to the federal LIHEAP office and REACh program managers.