Process Evaluation Research
Administrative Research: Assessment of the administrative context for the program.
Implementation Research: Analysis of the effectiveness of program recruitment, intake, service delivery, follow-up, and tracking.
Customer Research: Analysis of satisfaction, motivation, retention of information, and behavioral changes.
Non-Energy Benefits Research: Measurement of health and safety, economic, environmental, corporate positioning, and customer benefits.
Process Evaluation Activities
Data and Document Review
- Document Review: Review of program history, program goals, and program policy statements.
- Program Budget Review: Analysis of resources devoted to the project, including contract funding and staff time allocation.
- Program Statistics Review: Analysis of production and production rates by target groups, service delivery area, and other variables of interest.
- Quality Control Check: Assessment of the quality and reliability of program operations by direct observation or review of quality control inspection documents.
Interviews and Surveys
- Executive Interviews: Guided in-depth interviews with senior program managers regarding program goals, program resource requirements, program oversight, and metrics for success.
- Staff Interviews: Detailed interviews with program staff to document details of outreach, recruitment, eligibility, intake, scheduling, service delivery, information tracking, and follow-up.
- In-Depth Customer Interviews: Guided and open-ended interviews with program participants.
- Customer Surveys: Telephone, mail, or in-person interviews with customers who have received program services.
- Focus Groups: Group discussions with service delivery staff or customers who have received program services.
Observations and Inspections
- On-Site Observations: Observation of all aspects of program implementation including outreach, recruitment, intake, scheduling, service delivery, and follow-up.
- On-Site Inspections: Post service inspection of work delivered by the program and interviews with program recipients.
Process Evaluation Outputs
Program Documentation
- Documentation of goals, mandate, and funding for the program
- Documentation of program administration roles and responsibilities
- Documentation of program protocols
Procedures and Context Assessment
- Assessment of the feasibility of program protocols
- Assessment of the barriers to effective program operation
- Assessment of the barriers to program impact
- Assessment of the population served
Implementation Assessment
- Verification of actual program procedures
- Assessment of the effectiveness of each element of program operations in meeting the program objectives
- Assessment of program operations efficiency and quality
- Assessment of individual program experiences
Performance Measurement
- Development of performance measurement indicators
- Analysis and quantification of customer understanding, actions, and satisfaction
- Analysis of program achievements as compared to program goals
- Recommendations for program modification
APPRISE Process Evaluations
Below we provide a few examples of some of our recent projects that included process evaluation research. Click the tabs below to learn more about these projects, or see here for a full list of projects that included process evaluation research.
PECO implemented Universal Service Programs to help low-income customers maintain energy services and protect customers’ health and safety. The programs include CAP which provides reduced payments and arrearage forgiveness, LIURP which provides energy efficiency and energy education services, CARES which provides outreach and referral services, and MEAP which provides emergency financial assistance for overdue energy bills. The 2019 evaluation conducted by APPRISE included a review of program procedures and implementation; customer needs assessment; interviews with program managers and implementation partners; customer surveys; and an analysis of the impact of the program on energy affordability, bill payment, collections, and energy usage.
APPRISE assessed the fiscal integrity and operational efficiency of the NJ LIHEAP and USF Programs. This research included an assessment of the performance of the New Jersey Department of Community Affairs (DCA), Division of Housing and Community Resources (DHCR), Office of Home Energy Assistance (OHEA) and an assessment of the operations of the LIHEAP and USF programs. We reviewed program documents, and conducted interviews with HHS LIHEAP compliance staff, DCA management, USF managers at the NJ Board of Public Utilities, and managers at Community Based Organizations that administer the programs. We also reviewed program databases and statistics, conducted on-site interviews and observations with a sample of local agencies, and conducted a survey with LIHEAP recipients.
Sacramento Municipal Utility District (SMUD) has various programs to provide assistance to low-income and special needs households. These programs include rate discounts, home weatherization services, and bill assistance. SMUD contracted with APPRISE to conduct research on SMUD’s low-income assistance programs. The study included an assessment of the characteristics and needs of SMUD’s low-income population, a benchmark study of low-income assistance programs offered by other utilities and organizations, a review of SMUD’s current low-income programs, and recommendations for program assessment metrics and potential program modifications or additions.
The Illinois REACh project was an innovative pilot program aimed to help low-income households maintain energy service throughout the year and reduce the health and safety risks associated with energy service disruption. The comprehensive program worked with clients to increase resources available for paying energy bills, reduce the amount of energy used through effective energy-saving actions and participation in usage reduction programs, and develop the skills that are needed to maintain energy service through proactive management of their monthly budgets and energy bills. APPRISE conducted an evaluation of this program to assist program managers in assessing whether the program design is appropriate, whether there are any barriers to implementing the program as designed, and whether the program has achieved the intended impacts.
NYSERDA is supporting training for building code, design, and construction officials across New York State to help reach State Energy Code compliance goals. The training provides detailed information on current and planned New York State Energy Code requirements, building science principles, and technical guidance on appropriate application of these principles. APPRISE designed the survey, coordinated with multiple training providers to administer the pre- and post-training survey to all training participants, and processed the survey data. The surveys collected information on knowledge and awareness of specific code requirements and feedback on the provided training. This research helped NYSERDA evaluate the impact of training courses and to identify future training opportunities.