Process Evaluation Research
- Administrative Research: Assessment of the administrative context for the program.
- Implementation Research: Analysis of the effectiveness of program recruitment, intake, service delivery, follow-up, and tracking.
- Customer Research: Analysis of satisfaction, motivation, retention of information, and behavioral changes.
- Nonenergy Benefits Research: Measurement of health and safety, economic, environmental, corporate positioning, and customer benefits.
Process Evaluation Activities
Data and Document Review
Document Review: Review of program history, program goals, and program policy statements.
Program Budget Review: Analysis of resources devoted to the project, including contract funding and staff time allocation.
Program Statistics Review: Analysis of production and production rates by target groups, service delivery area, and other variables of interest.
Quality Control Check: Assessment of the quality and reliability of program operations by direct observation or review of quality control inspection documents.
Interviews and Surveys
Executive Interviews: Guided in-depth interviews with senior program managers regarding program goals, program resource requirements, program oversight, and metrics for success.
Staff Interviews: Detailed interviews with program staff to document details of outreach, recruitment, eligibility, intake, scheduling, service delivery, information tracking, and follow-up.
In-Depth Customer Interviews: Guided and open-ended interviews with program participants.
Customer Surveys: Telephone, mail, or in-person interviews with customers who have received program services.
Focus Groups: Group discussions with service delivery staff or customers who have received program services.
Observations and Inspections
On-Site Observation: Observation of all aspects of program implementation including outreach, recruitment, intake, scheduling, service delivery, and follow-up.
On-site Inspection: Post service inspection of work delivered by the program and interviews with program recipients.
Process Evaluation Outputs
- Documentation of goals, mandate, and funding for the program
- Documentation of program administration roles and responsibilities
- Documentation of program protocols
Procedures and Context Assessment
- Assessment of the feasibility of program protocols
- Assessment of the barriers to effective program operation
- Assessment of the barriers to program impact
- Assessment of the population served
- Verification of actual program procedures
- Assessment of the effectiveness of each element of program operations in meeting the program objectives
- Assessment of program operations efficiency and quality
- Assessment of individual program experiences
- Development of performance measurement indicators
- Analysis and quantification of customer understanding, actions, and satisfaction
- Analysis of program achievements as compared to program goals
- Recommendations for program modification
APPRISE Process Evaluations – Payment Programs
APPRISE Process Evaluations – Usage Reduction Programs
The Minnesota Conservation Improvement Programs (CIP) portfolios include low-income components to assist income-eligible households with energy conservation. APPRISE conducted research through the Conservation Applied Research & Development (CARD) Grant Program to provide a comprehensive review of the low-income efficiency programs, evaluate the program processes, and assess program policies and guidelines. The goal of the research was to facilitate more effective program operations and policymaking.
PECO’s Low-Income Usage Reduction Program (LIURP) provides energy efficiency services and energy education to PECO’s low-income customers to reduce their energy usage and increase the affordability of their energy bills. The evaluations estimated the impact of LIURP on participants’ energy usage, energy bills, and payment compliance and surveyed customers to analyze the impact of energy education and client behavioral changes. The evaluation of 2016 participants assessed the program process, examined the impacts of program treatments on energy usage and affordability, surveyed customers to assess program satisfaction, and observed program audits to assess delivery of customer energy education.
APPRISE Process Evaluations – Market Transformation Programs
The Data Mining Research included analysis of the program database and the eligible customer database. This research assessed which market sectors are highly represented among the SBEA participants over the last five years, which sectors are underrepresented and which are missing; determined how the market sector distribution of program participants compares to those of nonparticipants; characterized the mix of measures installed in the program; and explored predicted savings by sector.
The Low-Income and Limited English Study explored how to increase participation by small business owners who are low-income or have limited English fluency. The study contacted organizations in the state who work with these types of small businesses, characterized the businesses and assessed eligibility for the program, and determined whether the organizations were willing to facilitate participation of these businesses in the SBEA.