Washington Service Corps - Performance Measurement Training & Technical Assistance (PMT&TA) Program Evaluation

The Washington Service Corps (WSC) is a multi-focus intermediary program. As part of the AmeriCorps network, WSC oversees a consortium of approximately 200 partner sites across Washington state, recruiting, placing, and training between 500 and 600 AmeriCorps members at these sites each year. WSC developed, tested, and evaluated an intervention intended to build the organizational capacity for program evaluation and performance measurement among partner sites. The intervention included online training webinars, site visits with program staff, customized recommendation reports, and follow-up phone technical assistance intended to increase knowledge, change attitudes, and build organizational capacity around evaluation and performance management.

Study Goals:
The goal of the study is to measure the program’s impact on participants’ knowledge, attitude, and confidence relating to evaluation and performance measurement topics, and their capacity to carry out evaluation and performance measurement activities. Knowledge was measured through multiple choice quiz questions and scenarios applying key concepts. The evaluation included qualitative data from participant and staff interviews, as well as observations of program site visits to provide insights on the implementation of the intervention.

Research Questions:
The research questions were:

• Does participation in the training intervention result in greater increases:
  • Pre-post in knowledge about evaluation and performance measurement topics including a) evidence of effectiveness b) program alignment, data collection and use, and c) high-quality data? 
  • Pre-post attitudes about and confidence with evaluation and performance measurement topics?
  • Pre-post capacity to carry out evaluation and performance measurement activities?

Findings:
The evaluation found the following:

• The treatment group showed significantly greater increases in knowledge about "evidence of effectiveness" compared to the control group although both groups improved in their knowledge of other concepts during the study period. 
• The treatment group also demonstrated statistically significant increases in confidence and reported increases in organizational capacity for evaluation and performance measurement. 
• The intervention was implemented consistently and with fidelity, and was well-received by participating site staff. 
• The tailored technical assistance and specific materials, resources, and background research was noted by the evaluator as particularly helpful in supporting staff to advance their evaluation and performance measurement process.

For more information, download:

Full Report