Submitted by kwilkerson on
  • test wizard page

Step 3: Determining readiness to scale

SCALER Step 3: Overview

As outlined in the SCALER framework, there are five conditions an intervention and the organization implementing it should meet to be ready to scale successfully and with fidelity.

Specifying the intervention

  • Specifying the intervention ensures that new organizations, personnel, or sites can implement core activities consistently and with fidelity to the intervention’s theory of change.
  • A well-specified intervention clearly defines or identifies the following components: core activities, personnel roles, a setting, and completion requirements.

Step 3 of the SCALER consists of a series of checklists that help you determine where your intervention and organization are in terms of meeting the five conditions for scaling readiness.

Start

FAQ

We recommend designating one person or role to be the SCALER owner, such as the intervention’s program or data manager. Because personnel at all levels are involved in intervention scaling, the SCALER owner will want to involve others including:

  • Executive directors
  • Program managers or frontline supervisors
  • Measurement, learning, and evaluation personnel (data managers or internal evaluation specialists)
  • Administrative personnel (human resources, development, or purchasing personnel)
Completing Step 3 requires enough knowledge and understanding of the following aspects of the intervention and implementing organization to self-assess scaling preparedness:

  • How you define the intervention and target population
  • Intervention core activities, key personnel, setting, and how to determine completion
  • Implementation supports and the enabling context
  • Implementation supports related to monitoring, CQI, performance goals, and data systems
  • The organization’s implementation infrastructure

Organizations can take one of several approaches:

  • Involved personnel could complete the checklists on their own, discuss responses, and come to agreement on scores at a meeting organized by the SCALER owner.
  • Several personnel could fill out the checklists together during a meeting.
  • The SCALER owner could gather the necessary information and complete all checklists.

In taking any of these approaches, the SCALER owner should set up a timeline for completing the tool.

The printable Step 3 Results report summarizes your scaling readiness scores and suggests next steps. You can:

  • Share the results with key stakeholders, organizational leaders, and funders.
  • Use the results to identify gaps in readiness and prioritize next steps.
  • Complete Step 3 multiple times as you continue to prepare for scaling to gauge your progress.

Specifying the intervention

Use this checklist to assess whether you have a well-specified intervention. Choose the response option that best characterizes your organization’s status in defining the following aspects of your intervention.

FAQ

A well-specified intervention clearly identifies the core set of activities that are critical for achieving beneficial outcomes. Typically, core activities are identified in an intervention’s theory of change, as the mechanisms by which an intervention produces the expected outcomes.

Four components are needed to ensure the organization’s specification of the intervention is enough to facilitate successful scaling:

  • Clearly identified core activities
  • Clearly defined personnel roles
  • Clearly defined setting
  • Clearly defined completion

Each core activity must have a description of what it takes to produce the intended outcomes, including:

  • Its content (for example, the service focus or curriculum topics), how it is to be delivered, and how much of it participants should receive
  • The requirements for the personnel delivering it
  • The setting in which it will take place
Specifying your intervention ensures the intervention is delivered with fidelity both before and during scaling. Without such specificity, new organizations, personnel, or sites may not be able to implement core activities consistently and with fidelity to the intervention’s theory of change. Such a scenario would decrease the likelihood that the intervention will improve participants’ outcomes to the extent expected given the intervention’s prior success.
For a school-based intervention seeking to increase elementary-age students’ reading scores, a core activity might be delivering one-on-one tutoring using a specific reading curriculum in the classroom, whereas a non-core, or supplemental, activity could be disseminating monthly newsletters to parents or caregivers with book recommendations that could be read to children during non-school hours.
The specifications of the intervention’s core activities, personnel roles, setting, and completion should be aligned to the research that showed evidence of the intervention’s effectiveness. If an organization defines these components in a way that differs from how they were defined when the intervention produced beneficial outcomes, then the organization might not be able to produce the same beneficial results when scaling the intervention.
Not yet defined No, but making plans to develop a definition No, but actively developing a definition Yes, this is defined
For example, in person or online, one on one, or small group.
For example, amount, frequency, and duration.
Not yet defined No, but making plans to develop a definition No, but actively developing a definition Yes, this is defined
Not yet defined No, but making plans to develop a definition No, but actively developing a definition Yes, this is defined
For example, rural/urban, geographic region.
For example, a community center, participants’ homes, nonprofit organization, or school.
Not yet defined No, but making plans to develop a definition No, but actively developing a definition Yes, this is defined
For example, is everyone who enrolls considered a participant, or is there a minimum length of time a person must be in the program to be considered a participant?
For example, minimum services completed, certification acquired, particular score achieved on an assessment.
0
0/30
Back
Next

Defining the target population

Use this checklist to assess whether you have a well-defined target population for your intervention. Choose the response option that best characterizes your organization’s status in defining the following aspects of your intervention’s target population.

FAQ

A well-defined target population describes the characteristics of the population that the intervention was originally designed to serve. This can include, for example, age or grade ranges, participants’ locations, gender, and other demographic and background characteristics.

It is important to be specific when defining your target population. For example, describing a population as “at-risk youth” is inadequate because the meaning of at-risk and youth is unclear. It would be more specific to describe the population as “youth ages 12 to 17 who have been involved with the juvenile justice system.” If a program aims to serve more than one population, describe each with such specificity.

When scaling, an organization might want to offer an intervention to the same population for whom the intervention was designed and shown to be effective. Doing so makes it more likely that the organization will be successful in scaling, because interventions might not be as effective when used with populations or groups that are different from the target population for which the intervention demonstrated evidence. To do so, the population must be well defined. This definition must be specific and include the characteristics people must have to participate in the intervention.
Not yet defined No, but making plans to develop a definition No, but actively developing a definition Yes, this is defined
For example, students in 6th or 7th grade who scored between a lower cutoff score and an upper cutoff score on a reading assessment.

Some interventions do not have specific inclusion criteria for the target population that would receive services. If this is true of your intervention, select the “Yes, this is defined” response choice.
For example, a housing program that weatherizes or retrofits household units for low-income populations might exclude housing units constructed within the past five years.

Some interventions do not have specific exclusion criteria for the target population that would receive services. If this is true of your intervention, select the “Yes, this is defined” response choice.
0
0/9
Back
Next

Establishing implementation supports

Use this checklist to assess whether you have implementation supports in place for your intervention. Choose the response option that best characterizes your organization’s status in establishing the following implementation supports for your intervention’s implementation.

FAQ

These six implementation supports will help an organization implement the intervention with fidelity:
  • A monitoring system
  • Performance goals
  • Continuous quality improvement processes (a systematic process for improving implementation service delivery, processes, and outcomes)
  • Preservice and in-service training
  • Communication systems
  • Data systems
An organization must have implementation supports in place to ensure it can implement an intervention with fidelity during scaling. An intervention that does not have such supports might not be able to produce beneficial outcomes when scaled.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, a team consisting of supervisors, project managers, or project directors who will monitor the intervention’s implementation.
For example, the team will collect service delivery data through observations during site visits or review service data entered into a data system on a weekly basis.
For example, the monitoring team will hold quarterly meetings with personnel to discuss service delivery data and any arising implementation issues.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, the number of participants contacted per personnel member or the number of participants served per personnel member.
For example, creating procedures for how benchmarks will be measured and collected, who will review progress toward the benchmarks, and the frequency with which the benchmarks will be monitored.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, establishing a team that meets regularly to coordinate CQI efforts and having a plan to collect and analyze data on performance benchmarks to determine if implementation is occurring as intended.
For example, regularly testing the intervention’s implementation, adjusting as needed, and continuously retesting the modifications to ensure the intervention is succeeding as planned.
For example, making updates to the personnel trainings or providing extra technical assistance to partners as needed.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, specifying the initial training each job title will receive, which includes information on who is delivering the training and the training’s timing, content, and format (such as in person or online).
For example, ensuring that the initial training occurs in advance of service delivery so personnel have sufficient time to familiarize themselves with the intervention content and practice the methods for service delivery and the client intake procedures.
For example, specifying the ongoing trainings each job title will receive while the intervention is underway, which includes information on who is delivering the training and the trainings’ frequency, content, and format (such as in person or online).
For example, having annual refresher trainings on the intervention’s curriculum and the client intake procedures.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, a detailed plan may state that, for the first six months of the intervention, service providers must meet as a group with their supervisor for one hour each week to discuss topics related to recruiting and engaging participants; for the next six months, meetings must take place monthly in the form of hour-long sessions to discuss service provision.
For example, having a plan to obtain feedback from personnel about the communication system and tracking the communication that takes place (using logs of how often personnel met together or regular supervisor check-ins with personnel).
For example, having a plan to address issues with personnel not meeting with supervisors as often as intended or infrequent contact occurring between key intervention partners.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, a formal management information system, a case management system, or a custom database that will collect information on program implementation, costs, and elements related to the research evaluation of the program.
For example, there is a plan to check the completeness and accuracy of the participation data entered into a data system with paper records of attendance or case note file reviews.
For example, holding a standardized training to help personnel develop a common understanding of the data points in the data system and ensuring that the personnel analyzing the data have the necessary qualifications.
For example, using data to determine whether the intervention is meeting its implementation targets and then making decisions or developing new strategies based on those data.
0
0/57
Back
Next

Having an enabling context

Use this checklist to assess whether your organization has a context that will likely enable successful scaling. Choose the response option that best describes the extent to which your organization has the following characteristics.

FAQ

Two components will enable an organization to scale an intervention:
  • Support by organizational leaders and key stakeholders
  • A culture of innovation and learning
To foster successful scaling of effective interventions, an organization’s leadership and culture must support innovation, learning, and improvement. The organization’s structures, roles, and functions should also facilitate (rather than hinder) service delivery as well as the achievement of beneficial participant outcomes.
Not at all descriptive of the organization Somewhat descriptive of the organization Mostly descriptive of the organization Fully descriptive of the organization
For example, organizational leaders are engaged in planning activities or making implementation of the intervention a priority for the organization.
For example, stakeholders provide space for service delivery at local implementation sites or partner organizations ensure their personnel’s workload accommodates the time and effort needed for service delivery.
Not at all descriptive of the organization Somewhat descriptive of the organization Mostly descriptive of the organization Fully descriptive of the organization
For example, organizational efforts to identify, develop, and implement new ways of meeting community needs, such as offering new products or services or identifying novel ways of delivering regularly available services.
For example, the efforts to improve upon a previous version of the intervention or enhancing the current services being offered.
For example, successes related to implementing the intervention’s core activities, recruiting or serving the target population, or supporting the intervention’s implementation.
For example, difficulties with implementing the intervention’s core activities, recruiting or serving the target population, or supporting the intervention’s implementation, and the solutions to those challenges.
0
0/18
Back
Next

Establishing an implementation infrastructure

Use this checklist to assess whether your organization has the implementation infrastructure in place to support scaling your intervention. Choose the response option that best characterizes your organization’s status in setting up the following implementation infrastructure components.

FAQ

An implementation infrastructure has two components:

  • Financial and human resources: Financial resources cover the costs of intervention implementation, and human resources facilitate hiring the personnel necessary for scaling and supporting those personnel through a human resources management system.
  • Materials and physical space: These resources are needed for implementing the intervention.
To effectively support a scaled intervention, an organization’s infrastructure must have sufficient personnel for implementation, support supervision and development for those personnel, and provide funding and other needed resources, such as materials and physical space, for implementing the intervention.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, financial resources to pay the costs of implementing the intervention. These include costs of service providers and supervisors (such as salaries, wages, or benefits) and costs of infrastructure (such as rent for the physical space where the intervention will be implemented).
For example, having the necessary number of personnel with the required qualifications to work exclusively on the intervention’s delivery.
For example, processes and procedures for identifying, training, and developing personnel needed for delivering the intervention.
No plans in place No, but making plans to set up No, but actively setting it up Yes, this is in place
For example, materials for service providers, such as training manuals or implementation guides, or for participants, such as workbooks or supplies for activities.
For example, space for delivering the intervention such as classrooms or library space for delivering one-on-one tutoring to students.
0
0/18
Back
Next

SCALER Step 3: Results

Click on the tabs below to see your results. For each of the five scaling readiness conditions, the closer your total score to the maximum, the more ready for scaling your intervention or organization appears to be. Use these scores to identify conditions that need further work for your intervention or organization to be ready to scale. Then, draw on the next steps and suggested resources to develop procedures or processes to build your capacity in these areas. For conditions where you have higher scores, we suggest areas to monitor once scaling is underway.

Intervention readiness

0% Well-specified intervention Donut Chart: Well-specified intervention, 0% complete
0% Well-defined target population Donut Chart: Well-specified target population, 0% complete
0% Implementation supports Donut Chart: Implementation supports, 0% complete

Organization readiness

0% Enabling context Donut Chart: Enabling context, 0% complete
0% Implementation infrastructure Donut Chart: Implementation infrastructure, 0% complete

Specifying the Intervention

You should get started on specifying your intervention.

If you haven’t already, start by identifying your intervention’s core activities, meaning the specific services or curriculum that make up the intervention, as well as the delivery mode and intensity of those activities. Then think about clearly defining personnel roles and qualifications, the setting in which your intervention will be delivered, and criteria for participation and completion. All of these definitions should align to the research that showed evidence of the intervention’s effectiveness and can help bolster your intervention’s readiness for scaling. Changes in these components might not lead to the same beneficial outcomes during scaling. For more information, see “How to Fully Describe an Intervention.”

You’ve begun to specify your intervention, but there’s still work to do.

If you haven’t already, make sure you’ve identified your intervention’s core activities, meaning the specific services or curriculum that make up the intervention, as well as the delivery mode and intensity of those activities. Clearly define personnel roles and qualifications, the setting in which your intervention will be delivered, and criteria for participation and completion. All of these definitions should align to the research that showed evidence of the intervention’s effectiveness and can help bolster your intervention’s readiness for scaling. Changes in these components might not lead to the same beneficial outcomes during scaling. For more information, see “How to Fully Describe an Intervention.”

Your organization has mostly specified the intervention you will be scaling, but there’s still work to do.

If you haven’t already, make sure you’ve identified your intervention’s core activities, meaning the specific services or curriculum that make up the intervention, as well as the delivery mode and intensity of those activities. Clearly define personnel roles and qualifications, the setting in which your intervention will be delivered, and criteria for participation and completion. All of these definitions should align to the research that showed evidence of the intervention’s effectiveness and can help bolster your intervention’s readiness for scaling. Changes in these components might not lead to the same beneficial outcomes during scaling. For more information, see “How to Fully Describe an Intervention.”

Keep up the great work! Your organization has a well-specified intervention for scaling.

While scaling your intervention, assess whether your organization is adhering to your definitions for the intervention’s core activities, including the delivery mode and intensity of those activities, personnel roles and qualifications, the delivery setting, and criteria for participation and completion. All of these definitions should align to the research that showed evidence of the intervention’s effectiveness and can help bolster your intervention’s readiness for scaling. Changes in these components might not lead to the same beneficial outcomes during scaling. For more information, see “How to Fully Describe an Intervention.”

ResultsReturn to edit
Component Score*
Identifying core activities
 
0/9 points
Clearly defining personnel roles
 
0/9 points
Clearly defining the setting
 
0/6 points
Clearly defining participation and completion
 
0/6 points

*Each checklist item was assigned a score from 0 to 3, with higher scores meaning the item was further along in being defined. Scores were then summed up for each component listed in the table.

1
1
1
1

  • The intervention’s identified core activities are not defined
    An intervention that is ready for scaling should have core activities (the specific services or curriculum that make up the intervention) that are clearly defined. In identifying the core activities, the mode of delivery (such as online or in-person, and one-on-one or in a group) and the intensity (meaning the amount, duration, and frequency of each activity) should also be defined. These elements help ensure the intervention can be delivered with fidelity while being scaled.

  • The intervention’s personnel roles are not defined
    In specifying the intervention, you will need to define personnel roles. Be sure to:

    • Identify how many personnel you will need to scale the intervention
    • Define the titles and responsibilities that personnel will have in scaling the intervention
    • Define the personnel qualifications, such as the required education, prior trainings, or experience levels, for the personnel delivering services for the scaled intervention

  • The intervention’s setting is not defined
    In specifying the intervention, you will need to define the setting. You should:

    • Identify the geographic location where the scaling will take place (for example, which geographic region in the United States) and whether the setting is rural and/or urban
    • Define where the intervention takes place (such as at a school, community-based organization, or workforce center) and the number of these venues needed

  • Intervention participation and completion are not defined
    As part of specifying your intervention, clearly define intervention participation and completion. You should define:

    • Participation by the minimum amount of services a person needs to receive. For example, to be considered a “participant,” a person seeking work must attend at least one case manager meeting.
    • Criteria for completion to indicate what participants must do to be considered “graduates” of the intervention. For example, to complete the intervention, participants must attend 10 case manager meetings.

You have completed Step 3 of the SCALER. Based on your results, your organization needs to continue its efforts to build a readiness to scale its intervention. Proceed to "Planning your next steps" for guidance on how your organization can use the SCALER results to identify next steps for strengthening your scaling readiness.

Back
Print Step 3 Results
continue to Planning your next steps

Need to finish later?

Update or view your checklists in the future with your unique link:

return home view SCALER resources