Submitted by kwilkerson on
  • Overview

Step 1: Is there evidence your intervention is effective?

SCALER Step 1: Overview

The first step in the SCALER is to answer the question “Is there evidence that my intervention is effective?” The answer to this question is important because it determines what your organization needs to do to prepare to scale an intervention. If the intervention does not have evidence that it is effective, your organization should build that evidence before scaling. If the intervention has evidence that it is effective, your organization can then determine its readiness to scale the intervention.

Research rigor

Rigor refers to having a well-designed and well-implemented impact study that clearly separates the effects of an intervention from other factors that may have affected the outcomes. This also means impartial researchers have designed and conducted the study to ensure independence of the study findings.

Start

FAQ

We recommend designating one person or role to be the SCALER owner, such as the intervention’s program or data manager. Because personnel at all levels are involved in intervention scaling, the SCALER owner will want to involve others including:

  • Executive directors
  • Program managers or frontline supervisors
  • Measurement, learning, and evaluation personnel (data managers or internal evaluation specialists)
Completing Step 1 requires gathering knowledge about any research studies that have been conducted on your intervention. You can draw on rich information from external sources, such as the research clearinghouses listed on the SCALER Resources page, to understand if existing research shows that an intervention you want to scale has already been proven effective.

Organizations can take one of several approaches:

  • Involved personnel could complete the checklist on their own, discuss responses, and come to agreement on scores at a meeting organized by the SCALER owner.
  • Several personnel could fill out the checklist together during a meeting.
  • The SCALER owner could gather the necessary information and complete the checklist.

In taking any of these approaches, the SCALER owner should set up a timeline for completing the tool.

The printable Step 1 Results report summarizes whether evidence of effectiveness exists for your intervention. With this summary, you can:

  • Share the results with key stakeholders, organizational leaders, and funders.
  • Use the results to identify limitations in the research that has been conducted on your intervention.
  • Periodically search for research on your intervention and update Step 1 if you find studies that potentially demonstrate your intervention’s effectiveness. If you cannot find such studies, use Step 2 to help build such evidence.

FAQ

Because intervention scaling aims to improve lives for larger numbers of people and communities, organizations should only seek to scale interventions that they expect will maintain or surpass their beneficial impacts for participants after scaling occurs. Rigorously conducted evaluations can establish causal impacts—meaning that well-designed and implementation impact evaluations can show that the intervention, and not other factors, led to improvements in participants’ outcomes.
Research clearinghouses provide information to help organizations identify research on the effectiveness of a wide variety of interventions. Although the quality of evaluation research varies greatly, the research uses common principles that define the level of rigor that a study demonstrates to show that an intervention caused beneficial outcomes for participants. The SCALER Resources lists various research clearinghouses that you can visit to see if evidence of effectiveness already exists for your intervention.

Use this checklist to assess whether your intervention has evidence of effectiveness. Under “Research rigor,” “Intervention effectiveness,” and “Intervention relevance,” select the items that reflect the nature of the research that has been conducted on your intervention. Should one or more of the items in the checklist remain unchecked, it means your organization does not yet have evidence of your intervention’s effectiveness, and such research evidence must be built.

Research rigor
This means an independent evaluator who is external to your organization or the intervention and who has no vested interest in the evaluation findings conducted the research.
This means the research design included a comparison group, which shows what participants’ outcomes would have been had they not received the intervention.
This means the research design allowed the evaluators to distinguish between changes in outcomes that were due to the intervention and not to other factors.
This means few people in the treatment or comparison group left the study (low attrition), no people in the comparison group switched to the treatment group or vice versa (no reassignment), and the treatment and comparison groups used in the analysis did not differ on key characteristics at the start of the study, other than one group received access to intervention services and the other did not (baseline equivalence with the final sample).
Intervention effectiveness
This means the amount (such as number of studies) and rigor of the research on your intervention is adequate for you to consider the intervention effective.
Intervention relevance
For example, the locations or populations that were used to establish evidence for the intervention’s effectiveness are similar to or relevant to the plans you have for scaling your intervention.
Back
Next

SCALER Step 1: Results

Evidence of your intervention’s effectiveness that aligns to your plans for scaling does not yet exist. Your organization will need to build such evidence.

Before your organization invests its resources to scale an intervention, the intervention should first have demonstrated effectiveness. Such evidence should be based on rigorously conducted research that demonstrates that the intervention itself contributed to changes in participant outcomes. The evidence must also be relevant to the plans your organization has for scaling the intervention so your organization can expect the intervention will maintain its effectiveness when at scale. Because you were unable to identify any existing evidence that meets these standards, you will need to complete Step 2 of the SCALER to help build such evidence.

  • Research Rigor does not exist
  • Intervention Effectivenessdoes not exist
  • Intervention Relevancedoes not exist

What do your results mean?

Return to edit

Research Rigor

  • A third party did not conduct the research
    If internal personnel conducted an evaluation on the intervention you are seeking to scale, consider having a third-party evaluator conduct an evaluation as well. Using a third-party evaluator with no vested interest in the intervention to conduct the research study provides credibility and objectivity to the evaluation findings.

  • Research did not include a counterfactual
    Because the research design for the intervention did not include a comparison group as a counterfactual, it is difficult to conclude what participants’ outcomes would have been had they not received the intervention. If you are seeking to build evidence for the intervention in the future, work with a third-party evaluator to use a randomized controlled trial or a matched comparison group in your research design to create a counterfactual.

  • Research had confounding factors
    If confounding factors were present in the evaluation of the intervention, it is difficult to conclude that any changes in participants’ outcomes were due to the intervention and not to other factors. If you seek to build evidence for the intervention in the future, view the SCALER Resources for more information on common confounding factors so you and your third-party evaluator can design a study that will avoid them.

  • Research did not have low attrition, reassignment was used, and baseline equivalence did not exist with the final sample
    The presence of these factors shows that the treatment and comparison groups used in the final sample were similar to each other before the treatment group participants received the intervention. If differences between the two groups exist, then such differences, and not the intervention itself, can affect study outcomes. If you seek to build evidence for the intervention in the future using a third-party evaluator, view the SCALER Resources for more information on these factors to understand how a study should be designed and implemented to isolate the impacts of your intervention.

intervention effectiveness

  • The research did not show evidence of effectiveness
    If research has not shown the intervention to be effective, then it is not yet ready to scale. Review the conducted research to identify possible explanations for why it did not show evidence of the intervention’s effectiveness—for example, the evaluation might not have contained a large enough sample size to demonstrate impacts or the intervention was not implemented as intended—and consider how to improve upon these factors when seeking to build your own evidence in the future.

intervention relevance

  • The research is not appropriate to the context or environment in which scaling is planned
    This means the intervention might not be as effective when used with populations or in locations that are different from what was included in research that showed the intervention’s effectiveness. If this occurs, you may need to build evidence for the intervention using populations or locations that ultimately align with your plans to scale the intervention.

You have completed Step 1 of the SCALER. Based on your results, rigorous evidence that shows your intervention’s effectiveness and is relevant to your plans to scale the intervention does not yet exist. Proceed to Step 2 of the SCALER, which aims to help your organization build evidence for your intervention.

Back
Print Step 1 Results
continue to Step 2

Need to finish later?

Update or view your checklists in the future with your unique link:

Copy my unique Step 1 link to the clipboard
return home view SCALER resources