Strengthening Educate!'s Solutions With Rapid Impact Assessments

Educate! is impact-obsessed. For over ten years, we’ve invested heavily in measuring our solutions through rigorous impact evaluations, including external randomized controlled trials. While these longterm studies are critical to measuring our impact on youth life outcomes over time, we’ve also learned that collecting smaller datasets, faster, can ensure we’re able to improve upon our models in real time.

Applying Rapid Impact Assessment During COVID-19

In 2020, the need for high quality, rapid evaluation became even more apparent as we launched experiments and designed solutions for the new context. When the COVID-19 crisis closed schools in March of 2020, we paused our flagship in-school model and moved quickly to adapt all aspects of our work, from curriculum design and delivery to evaluation and operations, to support youth outside of the classroom. As our Design Team began iterating on a COVID-era version of our flagship model, the evaluation team set out to develop a new mechanism for rapidly assessing its impact. In order to iterate on our new distance learning model quickly, we needed to know how targeted decisions about program design and delivery affected our ability to support young people’s skill development. Recognizing this need, we built a Rapid Impact Assessment system that enables us to conduct rigorous, rapid tests — directly connecting changes in program design and delivery to tangible impacts on youth. While the outbreak of COVID-19 accelerated this work, the groundwork for Educate!’s Rapid Impact Assessment system began prior to the pandemic: we had recognized a need for simplified, actionable data to support program improvements and innovation. The first step in its construction was identifying early indicators of skills and then examining their progression from early knowledge (a basic understanding of the skill), to application of the skill, to a belief that the skill is available for use, to skill mastery.

Because important transferable skills like citizenship are developed in stages, articulating the progression of these stages allows us to understand if and how youth are advancing toward full skill formation. As youth move through the curriculum, we will know skill development is in progress if early skill indicators are present. If we can identify these indicators, we know our program design is likely on the right track. If we are unable to identify early skill indicators in participating youth, we can adapt our design to better target skills.

In the fall of 2020, when we launched the first iteration of Educate!’s distance learning model for youth, the VIP Bootcamp, we embedded our newly-designed Rapid Impact Assessment system into the pilot, collecting data about early skill indicators remotely from youth through phone-based surveys. Though the VIP Bootcamp was built by adapting components of our proven flagship model, many aspects of the remote model were new. As a result, we needed an effective way to learn quickly if these components were creating an impact. Using the Rapid Impact Assessment System, we aimed to understand whether our new remote model could create a similar experience to our in-person models, and would therefore generate positive impacts for youth.

Using the Rapid Impact Assessment system, we conducted a baseline survey of participating youth in the first iteration of Educate!’s VIP Bootcamp in fall of 2020. After the completion of the Bootcamp, we followed up with 771 participants to compare outcomes before and after participation. Using data and insights from the rapid evaluation, our team made swift modifications to increase engagement and participation as well as strengthen its impact on youth.

Preliminary Results From Our Distance Learning Model

We found that youth who completed the pilot cycle of our VIP Bootcamp were motivated to complete milestone actions: Earn, Save, and Act. These actions (earning an income, saving money, and launching businesses) demonstrate that young people participating in Educate!’s COVID-era model are engaging in an experience which promotes skill development and encourages youth to put these skills into practice. Young men and women who completed the VIP Bootcamp improved within key economic outcomes:

Using Rapid Evaluation to Scale Our Impact

With the accelerated evaluation and iteration made possible by the im - plementation of rapid impact assessment, we are eager to leverage this new tool to scale Educate!’s solutions and strengthen our impact on youth. As young people return to the classroom, we are determined to continue innovating and incorporating what works to build and refine effective skill-building solutions.