Applying the generalizability framework: Adopting Teaching at the Right Level in Zambia

Cross–posted from the J-PAL blog.

How can governments learn from evidence generated from randomized evaluations?

Over the last three years, J-PAL Africa has supported the Zambian Ministry of General Education to pilot and scale up the Catch Up Program, with the help of Pratham, Innovations for Poverty Action, UNICEF Zambia, VVOB – education for development, the USAID Zambia Mission, and USAID Development Innovation Ventures. Catch Up is a remedial education program modeled off the effectiveness of the Teaching at the Right Level (TaRL) approach but grounded in the Zambian context.

Our support to the Zambian government has been guided by a framework for how evidence can inform policy across contexts, as presented in Bates and Glennerster (2017). The framework describes how to apply evidence from one context to another by using a combination of randomized evaluations, theory, descriptive data, and process monitoring. To guide this process, it puts forward the following questions:

1. What is the disaggregated theory behind the program?
2. How strong is the evidence for the required general behavior change?
3. Do the local conditions hold for the theory to apply?
4. What is the evidence that the implementation process can be carried out well?

Teaching at the Right Level: Evidence, theory, and generalizable behavior

Several studies conducted by J-PAL affiliated researchers and others over the last twenty years on how to improve learning outcomes shed light on a key generalizable lesson in many contexts: teachers tend to teach to the top of the class.

Given the structure of education systems across many parts of the world, this is unsurprising. Many teachers are confronted with classes comprised of students with a wide variety of learning needs, dense and ambitious curricula, and high-stakes primary school leaving exams, which incentivize teachers to move at the pace of the fastest learners.

Teaching at the Right Level, an approach pioneered by Indian NGO Pratham, targets the root of this behavior. The approach groups children by homogenous learning needs; dedicates time to building basic skills rather than focusing solely on the curriculum; and regularly assesses student performance, rather than relying only on end-of-year exams.

Over the last 15 years, together Pratham and J-PAL have rigorously tested TaRL’s underlying theory of change through six randomized evaluations in India and a growing body of research in Africa, which find that when TaRL is successfully implemented, learning outcomes improve.

Catch Up: Context matters

Teaching at the Right Level is not an intervention that can be applied universally. Rather, it is an approach for working with education systems to help them adjust more flexibly to children’s needs.

Several important conditions existed in Zambia that allowed TaRL to take root:

• Zambia had a clear need for the program. Zambia ranked last in the 2011 Southern and East Africa Consortium for Monitoring Education Quality (SACMEQ) in numeracy and tied last with Malawi in literacy.
• Learning levels within each classroom are varied, and students have little recourse to learn basic skills if they have not mastered them in the foundational years.
• Education stakeholders were candid in their acknowledgment of the problem of basic skills and were open to thinking about solutions from other countries.

With support from J-PAL, the ministry chose three different evidence-based implementation models to pilot in the country.

While the evidence on TaRL is strong, it also shows how implementation failures lead to reduced impact. Before fully rolling out the program, the partner team supported the ministry to test whether the implementation could work effectively and sustainably in this new context. This work included:

• Piloting in 80 schools, with monitoring by government officials and independent monitors,
• Developing a data collection system and helping leaders react to this data, and
• Building a system of review meetings into the approach.

Given the demonstrated need for a basic skills intervention and the pilot’s implementation success, the ministry committed to scaling Catch Up to 1,800 schools by 2020. The scale-up plan includes options for iterative learning to ensure the program maintains effectiveness at scale and continues to improve over time.

Interest in Teaching at the Right Level continues to grow throughout Africa. From September 26-27, 2018, governments and organizations from across Africa interested or implementing TaRL-inspired interventions will attend a J-PAL and Pratham-hosted Teaching at the Right Level conference for the launch of the new TaRL website. The website will house useful implementation focused resources, as well as information on upcoming TaRL events and will provide a platform for those interested in TaRL to engage. The conference will provide an opportunity for the TaRL Community of Practice to share implementation lessons, strengthen design, and more effectively collaborate.