Chavi Jain, Author at

Teaching at the Right Level, Africa: Some Lessons on Monitoring, Measurement, and Review to date

Chavi Jain — Thu, 28 Oct 2021 06:27:42 +0000

Participants conduct a TaRL activity during a Programme d’Enseignement Ciblé (PEC) training session in Côte d’Ivoire. Photo: TaRL Africa

Over the last few years, the Teaching at the Right Level (TaRL) Africa team has worked closely with governments and other partners in Côte D’Ivoire (CDI), Nigeria, and Zambia to develop and grow contextually appropriate TaRL programs so that education stakeholders focus on ensuring that foundational literacy and numeracy levels of their children improve over time.

In this note, we share key principles underpinning our approach to monitoring, measurement, and review (MMR) in TaRL programs. We also share adaptations we have been making along the way to ensure we respond to systems’ needs and strengthen the capacity of relevant stakeholders to engage with and act upon learning outcomes data.

Data at the center of the TaRL approach

A key element of the TaRL approach is to have everyone focus on improving children’s learning outcomes. The main data that is collected is the learning levels of children for basic reading and maths. This data is collected by the program implementation team instead of external enumerators.

TaRL instructors assess each child using an oral, one-on-one assessment tool, which quickly and simply indicates whether a child has mastery of reading, number recognition, and operations, and if not, from which point below this level they need to progress. This information is then used by the instructors to organize children into groups according to their current learning levels instead of their grades or age. These learning outcomes and certain other complementary data flow up the system to inform action at different levels. See our measurement and monitoring page for more details on how to develop MMR tools and processes for TaRL programs.

Key principles that are kept in mind while setting up MMR systems

1. Measurement and data collection should be an integral part of the program and instructional activity, rather than a secondary “task”. This connection of data with activity reinforces the crucial importance of data for people who are both collecting the data and using it. Such data is as important for the implementers at the school/ local levels as it is for those who design and support interventions at the state/ national level. We try to collect only what can be meaningfully acted on.

As an example, children’s assessment data is used as below:

At the school level, by doing assessments one-on-one with every child in her class, a teacher is able to understand the real needs of every individual learner

Photo 1: A teacher in Côte D’Ivoire (CDI) assessing a child. Credits: Guy Edgar, TaRL Africa CDI Country Team

The teacher also records the level of every learner on a sheet. For each subject, the teacher marks the highest level task that the learner is able to do. For example, Obi here can read letters but struggles to read words.

When this data is summarized, a teacher is able to get a clear picture of a class. In the table above, a majority of students (7 out of 11) are at low learning levels (Beginner and Letter groups, i.e. they cannot read words)

All teachers in a school use the summarized data to then group learners by level.

At the cluster/ zone/ regional level, when the results are compared across locations, it helps the mentors/ government officials identify schools that have low learning outcomes or low improvements.

Lastly, at all levels, comparison of assessment results over time is helpful in getting a pulse check of the program. Are children making progress?

2. While more data may give more information, too many process indicators may sometimes shift focus away from the key ‘outcome’ – children’s learning levels in TaRL. Moreover, the capacity of many individuals to understand data is often limited. Therefore, we cut down on the “nice-to-have” indicators until we observe that the system is able to collect quality data and use it effectively. Lean but good quality data is better than large but questionable data.

In Zambia, the Catch Up Program has reduced the turnaround time of data from schools to the national level from 4 weeks to 2 weeks. This helps enable real-time use of data. This has been achieved through:

Improved collection templates and built-in data checks: This includes data validation mechanisms that highlight errors in the data. This means corrections can be picked up and addressed at the initial data entry point.
Simplified tools: Indicators that were not proving useful were removed.
Reduced data flowing up: Specifically, more detailed data that is most useful at the school level remains there to inform programming, whereas only the key measures useful to others flow up.

3. To ensure meaningful use of data for ongoing program improvement, it needs to be available easily at every level at the right time. Data entry tools should be easy to fill in so that more time is spent on using the data rather than collecting it. The systems and processes are designed keeping the ground realities in mind such as resources (e.g. smartphones, internet, laptops, etc.) available and the capacity of personnel. If it’s a government program, all processes are led by the government officials as far as possible.

In Kebbi, Nigeria, for example, data flows up from the teachers as follows:

In this process, teachers get a holistic view of their class, headteachers can easily monitor the whole school, and mentors (who are responsible for multiple schools) can easily track all schools under their purview. The data they need to make decisions is immediately available to them.

The data collection process is adapted to every context. For example, due to low smartphone and internet penetration in Kebbi, the data stays in paper format for the teachers, headteachers, and mentors. In another context, if the teachers can enter the data digitally themselves, the data collection can be digital from the outset and many steps can be reduced.

4. It is clear that having data at one’s fingertips is not enough to ensure optimal action. Many education officials are juggling multiple tasks with different datasets crossing their desks. Some will struggle to understand how to make sense of the figures presented and others will not see the possibilities of meaningful action among the many challenges they are grappling with on a daily basis. In addition, data has often traditionally been used as more of a stick. Those holding it have not necessarily been empowered to use it for improvement-focused change which should be possible at every level. TaRL Africa works with the relevant departments and people to build capacity on data collection, understanding, and use. We are constantly iterating on interesting ways to do this well. To promote capacity and confidence in data use, we have introduced the following:

Adding MMR-focused sessions in all levels of training: TaRL Africa has designed training resources to guide implementers on how they can make sense of the collected data and use it for decision-making. These include exercises to spot data errors in the tools and case studies to understand how to interpret and use data using real-life scenarios.

Encouraging action-focused conversations during mentoring visits and review meetings: TaRL Africa field staff members facilitate discussions on data in review meetings and whenever they visit schools.

Photo 2: School Support Officers in Kano, Nigeria looking at the midline data of the schools they are responsible for. Credits: Alex Banda, TaRL Africa Nigeria Country Team

Going forward

Learner outcomes data is a key feature of the TaRL approach and leads to immediate action by the TaRL instructors and others. As TaRL Africa, we see our careful iteration with our government partners working to enhance the data systems supporting TaRL as a key part of our broader education systems strengthening work. Ultimately, we hope to empower education stakeholders at different levels to focus on what really matters, and respond accordingly. We appreciate this is a process in and of itself, and like our other work, is most likely to have an effect if it is done together with our government partners at the pace at which they are able to absorb change.

We are excited to test new platforms in the near future and will continue to iterate and share on these.

The post Teaching at the Right Level, Africa: Some Lessons on Monitoring, Measurement, and Review to date appeared first on .

Developing a new numeracy assessment tool

Chavi Jain — Thu, 30 Jan 2020 12:22:51 +0000

Careful monitoring and observation of each component of the TaRL approach is an important aspect of successful TaRL implementation. In this post, we reflect on how the TaRL Africa team, along with country TaRL teams in Nigeria and Côte d’Ivoire, identified challenges with the TaRL assessment tool and used this information to tweak it. The new assessment tool is being tested in 2020.

How was the original tool developed?

The numeracy assessment tool used for TaRL programmes was originally conceptualised by Pratham in India and later adapted by country teams in Africa, with support from the TaRL Africa central team, to suit particular contexts.

What’s in the original tool?

The original tool used in Nigeria and Côte d’Ivoire comprised of two sections:

Number recognition: Tasks consisted of recognising numbers from 1 digit to 3 or 4 digits. First, children were asked to identify 1-digit numbers, if they could do this correctly, they were asked to identify 2-digit numbers and then, if identified correctly, they were asked to identify 3-digit numbers and so on.
Operations: 2-3 sums for each operation – addition with carryover, subtraction with borrowing, multiplication, and division. Children were asked to complete all operations.

In TaRL programmes the facilitator recorded each child’s highest number recognition level and recorded which operations the child could complete (writing either “can do” or “cannot do”). Children were then grouped based on their number recognition performance, as seen below.

Group 1: Beginner + 1-digit number recognition
Group 2: 2-digit number recognition
Group 3: 3-digit number recognition

Why did the tool change?

From field observations of the country teams undertaking government partnerships in Zambia, Nigeria, and Côte d’Ivoire and accumulated experiences of partner organisations and the TaRL Africa central team across contexts, it was found that the original numeracy tool suffered from a range of issues:

Time: The assessment process required each child to attempt all operations, making assessments time-consuming, especially in contexts where the child-facilitator ratio was more than 50. Observers noticed that many children were accustomed to solving operations using circles/tally marks, further extending the time needed for assessment. Let’s take the example of 57 multiplied by 9. A child would draw 57 circles/sticks on the paper 9 times and count all the circles/sticks again. Similarly, for 46 divided by 3, a child would draw 46 circles/sticks, count them again making groups of 3. This meant that facilitators spent a lot of time assessing children.

“Observers noticed that many children were accustomed to solving operations using circles/tally marks, further extending the time needed for assessment.”

Data recording: Since a child was marked on 5 competencies (number recognition and 4 operations), the facilitator needed to record 5 ticks per child, which meant that facilitators were more likely to make errors. Furthermore, the number recognition section was progressive (recording only the highest-digit number) while the operations section was successive (facilitators were required to record the child’s ability to complete each operation), requiring facilitators to be mindful of this switch when recording data. Often facilitators forgot to mark children on some operations which led to data totalling errors.

Grouping: The data emerging from many of the countries implementing TaRL in Africa revealed that the majority of the children are able to recognise at least 2-digit numbers but very few are able to solve operations. For example, 64% of the total children involved in the TaRL pilot in Borno, in northeast Nigeria, could recognise 2-digit numbers at baseline, but only 18% could solve an addition problem and only 4% could solve division. Since the grouping of children for a TaRL class was based on number recognition, most children were placed in group 2 or 3 (indicated in the grouping recommendations above). As classes started and children progressed in number recognition, they shifted to higher groups without necessarily making as much improvement in solving operations. This led to children with varying skills in mathematics operations being grouped in the same class. Sometimes this made it challenging for the facilitators to plan class sessions.

Given these challenges, the TaRL Africa central and country teams began to think – could the assessment tool be adjusted to cut down the time needed for assessment without compromising the ability to group children by level? They leveraged information from classroom observations, as well as the Annual Status of Education Report (ASER) survey assessment tool to create an adjusted assessment tool.

What’s in the new tool?

The new tool consists of 1-digit and 2-digit number recognition tasks followed by subtraction with borrowing and division problems. The tasks are completed progressively (moving from one-digit to two-digit, then from subtraction to division). The child is only asked to do a higher level task only if they are able to solve the previous task.

Each child is marked only at the highest learning level. Children are grouped based on the following recommendation:

Group 1: Beginner + 1-digit number recognition levels

Group 2: 2-digit number recognition level

Group 3: Subtraction + Division levels

Since this tool is progressive and does not require the facilitator to administer all tasks, it is much quicker and data recording is simpler. A drawback of this tool when compared to the old tool might be that the facilitator is not able to understand children’s progress in addition and multiplication and makes the assumption that they are proficient in these operations if they qualify the subtraction and division levels, respectively. Ongoing monitoring and classroom observations will help us to understand whether this is a problem, if it is we will need to come up with a solution.

Comparing the old and new tools

	Old tool and grouping	New tool and grouping
Time	More time consuming	Less time consuming
Data recording	More room for error	Less room for error
Grouping	Different learning needs in operations within a group	Similar learning needs in operations within a group
Understanding progress	Able to see progress in all operations	Able to see progress only in subtraction and division

Next Steps

Both tools have pros and cons. The suitability of the tool is contingent on the context in which TaRL is implemented. For example, if the classes are small, time is not a big constraint. If one facilitator is managing all learning level groups within one class, then grouping and progression needs to be considered accordingly. The new numeracy tool will be used in a pilot in Kano, northwest Nigeria, and was selected by government master trainers in the extension of the pilot in Côte d’Ivoire. As we receive feedback on the new assessment tool from these two pilots, the TaRL Africa team will continue to learn from implementing partners and iterate on different versions of the numeracy tool.

The post Developing a new numeracy assessment tool appeared first on .

TaRL Borno Pilot Wraps Up

Chavi Jain — Tue, 30 Apr 2019 12:59:39 +0000

The Borno State Government with support from the Department for International Development (DFID) through UNICEF, Plan International, and TaRL Africa recently concluded a small Teaching at the Right Level (TaRL) pilot reaching over 9000 children in twelve schools. The pilot provided the opportunity to mould and trial the TaRL approach in Nigeria, revealing critical insights into operations, model modifications, and highlighting components that require additional exploration. Furthermore, the team had the opportunity to examine how the approach worked in education environments affected by conflict.

TaRL Borno model

The teacher-led pilot centred on providing children in grade 4-6 with foundational skills in mathematics and reading. Teachers used the first two hours of the school day for two school terms (approximately six months) to accelerate children’s learning by grouping children by learning needs rather than age or grade and focusing on foundational skills applying the TaRL classroom methodology. Teachers were supported throughout the pilot with mentorship from head teachers, local government officials, and UNICEF and Plan International field officers.

What did we learn?

The pilot provided several interesting insights into TaRL delivery in Nigeria:

The master trainers and mentors, who taught their own TaRL classes ahead of the teacher training, played an important role in ensuring that challenges in the schools were quickly resolved and teacher capacity to lead TaRL classes was continually strengthened, as mentors and trainers relied on their own personal teaching experience. They met and interacted with teachers on a regular basis to review and course-correct.
Head Teachers provided a necessary coordination function at a school level by finding spaces for different groups of children to sit either in available classrooms or if there were an insufficient number of classrooms, under trees. This was vital given the large class sizes in Borno State.
The use of local languages in teacher training and for classroom instruction appeared to make teachers and children comfortable. This may have helped achieve rapid learning.

Moving to scale

In 2019-2020, TaRL will be scaled up within Borno and to surrounding states including Adamawa and Yobe. The model for scale is being co-designed with government officials from SUBEB and the Ministry of Education, and partners: UNICEF, Plan International and TaRL Africa.

The post TaRL Borno Pilot Wraps Up appeared first on .