There’s no question that the potential of the ‘data revolution’ first described in the U.N.’s “A World that Counts” report has captured the imagination of the international development community, especially data-wonks and donors concerned with how the Post-2015 Sustainable Development Goals will be measured. In the many discussions that have ensued, a consistent theme has been an aspiration to realize the potential of ‘non-traditional data sources.’ Non-traditional, for these revolutionaries, means not just the oft-cited Big Data, but forms of citizen-generated information that can shed light on real-world living conditions and public opinion.
Case in point—Senegal. In February, a team at Université Cheikh Anta Diop’s Laboratoire de Recherche sur les Transformations Économique et Sociales in Dakar revealed the results of their 2014 national assessment of children’s learning known as Jàngandoo (which means “learn together”)—part of a growing movement of civil society organizations who are carrying out independent, citizen-led efforts to measure basic learning of children ages six to fourteen in reading and Math in nine countries in East and West Africa, South Asia and Mexico. (Les resultats en francais).
Jàngandoo has only been up and running for three years, but policymakers in Senegal have taken notice. M. Serigne Mbaye Thiam, Senegal’s Minister of Education and no stranger to open data efforts, opened the national dissemination event, noting Jàngandoo’s contribution to providing policy makers and ordinary citizens timely data on the status of children’s learning that is easy to understand and interpret so that they can plan together actions for improving the quality of education. Since then, Jàngandoo has been rolling out regional dissemination events that have opened up a dialogue between citizen’s groups and local education officials about why children aren’t learning and what can be done about it.
Like other citizen-led assessments of learning, Jàngandoo takes place in homes rather than at school, and therefore captures data on children who do not attend regularly, have dropped out or never attended school, measuring learning earlier and more broadly than official national or regional assessments, which typically take place later in the primary cycle. Jàngandoo’s learning assessments are administered in multiple languages—French, Wolof, Pulaar and Arabic, depending on the child’s home language. According to UNESCO’s EFA Global Monitoring Report 2015, Jàngandoo is the only national-level assessment that has measured Senegalese children’s learning levels below Grade 9 in almost 10 years. The most recent PASEC, which measured children’s performance at the end of class 2 and class 5 in eight Francophone countries, including Senegal, is from 2007.
Unlike other citizen-led assessments, Jàngandoo also measures children’s knowledge of their own culture and environment, including certain aspects of sustainable development. Such locally-driven experimentation can also contribute to how UNESCO and others think about localizing how education for sustainable development gets measured in West Africa, a region where the effects of climate change and drought are an ever-present reality in nearly every child’s life.
Jàngandoo is an initiative of the Université Cheikh Anta Diop’s Laboratoire de Recherche sur les Transformations Économique et Sociales (Laboratory for Research on Economic and Social Transformation—LARTES). LARTES works with a network of twelve civil society organizations and a local technology firm that deploy a cadre of trained data gatherers armed with PDAs to reach all 45 departments of Senegal—all of whom make up what is known as Jàngandoo.
What’s striking about LARTES is that its staff is comprised mainly of young men and women representing a new breed of independent researchers in West Africa. If there is a data revolution, these are the revolutionaries. What’s their latest innovation? Ask Professor Abdou Fall, Jàngandoo’s leader, about his vision for on demand, a la carte data services for local governments who aspire to develop data-informed plans for improving their schools. Their first client: the Mayor of Dakar.
All this takes coordination and careful oversight, but also a particular vision for how data can be used to solve real development challenges. The sort of vision set out by the Africa Data Consensus for bringing different data communities together to generate “data for public good and inclusive development”.
While the global-level conversations about the data revolution are creating needed energy and networks, a cadre of data revolutionaries in civil society and tucked away in government agencies is also hard at work. Their progress often depends upon external funding that runs out before they can institutionalize their efforts. Perhaps what is needed then is a revolution in financing that will nurture the work of country-level data heroes like the LARTES team at Université Cheikh Anta Diop in Senegal. People who think differently about “data for whom,” “data for what” and the speed at which data can be made useable to change people’s lives for the better.
Despite the growing number of children in developing countries that are attending school, they are not reaching the levels of numeracy and literacy that they should. Some have called this a “learning crisis.” According to Rakesh Rajani, Head of Twaweza, previously “it [was] very hard to make that case and get policymakers to pay attention because we didn’t have data, we didn’t have evidence.” But now we do.
Now, civil society organizations across the world from East Africa to Southeast Asia are empowering communities to collect and use this data to demand better results for their children through citizen-led assessments. In eight countries, these groups are conducting household surveys to better measure all children’s learning, regardless of their school status.
The video below, produced by Made in Africa TV in Tanzania, describes the program and illustrates both the need for collecting household-level data on children’s learning, as well as the opportunities for better policies and action presented by this innovative approach.
The Hewlett Foundation announced a pledge of $18.5 million to the Global Partnership for Education’s 2015-18 replenishmenttoday. The pledge, which will be fulfilled through the Foundation’s grantmaking and technical support to civil society organizations and other key actors in global education, will support capacity building for better systems for learning assessment and the use of assessment data to inform planning and improve learning outcomes. It will also support civil society organizations that are using citizen-led, household assessments of learning to raise awareness about children’s learning status and to motivate action at the national, sub-national and community level for improved learning.
Last month, my colleague Dana Schmidt wrote a blog post about what the Hewlett Foundation and its grantees have learned about improving children’s early learning from the Quality Education in Developing Countries Initiative. Under this Initiative, our grantees implemented a variety of instructional models both within the school day and after school hours, with children enrolled and with those who were not. Many of these were evaluated using randomized control studies or other quasi-experimental evaluation designs to determine the impact of these interventions on children’s learning.
At the Hewlett Foundation, we encourage our grantees and also try ourselves to talk about and learn from failure, or at least from those things that don’t always go according to plan! As the (relatively) new kid on the block with the Global Development & Population team, I thought I would offer a few observations about what we learned about the evaluation process itself, recognizing that hindsight is 20-20 (or some approximation thereof). Most of this wisdom comes directly from conversations with our grantees and colleagues since I’ve joined the Foundation.
We underestimated the time that was needed for some of the instructional models to be more fully developed, and in hindsight, should have allowed our grantees more time to work the kinks out before carrying out some randomized control evaluations. In some cases, randomized control study designs were just not possible, and so we had to be flexible about evaluation methods. One of the smartest things we did was to encourage grantees and evaluators to work closely together—so evaluators could better understand the instructional model and the context, and grantees had input in framing the questions and helping determine how best to measure learning outcomes.
While the randomized control studies that we commissioned were able to measure the impacts of these instructional models on learning, they were not able to sufficiently unpack the effect or most essential elements of each instructional model. Practically speaking, this has meant that it has been difficult to tell which elements contribute most to learning improvements and thus are highest priority for scaling up, and which could be dropped or emphasized less, depending on resource availability. Despite these challenges, the Meta-Analysis by Patrick McEwan of Wellesley College goes a long way towards unpacking what is known about improving learning based on a review of dozens of randomized control studies conducted over the past 20 years or so. We think it is essential reading. We will continue to work with our grantees to help them better unpack the elements of their instructional models wherever possible; we encourage others to take this essential step before initiating randomized control studies in the future.
We also underestimated the challenges associated with completing cost analyses of these instructional models. But where we did succeed in doing so, it has produced valuable information that our grantees and policy makers can use for identifying possible areas for streamlining or improving efficiencies that will enable scale-up (e.g. rethinking or restructuring teacher training, mentoring and support, and options for getting instructional and reading materials into the hands of teachers and children most affordably).
Finally, we learned that it is especially important to spend more time and energy figuring out possible delivery channels and constraints to scale-up from the outset. We assumed that we and our grantees would be able to build ownership and political will for scaling up based on evaluation results coupled with brokering new partnerships and financing relationships. We did not fully appreciate all of the systemic barriers to change in environments where incentives and accountability are not currently structured around learning outcomes. We did not sufficiently plan how to manage the discontinuity in political will when reform-minded leaders and other allies left office.
We also could have done a better job of structuring our support to some grantees to enable them to work with policy makers and other key stakeholders to answer vital questions about scaling up. Such as: when to scale and whether this meant deepening services and impact in existing locations, viral spread of innovations, or scaling programs vertically through government or other key providers to reach more children with a basic package of improvements? How to do this without losing the basic integrity of these instructional models? Who would be responsible for quality assurance? Who pays for what and how to achieve the necessary commitment and clarify the roles of various key actors?
So what does this mean for the remaining nine months or so of this time-bound initiative that we started several years ago? The Hewlett Foundation is not funding the full scale-up costs of these programs—we don’t see that as our role in getting sustainable solutions in place, and simply don’t have the resources to do so. Rather, we are supporting our grantees to consolidate and expand the most promising instructional models, and also to use the results of evaluations and cost analyses to their best advantage. We are providing more intensive technical support and capacity building to a few of our existing grantees to assist them in pursuing opportunities that have emerged for scaling up. We also hope to document and share some of our experiences through our participation and support of the Brookings Institution Center for Universal Education’s “Millions Learning” work. Finally, we will be intensifying our support for household-based assessments of student learning, like ASER in India, Uwezo in East Africa and Beekunko and Jangandoo in Mali and Senegal, in order to better capture data on all children’s learning and inspire both communities and policy-makers to take notice and take action to improve children’s learning.