In June, we shared findings from an evaluation of citizen-led assessments we commissioned from Results for Development. Since the report’s publication, which showed how engaging citizens in large scale, household-based assessments of children’s learning can help focus education debates on learning, a number of colleagues have shared their own perspectives on the evaluation and my blog post about it (which was also translated into Spanish).
Charles Kenny from the Center for Global Development reflected on how Schooling Ain't Learning, and Learning Assessments (Alone) Ain't Reform. He concludes: “Assessments are the vital first step in learning the scale of the problem and finding out what works to help fix it. And the national and international attention that the efforts of groups like ASER and Uwezo have brought to the learning crisis has been invaluable. If they — or others — can help empower the legions of volunteers involved to act as agents of change, they might help foment an education revolution.”
Colin Bangay from the UK’s Department for Development offered his thoughts in: What role for Citizen Led Learning Assessments? – Moving beyond Measurement on the World Education blog. Colin praises the evaluation for its recommendations on how to increase the impact of citizen-led assessments and raises questions not tackled by the report about the relationship between external assessments and government assessment systems.
On the RISE blog, Rukmini Banerji from Pratham shared how, in setting up ASER in India, Pratham was Building a Movement - Assessment to Action. She talks about the motivation behind ASER, how they learned early on that “along with seeing a problem and realizing that there is a need to act, people seemed to need a clear demonstration of what can be done and how to do it,” and therefore how, through Read India, ASER has been linked to large-scale efforts by citizens for improving learning.
And finally, Modupe Adefeso-Olateju from The Education Partnership writes about How Nigeria is Beginning the Citizen-Led Assessment with the End in Mind. She offers examples of how LEARNigeria (Let’s Engage, Assess & Report Nigeria), the newest in the family of citizen-led assessments, is keeping the evaluation results in mind as they design their assessment.
The ultimate measure of success in education is not whether or not children attend school, but whether they learn. And creating a system in which learning is valued requires finding out what children are learning and building broad awareness about it. It was these two principles that inspired the organization Pratham in India to mobilize and train volunteers to conduct household surveys of children’s learning. Since 2005, Pratham’s Annual Status of Education Report (ASER, which means “impact,” for short), has provided estimates of reading and math abilities—the fundamental building blocks for more advanced skills—for children aged six to sixteen in every rural district, aggregated for every state and for India as a whole. What’s more, Pratham’s pathbreaking efforts have inspired similar assessment efforts in nine other countries across the developing world. This is a growing movement that my colleague Ruth Levineand I have written about before.
The Hewlett Foundation has supported citizen-led assessments since the early days of our Quality Education in Developing Countries initiative—part and parcel of our strategy to place learning at the center of the definition of success in education. By involving citizens in an assessment simple enough to be understandable by even illiterate parents, a broadly shared picture of progress in improving education, from the household to the national level, has emerged. This process has fostered the transparency, accountability, and participation in education that lies at the heart of our next phase of work in international education.
Given the centrality of this work to our past and future strategies, the Foundation commissioned an evaluation to better understand the extent to which citizen-led assessments have been successful in their aims. Have citizen-led assessments succeeded in building awareness of the learning crisis? Moreover, has assessment translated into meaningful actions that will ensure more children are mastering the basic skills they need to succeed later in school and in life?
The full reporton the findings from that evaluation, conducted by Results for Development, is worth a read—both by those who are engaged in efforts to improve learning and by those who are interested in harnessing data and engaging citizens to enhance government accountability.
My own top three takeaways from the evaluation are:
1.Thanks to citizen-led assessments, education debates are now focused on learning. Citizen-led assessments are by no means the first efforts to measure learning. They are, however, the first efforts to measure foundational skills of reading and math independently and at scale. Before ASER came along, the global community did not know just how many children were failing to learn even the most basic skills. Shockingly dismal results—like the fact that half of Indian fifth graders are unable to read—demonstrated in stark terms that challenges in learning are pervasive and start early. This wake-up call contributed to a shift in discourse within countries and internationally.
In Kenya, for example, the latest education sector plan cites the Uwezo assessment findings and includes a central emphasis on improving learning outcomes. Globally, deliberations on setting the Sustainable Development Goals are headed towards a goal for education that focuses on ensuring inclusive and equitable quality education and lifelong learning for all. Citizen-led assessments are changing the debate about education.
2.Citizen-led assessments provide clues for what and how to monitor the Sustainable Development Goals. Not only have citizen-led assessments helped to shape what is in the Sustainable Development goals, they also provide insights on how those goals should be monitored. Here three points are important:
First, focusing on ensuring that children are mastering foundational skills and are doing so early in their education is critical. If we continue to leave children behind on these skills they will not have a fair shot at developing other deeper learning skills that are critical for their life chances.
Second, the evaluation results show that citizens can be reliable data collectors. Citizens can play a central role in monitoring whether their governments are making progress against their development goals.
The third and final insight is that learning assessment can be done meaningfully and affordably. The evaluation also tells us that even simple assessments provide reliable estimates of skills that matter—and they reveal whether goals of inclusion and equity are being met better than school-based assessments, which fail to capture children who drop out, attend irregularly, and/or attend informal schools. And they do all this at a fraction of the cost of other assessments, providing a clear model of the kind of meaningful, affordable monitoring mechanisms that will be needed for the new Sustainable Development Goals.
3.There are several avenues for citizen-led assessments to explore that may help to deepen their impact and influence. Although citizen-led assessments have had a tremendous influence on the education debate, they have not resulted in widespread, effective action for learning improvement. The evaluation provides a road map for how these assessment efforts can intensify their impact in the years ahead. Recommendations to improve quality, including the comparability of the findings across time and languages, have already been taken onboard. Experiments are also underway to extend the roles of volunteers and partner organizations to try to more effectively stimulate actions by citizens and governments that improve learning. Insights about the ways in which information on social service outcomes can drive social accountability stands to benefit not only the growing family of citizen-led assessments, but also the broader field of transparency and accountability.
Citizen-led assessment have accomplished tremendous progress in putting learning at the center of education. The results of this evaluation are well timed to help inform the current and growing members of the newly-established People’s Action for Learning Network—which provides a platform for organizations to continue learning from each other, to draw on joint resources and expertise for improvement over time, and to nurture organizations who are interested in undertaking similar work—to iterate on a powerful design and move towards even greater progress.
UPDATE: The UNESCO Institute of Statistics, , a grantee of our Global Development and Population Program, recently took the initiative to translate this post into Spanish. Thanks to Alejandra Masa López and Ana Aznar Castillo of Spain's Instituto Nacional de Evaluacíon Educativa (INEE) for their work on the translation.
The Quality Education in Developing Countries initiative (“QEDC”) has been an eight-year effort to help ensure that all children have the chance to go to school and to learn. As the initiative comes to a close at the end of the year, this post tells the story of QEDC in a series of 10 photos. For more about the initiative, what the end of it means, and what we’re planning for the next phase of our work in international education, please see our website.
Ten years ago, Pratham developed a revolutionary approach to assessing children’s reading and math achievement when it launched a nation-wide household survey called the Annual Status of Education Report (ASER). As we’ve reported elsewhere, every year ASER tests children on their ability to read simple second grade level text and compute simple arithmetic up to fourth grade level. The entire process—from data collection to reporting—takes 100 days and engages tens of thousands of citizens in the process. The reports provide the only annual estimate of basic reading and arithmetic available in India.
This week, the Organization for Economic Cooperation and Development (OECD) recognized ASER as one of ten finalists for the DAC Prize for Taking Development Innovation to Scale, which acknowledges “development actors who take the step from supporting innovation to using it systematically and strategically to address development challenges, by taking it to scale.” In the ten years since it was first conceived by Pratham, ASER-like assessments have been implemented across six additional countries (Pakistan, Kenya, Uganda, Tanzania, Mali, and Senegal) and are currently being planned for and piloted in two more (Mexico and Nigeria). More than one million children are assessed every year by ASER and similar assessments in other countries.
This recognition of ASER’s success at scale presents an opportunity to celebrate the characteristics that have made ASER and similar assessments unique in the world of assessments:
Start with the basics. ASER focusses on basic skills like reading, number recognition and simple arithmetic operations. If children cannot read, they cannot move ahead—either in the school system or in life.
Collect data in homes, not schools. Since children may be enrolled in government or private schools, formal or informal schools, and may irregularly attend or drop out, the best place to find a representative sample of all children is the household.
Assess children one-on-one. To understand if a child can read, the assessment needs to be one-on-one.
Let lots of people participate. To own a problem, it is important to understand what the problem really looks like in practice—to get a feel for it. The architecture of ASER was designed to enable large scale participation of ordinary people so that people at all levels understand that there is a problem and can engage in bringing about solutions.
Enable action. To fuel possible future action, the ASER tools are easy to administer and easy to understand.
These features were developed with context in mind. India has a large proportion of children who still have not mastered basic reading, a large share of children who do not attend regular schools regularly, and an emergent culture of measurement. Given many other countries share these features, there may be more scaling of this approach on the horizon!
Three years ago I was in rural India directly participating in one of these surveys in the almost unbearable heat of summer. I was with two Indian colleagues who were hard at work mapping a village in Uttar Pradesh under the direction of the village headman. We were squatting in the shade but there was no escaping the hot, humid air.. A crowd formed around us as my colleagues sketched the village’s different neighborhoods on the pavement with chalk. The crowd moved quickly from bystanders to participants, offering advice with hand gestures and rapid Hindi—“No, no, the school is across the road from the well, not next to it!” Or at least that’s what I imagined them saying. My Hindi was—still is— next to nonexistent, but there was no mistaking the interest they took in us and our map.
Eventually the villagers asked us why we were there. Why were we drawing a map of the village? What had we come here to do? My colleague explained that we were doing a survey. We wanted to see if the children of this village could read.
That day was actually a dress rehearsal for a much larger survey. My colleagues were preparing for the Annual Status of Education Report, or ASER (pronounced ah-sir) for short. ASER means “impact” in Hindi and is aptly named since the goal of the survey is to see what impact education is having.
This year, it’s likely that more children will finish elementary school than ever before. This is not just because there are more fifth graders in the world than ever before. It is also because today most children go to school, no matter where in the world they were born. In an extraordinarily short amount of time the world has managed to make the opportunity to go to school nearly universal.
But what about the opportunity to learn?
For a long time people were so focused on counting whether or not children were attending schools that they did little accounting for how much learning was going on. ASER is an effort to address that gap. So are similar surveys like Uwezo in Kenya, Tanzania, and Uganda, Bèekunko in Mali, Jàngandoo in Senegal, and ASER Pakistan. Collectively these efforts assess over one million children, every year, in their homes. As my colleague Rukmini Banerji, Director of ASER, points out, the surveys help “demystify learning for mothers, fathers and family members—especially those who are not literate or do not have much schooling—and make it possible to see what learning looks like.” The findings show that a mere half of fifth graders in India and Pakistan can read, and a mere 20% of fourth graders in Mali can subtract. This has put pressure on governments to place due attention not only on whether or not children finish school, but whether or not they leave with the skills they need to thrive.
To complement the video released this week, the photo essay below explores these assessments in progress, as captured during that sweaty summer day in Utter Pradesh and other visits to the field.
Louise Bay Waters is the Superintendent of Leadership Public Schools, a network of four charter high schools serving predominately low income students in the San Francisco Bay Area. She is committed to ensuring that her graduates are fully prepared for college. Given the academic level of entering ninth graders, in order to attain this goal she and her faculty face the formidable task of helping students catch up on two years of academic content for every one year they spend in Leadership Public Schools. They have been impressively successful at their task: 97% of graduates have been accepted to college.
To facilitate these remarkable learning gains, Louise and her staff built a deep understanding of the needs of their students and committed to adapting their curriculum to meet those needs. They realized that the vocabulary in traditional textbooks was too advanced for their students, many of whom were English language learners. In searching for an alternative, Louise discovered that CK-12 provided high quality textbooks under an open license, which meant that anyone could download them for free and modify them without violating copyright law. So that’s exactly what Louise and her staff did. They created their own college access readers that had embedded vocabulary support to help their students develop reading comprehension while learning grade-level content. CK-12’s open license saved Louise and her team from starting from scratch while also enabling them to create unique material that met their needs.
Since 2001 the Hewlett Foundation’s Education Program has supported a movement towards more openly licensed materials, also known as Open Educational Resources (or simply “OER”). Over the years, the Foundation has supported organizations to develop high quality openly licensed education content like MIT’s OpenCourseWare, helped to develop the standards that enable OER like Creative Commons’ open licenses, and promoted the broader development and use of OER worldwide through organizations like UNESCO and the Commonwealth of Learning. Through this stream of grantmaking we have seen that open licenses can dramatically increase access to valuable knowledge. As just one example, an open license on the PhET science simulations created at the University of Colorado has allowed others to translate them into more than 40 languages and reach thousands of students around the world. We have also seen that the ability to adapt open content means it can easily be reinvented for new contexts, as exemplified by the experience at Leadership Public Schools.
Until recently, our work to promote open licenses was restricted to our OER grantmaking. But the Foundation supports organizations producing valuable knowledge across many domains—knowledge that could have broader and deeper impact if it, too, were openly licensed. That is why the Education Program has started encouraging all of its grantees to license work created with funding from the Hewlett Foundation under the most recent version of the Creative Commons Attribution license (CC BY 4.0). Although this is already common practice among our OER grantees, it is new to many of our grantees working on Deeper Learning. Our initial conversations with grantees new to using open licenses have revealed that while in some ways open licensing is attractive, in other ways it is intimidating.
Our grantees aspire to have the greatest impact for their work, which can happen when their materials are used by as many people as possible as productively as possible. Some of them have seen how an open license can pave the way for this to happen. For example, after posting its Common Core-aligned curriculum under an open license, Expeditionary Learning has seen 1.6 million downloads from across the U.S. On the flipside, a concern among our grantees has been the possibility of for-profit companies taking their content and profiting from it. Another concern has been that an open license will decrease opportunities for generating revenue. For these reasons, organizations like the Teaching Channel have chosen to only openly license videos of inspiring teachers in practice that have been created with Hewlett funds. Most of the rest of its videos will remain openly accessible but under a proprietary license. We think this mixed approach will be informative for the Foundation and the wider field, allowing us to test the downside risk of commercial appropriation against the upside potential of brand-building and wider uptake. We recognize that organizations face real and important tradeoffs when it comes to open licensing and will be monitoring the impact of these decisions alongside our grantees.
At the Hewlett Foundation we have a bold mission of “helping people build measurably better lives.” The organizations we fund have equally bold visions. To achieve audacious goals like these we need to improve on what we know. By openly sharing the knowledge and resources we produce we can not only expand access to knowledge, but also enable adaptation and improvement of that knowledge. For the Hewlett Foundation, openly sharing resources is one important component of our commitment to transparency. We will be building off our experiences in the Education Program to expand a policy around open licenses in other programs as well over the coming year and hope to continue learning along the way about the constraints and the opportunities this presents to amplify the work of our grantees.
If you’re like me, you probably haven’t used the quadratic formula since high school. You may remember bits of it (there’s a “-b”, and a “4ac” involved) but struggle to put them all together. Your first instinct, like mine, is probably to Google it. (In fact, maybe you already have Googled it!)
Last year, I found myself struggling to remember the quadratic formula, but I didn’t have access to Google. I didn’t even have any textbooks to consult. All I could refer to was my own fuzzy memory and a couple of colleagues with equally fuzzy memories. I was with Ruth Levine and Margot Fahnestock in Kitanga, a rural village in Tanzania about 60 kilometers outside of Dar es Salaam. We were sitting in the living room of the local primary school teacher, struggling to help her 14-year-old twins prepare for their examinations by the dim light of a kerosene lantern with 60 goats bleating loudly outside. While we strained our eyes (and our brains), about a kilometer down the road Eric Brown and Hewlett Foundation Board member Rakesh Rajani were in deep conversations with the village chairman, learning about the local party politics that prevented the community from investing the $20 it would take to fix the village's only well. Nearby, Foundation President Larry Kramer was enjoying his first taste of mandazi and watching a cheaply-produced Tanzanian movie that villagers paid a few shillings to view in the local video shack.
We were all in Kitanga for an “immersion” visit organized by our grantee Twaweza. Every year Twaweza staff spend one week in a village in East Africa in order to gain insight on issues like how people get and share information, how they access fundamental services like education, and how they make change in their lives. We were in Kitanga not for one week but for one day and one night, at the start of our weeklong visit to the region to give Larry a more tangible sense of the work of the Global Development and Population Program. Our program aims to help people around the world reach their potential as individuals, citizens, workers, and parents through grants that seek to expand women’s choices and amplify voices calling upon governments to deliver better results for their citizens.
While visiting one village for one day by no means gave us a deep and comprehensive understanding of the complex realities in which our work occurs (indeed, one must consciously fight against giving undue weight to anecdotal experience), it did provide a glimpse into the daily lives of the individuals whom our grantmaking is meant to affect. For example, I emerged with a renewed appreciation for the constraints students and teachers face that limit opportunities to learn: from inadequate light for nighttime studies to teachers struggling to facilitate learning for an entire class with only two books.
Over the course of the week that followed our immersion—joined also by Kevin Bohrer and our Nairobi-based consultant Peter da Costa—we had a chance to deepen our village experience with site visits and conversations with other grantees in the region working to improve education quality, expand access to reproductive healthcare, and improve governance. It was a good opportunity to see the problems we work on through a fresh set of eyes as Larry engaged with grantees and as many of us met grantees from other parts of the program portfolio for the first time. This was just the sort of deep dive that helps us gain insights on our work—and brush up on our high school algebra! The photos below help tell some of these stories.
For the first time in history, nearly every child has a chance to go to school, no matter where they are born or how wealthy their family. On the face of it, global efforts to educate the world have been a huge success. But although classrooms are brimming with children, the students are, more often than not, struggling to learn. Even basic literacy—arguably the fundamental skill acquired in school—is elusive for many students. In India, half of fifth graders cannot read a simple story. In Uganda and Mali, only one in fifty second graders can read. Tests administered internationally reveal that the average learner in a developing country performs at the level of the worst performers in countries like the U.S.
In June I visited Aber Primary School on the outskirts of Lira, in northern Uganda. The first grade classroom I visited, pictured above and here, had over 100 eager children tightly packed in, five or six sharing each small desk. The walls were mostly bare, save for the blackboard in the front of the room. I could tell that many of the children, barefoot and in well-worn uniforms, came from poor households. Seeing a classroom like this makes it easier to understand why national achievement data suggests that only two of these first graders are likely to be reading by the end of second grade.
Though their room looks like a typical Ugandan classroom, the instruction these students are getting is unique. Their teacher was given special training and support in teaching children how to read by an organization called Mango Tree. As a result, these first graders are reading at levels a year ahead of their peers in neighboring schools that haven’t gotten this support.
Mango Tree’s pioneering work in northern Uganda is part of a growing body of experiences across the world that show what it takes to improve learning outcomes for children in resource-constrained settings. Since 2007, the Hewlett Foundation has funded eleven school-level approaches to improving early learning, including Mango Tree’s. We have also funded ten external evaluations of these approaches to build evidence on what it takes to improve learning. Overall, these experiences have shown that when you teach children well, they learn. Although the particularities of local contexts will define what “solutions” look like in any given place, at least four broad principles seem key to giving all children a fair chance at learning:
Aim at students’ learning levels. Too often children are being taught with curricula that do not target their learning level. For example, despite the fact that half of Indian fifth graders have not mastered what they were expected to learn in second grade, their instruction moves in lockstep with a curriculum that is far more advanced. This has devastating consequences as children simply fall further and further behind. Rather than teaching the curriculum, we should be teaching children. This requires setting clear goals for the skills we want them to master, measuring where they are now, and providing instruction that bridges where they are today to where we want them to be tomorrow. This can be as simple as grouping children according to their current learning levels and concentrating instruction on activities that will help move them to the next learning level—a concept that has been demonstrated powerfully by Pratham in India (e.g., in Jehanabad), and replicated elsewhere.
Build structured lessons and teaching tools. In most classrooms in Uganda—and many other countries—teachers are given the official government curriculum and textbooks to follow, but are expected to figure out the exact lesson plans that will help children learn critical skills, like how to read. Organizations like Mango Tree heave made teachers’ jobs easier by giving clear guidance on steps to follow and activities to use to teach reading. They have sequenced the curriculum to show teachers exactly what letters to teach when. And they have given teachers low-cost reading materials for every student in the class that map to the lessons. Instead of leaving teachers with a vague charge to teach the children how to read, we need to provide a clear approach, alongside materials to help children learn.
Communicate in a language students understand. Children at Aber Primary School have another enormous advantage over many of their peers in the developing world: they are being taught in Leblango, the same language they speak with their families at home. They are not being asked to master a new language and a new skill at the same time. Furthermore, the words they are learning to read have meaning for them. This greatly increases their chances of success at learning to read in Leblango and improves their chances of reading in another language, like English, down the line. Fortunately efforts to instruct children in their home language are expanding, including through a major initiative in West Africa. In Senegal, Associates in Research and Education for Development is looking for opportunities for the government to scale up their tested model for instruction in three local languages.
Develop teachers. Making any of the first three principles possible—to aim at students’ learning levels with structured lessons in a language students understand—rests on what teachers do in their classrooms. The training that teachers currently receive is often theoretical, superficial, and stops before they ever enter the classroom. Giving teachers training and mentoring to develop these skills requires practical hands-on training and regular visits to the classroom to observe teachers in practice and offer suggestions for improvement.
In the synthesis report on Learning to Improve Learning, we call these principles the ABCs of improving learning and provide more details on the research underlying each of them, drawn from these individual studies. The principles are corroborated by numerous other studies and implementation experiences from around the world, including in a review of randomized evaluation findings completed in 2013 by Patrick McEwan. They point to several urgent policy issues, including the need to simplify the curricula, develop a structured approach to teaching, reorient teacher training, and align teacher incentives to student learning.