Measuring IMPACT; Are We?
It seems we have been fixated on Impact for a couple of decades now. There is very rich and high quality literature available for one to read and learn from. Many research institutions are leading the way to clear the ambiguities about impact and its measurement. I am attempting to write a few observations from my experiences.
My thoughts are based on the premise of these two Definitions
Impact: A marked effect or influence; Oxford Dictionary
Impact: The force of impression of one thing on another; Miriam Webster
Focused heavily on having an impact, not-for-profit organizations, donors and the private sector are eager to achieve and demonstrate the impression their efforts leave on their surroundings. They want to measure the tangible, almost forgoing the measurement of the intangible effects of their actions and influence. Individuals and professionals also want to have an impact: they want to improve lives, change people and the environment, adopt fair trade options, find jobs in Global Development, join community tourism initiatives or use their vacation to volunteer for bringing meaningful change and leave an impression. In the Global Development arena, we can claim outstanding results within the wide range of sectors that we have framed. We have many definitions of impact to work from, however a deeper understanding of what we are aiming to achieve and measure needs continuous discussion and learning.
The pivotal role of impact assessment in shaping future program design, management, resources acquisition and success of Global Development makes it critically important that we gain a simple and widely applicable concept of impact and its measurement.
The meaning, definition and interpretation of impact have been different across different entities. In the Global Development sector it has often been misused to imply minimalist change, drive small shifts in indicators and has even been overlapped with monitoring, evaluation and learning. The concept of impact has changed and evolved so much over time that at many junctures we find that everyone at the table is not talking about the same thing. Measuring the difference in predefined indicators during and after programs denotes good intentions and evidence of success in short-term results of global efforts and programs, but it does not inform us about the impression, the marked effect and influence on the communities. We are not confident to say that what we achieved would be replicable in the coming years.
I have a very vivid memory of celebrations after the restoration of a bridge and the attached roads. It was an immediate evidence of the program success. However the real life change, shifts and benefits on the lives of people and the cities that got connected after the bridge was rebuilt would have been evident only years down the life of the relevant programs. I do wish that once the ribbon across the bridge had been cut, the assessment of the change had started. There was evident ease of inter-city travel and improvement in transportation, leading school children to spend less time in the van to and from the school; I wish that could have been followed as a trend in the education sector. I wish we had the luxury of documenting the effects of timely transportation on the birth outcomes among pregnant women who could travel from the village over the other side of the bridge and show its impact of health sector. I do hope there were some entities to take the credit for the improvements or others ready to take the blame for the risks to the natural environment, the local biome and habitat in the river. I do hope that these changes if studied would not unduly muffle or over amplify the story; how the bridge changed the city.
Talking of attribution; we know that, no one intervention or practice within a package of interventions is likely to have a major impact on the birth outcomes within a five year program. I do hope that one day we will have proven population health strategies that address the entire range – from remediation of problem to prevention of risk - of factors influencing the birth outcomes. I do hope there will be proven corresponding measurement approaches to show the impact of improved birth outcome, better roads, a bridge on Newborn and child health. Approaches and measurement across the entire range of factors and sectors.
Here I would like to talk about experiences and observations from one of the programs that I have done over last 25 years. Starting with one of my interns, who was very talented, diligent, and passionate about bringing change and improving lives. One day she was ready to jump on the plane and return home from her assignment because she was troubled by the fact that she was not having an impact. All I had to say was “but you have been here only 6 months”. Like many others of us she was fixated on impact but without understanding its complexities. I think we need to recognize that we can only show the “novelty” of a program in the first 6 months, how long does it take to show long-lasting effect or shifts in life, the true impact, is a question that still stands to be answered.
The intern ended up staying and preparing reports for her program. There was incremental progress; including daily peer educator visits for community sessions on HIV & AIDS education, more and more condoms were being given out, more than planned leaflets were distributed, more home care visit were taking place as well. The project had positive results to demonstrate and share at the monthly NGO meeting. The senior management team were satisfied with the results and approved a wider program area at the end-of-term. We got more money and expanded the program area to work for another 3 years.
One day driving to the project site, we saw kids running around with a bunch of peculiarly shaped balloons, ah they were so happy and having loads of fun. Did we have an unknowing benefit, an unintended result of improving the cognitive development and emotional competence of the children playing with the condom-balloons? In reality this sighting was documented as an anecdote of misuse of a project input distributed as a prophylactic for HIV harm-reduction. The indicator of actual condom usage was not being measured at that stage of the program. As a matter of fact in spite of good results deep, significant and lasting effect was still an aspiration. It was expected to happen 10 – 15 years down the line. The program aimed to see lesser and lesser girls going into sex trafficking, more boys and men changing their sexual behaviours, trained community health worker in numbers ample enough to meet the increasing demand of home –based care. These sounded like audacious aspirations only at that time.
Measuring number is not useless and we definitely need quantifiable results to justify the successes and achievement of the program, but I want to reiterate that numbers are just a small part of the whole. I must argue that measuring the shift is hard. Human Development cannot just be reduced to quantities and numbers. We are guilty of sticking to measuring what is SMART as the proxy for the actual shifts, the real life transformation and evident effects on society. We must admit that we are unable to adjust for the complexities involved and fall short of measuring impact in its true sense. Many a times the context has evolved in a blink and we are left catching-up.
The real time change that many others and I witnessed was in the lives of a program staff. Bia worked hard for her job and got timely program results. She had a family owned handicraft store, which showed improvement by leaps and bounds. It was renovated, there was more inventory, and more variety of items to sell. Increased quantitative results enabled Bia to help her village cousin and her orphaned niece was back, and became a regular at school. The family life was improving there was evident feeling of fulfilment and satisfaction at work. Deep significant and lasting change had started to appear after a deep movement of status quo - finances. Forgive me if I gave the impression that this was a direct result of the program. I am telling this story to highlight that change happens when it is aimed at, for and by the people themselves for a very specific target. Only then it will show the strong effect - the lasting impression.
So are we ready to allow ourselves to say that we do not understand impact and its measurement in its true sense. We are still battling to get rid of the baggage and the weight that comes with it. We are only good at creating “novelty” in Global Development. When a dent – nothing even close to a strong impression - is just about becoming visible we are bound by the changing context and by our own strategic priorities to move on to a new novelty.