How to Measure Impact

From Ni Wiki

Revision as of 23:01, 19 July 2007

But how do you measure it?

Measuring Impact is an essential component of every development project. Without adequate metrics it will be impossible for you to know whether or not you have helped the communities your in or hurt them.

The Joy of Economics

Economics may be the dismal science but you're going to need it if you want measure the impact of your projects. Economics has come in for a lot of abuse over the years but for the most part it has unjustly. Economics is worth celebrating. Economists may often get things wrong but economics at its most effective has changed the world for the better, and can continue to do so.

"Economics is what economists do", reckoned Jacob Viner, a leading 20th-century economist. For a more helpful definition try "the study of how society uses its scarece resources" or more snappily, "the science of choices", Without scarcity - of land, labor, raw materials, capital, entrepreneurial spirit, time - there would be no need to make choices about how to use those things to greatest effect, and thus no need for economics. At its best economics helps people to make the right choices; at least, it shows them the most efficient way to use scarce resources in the process of achieving their goals.

How you can apply this to projects

The science of economics is most useful when you give it something to measure. Presumably there will be some goal for each project, some area in which you hope to make a lasting improvement or change. It may be building educational gardens for schoolchildren, it may be that your trying to diffuse labor saving agricultural technology like the Universal Nut Sheller. Either way there will be people that are affected by what you are doing and your goal should be to find out how the change in a specific factor affects their livelihood.

With the Universal Nut Sheller project in Uganda students were introducing labor saving peanut shelling technology to rural communities. In this case the introduction of the sheller was the isolated factor and what needed to be measured was welfare. For each project there will be different objectives and different things that you expect will change with respect to welfare. In this case it was expected that with the Nut Sheller households could now shell nuts 50 times faster than they could before. This meant that they could either save time, or earn more money since shelled nuts are worth twice as much as nuts in the shell. An inventory (survey) was developed to quantify this change welfare in the form of income, expenses, time use, health and gender roles.

Randomized Controlled Trial

The next step is administering the survey. Doing so requires a randomized controlled trail. A randomized controlled trial is a scientific procedure that uses randomized control. This is considered the most reliable form of scientific evidence because it eliminates all forms of spurious causality.

As you might imagine there are two essential components to this. The first is that an effective survey is sufficiently randomized. The basic idea is that a factor is allocated to subjects at random. This ensures that the different treatment groups are 'statistically equivalent'. Randomized surveys protect you from systematic biases in one direction or another. The second aspect is that you need a control group. The basic idea is that you're not testing a group in isolation but you're comparing it to a like group that is similar in every aspect except they have not received a 'treatment' (in this case a sheller). The importance of having a control group cannot be overstated. A good example of this can be found in clinical trials. Merely being told that one is receiving a miraculous cure can be enough to cure a patient—even if the pill contains nothing more than sugar. Additionally, the procedure itself can produce ill effects. For example, in one study on rabbits where these subjects were receiving daily injections of a drug, it was found that they were developing cancer. If this was a result of the treatment, it would obviously be unsuitable for testing in humans. Because this result was reflected equally between the control and test groups, the source of the problem was investigated and it was shown in this case that the administration of daily injections was the cancer risk—not the drug itself.

The same applies to economic impact studies. You might have introduced a sheller to a community and then come back a year later to find that incomes have gone down & illness has skyrocketed. This may or may not have anything to do with the sheller but you'd have no idea if this is a country wide trend unless you had another community to compare it to.

Baselines & Return Trips

When you're measuring how a project has affected a community you have to define a time frame in which you'll be trying to measure this change. Every study begins with a baseline; or a initial measurement of what the situation is before you introduce

A sample instrument

To give you a better idea of what an instrument might look like