Are expert and key informant judgment evaluation designs types of ‘impact evaluation’

Up on the American Evaluation Association Linkedin group, I’ve started a discussion about what are the range of evaluation designs which can be regarded as impact evaluation designs.

I have a typology of seven major impact evaluation design types used in Duignan’s Impact Evaluation Feasibility Check. http://doview.com/plan/impact.html.

At least two of those design types – expert judgment and key informant judgment design types – are not seen by some as being appropriate to be called ‘impact evaluation’ designs. Some want to restrict the definition of impact evaluation designs to types such as Randomized Controlled Trials. Key informant designs are where groups of people ‘in the know’ about a program are asked questions about the program.

My definition of an impact evaluation design is one where someone is making a claim that they believe a program has changed high-level outcomes. In my Types of Evidence That a Program ‘Works’ Diagram (http://outcomescentral.org/outcomestheory.html#4), impact evaluation is conceptually distinguished from implementation evaluation on the basis of it making such a claim.

In contrast, non-impact, implementation evaluation (where you do evaluation for program improvement even in situations where you cannot measure impact) is not trying to make such a claim. I am not saying here that every type of key informant or expert design is impact evaluation, just ones where a question is asked along the lines of: ‘In your opinion did the program improve high-level outcomes’.

I think that if this question is asked, then the evaluation is trying to ‘make a claim about whether a program changed high-level outcomes’. The question of whether particular stakeholders believe this to be a credible claim in a particular situation is a conceptually different questions. And there are many stakeholders who would not regard it as such. However, this does not detract from the conceptual point that, if you can find stakeholders who in some situations would regard key informant or expert judgement designs as sufficiently credible for their purposes, then these designs can be regarded as a type of impact evaluation.

My broader purpose with this thinking within outcomes theory is to get the full list of possible impact evaluation designs considered in the case of any program so that we don’t just get obsessed with a limited range of impact evaluation designs, useful though things like Randomized Controlled Trials (RCTs) may be in some circumstances.

Moving past the debate about randomized experiments

A colleague Bob Williams recently drew attention to articles on the New Yorker about the use of randomized experiments and particularly one from an economist advocating their widespread use in a range of program areas.

I’ve been involved in a number of seemingly endless discussions and presentations about the pros and cons of randomized experiments and the rise of what are being called the Randomistas – those advocating for a much wider use of randomized experiments. In this post I want to talk about how we can move beyond these seemingly endless discussions.

Can an exhaustive list of impact evaluation designs be developed, or is my mission on this futile?

I have set out on a mission as a part of outcomes theory to attempt to develop an exhaustive list of impact/outcome evaluation designs – evaluation designs which make a claim that changes in high-level outcomes can be attributed to a particular intervention. If we could pull off developing such a list that most people are happy with, it would be very powerful. First it could be used in evaluation planning to work out if all of the possible impact evaluation designs had been assessed for their appropriateness, feasibility and/or affordability. At the moment I think that almost every evaluation planner walks around wondering if there is some sort of impact evaluation design they have not considered.

Randomistas Rule

Just read and commented on an interesting article referred to on the 3IE site – a site dedicated to improving evidence about what works in international development. The article was by Martin Ravallion and was about the rise of the Randomistas in international development economics. Randomistas are those who promote much more use of randomized trials to try and work out what works in international development. It is a good article which points out the fact that randomized trials are not feasible in many important types of development interventions. This debate is the same one which is occurring in many sectors at the moment and one which has been debated on and off in the evaluation field for many years. My take on it is that we need to develop some underlying principle which we can debate and generally agree on so that we don’t need to have this debate endlessly without seemingly making much progress on it.

Damascus – YEN Evaluation Clinic

Apologies for not blogging for a while, I’ve been involved in considerable travel and lots of other work – but that’s really no excuse. Maybe I just got all blogged out. What with Knolling, Blogging here and Twittering, maybe it all just got too much. Anyway, I’m back in the saddle now as they say! Last month I was fortunate to be an evaluation expert at the YEN Evaluation Clinic in Damascus. YEN is the Youth Employment Network – an International Labour Organization, World Bank, United Nations collaboration. A site for the evaluation clinic has been set up at yenclinic.groupsite.co.

The Evaluation Clinic took two examples of youth employment programs and worked through designing an impact evaluation for them. It was a fascinating experience. I’ll blog about what it was like being the sole psychologist evaluator working with  five economists evaluation specialists (from the ILO and the World Bank) another day!

Tracking jobs created under the U.S. Recovery Act – when should the attempt at measurement be abandoned?

The default expectation in at least some sections of the U.S. public sector seems to be that it should always be feasible and affordable to both measure and attribute the results of interventions. This is using the term attribution to mean being able to actually demonstrate that a change in an outcome has been caused by a particular intervention rather than being the result of other factors (see here for more on attribution). The recent U.S. Recovery Act is a case in point.  While it’s reasonable to start from the position that you should routinely assess the possibility of measuring and attributing changes in outcomes of particular interventions, you can’t start by just assuming that it will always be feasible or affordable to do this. Clinging to such an assumption, where it is untrue, can result in you either measuring an outcome when the data you are collecting is not accurate, or acting as though what you are measuring (even if it is an accurate measurement of a change in an outcome) is demonstrably attributable to a particular program, when in fact it may not be. 

Impact evaluation on full program roll-out versus just on piloting – two paradigms

I’ve just posted an article on the two paradigms in regard to impact/outcome evaluation and full program roll-out. What this is about is making a distinction between designing an evaluation which can provide impact/outcome evaluation information about full program roll-out versus a paradigm where you do impact/outcome evaluation just on piloting and then in regard to full program roll-out you just make sure that best practice is implemented. I once was involved in the evaluation of an overall program which had over 900 component programs. The way that we went about evaluating it was, in my view, wrong.

Being sector-centric not program-centric in deciding on program evaluation priorities

I have been blogging less in the last week or so due to going on holiday – blogging will still be less frequent for a week or so. I was recently involved in assessing a set of projects summaries to help determine which of them should be selected for more intensive evaluation input. This was not being done directly to determine project evaluation priorities, however the exercise reminded me once again of the general issue of how we determine what types of evaluation should be undertaken for particular projects. My blog comments below are about the general issue rather than the particular exercise of project selection I was recently involved in.  The set of information we typically use to work out what type of evaluation should be undertaken for a project is information such as the nature of the project, the proposed evaluation questions, and the proposed evaluation budget. I think that we need more than this when determining evaluation priorities.

Don't assume that impact evaluation should always be done

Impact evaluation – evaluation which looks at whether changes in high-level outcomes can be attributed to a particular program, organization or other intervention – is a particularly useful type of evaluation when done properly. It clearly tells us what works, and what doesn’t work, and this information can be used in decision-making about which programs should, and should not, be funded in the future. However, particularly at the present time, with all of the enthusiasm for evidence-based practice, many people mistakenly assume that impact evaluation should always be attempted in regard to any program, organization or other intervention. Assuming this is a serious mistake. I’ve just put up an article in the Outcomes Theory Knowledge Base which sets out in detail why it is and the way we should approach assessing when impact evaluation should be done.

Social Innovation, evaluation and outcomes

I attended a launch of the New Zealand national Center for Social Innovation last night. Geoff Mulgan from the Young Foundation (a similar center in the U.K.) talked about social innovation. The social innovation movement is about getting stakeholders and sectors together to do things differently to achieve better social outcomes. Already a dynamic movement, it has recently received a shot in the arm from the global economic melt-down – traditional ways of doing things are increasingly being questioned and people are looking for new solutions. A number of points made by Geoff and in the subsequent discussion are particularly relevant to outcomes and evaluation were: