Campbell’s Law and why Outcome Measurement is a Dead Cobra.

Target setting and ‘outcome measurement’ are part of a vast industry of checking and ‘holding to account’ but basically it doesn’t work, and probably causes more problems than if you didn’t bother in the first place.

That’s a bit controversial…, particularly if you are in the game of; measurement, checking and ‘holding to account’. Let me explain, but first have a look at these images from the Foresight Tackling Obesities: Future Choices Report from 2007 (via the GOV.UK website).

Foresight Obesity Map. Individual Energy Balance at the center with major influences surrounding it; Education, Healthcare through to Food Production and Media. It’s Complex!
Just in case you need any more convincing that it’s complex. Extract of the Weighted Causal Linkages map – Solid line = Positive, Dotted = Negative, Thickness = Strength of Impact

Measuring outcomes in Adult Social Care. This week I was at a session organised by Monmouthshire Council Social Services and Nick Andrews from the Swansea University, School of Social Care Research, where Toby Lowe from Northumbria University introduced the The Foresight Obesity Map.

The point Toby was getting across was that in a complex system, like society wide obesity, specific interventions on discrete areas are unlikely to have the outcome you hope for.

There’s too much going on; the system is not predictable and it is constantly changing. This means that targets and carefully measurements of progress against them is pretty much pointless. You are trying to define the unpredictable (by setting targets) and measure things over which you have no control. A complex system is controlled by many things beyond what you are concerned with.

The point that Adult Social Care operates in a complex system was not lost on most people in the audience – even the people from regulation/audit/performance management and data. We had an existentialist crisis in a group discussion on our table… ‘What is our purpose…?’ Always a good one to ponder just before lunch.

The key point from all of this is that – we live in a complex world and attempting to control and measure it by breaking it up into smaller parts (reductionism) doesn’t work like you think it does.

If you want to get deeper into complex systems have a look at Toby’s blog, Embracing Uncertainty and what Dave Snowden has to say in posts like; ‘the banality of measurement’ and,‘the myopia of metrics’.

Campbell’s Law and Corruption Pressures. This is my second big point. Basically, people will change their behaviour to achieve targets that are being used to measure their performance. Many people will have heard of the expression, “What gets measured gets done” – attributed to many people – and basically true as far as I’ve observed.

Image Source: https://jl8910.wordpress.com/2011/03/01/quote-poster-campbells-law/

Campbell’s Law describes this change in behaviour as ‘corruption pressures’ which in extreme cases can be ‘corrupt’. Deliberately acting to falsify results or in less extreme cases ignoring some things (like a small act of human kindness) whist you focus on getting the job done on time (because that’s what you are measured on).

If you search the internet you’ll find many examples of people and organisations changing behaviors and ‘corrupting’ measurement processes to make sure they deliver against what is being measured. Examples include; education in schools, hospital waiting times, police records of crimes solved etc etc etc. The evidence for this practice is so widespread that I’m a bit surprised that we don’t talk about it more – it’s a bit of a problem in my view.

You can read more about the work of Donald T. Campbell (a Social Scientist) here. Also, there are people like Charles Goodhart (an Economist) and Marylin Strathern (an Anthropologist) who pretty much said the same thing which you can read about in this post by Dave Snowden, The Strathern Variation. In the post Dave offers his own variation: Anything made explicit will sooner or later be gamed for survival purposes and that need will corrupt practice and people (Snowden)

Dead Cobras (finally). My third point – the law of unintended consequences. When people are ‘gaming’ (a less threatening word than corrupting) measurement system, you often get unintended negative consequences.

Known as the Cobra Effect one of the best known examples comes from time of British Colonial rule in India. The Authorities wanted to reduce the numbers of Cobras so offered a reward for any dead Cobras delivered to them. People saw the financial opportunity in this and started to breed Cobras so they could claim the reward for the dead ones. The Authorities got wise to the practice and stopped offering the reward. The result was Cobra breeders releasing the now worthless snakes and the problem was much worse than when things started.

I suspect that if most people sit and think for a while they will be able to find an example of The Cobra Effect from an aspect of their working life. Maybe not as dramatic as a plague of poisonous snakes, but an unintended negative consequence of people chasing targets.

Outcome Measurement is a Dead Cobra. I mentioned that just before lunch these was a bit of an existential crisis in the discussion at the regulators/auditors/performance and data people’s table.

Basically this is where I was at: “in a complex system target setting and measurement of progress against those targets is pointless, and people will corrupt the system (change behaviour or full on cheat), and you are probably going to get unintended negative consequences – like a plague of poisonous snakes”. Oh happy days!

The Story is The Measure*. Like any good story format I’m offering you a bit of hope here. There was an afternoon session and we came away with some ideas on what you can do to ‘measure progress’ in a complex system.

I promise I will write that post soon (on the train to Bangor on Wednesday) and walk away from the pit of poisonous snakes. All I’m going to say for the moment is that core to it is the idea that ‘the story is the measure’ of what is happening in a complex system. Oh, and by the way I don’t mean manufactured ‘corporate stories’. I mean real stories told be real people in their own, unfiltered words – a bit like what I’m talking about in this post; Cool Story Bro… be cautious of Organisational Storytelling and Silver Bullets

*I’m attributing ‘The Story is the Measure’ to Matt Wyatt @ComplexWales if he didn’t invent it, he will certainly have a view on who did.

So, What’s the PONT?

  1. We live and work in a complex system. Simple, traditional linear models do not work in complex systems.
  2. Campbell’s Law is a real thing – people change their behavior to meet targets. These ‘corruption pressures’ often have unintended consequences.
  3. Unintended consequences are often negative like the Cobra Effect – things are far worse than when you started.

About WhatsthePONT

I'm currently in the middle of a Winston Churchill Travelling Fellowship. Looking at big cooperative enterprises and social businesses in NE Spain and the USA. You can find out more here: https://whatsthepont.com/churchill-fellowship/

9 Responses

  1. David Manheim

    This is a great summary. I’d note that the dynamics behind Goodhart’s / Campbell’s law are a bit more complex, and it’s worth breaking it down somewhat. I did so in a (very long-form) blog post here: https://www.ribbonfarm.com/2016/06/09/goodharts-law-and-why-measurement-is-hard/

    I also want to take exception to the claim that measurement setting is pointless or harmful – it can be, but does not need to be. More specifically, there are a number of specific strategies that are useful for avoiding the critical pitfalls of naive application of metrics that you highlighted. My more recent paper on that topic is here: https://mpra.ub.uni-muenchen.de/90649/

  2. Gary Thomas

    I work within Adult Social Care in Wales and you are definitely right, gaming happens and the perverse incentives from target setting is very evident. I believe that our system is complex non-linear for ever changing where attribution of cause and effect cannot be laid against a single organisation, process or service. There are many interactions and factors at play all unique to the person. This makes comparability difficult almost impossible at the organisational level. Therefore, it becomes measurement for judgement not measurement for learning and improvement.

  3. Good points being made in this article. Working with adult social care and community health, I have a team that is in the process of coming up with new measure that do not drive the wrong behaviours, and that are focused on what matters to patients.
    At present, in their experiment, they are recording what the actual demand and what matters to the patient is, compared to the demand that we are presented with. The time sprent dealing with his to conclusion, the waste steps, the skills required, the outcomes, and if we think they will return to our service because have not solved the right thing.

  4. Good points being made in this article. Working with adult social care and community health, I have a team that is in the process of coming up with new measure that do not drive the wrong behaviours, and that are focused on what matters to patients. We have found that the measures and KPIs are the main drivers for behaviour in this system, bit not always the direct ones. For instance, completing the assessments correctly is one measure and it drives us to do the wrong thing by not understanding the person – because were focused on completing the assessments.
    At present, in their experiment, they are recording what the actual demand and what matters to the patient is, compared to the demand that we are presented with. The time sprent dealing with his to conclusion, the waste steps, the skills required, the outcomes, and if we think they will return to our service because have not solved the right thing.

  5. I’m currently doing some work in this area which I have shared with Nick to a limited degree. Lots to say but in essence it is about taking the best from person centred planning and strength-based theory and applying it to some of the concepts already out there like Outcome Star. However, we (there are three of us) are also adding some App technology that allows individuals to collect evidence of achievement through words and pictures. It is a totally bottom-up process that focuses on what is important TO the individual, as well as what is important FOR them. Lots more to say about this but contact me if you’re interested. We have agreement from Denbighshire CC to pilot the system later in the year.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s