On measuring stuff and how metrics can go wrong

#1

I spend a lot of time with clients talking about how they have to be really careful what they measure, because when you start measuring a behaviour you change it. This is a really interesting piece by Ev Williams (co-founder of Twitter) about how Medium measures the impact its articles have, and what the limitations of those metrics are: 


https://medium.com/@ev/a-mile-wide-an-inch-deep-48f36e48d4cb

This stuff applies not just in technology though but in any area where you’re measuring a behaviour in order to reward or punish it. As soon as you start rewarding or punishing people for the results of your measurements, you encourage them to change the measured behaviour, which might not be the behaviour you want to change. In short, you encourage people to try to game the system and this can have massively detrimental effects. If only government understood this. :/ 
0 Likes

#2

From a different perspective as someone who is trying to change farmer behaviour. One of the practices we use is a process that involves collecting data on farms and then having the group of farmers talk through it. One thing I have noticed, but I don’t think has ever been formally documented, is that the simple fact of taking the measurements from a paddock changes farmer behaviour - even though they are not involved in the measurements.


Farmers have a tendency to pay more attention to what is going on simply because we are looking at their paddock. I suspect it is in part due to subconsciously wanting to impress us and their peers with how well they farm. What it means of course is that you should be careful what you attribute the outcomes to. How much  is what they learned from the process and how much is due to taking the extra time so they look better in front of their peers?
0 Likes

#3

One of the best examples of metrics affecting behaviour is academic monitoring such as REF, and HE league tables (one for KenMcK too). Playing to league table position has changed the way depts do things., and not always in a good way.

0 Likes

#4
One of the best examples of metrics affecting behaviour is academic monitoring such as REF, and HE league tables (one for KenMcK too). Playing to league table position has changed the way depts do things., and not always in a good way.
A very simple example of the law of unintended consequences is that one of the most effective way for a UK HE institution to rise up university league tables would be to cut their widening participation intake. For a host of depressing, largely class-based reasons, WP students have worse outcomes than non-WP students and as institutions are judged on employment and not so much on their social responsibilities, the logical thing to do is to ditch the students that lower their scores. I don't know of any that have deliberately set out to do that, though (which is reassuring).


0 Likes

#5
From a different perspective as someone who is trying to change farmer behaviour. One of the practices we use is a process that involves collecting data on farms and then having the group of farmers talk through it. One thing I have noticed, but I don't think has ever been formally documented, is that the simple fact of taking the measurements from a paddock changes farmer behaviour - even though they are not involved in the measurements.

Farmers have a tendency to pay more attention to what is going on simply because we are looking at their paddock. I suspect it is in part due to subconsciously wanting to impress us and their peers with how well they farm. What it means of course is that you should be careful what you attribute the outcomes to. How much  is what they learned from the process and how much is due to taking the extra time so they look better in front of their peers?

Reminds me a lot of the results of productivity studies, which eventually found that actually any change increased productivity, even if it was trivial. 


Ah, humans. So messy! :wink:
0 Likes

#6

note to self - logged]

0 Likes