A false positive has become better-known to the general public during the pandemic and with COVID-19 testing particularly with home based quick tests. We know these are 90-95% accurate but this means that in 5-10 times per 100 they will record a false positive, a positive result when there is none. This can be checked with a further more accurate test. The more worrying one, in the case of the COVID-19 testing, is the false negative. Worrying because someone could then feel safe and infect someone else.
But back to change or corporate initiatives. We probably all know the process. Someone has a cool idea and manages to get a leader, or more, on board. Everybody says, “let’s run a trial/pilot” the trial works well and we trot along to senior leadership saying we have a great idea, the trial went really well, and this will have a positive impact: more engagement, or save money, or earn more money. Senior leadership is impressed – you have done your background work, the trial is good, the budget is therefore approved and off you go…to unspectacular results.
One reason is that one single trial could have been a fluke (the other reasons are listed here). One single trial with a few people is not enough. This is well known in statistics – the smaller the population the more unreliable the data. In psychology we would look to a minimum of 100 participants but that is bare minimum, preferably 1000 participants. In the corporate world that may be unrealistic with time and resource restrictions, but the general advice is run three independent trials.
If you run three independent trails you will have a much better grasp on what is happening and whether your idea can be successful. You may also be able to identify further with who and in what context it could be successful, which are the representativeness problems. Alternately you can run a trial which can then be expanded to a larger scale pilot, to an even larger scale feasibility exercise.
However, the advantage of three (or more) trials is it gets a better grip on false positives but also gets a grip on false negatives which are lost opportunities.
The Simple Takeaway
-
- Aim to run multiple independent trials
- Trial / pilot on large groups
© leading brains 2022
References
More articles on limitations to behavioural change
Limits to Scalability – Voltage Drops
Voltage Drop is a useful term and analogy to the problems of scaling ideas or change in any environment. Voltage drop refers to how electrical current can decrease across distance it travels.
The Equilibrium Effect
Things in life tend towards certain balances. This is particularly true in large systems and this is why change can happen in small contexts and be very effective or successful but in large systems different rules apply.
The supply side of scaling
There are a number of problems with the supply side – being able to supply the resources and competencies to drive change.
Unintended consequences and negative spillovers
We all know that any change – well, obviously, changes something. But this also means there will be various knock-on effects
Representativeness of the Situation
“You had to be there” is an expression that says you had to be in a particular situation, in that particular vibe, to fully understand a situation.
Representativeness of the Population
When we get a good idea, we may know it is a good thing. And because we know it is a good thing we may then falsely assume – without really thinking this through – that everybody thinks this is a good thing.