False Positives – The inference problem

Article

A false positive has become better-known to the general public during the pandemic and with COVID-19 testing particularly with home based quick tests. We know these are 90-95% accurate but this means that in 5-10 times per 100 they will record a false positive, a positive result when there is none. This can be checked with a further more accurate test. The more worrying one, in the case of the COVID-19 testing, is the false negative. Worrying because someone could then feel safe and infect someone else.

But back to change or corporate initiatives. We probably all know the process. Someone has a cool idea and manages to get a leader, or more, on board. Everybody says, “let’s run a trial/pilot” the trial works well and we trot along to senior leadership saying we have a great idea, the trial went really well, and this will have a positive impact: more engagement, or save money, or earn more money. Senior leadership is impressed – you have done your background work, the trial is good, the budget is therefore approved and off you go…to unspectacular results.

One reason is that one single trial could have been a fluke (the other reasons are listed here). One single trial with a few people is not enough. This is well known in statistics – the smaller the population the more unreliable the data. In psychology we would look to a minimum of 100 participants but that is bare minimum, preferably 1000 participants. In the corporate world that may be unrealistic with time and resource restrictions, but the general advice is run three independent trials.

If you run three independent trails you will have a much better grasp on what is happening and whether your idea can be successful. You may also be able to identify further with who and in what context it could be successful, which are the representativeness problems. Alternately you can run a trial which can then be expanded to a larger scale pilot, to an even larger scale feasibility exercise.

However, the advantage of three (or more) trials is it gets a better grip on false positives but also gets a grip on false negatives which are lost opportunities.