ML fails: Loyalty prediction? Not really. And bonus prediction? Oh dear

Unexpected consequences - or drawing the line at non-linear relationships

Car sales person walks away, smiling brightly, from couple looking at car...
Keep smiling, keep smiling. Ugh! I shouldn't even bother making this sale.

Hello, wrong number You may never have heard of the NPS but if you have ever answered a question such as: "How likely are you to recommend our company/product/service to your friends and colleagues?" you have entered the Net Promoter Score zone.

The Net Promoter Score is used to gauge the loyalty of a firm's customers and, since its introduction in 2003, has been adopted by more than two-thirds of the fortune 1000 companies.

The original idea of the NPS was that a large number of people should be asked that single question and would answer on a scale of 0 (not likely) to 10 (extremely likely). The number they choose defines how they are classified.

A person who answers with anything from a zero to a six is classified as a "detractor" and contributes an NPS score of -100. Anyone answering with seven or eight is a "neutral" and contributes an NPS score of zero; finally choosing a nine or a 10 makes them a "promotor" and their NPS score is +100.

Net Promoter Score Classification

Here we see how the customer score relates to the NPS score

The NPS scores from all respondents are used to produce a Net Promoter Score Classification. Scores are averaged and the answer can range from -100 (all detractors) to +100 (all promoters) but in practice a score of around 10-30 was originally considered to be "normal". Once a base-line average has been obtained, the company can change its business in some way and then rerun the survey a couple of months later. If the NPS goes up, then the change would be deemed a success.

Even if used as originally intended, there are significant doubts about the efficacy of the NPS. For a start, the asymmetrical bucketing throws away a vast amount of information; surely we want to be able to distinguish a zero from a six and yet both contribute exactly the same NPS score. In addition, other researchers have argued that "NPS measurement does not necessarily correspond to actual behaviour". This has been written about in Fallacy of the Net Promoter Score: Customer Loyalty Predictive Model (PDF).

But even more problems arise because in practice many companies have chosen to implement the NPS very differently: applying the NPS to individual employees instead of the business as a whole.

As an example of this more granular application, whenever a salesperson sells a car, each individual customer is asked the question, and if the salesperson doesn't get the required NPS score, they fail to receive a bonus. And this really does happen; the last time one of the authors bought a car they were taken aside by the salesperson and told: "When you are asked about recommending our company to a friend, please give me a 10 otherwise I don’t get my bonus." We have also seen very similar actions in employees of Sky and Wetherspoons.

The major problem here is that the NPS score is being “averaged” across tiny groups. In addition, the “average” NPS value required seems to have crept up and up. In the early day,s values of 10-30 were regarded as normal, but organisations don’t want to be normal they want to be the best of the best, so NPS score expectations can be set at much higher values such as 60.

So, now we know the background, we can apply some very simple calculations to this implementation of the NPS and expose the consequences. Suppose you are a car salesperson and sell a few cars a week. Your NPS score is averaged over the week and, if you fail to hit 60 you don’t get your bonus.

Your employer explains that you can get your bonus if you achieve 70 per cent Promoters, 20 per cent Neutrals and 10 per cent Detractors (70P,20N,10D). Ignoring the fact that an NPS of 60 is very high, the reasoning sounds intuitively right because (70P,20N,10D) does, indeed, average out at an NPS score of 60. You feel you can hit this required ratio of P:N:D and so you think you will get your bonus every week; but you won’t. Let's run the numbers.

Imagine, on a given week, that you sell two cars. Each buyer can be a P, N or D so there are nine possible permutations of buyers (see the table below).

Row Buyer combinations Probability calculation Probability Average NPS Score Bonus Paid
1 PP 0.7x0.7 49% 100 Yes
2 PN 0.7x0.2 14% 50 No
3 PD 0.7x0.1 7% 0 No
4 NP 0.2x0.7 14% 50 No
5 NN 0.2x0.2 4% 0 No
6 ND 0.2x0.1 2% -50 No
7 DP 0.1x0.7 7% 0 No
8 DN 0.1x0.7 2% -50 No
9 DD 0.1x0.1 1% -100 No

The first of the nine options is two promoters (PP) and we can work out how often, if you have two customers in one week, both will be promoters. Since the probability of any one of your customers being a promoter is 0.7, it follows that, assuming that the two events are independent, the probability of you getting two promotors in one week is 0.7 x 0.7 = 0.49 = 49 per cent. Two promotors mean you get an NPS of 2x100/2 = 100 and so, 49 per cent of the time, you will achieve your bonus! The bad news is that every other pair of buyers gives you an NPS of less than 60, so you are punished for the remaining 51 per cent of situations.

Bonus achievement

Changing the bonus threshold has a non-linear effect on the bonus being paid

As you can see, with small sample numbers, the NPS is very coarsely grained. In our example the only possible NPS scores are 100, 50, 0, -50 and -100. Four out of five of those scores are below the magic 60. Note also that if the NPS bar was dropped to 51, you would fare no better; but drop it to 50 and you suddenly get your bonus 49 + 14 +14 = 77 per cent of the time instead of 49 per cent. The setting of the bar is really, really important because the relationship between that and the bonus payment is wildly non-linear so the consequences of upping the bar a little, in a non-linear relationship, can be catastrophic.

And it isn’t just the bar setting that produce a non-linear result; it is the number of customers in a week as well. If we keep the bar at 60 and vary the number of customers, we find that the chances of getting a bonus can vary from 70 per cent to 47 per cent in a very non-linear way (note particularly the values for five to seven customers).

Probability of bonus 2

The number of customers has a non-linear effect on the probability of getting a bonus

Smart salespeople (the ones who can do probability calculations) will realise that if they have only closed one sale by the week’s end then they should desperately avoid closing another because their chance of a bonus will drop from 70 per cent to 49 per cent if they do.

Was that really the intention of the company when it implemented the NPS in this way? Doing the initial calculations for this is painfully simple (how hard is it to multiply 0.7 x 0.7?) but people don’t do them and, because they don’t see the non-linear relationships, they assume linearity with increasingly ludicrous consequences. ?

Biting the hand that feeds IT ? 1998–2017