We measure user loyalty: is Net Promoter Score so good?

Vasily Sabirov, a leading analyst and co-founder of devtodev, talks about how Net Promoter Score can be useful for measuring user loyalty.

Vasily Sabirov

The other day, another service asked me to rate on a scale from 0 to 10 the probability that I would recommend it to friends.

Every time I am asked this question (and it happens on average once a week), I feel great curiosity: what decisions will be made in the service based on this survey? Can he really change anything?

This method of measuring user loyalty is called Net Promoter Score, and it is very popular (we even wrote a little material about it for App2Top.ru a year and a half ago). However, as an analyst, I am slightly skeptical about this method. That’s what I want to tell you about.

What is Net Promoter Score?

I’ll briefly tell you how it counts. Depending on what rating the user has set, it is classified into one of the following categories:

Promoters (9 or 10 points). These guys are ready to go into fire and water for the product, they are the most loyal audience to the product.

Neutral, indifferent users (7 or 8 points). In general, they like the product, but they are not ready to recommend it.

Critics (from 0 to 6 points). They give a negative assessment, they don’t like the product, and they are ready to talk about it.

Accordingly, the share of promoters and critics among all respondents is calculated further. The share of critics is subtracted from the share of promoters, and so you get a Net (that is, cleared of critics) Promoter Score. The higher it is, the more loyal the audience is considered to be to the product.

Advantages of NPS

Simplicity of calculations. To explain how it is considered, I didn’t even have to resort to formulas, everything is told in words. And in fact, counting the totals takes less than a minute.

Versatility. Whether you’re an airline or a doughnut delivery service, you can count the NPS. The calculation rules are uniform and simple, the collection of information is done quickly and easily, the user only needs to press one button. By the way, here is a list of companies (of course, incomplete) that use NPS in their practice.

Prevalence. NPS is considered by many, if not all. Accordingly, many share the results and on the Internet you can find approximate values of NPS for different industries (free access, paid access).

Disadvantages of NPS

NPS is a thought experiment, not an action

This is the first and main disadvantage of this method. Not all critics actually criticize and not all promoters actually tell friends. You’re just asking the user to perform an abstract action in their head, nothing more. Everyone has different relationships with friends, different social behavior, and not always true loyalty to the brand coincides with the desire to share information about it. Virality is just one of the optics with which you can look at a broader indicator of loyalty.

NPS throws out some of the users from the calculations

If we assume that the probability of putting each of the scores is evenly distributed, then, without taking into account those who put scores 7 and 8, you are not looking at 18% of the users who answered. Not the smallest error for the quantitative method.

NPS depends on the cultural aspect

We all perceive reality differently. The scale from 0 to 10 is not unambiguously interpreted in different cultures.

For example, in the USA, user ratings are more polar, there is much less of a neutral attitude: they either like the product, and then the ratings are set to maximum, or not, and then the ratings are set to negative.

In Russian culture, there is a certain desire for “normal” (how are you? it’s fine!), and to put a score of 0-1 or 9-10 means really recognizing your bright emotion, which not everyone is capable of. There are much more ratings of 5 and 6, which mean an average “normal” attitude to the product in Russia, but are considered negative according to the NPS methodology.

NPS does not take into account the structure of the audience

In free-play games, for example, 90-95% of income is brought by old players who have registered for a long time and have made more than one payment. Such players are usually no more than 5% of the total audience. That is, in the context of NPS, they will hardly be able to significantly affect the index values. Although the true loyalty expressed in money, these users have much more. By giving less importance to the core audience, you risk making a decision that will appeal to young non-paying players and will not be accepted by those who bring you money.

Case

One of the companies (which one, we can’t tell: NDA), which turned to devtodev for analytical advice, released an update that changed the economy of the game quite seriously.

To assess how users reacted to the update, we used NPS. And we found out that the NPS has grown. And the income has fallen!

They began to understand and realized that this update worked perfectly for a non-paying audience, but the paying players did not like it. There are much fewer paying players, and their share in the NPS turned out to be unnoticeable.

If we calculate the NPS in that case separately by paying and non-paying, then we can see how differently different segments of users reacted to the change.

To avoid such cases, it is necessary to make NPS weighted average, where the total payments for each user should be indicated as a weight. It may not be universal, but it is more fair.

The NPS depends on whether the client has passed the survey before

When a user is faced with the question “Estimate the probability that you will tell your friends about the product, from 0 to 10” for the second or third time and he remembers his previous answer, two factors can affect his current answer:

• “They’re asking again, I answered last time.” And the score in this way may decrease simply because of the repetition of the question.
• “And what has changed since the last time?”. The user, assessing loyalty, will focus not on his entire experience of playing this project, but only on the interval that passed between the two questions. It’s not really loyalty, but rather a reaction to recent changes.

NPS requires periodic recalculation

Let’s say you calculated your NPS and it turned out to be equal to, say, 60%. A lot of water has flowed since then, many new releases have been released, and you rightly want to know how user loyalty has changed since then.

Here it is worth being as careful as possible: in the previous paragraph we said that repeatability does not play into the hands of accuracy, which means that it is desirable to choose a new audience (and preferably randomly) for the survey and choose from among those who have not passed it before.

This is not always easy to do while maintaining the representativeness of the sample. On the one hand, you need more users so that the sample is representative, on the other hand, the probability that users will take the survey again increases. And you do not always have enough technical means at hand to cope with this task.

NPS is an inertial indicator. Considering how different layers of the audience get into the surveys, you can’t tell how often these people use the product on average. More precisely, if you look at the average, they are probably used less often than you think. This means that there is inertia in the attitude of people to the product: not everyone has time to follow your updates and make a decision from 0 to 10 based on a previously developed attitude to the product.

NPS does not answer the question “why”

Again, NPS is a simple thought experiment measuring the current surface attitude to the product. And, armed with only one NPS, you will not be able to answer the question why this or that user gave a negative or positive rating.

What conclusions can be drawn?

So what now, not to use the established NPS mechanism?

• Firstly, user loyalty is not alive by single surveys. In NPS, the user conducts a thought experiment, and users demonstrate true loyalty by voting with their feet (returning to the product) or with a ruble (making purchases of the product). Therefore, it should be considered together with retention and monetization metrics (ARPU, ARPPU, LTV). All together, these metrics can tell a lot more about the product.
• Secondly, if we still talk about surveys, then NPS has alternatives in the form of other survey methodologies. In particular, the American ACSI index or the European EPSI index. They are not devoid of all the disadvantages of NPS, but their questions imply slightly deeper answers.
• Thirdly, the same NPS, if carried out throughout the technology, can give more information about the product if segmentation is applied to it. Separately counting NPS by paying and non-paying, by country, by age in the product and other types of segmentation, you will understand much more about how user loyalty is actually distributed and how it works. Moreover, I would say that segmentation must be applied to NPS, and without it, an aggregated estimate is a spherical horse in a vacuum.
• Fourth, the only benchmark worth focusing on is the previous NPS values for your product. Other guidelines obtained from open sources can help you only indirectly. If you know that other companies, even competitors, have NPS more or less than yours, then this knowledge will not give you anything: you do not know if they think it is right, you do not know how NPS is distributed across different user segments. The most important thing is that the NPS of your product increases over time. And even better, retention and monetization metrics should also grow. Only then will you be able to say more or less unequivocally that the loyalty of users in your project is also getting better.