Study: how to predict the success of a game on Steam by the number of reviews

As far as the number of reviews of the game on Steam indicates its good sales, the author of the indie game Starcom: Nexus investigated. App2Top.ru publishes a translation of his post from Reddit, where the developer is known by the nickname JustKevin.

I’ve seen a lot of discussions on Reddit about what factors determine the success of the game. They discussed: how important is the quality of the product? Is the recognition of the game before the release enough for its good sales? Will the demo help sell the game? And if the release didn’t go according to plan, is it possible to save the project? Is it possible to predict at least approximately the sales of a game that has not yet been released?

I tried to answer each of them before the release of my own game. To do this, I had to spend a lot of time monitoring future new products. I have compiled a table of games, where I indicated:

number of subscribers of the game;
did it come out in early access;
how many reviews I received in the first week, month and quarter, as well as other factors.

I share this data in the hope that it will help other developers understand and predict the sales of the game.

Fear the Night

To begin with, a few notes.

The number of reviews on Steam has become one of the main sources of data for the study. There is evidence that they are related to the number of copies sold. It is often mentioned that there are 50 copies sold for each review. But this ratio is not always specific. It is more likely that most games on Steam fall into the “fork” of 25 to 120 sales per review. Although there are exceptions. Let’s call them “outliers”*.
* In statistics, the concept of “outlier” means a measurement result that is very different from others. By the way, it is most likely that the “outliers” will be games with a very small number of reviews.
I know the actual sales of only one game — my own. So, in the first week I sold 1,587 copies of the game, and in the first quarter — 3,580.
The sample included 115 games. I selected them randomly by looking at “Popular Future Products” and “Future Products” on Steam. There are more games from the first rating in the table, since I wanted the sample to be diverse, but not consist entirely of poorly sold games.
The games are ordered by release date — from October 26, 2018 to December 20, 2018.

Explanations to the table

The indicator 0.25 in the column “Release discount” means a discount of 25% in the first week of sales.
The forecast of reviews for 1 week is my guess about how many reviews of the game will be left by users after the release.
Accordingly, the actual reviews for 1 week are how many reviews the game actually received in seven days of sales.
Actual reviews for three months — the number of reviews for three months of sales.
Number of subscribers — the number of subscribers of the game before its release. In some cases, I fixed the number of subscribers right before the launch. In others — a week before it.
Score — the percentage of positive reviews for the game on Steam. Let me remind you that the game needs at least 20 reviews for the store to calculate and display the percentage.

Hitman 2

Question 1: Is it possible to predict the success of the game based on its quality?

Quality is a subjective concept. On Steam, the most objective indicator of the quality of the game, obviously, is the percentage of positive reviews on it.

I excluded from the study all games that did not have at least 20 reviews in the first month. As a result, my sample was reduced to 56 projects.

Based on the data on these games, I calculated the relationship between the ratings of the games and the number of reviews for the first three months of sales using the Pearson correlation coefficient*. It was -0.2.

* This coefficient is used to study the relationship of two variables. The value of the correlation coefficient varies from -1 to 1. Using the Pearson correlation coefficient, only the strength of the linear relationship between variables can be determined.

In general, 0.2 — it does not matter, plus or minus — indicates a not too strong relationship between the two factors.

Most importantly, the Pearson correlation may be inaccurate due to “outliers”- games whose results stand out strongly compared to the rest. Because of them, the correlation discrepancies can be enormous.

Take, for example, Artifact from Valve. She received the largest number of reviews in the first three months — and at the same time one of the lowest ratings (53% at that time). Removing this game from the sample made the average correlation of the indicator in the table almost zero.

A similar result is obtained by calculating the Spearman rank correlation coefficient, which minimizes the impact of huge outliers.

Conclusion: if there is a relationship between the quality of the game (measured in the percentage of positive reviews on Steam) and its sales in the first quarter (measured by counting the total number of reviews), my research does not reveal it.

Artifact

Question 2: Do the availability of demos, early access and release discounts affect the success of the game on Steam?

Unfortunately, in this sample I met only 10 games with pre-release demos. It is possible to draw conclusions in such cases only if there is a very strong correlation. Here, as expected, it was insignificant.

There were more games released in early access — 28. But here, too, the correlation was too small, and therefore useless for research.

More than half of the titles were launched in the store at once with weekly discounts. The correlation coefficient between the availability of such a discount and the number of reviews in the release week was -0.3 (moderately negative). But this is due to the lack of release discounts for AAA titles that sell best. If you remove from the sample games that could earn more than $ 1 million in the first week, the correlation will be zero.

Conclusion: there is clearly not enough data. Demos, early access and release discounts do not have a clear relationship with Steam ratings. And the effect of their benefit or harm to the new game is not so pronounced that it can be evaluated.

GRIS

Question 3: How does the expectation rating of the game (its recognition before release) correlate with possible success?

Here, the expectation rating is measured by the number of players interested in a new title. How many of them, you can find out if you type any game into the Steam search and go to the community dedicated to it — the game center (the store creates it automatically). The number of users in the game center is a good indicator of the game’s recognition.

The correlation between users shortly before the release and the number of reviews for three months was 0.89 (very strong positive). The rank correlation was also high — 0.85. This suggests that the relationship between data from game centers and reviews exists for all games, not just for the most anticipated ones.

With the exception of one outlier (more on this below), the ratio of the number of reviews for three months to the number of subscribers ranged from 0 to 1.8 with a median value of 0.1. The lower threshold refers to a number of titles that had no reviews at all.

Accordingly, if your novelty has 1000 subscribers, then by the end of the first quarter the game may receive about 100 reviews.

The outlier I mentioned is as follows.

Some games had a very large number of subscribers compared to secondary indicators of market recognition — mentions on forums and on Twitter. It turned out that Steam automatically places users in the game center who have activated the game key for release. For example, Kickstarter backers or beta test participants. Thus, the game can gain more “subscribers” than organically.

Conclusion: the number of organic users in the game center really reflects the possible success of the novelty.

Just Cause 4

Question 4: What about the cost of the game?

The correlation coefficient between the price tag of the game and the number of reviews for three months is 0.36 (moderate). I am not sure about the usefulness of this data.

There is a correlation of -0.41 between the price of the game and the user rating. Apparently, players take into account the adequacy of the price tag when writing reviews for the game. A $60 title is more likely to get a review than a $10 title.

Question 5: Is it possible to predict quarterly sales of the game by the sales indicator of the first week?

The Pearson correlation coefficient between the number of reviews in the first week for three months is 0.99. Spearman’s coefficient is 0.97. These are the highest figures for the entire study.

For most new products, the number of reviews in three months is twice as high as their number in the first week. (Here we do not take games that have sold a very small circulation in seven days and received less than five reviews.)

It turns out that in the release week we sell the same number of copies of the game as in the next 12 weeks. The ratio of the number of reviews received in the first week to the number of reviews received in the first three months of sales ranges from 1.3 to 3.2.

The developers of several games, whose release in the Valve store was rather weak, were interested in my opinion on how to improve sales. I recommend looking at the number of reviews for the first week and just double that figure. This will be your result by the end of the first quarter after the release. Although I am sure that proper post-release marketing will help improve sales.

Conclusion: yes, there is indeed a direct link between sales in the first week and sales in the first three months.

Pro Fishing Simulator

Question 6: Does the quality of the game affect the tail of sales?

As just mentioned, there is a direct relationship between sales in the first week and sales in the first quarter. However, the range of this dependence is very wide. Here we introduce the concept of “tail coefficient”. This is the ratio of the number of ratings of the game for three months of sales (without the first week) to the number of ratings at the end of the first week of sales.

The lowest coefficient value — 0.95 — was at the Pro Fishing Simulator. This game received at the end of three months one review less than it was at the end of the first week.

The highest coefficient — 6.9 — was recorded in another game, which I will tell you about below. This game itself is a huge outlier.

It is hardly a coincidence that the game with the worst coefficient is rated at 22%, and the game with the best is rated at 96%.

Pearson’s correlation between the tail coefficient and the evaluation of the game on Steam is 0.42 (moderate coefficient).

Conclusion: although there is no clear link between the quality of the game and its reviews and/or sales, there is one between the “tail coefficient” and the percentage of positive reviews. Ultimately, quality games sell better (but let’s not forget that quality is a subjective concept, — approx. editorial offices). But the relationship is insignificant against the background of such a factor as the recognition of the game at the pre-release stage.

Overdungeon

Question 7: Is it possible to guarantee the success of the game before the release without having access to the vishlist of users?

I collected data on each game, often even before the scheduled release date, so that I could predict how many reviews I thought they could get in the first week. I relied on the number of subscribers of the game, although in the case of some games, data on the activity of the audience on the Steam forum and on Twitter would be useful.

The correlation between my forecasts and the actual results reached 0.96. Almost all of my forecasts in the table hit the mark (only in rare cases they turned out to be far from the real values).

My experience suggests that multiplying the number of subscribers by 0.1 in most cases gives an approximate understanding of how many reviews the game will have at the end of the first quarter. If the forum dedicated to the game does not have at least one question for every hundred subscribers, then this game has too many “inorganic” users, and the forecast needs to be adjusted.

Conclusion: it is possible to estimate approximately the success of the game in the first week, using data on the number of users, even before the release (with some exceptions). And you can also imagine the results of the first quarter, given the large relationship between sales for the first seven days and for three months.

11-11 Memories Retold

Last question: what emissions were mentioned?

During the research, I came across games whose indicators stood out significantly. These are the outliers.

The first of them is Overdungeon. Shortly before the release, this game had 77 subscribers in the game center. Quite a small number, so I was expecting less than 10 reviews in the first week. But 86 of them came. And the game finished its first quarter with 572 reviews. The game turned out to have the best “tail” among the entire sample.

It seems that Overdungeon is the only case in the sample when a game with a small pre—release number of “subscribers” came to solid sales for the entire quarter.

The second outlier is the title 11-11 Memories Retold. With 767 users in the game center before launch (10 times more than Overdungeon), it received almost zero reviews on the first day and only 10 in a week. Although her team had everything to unwind. The head of the project is Yoan Fanise, co—author of the popular Valiant Hearts, and the publisher is Bandai Namco Europe. Sebastian Koch and Elijah Wood took part in the voice acting. The game has earned dozens of positive reviews in the gaming and regular press. Now her rating on Steam is 96%.

And by the end of the first quarter of sales, 11:11 demonstrates the largest tail coefficient — but even then the number of reviews for it was 69. Now there are about 100 of them. The tail coefficient is incredible, but the game has almost certainly become a commercial failure.

This is a great example of the fact that a good and high-quality game does not necessarily sell well.

To summarize.

Main conclusions:

the success of the project on Steam strongly depends on its results for the first week of sales;
and the results of the game for the first week strongly depend on its recognition before release;
the quality of the game does not affect the performance of the first week too much, but it can have a positive effect on the “tail” of sales;
all conclusions concerning sales are based on correlation with the number of reviews.

Also on the topic: