The methodology is good, but the sample is incorrect — Alexander Barabanov from Sad Cat Studios on the perelesoq study

Last week, the studio Perelesoq published a study suggesting that Steam wishlists are becoming outdated, and strategies focused on collecting them are ineffective. Some members of the community criticized this work. Alexander Barabanov, lead designer at Sad Cat Studios, explained to the editors why one should approach Perelesoq’s conclusions with caution.

Alexander Barabanov

I will start by saying that the methodology used by the team at Perelesoq is good and sound. Utilizing cubic regression to assess the rise and fall of interest in games on Steam is a smart approach.

However, in such studies, not only the method is important but also the accurate sample. Without it, no conclusions should be drawn — only hypotheses shared for future testing.

The sample in Perelesoq’s study is flawed for several reasons. The key issue, which many have already pointed out, is the small number of projects included. It's not possible to assess algorithms based on just 5-6 games.

But the problem isn't just the number of projects. The sample is heterogeneous. Projects have been on Steam for different lengths of time. They collected wishlists over varying periods. Some projects took five years to release, while others launched their page six months before launch. Not to mention that they released at different times.

It's incorrect to choose such diverse projects. I understand that not everyone is willing to share data, and the study had to make do with what was available. However, a limited amount of information shouldn’t be an excuse.

Additionally, for most projects in the sample, the majority of wishlists come from Russia. This audience is currently facing significant purchasing challenges: older Russian wishlists might not have converted for this reason.

Was this factor taken into account? Unfortunately, judging by the text of the presented study, it was not. In other words, external factors that could affect the data's representativeness were ignored during the study.

Another reason why the sample is problematic is the insufficient number of successful projects within it. Only one project out of six has more than 1,000 reviews, while two others have slightly more than 500 reviews.

As a result, Perelesoq's sample is not only very small but also heterogeneous. Due to this, one cannot extrapolate the data from these projects onto the entire platform. Furthermore, it’s inappropriate to make global conclusions and give advice.

To be fair, American marketer Chris Zukowski, whose article kicked off the discussion (in which the American marketer claimed that wishlists "are not outdated"), also presented a very small sample that shouldn’t be trusted.

In conclusion, I want to address the following quote from Perelesoq's study:

“Blindly collecting wishlists” is an ineffective strategy. The approach of "opening a page as early as possible and gradually accumulating wishlists," according to our data, does not yield significant results. It is better to focus efforts closer to the release and post-release period, where the most active and solvent audiences are.

I believe this is the main reason for the criticism in the game development community. This conclusion from the Perelesoq team reads as if the reach of an advertising campaign doesn't change depending on its duration.

This scenario will only work if the campaign's reach in the pre-release months is definitely greater than, say, launching a page much earlier with passive wishlist accumulation. Unfortunately, no one can guarantee or predict this.

Therefore, I consider this conclusion to be bad advice.

As for my opinion on the obsolescence of wishlists, I do think older wishlists indeed have lower conversion rates. However, the data presented by Perelesoq cannot serve as evidence here. The sample is too small and flawed.