Pixonic: "How we tested and tested our hypotheses"

About the importance of events and the experience of working on them, — says Vadim Charugin, game designer at Pixonic.

The material is an adapted version of Vadim’s report, read at the White Nights autumn session in Moscow.

Vadim Charugin

Events: concept, pros and cons

For us at Pixonic, when it comes to hypotheses and their verification, one of the key terms is “events”. This word can mean different things, so I would like to start with what we usually mean by it.

Events are in–game events that are limited in time. As part of these events, players are given any additional functionality or content that is not available in the product at any other time.

Among the advantages of holding events:

— the ability to increase the user base of players;

During the holidays, users have much more time to watch the game, play it, participate in some activities. They also communicate more, watch more videos.

For example, on our YouTube channel, all videos related to events collect much more views than usual (the difference sometimes amounts to millions of views).

— high efficiency for retargeting;

Users respond better to those campaigns that are associated with events. Advertising for former users works especially well if the game promises to reward in case of return with any content.

— increase in the average check;

As part of the events, users spend more than usual. This is due both to the fact that during the holidays people are more prone to emotional purchases, and to the fact that sales often take place during events.

— a way to test hypotheses and test new mechanics.

With the help of events, we add new functionality or content to Pixonic, and then monitor how players react, collect feedback. At the same time, if the hypothesis is unsuccessful, we do not need to tell the players that this is some kind of temporary activity, it is initially declared as incoming. If they like it very much, then they themselves actively ask for it to be introduced on an ongoing basis, which we do.

The disadvantages of events include:

— temporary drop in sales after the event;

Within the framework of events, users play more actively than usual. There is a risk of being fed up with the game. Over time, it goes away, but within this period of “recovery”, the metrics of the game may be worse than the average.

— getting used to events;

If events happen too often in the game, players can get used to them. It will seem to them that an event with special content and temporary promotions can start at any moment. And instead of spending here and now, they will wait for the event. This will lead to a drop in revenue from the game as a whole.

— tight time limits regarding seasonal and festive events.

Many events are associated with specific dates. For example, you can’t release a New Year’s event in February. This keeps the team in constant tension. To keep up, you have to start cooking them sometimes six months before the release. The impossibility of delays leads to the fact that they can come out with bugs.

Our first event

We took up events in War Robots only in 2016. Then we clearly understood not only their pros and cons, but also that they had become a standard for the industry, and therefore a necessity for us.

However, there were two more important circumstances why we launched our first event only that fall.

First. By that time, the game already had enough content for all types of users. And we were ready to do more.

Second. We wanted to add something new to the game.

Event launch time and duration

Since the launch of the first event, we have experimented a lot with the start time and duration of the event. As a result, we came to the following conclusions.

First. The event on all platforms on which our game is released must take place at the same time. If it starts on one platform on the 20th, and on the other on the 22nd, it will be very inconvenient for players and may confuse the work of the developers themselves.

Second. The event should not be too long. Otherwise, it may tire users.

For example, we had an event that started in early December and ended in mid-January. In the first weeks, he was perceived positively by the players. But when the event went to the fourth week, the players were overworked. As a result, by mid-January we got a drop in activity. It reached the previous level for the rest of January.

The third. You can’t make the event too short either. It is important that as many users as possible can participate in the event. We have come to the conclusion that the optimal time for the event is 17-18 days. It is better to schedule the start on Thursday, when the working week begins to come to an end.

What we are doing in the framework of events

We have changed a lot of things in our events during their holding.

Icons

This is a change that any player sees as soon as the game is updated. Many people realize that the game has somehow changed, and this encourages them to return to the project.

You should not change the icon too often or too much. In this case, a scenario is possible when the user either will not be able to understand what kind of game is in front of him, or will guess: is this a new icon or is it still an old one?

New visual elements on maps

A good solution is to add new elements within the event that do not affect the gameplay, but bring visual diversity. Such things show the players that the event has begun.

The new design usually causes discussions in communities, because a new visual always comes to someone, and someone does not. Players go to forums, share their opinions. The latter has a positive effect on engagement.

The main thing is that the thematic design does not work for too long. The latter can tire and thus cause a negative reaction in the spirit of “it’s spring outside, and you still have festive Christmas trees on the cards in the game.”

Our most striking case here is the first Halloween event. We came up with the idea to add ghost robots to the maps. It sounded very logical, because Halloween is associated with ghosts. Our ghost robots were supposed to appear here and there, run around the map, scare the players, make a slight commotion on the battlefield.

Before the release of the event, we released a video about this temporary innovation. The video gathered a lot of views and was warmly received. The players were really waiting for this feature.

However, when we rolled out the change, the game immediately began to fall due to a very large number of players. Therefore, on that Halloween, not everyone could see the ghost robots. So this event has become something of a legend that many are still asking for and waiting for.

New visual elements on the main screen and in the UI

Today, as part of the events, we try not to make big changes on the game maps. Instead, we focus our efforts primarily on the hangar and pop-ups. The reason is simple. Any player who enters our game, first of all sees the hangar and gets a pop-up.

The main difficulty here remains for us — maintaining productivity at the same level. If you start the game, and it starts to slow down, then instead of positive emotions, you will get negative ones. And the event will work with the opposite effect.

Additional tasks

Now we are adding quests as part of the events without fail. One of their goals is to increase the time spent by users in the game during events.

Initially, we had quest chains. They worked as follows: once a player completed the first quest, he could complete the next one. At the end of each block of these tasks, the player received a reward. The more blocks passed, the steeper the reward.

The system has had a positive impact on user activity. The average number of fights per day per person during the events has increased significantly. Moreover, after the end of the event, activity began to return to its previous level. We got rid of the downturns.

We also found out that some quests were performed much more often than others. So the team came to the conclusion that we and users evaluate the value of game items differently. Subsequently, we corrected the situation. To do this, it turned out to be enough to redistribute quests in such a way that players first receive weapons, and only then robots.

We also noticed that the players who go the furthest in the quest chains are those who:

they pay most often;
they pay the most;
they play the most.

This system had one big drawback: if you entered the game at the end of the event, you did not have the opportunity to complete all the quests in the chain: you simply did not have time.

So we redesigned the system. We decided to abandon quest chains, made quests easier, limited their execution by time and began to issue them every day.

Regardless of whether you have completed the previous quests or not, if you log in on Tuesday, you will receive a quest for Tuesday, and on Wednesday you will receive the next one.

We also reduced the total number of rewards that players could receive as a whole. But we have increased the percentage of people who have received at least one. As a result, some awards were received by 20 times more people than at previous events.

Testing within the framework of events

In this chapter I move on to a new topic. In it I will tell you what we have tested and are testing within the framework of events.

New Robot Painting system

We initially launched the painting system as a temporary innovation. The new mechanics allowed players to buy paint schemes and use them to change the visual appearance of robots.

Our goal here was to check how many players would be interested in it.

We didn’t really like the result. This mechanics interested a small part of the audience. Perhaps the fact is that the players in War Robots do not have time to study the skins of allies, and the coloring of enemies due to the predominantly long-range gameplay we can not see.

Increasing the limit of players in clans

Another feature that we tested as part of the events was the ability to increase the size of clans.

Our clan system allowed players to accumulate activity points. The more players play, the more activity points they have and the higher their clan is in the league.

We wanted to check how many people would be interested in this. As a result, we saw that our clans as a whole are not very interesting to players. Therefore, in the future, we firmly settled down to improve the entire system.

Game Modes

The most interesting thing that we are rolling out as part of the events is the new game modes. Each of them greatly changes the gameplay, gives a new experience to the players. If the mode becomes widely popular, it remains after the end of the event.

One of these was the “Skirmish” mode (Skirmish). This was our first non-rating mode in which players fought each other not on their robots, but on ready-made builds from developers. The builds themselves change regularly.

This regime had three tasks:

advertise our new content among high-level players;
give new players the opportunity to evaluate the entire game arsenal;
increase retention.

As a result, interest in the content increased slightly. However, it turned out that players enjoy playing in this mode due to the fact that the rules are constantly changing there (for example, some types of robots come with weapons that are not normally put on them).

Thanks to the regime, the ratio of DAU to WAU increased by 5%.

A/B testing within the framework of events

Thanks to the events, we also got the opportunity to regularly conduct A/B tests.

Pop-ups (pop-ups)

For example, we compared the effectiveness of various pop-up windows warning about the end of the event.

We showed Group A a pop-up, in which there was a plain text with a description of what awaits the players during the event, as well as a “Go” button to the social network to find out the details.

Group B saw a pop-up with stylized pictures, and group C saw beautiful pictures, some of which were even animated.

We thought that Group C would show the greatest conversion. However, a picture with stylized pictures turned out to be more effective. The lowest conversion was demonstrated by group A.

Alerts

We also conducted A/B testing of push buttons.

We divided all the players into 8 groups. We sent three push messages to each of the groups during the event:

notification that the event has started;
notification in the second week that the event is ongoing;
notification before the last weekend that the event will end soon.

Pushy was very different in subject matter. In some, in addition to the main information, it was reported about gifts, in some it was about quests or special offers.

The largest conversion to launch was shown by the group where each new push was significantly different from the previous ones (both in topic, content, and suggestion).

Interesting fact: all the pushies from the groups that had emojis performed better than the others, so since then we have been using emojis in our alerts as often as possible.

Current experiments

Another thing I would like to share with you is our current experiments.

For example, last fall we tried to add a regional event to the game that would be visible to all players, but focused primarily on a specific country.

In October we had an event dedicated to the Day of the People’s Republic of China. This holiday falls at the beginning of the month. Usually, no one works in China all week within its framework. Obviously, users have more time to play.

We were interested:

how will Chinese users react to the event?
will they really play more on the weekend?
will such an event be interesting to users from other countries?

The answers to all these questions were positive.

***

This spring, War Robots turned five years old. By this point, the game has passed the threshold of 120 million installations. And today she is not going to slow down.

Also on the topic: