How is the analytics department built in Belka Games

Evgeny Gilmanov — Head of Analytics studio Belka Games — in his column for readers App2Top.ru he tells in detail how he built the analytics department in the company.

Intro

Evgeny Gilmanov

The last year at Belka Games turned out to be very busy. Including in the analytics department, the chance to form which I was given.

Even before working at the company, I had experience in analytics of browser games, as well as in analytics of web and mobile projects. Belka “bribed” a huge number of interesting and ambitious tasks.

There really were a lot of them. First, it was necessary to improve the analytics system as a whole. Secondly, in a number of moments it had to be rebuilt. For example, we wanted the system to be able to give advice on improving features both in terms of user experience and in terms of metrics.

Now there are enough tasks too. We have a place to grow.

The first time

1) Selection and implementation of an analytics system

When I got a job at the company, the first thing I had to do was bring all the people here to one “analytics window”. To do this, it was necessary to select and implement an analytics system.

The process took about two months and deserves a separate longrid. In short, there are many solutions on the market, all of them have their own characteristics both in terms of features and in terms of prices.

While the choice of the analytics system was being made, server developers were transferring the database from MySQL to Greenplum, a more modern and suitable tool for working with large amounts of data.

When choosing a system, you need to understand the needs of the product and the team. You don’t always need a spaceship with a bunch of features. Also, a paid solution is not always the best.

The most important thing to understand is that you choose the system for a long time. It will be difficult and labor-intensive to create a large number of reports and then “transport” them to another system.

When choosing an analytics system, the following was especially important for me:

the ability to flexibly configure what data I want to receive;
the ability to quickly receive and visualize this data;
the ability to give access to data to a large number of colleagues;
high speed of the system;
all data must be on our servers.

As a result, after researching more than 10 solutions, the project team and I settled on the re:dash analytics system. It allows you to use a simple code editor to get a visualization of queries, collect dashboards from them, and hang variables/filters.

We tied the system with html code and got a platform for a single “analytics window”, which we use now in the company.

Belka Games analytics window (edited, without exact values)

You can read about both Greenplum and re:dash via the links.

2) Setting up processes between the product department and the analytics department

Analytics should not only be accurate, but also bring answers and insights. And most importantly, analytics should be timely. Getting a cool insider in six months, when the project has already changed several times and acquired new features, is sometimes completely useless.

In order for analytics to be timely, it is necessary to instill in the team the habit of receiving feedback within a strictly stipulated time frame.

For example, we change the level design of the range of levels. After that, it is necessary to measure the effectiveness of changes on the part of the audience. Based on the conversion rate and a number of other internal metrics, we calculate the minimum sample size and the number of days for which we will achieve it (if the number of days tends to an indecently large value, the experiment is sent for rethinking).

After receiving the sample size, the test results are calculated, and a decision is made to roll out a new level design for the entire volume of the audience.

3) GIGO and self-test

There is such a principle in analytics: garbage in — garbage out (GIGO)*.

* According to Wikipedia, this principle means that incorrect incoming data will produce incorrect results, even if the algorithm itself is correct. In Russian—speaking culture, the analog of the principle is the expression “What you sow, you will reap.”

Therefore, it was important to make sure that the data was clean and correct during the first months of work.

Here are a few points for self-checking when preparing reports:

you need to have several analytics systems as data sources at least for the first time;
when analyzing, you need to thoroughly study the feature in order to understand for yourself how it works (including a brief description of the feature in the reports is also a good habit);
the analyst must understand what is being written to the database (it is better to check the logs with your hands if this entity is new to the database);
the information provided (graphs, tables, and so on) should be clear and visual: the more difficult it is to convey information, the less trust there is in it.

***

When the main tasks for the first time (creating a single “analytics window” and debugging processes between departments) were solved, internal development teams realized that analytics is an excellent tool for getting answers and insiders. The understanding was followed by a geometric growth of requests.

It’s time to expand

1) How did you decide that we would grow

A large volume of tasks is very nice and cool, but there is a nuance. Human possibilities are limited: even if you are a workaholic and like to recycle (we do not do this and do not advise anyone), this will not solve the problem. It is difficult for even a small team to keep analytics of several projects at a good level (especially such large ones as “Watchmaker” and “Funky Bay – Farm and Expeditions“).

So we came up with the idea of expanding the team.

It is difficult to find people in the CIS market, but it is possible. There are several main points for the candidate (I am not consciously writing about the default things from the job description right now):

relevant background (analytics, but not necessarily in game development);
independence and accuracy in working with data;
proactivity.

Since the beginning of 2019, we have started looking for people. I talked about Belka Games analytics on DataTalks and DevGAMM, we actively interviewed candidates. Now we have a fully equipped team, but we are always open to bright minds, there is more work every month.

2) Remote

The most difficult, but necessary for the team is to decide on a remote. Sometimes, due to procedural peculiarities, this is not possible at all.

We at Belka Games had doubts about this format. An analyst who does not work as closely as possible with the team may miss important information somewhere, or be insufficiently motivated and productive somewhere. The cons were clear, but the pros outweighed:

the distributed state of the team gives the employer more flexibility;
overcoming the fear of inefficiency of the remote, it is possible to scale the team faster and better;
building a distributed team is my personal challenge.

Now 60% of the analytics team work distributed in other cities and only 40% — in the Minsk office.

I consider the key success factors of the remote team to be:

a person’s high interest in what we are doing (this interest is a process that begins with an interview and a trial period, it is the task of the company and the lead of the team to maintain interest);
work according to the same rules with all teams (we have Asana as a binding tool for the entire development, and no significant task passes by);
careful documentation (all analytical notes are kept in Confluence, a knowledge base, without which it is very difficult to live);
transparent team dynamics (everyone in the analytics team knows who is doing what, there are processes and they are followed);
regular one-on-one meetings/calls with the manager (within their framework, they discuss: traffic on the roadmap, any corporate issues, difficulties on tasks, and so on).

3) Adaptation

Despite the distribution of the team, all the guys, without exception, go through the adaptation period in Minsk. It usually lasts a month and a half.

The first couple of weeks of a new employee is scheduled by the lead almost by the hour. Then he becomes much freer.

The stage of personal adaptation is very important. This is acquaintance and lapping with the team, close work with the manager, understanding what result is expected from him at Belka Games (we are waiting for the results already during the trial period).

After the adaptation period, the team can be confident in the person. It is important.

How processes are built

1) Prescribe the processes

The system will not work correctly until the rules of its operation are prescribed.

Now we have templates for routine tasks (analysis of updates and features). Templates are needed so that there is no circus with the design and harmony of analytical documentation.

We have all the processes registered. What is meant here is that there is documentation: when, what and how should be analyzed. For example, there is a separate dashboard for all features/updates to monitor only the necessary indicators.

Belka Games analytics window (edited, without exact values)

I consider the process of setting tasks by analysts to be important. While it was not prescribed, at times it was difficult even to start the task, the flow was stretched to clarify the details / deadlines / desired result from the customer, which affected the speed and even quality.

To prescribe this process, we conducted several internal meetups where colleagues asked questions to the analytics department. In response, analysts told how it is more convenient for them to digest requests in the spirit of “count and tell”.

2) Keep our finger on the pulse

The team has an open roadmap for each month, which indicates how many percent each task is completed. Here you can track the dynamics of the team.

Every week we form a new sprint based on the roadmap and the needs of the product teams.

Every day we all write to the closed Slack channel what was done yesterday and what is in the plans for today.

From the outside, this may seem like excessive micromanagement. However, it is very useful to understand in the morning what you did yesterday and what you will do today. Plus, there is always the opportunity to share this with colleagues. This is a cool ritual that allows the brain to wake up.

3) We formulate short and succinct answers

The answers to the product teams to their requests should be formulated as accurately and as soon as possible. It is not enough to give an answer AS IS: it is necessary to offer recommendations on how to improve/ balance the feature, what worked well in the update and what did not.

Here I want to stop a little more detailed. Analysts (as it happens) have one trait. I will not say that it is good or bad, but we really like to “pour” a dozen graphs and tables seasoned with terms on a person. This approach causes discomfort even for a prepared reader, and what can we say about a busy manager. Analytics turns into multi-page reports that no one reads.

Detailed analytics are needed, and we have not left it, but we have introduced a “report rule”. Its essence boils down to the fact that the analyst must disclose the results of the feature test in one or three sentences and suggest improvements within the same limitations.

Example:

Conclusions:

we leave the feature on the prod;
the change in total increased the retention of the 7th day by 20%;
the change affected the iOS audience more than the Android audience (an increase of 25% versus 15%).

Offers:

explore deeper the difference in effect on platforms;
the feature has potential that can be developed by refining the balance.

The conditional producer receives several results on the implemented features and a list of proposals for “discuss”, as well as a link to a large report with graphs, tables and a bunch of text.

4) We take part in inventing features at all stages

Analysts often complain to each other that developers come to them and say: “The feature was released in the prod a month ago — do an analysis.” Or: “Our metrics have dropped, find out what happened.”

I wanted to avoid this from the very beginning of my work at Belka Games. Finding out such things in hindsight is often very difficult. The necessary data may no longer be available at this stage. Therefore, our analyst’s work with the new feature is structured differently. The analyst participates in all stages of its implementation.

Design and game design. Designing new logs (if necessary), a description of what and how we will analyze. It is important to schedule this before the release, and also think in advance about which metrics the feature could potentially affect. This will allow you to more accurately predict which logs may be needed, in what time frame it is possible to evaluate the feature.
The appearance of logs. Here the job is to control what reaches the databases and in what form.
A/B test. To make a decision about the need, calculate the necessary volumes and produce results is the task of an analyst exclusively.
As soon as the feature is released to the prod, the analyst creates a dashboard for it in such a way that the DG, producers and all concerned colleagues understand what is happening with the game and how the feature affects the balance. You need to create it on the same day or the next.
If we are talking about an update, then we prepare the main metrics in the dashboard separately for cohorts of new and old audiences.

5) Tracking metrics

We separately monitor how releases and features affect different segments of users, how they affect old players and new ones.

For beginners, we primarily track the retention curve (Retention) and LTV.
For oldies, we track the dynamics of churn rate and ARPPU/ARPU.

If we are talking about the introduction of a new feature, then we look at the metrics associated with it, as well as those that it may affect.
For example, we change the balance of personal goals in the “Watchmaker” side events. What interesting things can I see here?

Macrometrics:

engagement (time in the game, number of sessions);
revenue.

Custom metrics:

how has the proportion of people achieving the goal changed?
have the players involved in the feature started spending more time in the game while this feature is running?
how did the feature affect the balance of in-game currency received/spent?
how will the feature work together with other in-game events?

6) We are implementing achievements for analysts

We have recently introduced a new ritual — the achivok system. At the end of each month, the analyst selects from two to five completed tasks and reveals them as follows:

task description;
brief conclusions;
further actions;
exhaust for the company.

It seems to be nothing new, just a regular report. But when you perform an analytical task with an eye to filling out the “Further actions” and “Exhaust” items, you have a completely different attitude to how to submit information to the team and what result to demand from yourself. “In the table” will not work.

7) We take part in game decontracts

In many companies, the practice of deconstructing games is put on stream. But usually this is done by producers and game designers. We have them doing it too, but not only them.

I advocate that analysts also actively participate in this process. The reason is simple. The coolest ideas arise at the junction of user experience, numbers and knowledge of your game.

What’s next

In principle, it is impossible to come to an ideal system, an ideal analytics department. New challenges will constantly arise, and with them fakaps, breakthroughs, and powerful ideas.

However, you can strive for the ideal. For this, it seems to me, it is important to keep the system in working order. The latter requires:

clear and working processes;
love of games and interest in the industry (in game dev, you can’t come to work just to do work);
work closely with the product, marketing and business development teams.

Also on the topic: