How to test art correctly?
Why do we need a split art test and how to conduct it correctly, – says marketing analyst at Kongregate Heather Gainer (Heather Gainer).
Early testing helps to assess how your creatives (in this context, creatives mean advertising art for the game – editor’s note) affect the attraction of players. Art is usually tested for two reasons:
- when the visual component of the game has not yet been decided and the team is in search of an artistic style;
- to understand how cost-effective the chosen style is.
What to look at?
Let testing have only one task. If you need information on several points, run several tests. Never try to cram several tasks into one test at once: it will all end up with the results being useless. Art is tested according to various parameters; four main ones are listed below.
1. Which setting is best combined with the chosen artistic style?
When testing the setting, try to keep everything depicted in the picture within the chosen subject. Notice how the four characters in each of the pictures below are united by one setting.
I note that the CTR is often very influenced by pop culture. For example, the premiere of a new Marvel movie may make people more willing to click on superhero art. Unfortunately, what is popular today will not necessarily remain popular in a month.
2. Which art style is best suited to the chosen setting?
There are a huge number of art styles: 3D, pixel, low-poly, cartoon, and so on. The style is able to tell the user a lot about the game. When someone watches Dasha the Traveler for the first time, he does not need to explain that this is a children’s cartoon: it is immediately clear from the art that the series is for kids. When deciding which artistic style to test, it is important to imagine your audience. If you want to learn more about art styles, here are some resources on the topic: Creating an Art Style and The Aesthetics of Game Art and Game Design.
3. Which color palette best suits the game?
The color palette is sometimes also tested, but most often it is determined by the audience of the project and the emotions that the game should cause. Here are some pretty interesting guides for those who are trying to find a color solution for their game: Color in Video Games: How to Choose a Palette and Picking a Color Palette for Your Game’s Artwork.
4. How much is the selected art in commercial demand?
When it comes to game art, this is probably one of the main issues. In general, the trend is as follows: the more casual the game looks, the cheaper the purchase of users. But at the same time, the cost of users within the three main categories (casual, midcore and hardcore) still varies greatly! To determine commercial demand, the CPI test is much more reliable than the CTR test.
How to measure the results?
Is the build ready? If the game already exists (even in the form of a prototype), then you can measure CPI, CTR and conversion rate. If there is no game yet, then it will be possible to measure only CTR.
CTR test: allows you to determine the general interest in the game.
CPI test: gives more complete information about the commercial potential of the game. If the CTR test allows you to measure the overall interest in the game, and the conversion rate is how many users are willing to install the game, then the CPI test reveals how much the purchase of users will cost.
CTR test:
- select one parameter to be tested (setting, style, palette, etc.);
- create 2-4 creatives that are very different from each other, but at the same time focus on the elements that you are going to test; add 1 creative from another project for which the CTR values are known. It will be used as a reference point;
- create Facebook ads with the selected creatives (to do this, use the Facebook ad creation guides). Make sure that when clicking on each of the creatives, the user goes to the same application (make sure that no other factors affect the downloadability of the application);
- ensure that each ad has the same statistically significant number of impressions.
The most important thing is confidence in the result! The smaller the difference between the two indicators, the wider the audience of the test should be.
For example, if one creative has a CTR of 1.82% and the other has 2.18%, it will take 96.2 thousand impressions to figure out which is better. If the CTR is 1.70% versus 2.30%, then only 35 thousand impressions will be required. To calculate exactly how many impressions you need, use a statistical significance calculator or something like that.
CPI test:
- select one parameter to be tested (setting, style, palette, etc.);
- create 2-4 images that are very different from each other, but at the same time focus on the elements that you are going to test;
- if this is possible, then for guidance, run another game in parallel in the same countries as the project under test. CPI can vary very much from month to month. It is always useful if there is something to compare it with;
- create Facebook ads with the selected creatives. Make sure that when clicking on each of the images, the user goes to the same application (make sure that no other factors affect the downloadability of the application);
- ensure that each ad has the same statistically significant number of clicks (see the previous paragraph).
Please note: in the case of the CPI test, you need to count the number of clicks, and for the CTR test, the number of impressions is counted.
Why Facebook?
I will answer in one word: targeting! In the campaign settings, you can set both age and gender – so that only your target audience will see the art for sure. If you run the test without targeting and get, for example, a high CTR for some art for preschool children, despite the fact that the target audience of the game is men 18-35, then the results of such a test are unlikely to tell you anything. If the game is for everyone, then you can test it on any network.
How do you know if art’s performance is good or bad?
Unfortunately, I don’t have any reference metrics with which to compare the project’s performance. The market is constantly changing. That’s why I recommend using a reference point – an image that was used for another game and that you can rely on when testing new art, so that you can evaluate the results in comparison.
The test is completed. What now?
- Make sure that the difference between the test results is statistically significant. If it is too small, you may need additional data or images that are even more different from each other.
- What are the results of art – the same as the landmark image, and better? If not, then you may have to make additional changes and run the test again.
Source: Kongregate blog
