The Pitfalls of Testing: What Mistakes Do Developers Most Often Make When Testing Games?
Continuing with the publication of transcripts from the “Gaming Industry” conference. Up next is the presentation by Alena Karaseva, the head of the testing department at "GostTeam." She shared the most common issues encountered when testing gaming products.
In this talk, I will share cases from my experience.
But first, a little about myself. I entered IT 25 years ago when I was an engineering student. I've been in game development for 20 years, 15 of which I've dedicated to testing.
At GostTeam, I established the QA department from scratch and grew the team to 250 employees.
My team and I participated in testing games (including AAA titles and children's programs) and a variety of non-gaming solutions (online cash registers, online cinemas, and more).
In this talk, I will discuss:
- working with requirements;
- working with test strategies;
- developing test documentation.
I won't delve deeply into testing itself. It's just too vast a topic.
Working with Requirements
This is arguably the most challenging phase for any testing team because it involves the most communication.
At this stage, it's essential to:
- discuss which elements need testing;
- indicate the types of testing required;
- miss no details and maintain communication;
- when examining requirements, highlight and clarify all uncertainties and contentious areas.
Quite often, the first problems arise at this stage.
I've encountered situations where the team requesting our testing had no project documentation. Occasionally, I was told it existed, but only in the head of an expert who left the company months ago.
As a result, it becomes necessary to independently analyze the project, prepare documentation, and then ask the client for clarification on each element or feature to ensure we've understood how it should function.
There are times when documentation exists but is spread out as a collection of separate documents held by various specialists in different formats and stages of completion. This typically occurs when the client lacks a unified platform for maintaining documentation. Consequently, handling it generally proceeds as follows:
- part of the document's sections are under development by one expert;
- another part by a different expert;
- as soon as any of them finishes their portion, the document is shared as a file in a unified communication channel;
- the team ends up with several files with the same name but different contents.
The next problem is the roadmap.
But it's essential to understand: a roadmap is either a standard or a guide primarily for management and leadership.
It is quite common to receive a document with a roadmap only to find that timelines have shifted, and an entirely different feature is under development without any updates to the document.
Another issue encountered at the requirements phase is when the gaming platform is undefined.
For instance, a client might say: “Here’s a game, go through the tutorial, explore the map, collect loot, eliminate enemies, observe any defects and document them in bug reports.” As you start asking how various features operate, since the project documentation is lacking, you end up reaching out to developers to find out how a particular element should function. Subsequently, it turns out that the game requires more than just functional testing. Beyond that, it’s essential to define system requirements, test performance, and, most importantly, conduct regression testing.
Why is regression testing necessary? The product is in active development, sprints are ongoing, new features are being introduced and checked, yet nobody reviews how older features function post-updates — regression was forgotten.
What are potential solutions?
Firstly, it’s necessary to choose a unified platform for managing documentation and storing data.
Collaborative work on documents simplifies the workflow for all departments: analysis, marketing, and, of course, the QA department, where team members can immediately see feature owners and descriptions.
Obviously, it’s crucial to facilitate documentation maintenance. Keeping it updated helps identify discrepancies in testing requirements.
It’s very common, for instance, within project documentation to see, say, 25 sections. In section N1, there’s a description of a feature operation, and in section N25, there's a reference to the same feature but with a different description.
One of the testing department's tasks is to analyze such discrepancies, identify parameter mismatches, clarify the correct operation, and inform the feature owner of the inconsistencies.
Thus, the QA department identifies functional defects and aids in refining the feature launch protocol within the roadmap, helping to keep it updated so that resources can be aligned appropriately with current work priorities. As a consequence, resources and their volume are reallocated to tackle specific tasks.
All of this leads to the next stage.
Developing a Test Strategy
This is a crucial step for any team as it determines the necessary resource allocation for development.
We’ve faced situations where a game in release introduced a new feature monthly, yet lacked a test plan and clarity on what precisely needed testing.
It happened that a client allocated only two to three weeks for QA before a release. During testing, numerous critical bugs were discovered, forcing the team to postpone the release.
How to avoid that?
It is essential to clearly understand which types of testing are necessary and why they are essential within this development cycle, and to set priorities.
For instance, there was a scenario where we were asked to conduct performance testing at the prototyping stage. This was economically impractical since the project’s goal was to attract investors and secure additional development funding. We explained this to the client and advised focusing on functional testing.
Other recommendations include, of course, developing a test plan, which allows determining the testing budget.
The next phase, significantly affecting test plan budgeting, is determining the testing environment.
As I mentioned earlier, clients sometimes overlook which platform or equipment the testing should be conducted on.
For instance, we once had a client developing a web project. They anticipated the game primarily launching on desktop devices, so they only wanted it tested in a few browsers on Windows and Mac.
During negotiations, it became clear that the client wasn't aware that a significant portion of web traffic now comes from mobile devices. Consequently, our configuration range doubled. There arose a need to test the game on mobile devices as well.
In my practice, I’ve also encountered situations where clients wanted to save money. They suggested gathering the minimal possible configuration for testing, believing that a specialist could immediately conduct both functional testing and performance assessment.
What did this lead to?
The tester launches the game on minimal specs. It takes three minutes to load. Performance is at a slideshow level. Instead of catching bugs, the specialist continually experiences freezes. Consequently, testing time drags on, inflating its budget.
The takeaway is: do not skimp on the test environment. Yes, devices with minimum configurations should also be present, but as separate test rigs used, for example, once every two weeks to check how changes in the game have affected performance.
Let us move on to working with testing tools.
These days, it’s very challenging to select any tools at the start of a project due to sanctions (not knowing if the tool will function in a country half a year or a year later). Plus, generally, these tools' costs are quite high.
I mentioned the cost for a reason. Let me share one of our cases. About three years ago, a development team approached us needing to test a product’s compatibility and performance over a year. They allocated around 100 hours for this testing. Meanwhile, they wanted to use a specific niche tool that performed certain game metrics. The tool’s license was $5000.
What happened next?
The company spent money on 100 testing hours and purchasing this tool. At year’s end, we provided a report, which the team didn’t need after all.
That's why I recommend weighing all the pros and cons before collecting data and spending money, and considering whether the data is genuinely necessary.
Moreover, given the sanctions, I recommend considering domestic tools to reduce risks.
Testing
The testing phase itself is multifaceted and includes many stages.
I would like to focus on quality management and the issues we've identified during game product testing.
We often encountered a lack of defined product quality criteria. We had to create our own verification matrix and determine what constituted a release blocker for the project.
Another problem was the absence of simultaneous management of two branches — the development branch and the stabilization branch. Ideally, it should work as follows: development of a feature happens in the development branch. After release, the build with the new feature should move to the stabilization branch. Unfortunately, many developers do not follow this practice.
Moreover, the practice of maintaining a decision table has been neglected. This table records which objects need testing and which have already been checked.
We’ve already mentioned ignoring regressions. During active development, teams often lack the time to revisit and check other parts of the content within the game. More focus is placed on new features, which can cause other previously implemented elements to suffer. To prevent this, it's important to allocate time and resources for regression testing.
Additionally, when focusing on functional testing, some forget that the game primarily needs to be engaging. This is where focus testing comes into play and should also be included in the plan. It's necessary to define the target audience and determine if the functionality appeals to the users the game is meant for.
A common problem is developers forgetting to update test documentation. Many believe it's not important, or there's no time, thinking it’s better to just fix the bugs. This way documents remain outdated for months, then years. Consequently, when specialists change, the documentation no longer reflects reality, leaving no clear direction on what to tackle.
Alongside that, regular reviews of test plans are vital since any project is dynamic. Any introduction of new content or features necessitates changes to the test plan.
Often, I find that clients don’t analyze received reports. The reason is unclear; perhaps it's from fear of having to delay the release. So, it's often necessary to remind them: guys, notice that there are certain issues here, and they need to be addressed.
The next problem in quality management is communication.
Recently, we had the following case: the testing team spent five hours of their eight-hour workday on calls and discussions. Because of this, deadlines suffered.
I also often encountered situations where development teams would gather everyone for every call—developers, analysts, marketers, and designers. Consequently, a massive amount of time was wasted on trivial discussions.
Conversely, there were situations of complete interaction absence. Teams refused to call each other and discuss anything. This was equally detrimental to the project. Management didn’t understand what was happening with the product. Testers didn’t understand how game elements worked. Developers didn’t understand what to do next or what to fix.
To avoid such issues:
- clarify the channels and formats of interaction for team members;
- determine beforehand which topics will be discussed in which meetings and with whom.
This approach reduces time spent on communication, enabling tasks to be resolved more quickly. No longer will half a day be wasted discussing tasks.
And accordingly, detailed preparation of bug reports is crucial. The more detailed a bug report is, the fewer questions the feature development specialists will have.
That's all from me. Thank you very much for your attention.
