Adaptive soundtrack - what is it, why and how it works

Mark Brown, an indie developer and editor at Pocket Gamer, told what an adaptive soundtrack is and how exactly this technique is able to enrich the game – as part of the video cycle Game Maker’s Toolkit. With the author’s permission, we have prepared a printed version of the material and are sharing it with you.

In most video games nowadays there is a kind of dynamic soundtrack: the music reacts to what is happening on the screen. For example, the melody smoothly transitions from calm to tension when you engage in battle – and then becomes calm again when the last enemy in the location is killed.

For most games, this is the limit of dreams. Some games in this regard are a little better worked out – like Red Dead Redemption, where musical themes are intertwined, or like Dead Space with its scraps of orchestral melodies, or like in the soundtrack of Amon Tobin Splinter Cell: Chaos Theory, in which melodies seem to overlap one another in layers as the enemy notices the hero and the battle begins.

But I want to talk about a more inventive use of music in games. In some projects, the soundtrack was made interactive. As a result, the games became even more dynamic, users began to receive additional feedback and it became possible to create a different mood with only small changes in the soundtrack.

Nintendo loves this technique. This is especially evident in the Mario and Zelda series. Take at least the cards from the new Super Mario Bros U, where the same melody is used in each world, but at the same time musical instruments imperceptibly change as you move between locations.

Nintendo did the same in Skyward Sword. In Skyloft’s flying city stores, the music changes key as you move from store to store. And in certain dungeons, such as Sky View Temple, the main motif remains unchanged, but new elements are added to it as soon as you enter different rooms.

There are many successful examples in the game Spirit Tracks from DS. The tempo of the music accompanying the movement of the train changes as you accelerate. And in the Spirit Tower level, more and more new instruments are added to the melody as you climb the central staircase, which becomes higher the further you move along the plot. And when you finally get to the top floor at the end of the journey, it gives an absolutely incredible feeling.

The Super Mario Galaxy series also has a bunch of great examples. For example, objects that perform a melody while you pick them up. Or a level where you need to ride a ball – where the tempo of the soundtrack changes as soon as you accelerate or slow down.

There are also many examples in other Mario series games. In one of the projects, as soon as Mario sits astride Yoshi, drums are added to the main melody. There is an interesting musical illusion in Super Mario 64 – it seems that when you run up an endless staircase, the tone is constantly rising. Luigi in Luigi’s Mansion hums to himself or whistles, depending on how dangerous it is around. In Mickey Mousing, enemies dance to the music.

And yes, have you noticed how perfectly matched the music in the Mount Wario level of the Mario Kart 8 game is – like a live orchestra accompanies your every move? This is actually a very simple trick – there are four different soundtracks in this level and each one flows smoothly into the next one as soon as the player crosses an invisible border on the track. So while the player is going down the mountain, he hears one music when he enters the cave – the next, when he tries not to crash into trees – crazy violins come in. The finish line is also accompanied by a certain melody. The result is incredibly cinematic, and everything works perfectly, no matter how fast you drive or slow. This is the magic of the adaptive soundtrack, and Nintendo is using it to the fullest.

Almost none of these examples have a real impact on the gameplay, but they all add atmosphere and volume to the game world. Because of them, the Zelda series seems even more epic, and the Mario series is even more fun and toylike.

Of course, not only Nintendo uses these techniques. The adaptive soundtrack has been known since the days of Monkey Island 2, in which LucasArts used the iMUSE interactive music system to harmoniously combine tracks for dramatic moments, jokes and exploring the game world. This is most noticeable in the Woodtick location, where musical themes change as you move from room to room. And as soon as you wake up the sleeping pirates, an accordion is woven into the melody. As with everything else in the game Monkey Island, the inspiration for this technique was the rides at Disneyland. Those in which one musical theme is replaced by the next one by itself.

In the good old Rare series from Nintendo, this is also common. In the game Banjo Kazooie, for example, there is music that changes from location to location or becomes muffled when you dive under water.

A fresher example is Portal 2. Music is not only enriched with cool effects when you jump on a blue gel or run on an orange gel, but also some elements of the game world emit their own melody when you activate them. For example, in one of the locations, lasers, if redirected using sensors, create a new melody. This is not just another entertainment for players. The melody informs the player that he is moving in the right direction. And if you mess up somewhere, you will immediately understand, because it will be heard. Perhaps the creators of the game were inspired by the Auditorium game, where something similar is happening.

All these examples prove that developers often forget about sound as a way to give a negative or positive feedback to the player when he solves puzzles.

I also like how the theme music in Luftrausers changes depending on what elements the ship consists of. This enhances the sense of uniqueness of its design.

Or Wipeout HD, when a low-pass filter is applied to the background music as soon as you use the shield. As a result, the feeling that you are in a cocoon increases.

Or LA Noir, where the music stops as soon as you leave the crime scene. Because of this, it becomes clear that there is no evidence to be found outside of it.

The bottom line is that the soundtrack is a completely invented thing. I mean, if you get to a tropical island and start shooting at the locals there, there won’t be crazy war drums from heaven. Therefore, developers should not be afraid to experiment with sound – it will not harm the authenticity of the game. And you can also work hard and come up with something more interesting for an adaptive soundtrack than just amplifying noise during a fight. For example, in a battle scene, assign each enemy a specific tool. And as the opponents die, the music will fade away. I don’t know, I’m not a composer. But an adaptive and interactive soundtrack is found only in games, and this is very interesting. So let’s hope that in the future developers will start experimenting with sound more often.

Source: Mark Brown’s YouTube Blog