Adaptive Music for Video Games - How does it work?

Adaptive music is a growing trend due to computer/video games, to enhance the user experience. But how does it work really? And what are the most important things to learn, and to implement, as a composer? :slight_smile:

Please share your thoughts and insights into this exciting field of “adaptive music”.

I’m tagging Vicente @VSHDEL, as he wrote in another post that he was focusing on learning more on this field in music composition for video games, and I find it a very interesting topic too! :slight_smile:

3 Likes

I think that the core part of video game music is quite easy:

Your major job is to compose cues with a lot of stems that work harmonically perfect together when the game triggers it to choose a loop again.

Film Music is linear music that goes from A to B and never comes back. But game music is linear and adaptive at the same time and you can only achieve it with either loops/stems or harmonies that work together.

If you play games like Call Of Duty WW2, you will hear that music is not playing all the time, only when it makes sense to serve and support the story, but you don’t hear the same exact music, you hear the same melodies and motives but with variations. That’s why you are never bored to play the same game or level again, it always feels different even though you see the same pictures. Music one of the most important key elements in games today, as if music would be the same with no variety, people will be subconsciously be bored and would not recommend to buy it. It has to be an experience for many playing hours, and one of the best examples are GTA and Last Of Us. Story and the musical support make these games awesome and you never get bored, even if you need to replay a level because a Zombie :zombie:‍♂ has eaten you up :joy:

3 Likes

Thanks Alex, very good explanation. I wonder if a good thing would be to create groups with variations for each section, and then stem them out so they are interchangable. That is probably what I would do, but I never worked with a video game so I am not sure what their demands for the music would be. :stuck_out_tongue:

As mentioned above: Film is a linear media where music is synced to the picture. A game is an interactive medium where we need to give the impression that music is written to accompany your performance. One of the main challenges is not to make the music repetitive to avoid boring the player.

Similar to the filmmaking field, game composers must understand some basic concepts of how a video game works: What’s a game engine, an event, a map, a cinematic/cutscene, a trigger box, etc.

You need an audio middleware since game audio engines are simply not as powerful as their video engine. One of the most used ones, especially for AAA games is WWISE. There’s a list of games that use it on AudioKinetic’s website. Other middlewares such as FMOD are also used in the industry.

The middleware’s job is to manage how audio behaves in a game as well as the resource budget, dialogue in different languages, audio settings per platform, etc. For the music, you can set a BPM and program its behavior (for example: Switch from Track A to Track B on the next beat/bar/marker and use this cymbal swell file as a transition).

As for the music writing part, it’s crucial that the composer discusses understands how the music system works within the game in order to adapt their writing techniques to so it works with the interactive music mechanics. There are different writing techniques. Here are a few:

  1. LAYERING

You write and orchestrate your music as you do normally and stem it out. For example, Strings High, Strings Low, Brass High, Brass Mid, Brass Low, etc. By doing so, every stem is also filling the appropriate sonic space in the frequency spectrum too.

Instead of adding more elements so they can work together, you simply strip down your arrangement into submixes but making sure the appropriate layers are grouped together (I learned this technique from Tom Salta’s conference at GameSoundCon).

The middleware will be programmed to add/remove layers according to different events. It could also be controlled with a variable between 1 and 100 according to your evolution in the game, or stress level, or anything really.

  1. BLOCKS

You write in small loopable section blocks and switch from block A to B to C on specific events (for example Change of location, collecting x number of points, etc.)

  1. STATES You can write two pieces of music that play simultaneously, one on mute, the other at full volume and they crossfade to switch. A great example is Mario Odyssey on Nintendo Switch when he goes from full 3D to 8Bit and the music goes from a full orchestra to chiptunes. This system could work in different contexts such as Exploration vs Combat music, for example.
  1. RANDOMIZED BLOCKS

Similar to point #2, but the blocks are contained within a looping random playlist (similar to “shuffle mode”) and they always transition well and logically, regardless of the order, to prevent the player from being bored.

  1. AMBIENT MUSIC WITH LAYERS OF RANDOM PHRASES

Similar to the blocks in point #2, but each one has ambient layers + an additional layer that consists of musical phrases triggered randomly every x number of seconds.
Example: The exploration music in UNCHARTED: The Lost Legacy

  1. MUSIC BY STINGERS

When the player reaches certain areas, short musical pieces/phrases are triggered and separated with gaps of silence.

For example, God of War:

Obviously, techniques can be combined together. You can have randomized blocks + different layers per block + randomized music phrases per block, etc. The sky is the limit and it depends on so many factors. These discussions are usually made between the composer, the audio director, game designers, and programmers prior to writing the music.

There’s a great article from Designing Music Now that I highly recommend.
Other sources:

6 Likes

@medhathanbali Medhat, you always amaze me with your breakdowns full of depth and details in such a readable and enjoyable style! :smiley:

This post is so great in fact, that if you don’t mind, I would like to share it on my blog. Please let me know, of course you will get full credit and links to your website etc. :slight_smile:

2 Likes

Everything that I was going to add has already been said. But it’s probably useful to add how the origin of video game music started. Originally games worked on an 8 bit system. That was comprised of 8 small cores of memory. Usually one of those cores was responsible for the music and another core was used to sync the music to the game. The rest was gameplay information.

Then the music was comprised of whole sections of music that looped. Space invaders would be good examples of this. As we moved to 16 bit this just expanded, the phrasing and voices got bigger too. Mario and sonic were testament of this. Now we had big pieces of music and every level ha xxz finished piece that looped. It wasn’t until the late 90s that music started to become more thematic, legit motifs were introduced along with better graphics (resident evil and tomb raider did this very well).

At this point another element was introduced that was equally important in video game music… the sound effects. They got increasingly more complex and this pushed music back into smaller phrases. If I’m not mistaken this is where the whoosh really took hold, originally taken from hip hop.

Always thought it was useful to understand the origin of something :slight_smile: plus I love games, I just don’t have as much time to do it nowersays.

2 Likes

Thank you @Mikael! I’ve been teaching composition for games at the college level in Montreal for a while, so I try to be as clear as possible :slight_smile:

Definitely, I did some edits on my post to make it clearer and added some extra book resources that would hopefully be useful for someone who wants to go further.

1 Like

Thanks Medhat, may I refer to you (title you) in any particular way of your choice and link to your new website for 1-1 consulting? :slight_smile:

Sure, I’ll private message you :slight_smile:

1 Like

Yes, please do! :slight_smile:

do you think it’s okay to composer for video games if you don’t play video games yourself? But maybe you just get familiar with the video games out there just for general knowledge?

You don’t need to be a gamer but at least be up to date with the releases on the market, understand the music styles, the narrativity throughout the game as well as the interactive audio system.

Then again: Maybe you should ask yourself why someone would want to work for many months on a project in a field they aren’t passionate about? :slight_smile:

1 Like

I see. Makes sense, I mean if you enjoy the process of writing music to picture and for video games pictures too, then would it be okay still if you don’t play the games yourself. Seems you think it’s okay as long as there is an interest for the stories and what the picture is.

A good friend of mine writes music for high-profile games and franchises, he once told me as a joke “I’m not much of a gamer. When I’m playing, I feel like I’m working” which I found pretty amusing :grin:

1 Like

Funny that we just started talking about this for the game OST I’m working on, and now it’s being discussed here. We’re experimenting with multiple versions of the main theme that are all proximity-based. There’s a main, constant ambience that loops, and a handful of different versions of the theme that are all always “on”, but depending on where the character is in the game, one or another of the versions naturally fade in and out. Work-in-progress, and we’re also trying to figure out how to have a trigger that shifts naturally from the main theme into some tension-building offshoots.

1 Like

Reminds me of Ocarina of Time’s maze where 2 layers consisting of the percussion and melody are playing and you need to follow the melody to find the right path (This video at 2:15). Obviously your system is more advanced :smiley:

I don’t know what middleware you’re using, but in WWISE, you can insert multiple custom cues as markers on a music track, and they serve as transition points.

The transition will only happen once the play head reaches these cues to keep the musicality. This could be the solution to your problem.

1 Like

I personally like if there are also ‘stingers’ meaning short almost sound effect style music phrases, they can add a lot of that immediate immersion into the world you interact with, just like sound effects do.

Did you experiment with adding those musical stingers?

Stingers are a great way to hide transitions if they’re set to transitions immediately. But if there’s a lot of shifts, stingers can quickly become annoying like in Metal Gear Solid

1 Like

Yes, but those stingers were too loud and “annoying” to be honest. :stuck_out_tongue:
I also love when games have interaction based stringers. Perhaps that is not called stingers?

For example, I loved Warcraft (the strategy game, not the MMO), and when you clicked the church in Warcraft 2 there was a “divine choir” stinger playing. I could click on it many times because I loved the sound so much lol.

Hahahaha Warcraft. “Yes, m’lord?” “Of course, m’lord”. Starcraft’s, “Remain vigilant” still pops up in my music / movie / game quote Tourette’s on a regular basis.

1 Like