How does audio work in a game versus in music and other media?

In a series of articles, PXL-Music attempts to paint a picture of immersive audio and where we stand in terms of audio in today's virtual landscape. In doing so, we look at the gaming world, streaming, the music industry, and the internet, among others. In this article, we outline how sound works in gaming and how that differs from how we typically view sound in the cultural sector.

The cultural sector: common with time as a reference

In the cultural sector, we have many different ways of working, each with its own unique workflow and experience. Just think of a recording studio, a band or solo artist, a concert hall, a post-production company, a festival, a classical orchestra... We all create experiences in which we are very specifically concerned with the progression of that experience. Albums, films, and series are developed on a timeline and are given a final, fixed duration. Festivals and concerts are concretized with timetables and scripts. The spectator or listener is seen more as part of the audience and has little influence on the experience, which can be viewed more as a shared one. There is a defined linear progression from start to finish.

Gaming: individual experience with events as a reference

The experience in a game is radically different. Instead of looking at a single experience for a large audience, the focus is on the individual experience of what constitutes a player. That player is in control and fully influences the experience through how they play. Countless possible timelines can occur, so to speak. There is action and reaction: sound and visuals follow the player's actions, while simultaneously the player reacts to what they experience through sound and visuals. We speak of interactivity. But how do you build that experience auditorily?

The passage of time is not the basis in games; instead, the experience is shaped by the events and building blocks of the 3D world. These form a complex set of parameters that determine the final audio mix, where a relevant mix is important. If the player applauds or eats something in the game, you want to hear those sounds. If the player walks on a wooden bridge over water, it will sound different than when they walk through the grass. If the player is in a crowd, you want to be able to hear individual voices all around. But you would rather experience a group of people far away as vague and impersonal. In terms of music, specific changes in the game (by the player) will signal changes in the music. But as long as those do not occur, the music must continue playing.

This naturally affects the way the composer and sound designer have to work. They can still build their sounds in the DAW, but will then have to export them as individual audio assets in a specific way. The composer needs to work more with loops, while the sound designer has to start building sounds individually and provide variants for the appropriate context.

How do we create the right mix in a game?

The classic way of working (still used for games with a limited scope or budget) involves an audio programmer collaborating with the composer and sound designer. They establish an audio feature set for which they must provide sound and write code to link their audio to events and objects in the game engine. In this way, the link between visuals and sound is established.

For a larger game, this entails a considerable amount of complexity and precision. If you want a specific sound effect, you will have to build it. Adjustments and mixing must also always be handled by that programmer, unless your sound designer has also learned to program and is therefore the programmer as well. Many agreements need to be made, and everything must be thoroughly tested for all scenarios. In large games with an open world to explore, there are too many parameters and functionalities to take into account. That is why audio middleware was developed.

Enter: audio middleware

Middleware consists of software packages that handle a specific element of a game. Unreal Engine and Unity are examples of middleware that handle the implementation of visuals. Audio middleware, the most prominent examples of which are Wwise and FMOD, handles the implementation of sound. They provide their own standardized, dedicated audio engine that runs in real-time behind the scenes of a game.

This audio engine listens to so-called game calls , data that the game transmits to indicate exactly what is happening in the 3D world (the parameters discussed previously). Think of information such as the space the player is in, the surface, whether there are other people or objects in the room making noise, information about the player's health, when they take footsteps, and so on. Through a software environment, you can then use tools to respond appropriately audibly to those game calls, so that you can link a 'footstep' and 'space' game call to the correct sound of a footstep in that space.

With audio middleware, no programming work is required anymore to integrate sound and visuals. You get the engine; you have to build so-called soundbanks from the audio assets and define when and how the engine should create a mix from them. The job of audio programmer (now: audio implementer) shifts towards creative thinking about applications to link game calls to sound. As a result, the job suddenly becomes very accessible to a sound designer, who can even partially replace the DAW with the middleware. After all, nowadays you can perform recording, editing, and mixing within the audio middleware. And although the game itself has no timeline, you can, for example in FMOD, build music from individual tracks on a timeline.

The benefits of audio middleware are clear, and as a result, an audio middleware engine now runs behind all major games. It allows for the development of a game's sonic and artistic identity more than ever before, especially now that both Wwise and FMOD have been free for indie developers since 2023. Support for new technology in middleware also ensures greater availability of that technology for use in games. For instance, Dolby Atmos is now integrated into Wwise, FMOD, and Unreal Engine , allowing you to immediately develop your game with immersive audio.