Immersive audio and spatialization in games

In a series of articles, PXL-Music attempts to paint a picture of immersive audio and where we stand in terms of audio in the virtual landscape of gaming, streaming, and the internet, among others. In this article, we look at immersive audio in gaming.

Recently, we organized a workshop with Dolby for students at PXL-Music and Howest DAE. The use of Dolby Atmos in gaming was discussed there. Dolby confirmed what we already suspected: immersive audio is a very logical next step in improving the gaming experience.

The world is immersive

Anyone familiar with PXL-Music knows that we conduct groundbreaking research into immersive audio. Our motivation in that research is simple: immersive or spatial audio offers us an experience that corresponds to how we perceive sound in real life, namely in 3 dimensions.

“There is no non-spatial hearing,” wrote the German scientist Jens Blauert in 1997. Wherever you are, there are sound sources all around us, the position of which we can often clearly deduce.

... but what about in a virtual world?

In the previous article, you could read how sound works in a game . An audio engine runs in real-time in the background of a game. That engine listens to so-called game calls , data that the game transmits based on the player's position and actions in the virtual world. Audio middleware provides you with the tools to develop that engine primarily creatively, rather than having to build all the technical aspects.

Most gaming experiences take place in 2 dimensions on a screen. But the world in a game naturally consists of 3 dimensions. Objects give shape to that 3D world, and precise positional data is available for every object at a given time. With immersive audio, that positional data can be fully utilized for the first time to offer an accurate, high-quality 3D sound experience.

Spatialization

We generally speak of immersive audio rather than spatial audio. In gaming, it is clear why: people began experimenting with spatializing audio in games as early as 15-20 years ago. This was driven by the improved quality of home sound systems. The average gaming experience was in stereo (i.e., 1 dimension), and some people even invested in surround sound (2 dimensions). In doing so, the added value of audio that adapts to your perspective as a player became apparent. Let's call this horizontal spatialization.

Specifically, you want to hear movement, albeit limited to the horizontal plane and linked to what you see on the screen. If a sound source is visible on the left, it sounds on the left; if it moves to the right on the screen, you hear it move to the right. Positional data suddenly becomes relevant, but using it adds extra complexity and work.

With the rise of audio middleware, horizontal spatialization is becoming much more accessible because it is offered by default in the middleware. Nowadays, in the Wwise software, you only need to check 'spatialization' for a series of sounds. When the audio engine sees the positional data of the corresponding objects, it will move the objects based on your perspective. You don't have to do anything else.

Towards 3D Spatialization

But no matter how you look at it, horizontal spatialization is a translation of 3D positional data into 1 or 2 dimensions, and therefore far from perfect.

There are two major actors accelerating the push towards immersive audio for games: the introduction of home VR headsets and Dolby. VR headsets change how the image is displayed in a game: you no longer need the classic 'camera' perspective, but a 360-degree spherical image of which the player with the headset views only a part.

But Dolby is also already working on 3D audio via Atmos in film and music, and they are looking at the next market. They recognize that if the audio engine supports 3D audio, it is possible to work directly with the 3D coordinates of objects. This makes it simple to place objects in an exact, accurate location around a player and offer a high-quality experience. Therefore, starting in 2015, Dolby entered into partnerships with game developers, console and computer developers, and middleware developers. The result is broad support for immersive audio not only in consoles and games, but also in middleware such as WWise, FMOD, and Unreal Engine. As a result, the barrier to working with immersive audio from the start is much lower today.

Benefits of immersive audio in gaming

Theoretically, it sounds logical, but what is the added value of immersive audio for the player?

Thanks to the highly accurate 360-degree experience, the game world feels more believable. This creates a suspension of disbelief : the player feels that the auditory experience is influenced by movement, allowing them to become more immersed in the environment. The gaming experience is thus more intense and allows a player to truly become aware of the virtual environment, to feel more in the middle of it.

Research also confirms this, according to Dolby: immersive audio reduces reaction time. With immersive audio, users get a much faster and better sense of their virtual environment and will perform better in the game. If you can clearly hear when an enemy is behind you, you can react better to it. "A proper rendering in Atmos gives you a much better environment to game in."

As with music and film, the final mix is adjusted via a renderer to the ultimate target system—stereo, surround, immersive for speakers, binaural for headphones—and platform. For instance, renderers for Dolby, DTS, and Windows already exist. This makes the audio engine less complex and saves time in game development.

"But how can we easily do immersive gaming then? People aren't going to fill their homes with speakers en masse just to game, are they...?" A valid point. For gaming, the most important experience is actually the headphones. According to Dolby, more than a billion people game via headphones. Consequently, the renderers mentioned above all offer a binaural 'immersive' headphone experience. This way, you don't need anything extra except headphones, and we all have those, don't we? At PXL-Music, we are not yet convinced by the existing solutions, but we do believe that the future of immersive audio for gaming lies there.