Samsung and Google IAMF

You may have come across articles recently about the so-called ‘open-source 3D audio format from Samsung and Google that is taking on Dolby Atmos’. It’s called Project Caviar or IAMF, which stands for Immersive Audio Model and Formats. But what exactly are Samsung and Google building? Here’s a hint: it doesn’t appear to be a competitor to Dolby Atmos.

With a view to what IAMF might mean for music, film, TV and gaming, we at PXL-Music have gone through the IAMF design document to outline what to expect. But to be able to compare Atmos and IAMF, first a brief explanation of Dolby Atmos.

What exactly is Dolby Atmos?

The raison d’être of Dolby Atmos is to provide a means of mixing and listening to immersive audio. To this end, Dolby has developed an ecosystem comprising three key components: a file format, a renderer (consisting of an encoder and/or decoder) and a standard/model (which defines speaker configurations, workflows for working in Atmos and more).

The technology in more detail:

• File format: a Dolby Atmos ADM file is a .wav file with additional metadata. This metadata, an Atmos-specific implementation of the open-source Audio Definition Model, provides information on how the audio should be interpreted by the renderer.

• Renderer: there are two types of renderers: encoders and decoders. Atmos renderers are built into DAWs such as Pro Tools, Logic and Nuendo, and are also available as standalone software. At the hardware level, Atmos is integrated into immersive engines for PA systems, soundbars, TVs, AV amplifiers, etc.

  • The decoder converts a Dolby Atmos file into sound using the metadata contained within that file. This data specifies how many channels are used in the mix, whether there are free objects in the mix and what their movements are at specific moments, and so on. The decoder analyses the listener’s system (speakers or headphones) and translates the mix as accurately and transparently as possible to that system. Even if you do not have as many speakers as there are channels in the original Atmos mix, you will still hear everything in the most accurate position possible.

  • The encoder creates Dolby Atmos files. In an Atmos-supported DAW or game engine, you can position (‘pan’), balance and move audio sources within a 3D environment, and all that data is written to the ADM file alongside the audio.

So what is IAMF?

IAMF has a fundamentally different purpose to Dolby Atmos: it’s being developed to enable better streaming of immersive audio over the internet. There isn’t a good open-source infrastructure for efficiently streaming in Dolby Atmos or other formats such as NewAuro, Sony360 and MPEG-H. Much more data needs to be transmitted, regardless of the end-user’s system. IAMF aims to handle everything behind the scenes without you even noticing.

When compared to Dolby Atmos, the technological components take on a different meaning, as the objective is entirely different:

• File format: IAMF does not define its own file format for immersive audio; it builds on existing immersive audio files. Support for immersive formats (without naming specific ones) and ambisonics is clearly stated.

• Renderer:

  • IAMF has developed a kind of decoder-encoder to stream immersive audio as efficiently as possible. The listener’s sound system is detected remotely, so that the audio can be decoded for that system before it is sent to the listener. This ensures that only relevant data is sent, which saves costs and is faster. The audio is then encoded into a stream capable of handling the typical issues associated with streaming: unstable internet connections, latency, data loss, etc.

  • Once it reaches the user, the immersive stream is reassembled by a second decoder and converted back into an audio file. The conversion to the listener’s system had already taken place in the encoder. There is just one extra step.

  • Next comes the mixer, which offers personalisation and optimisation:

    • Personalisation: you can turn up the commentary during a sports match, change the language – the sort of personalisation that can be particularly relevant for live events – or have voices dubbed, etc.

    • Optimisation: an AI that dynamically adjusts the balance of the mix so that it sounds ‘better’ on the device you’re using, taking into account what the device is capable of reproducing, and normalising the volume to make quiet sections louder and vice versa.

Why IAMF?

Samsung and Google have a long-standing, successful partnership and complement each other well with hardware and software. IAMF does the same: it is software that takes the listener’s available hardware into account. But why the focus on IAMF?

It seems to us that Google sees a clear future in immersive audio, and IAMF is a way for them to enable immersive streaming on a large scale on platforms such as YouTube. This is hinted at by the fact that the codecs YouTube uses for audio – Opus and mp4a/AAC – are included in the design. IAMF provides a technical framework, in combination with these codecs, to deliver immersive audio to the end user and lowers the barrier to entry (both financially and technologically) by streaming only what users can actually hear.

This is necessary because immersive streaming remains challenging. On YouTube, ambisonic support is limited to 360-degree video, which does not offer the same capabilities and experience as formats such as Atmos.

On the other hand, support for immersive audio depends on the format. Dolby Atmos is implemented and offered in slightly different ways on Apple Music, TIDAL, Amazon Music, Netflix and Amazon Prime, each with platform-specific limitations. NewAuro now also has a codec for streaming, which in turn needs to be specifically integrated for each streaming platform.

With IAMF, the format is decoded before it is streamed, and it seems to us that this makes it much easier to deliver immersive audio to the listener. This is, of course, subject to the important caveat that the decoding must work properly for all formats.

As far as Samsung is concerned, we suspect this is also linked to the Samsung XR headset announced in August, their counterpart to the Apple Vision Pro VR headset.

The IAMF design places a great deal of emphasis on scene-based audio, or ambisonics. However, ambisonics is currently being sidelined in favour of binaural rendering when it comes to the immersive experience on headphones. Examples include Apple Spatial Audio with head-tracking via AirPods, Dolby Atmos in gaming via Microsoft Spatial Sound, and the KLANG system, which provides musicians with immersive monitoring during their concerts. Binaural rendering is not specifically incorporated into IAMF, perhaps because its scope is considered too broad for IAMF, as the illusion of speakers through headphones promised by binaural technology is not yet fully refined.

Ambisonic, on the other hand, is conceptually quite simple and has proven itself for years as a good match for 360-degree video, such as in VR. Ambisonic and VR video represent a 360-degree environment in an equivalent way, meaning that a head movement simply results in a rotation of the image and audio. The combination provides a reasonably immersive experience on headphones when used with VR headsets.

It’s possible that Ambisonic is being considered as a format for the VR content Samsung intends to offer for its XR headset. Or perhaps they’re keeping their options open for both binaural and Ambisonic. Only time will tell.

Conclusion

Will IAMF replace Dolby Atmos? No, because IAMF is a framework designed to better support immersive audio, not an alternative to a format such as Dolby Atmos. IAMF is a system that operates in the background during streaming and which you’ll hardly notice, just as you aren’t confronted with codecs. Dolby Atmos and IAMF operate in completely different areas of immersive audio technology, and the notion of fierce competition is, unfortunately, seriously exaggerated.

Can Samsung replace Dolby Atmos with IAMF in its devices? In principle, Samsung could replace the Atmos renderer in its devices with a version of IAMF – but only if IAMF is also capable of reading Atmos files and translating them for the end-user’s sound system. Otherwise, that would be a declaration of war on everyone who has invested in Dolby Atmos.

With IAMF, immersive streaming will become possible on a larger scale within platforms such as YouTube and other streaming services. That is something we are, of course, enthusiastic about.

We are less enthusiastic about the mixer module. There is a risk that this could drastically alter the artistic choices regarding the balance and dynamics of a mix. These are often driven by emotion and the build-up of tension, and this could negatively impact the listening experience. A similar situation on radio stations suggests that this wouldn’t spell the end of high-quality immersive audio, but we must ask ourselves to what extent it is necessary.