An immersive audio digital twin

There is a clear demand in the sector for digital twins for all kinds of applications —virtual soundchecks, running simulations, creating digital extensions of a physical space, ...  At PXL-Music Research, we observe that audio is always secondary in these types of applications. So we looked for the next-level audio experience in this context. Given our strength in immersive audio, it quickly became clear: an immersive digital audio twin. But how do you achieve that?

Concept

How can you create a digital representation of a venue and have an artist perform in it? For the audio section, we need to break that down into a number of other questions:

1. How do you define a venue auditorily?

2. How do you record a venue auditorily?

3. How do you process that recording?

4. How do you create an interactive dynamic model?

5. How do you place an artist in the digital twin?

How do you define a venue auditorily?

A space reacts to sound that resonates within it. Sound is energy that travels through space at a constant speed and collides with the surfaces in the space. The energy is absorbed and/or reflected. These reflections are chain reactions that continue until the energy is used up. The sound fades out.

With such a formulation, you might think that we are going to fully physically model a space using complex mathematics, in order to algorithmically approximate how the space would react to sound.  Systems of that kind are certainly being built, but frankly, this approach is not yet sophisticated enough to create a truly convincing copy of the space—let alone one that is also immersive and high-performing. Unfortunately, at PXL-Music Research, we also do not have the resources to develop a complete modeling engine using such complex mathematics.

There has been an established practice for decades: creating an impulse response and using it in a convolution engine.  We use a specific sound signal called a sine sweep. This is a continuous sine frequency that gradually rises in pitch.  You play that sound through a loudspeaker in the room and record the result. The recording contains the sweep and the room's response to it, which can be converted into an impulse response via deconvolution. You extract the sweep from the recording and are left with only the room's response. That response is then aligned in time (compressed) so that all frequencies start simultaneously instead of sequentially. It sounds almost like a gunshot.  Via convolution, you can then send new sounds through the impulse response, which then sounds as if you are sending it through the virtual space. Sounds simple enough, right?

How do you record a venue auditorily?

We now know how convolution works, but how does that work in Immersive?

We wanted a result that would work via both speakers and headphones. Therefore, we decided to use a microphone setup that is paired 1-to-1 with specific speakers. Specifically a 5.0.4: front left, front center, front right, rear left, rear right, top front left, top front right, top rear left, top rear right (LCR-Ls-Rs-Ltf-Rtf-Ltr-Rtr). Many immersive systems can convert channel-based to binaural, so this was a logical choice.

Other systems, such as ambisonic, also work for speakers and headphones, but because all capsules in ambisonic are positioned at the same point, you miss time differences between the capsules. We have already experienced in the past that these time differences are crucial for creating a sense of space.

There are various approaches regarding microphone selection, placement, directivity, and more. We have not yet found the ultimate setup. Omnidirectional microphones are important in any case, because they can pick up low frequencies well. However, cardio and hypercardio microphones can also be of added value because they capture sound very directionally and thus create greater mutual contrast. Further research is needed!

Because we didn't have enough speakers for every microphone, we placed one speaker at nine positions so that each microphone could capture at least one sweep specifically directed at it. That is a necessary minimum for our workflow in Immersive. Also, keep in mind that the speaker should remain at the same distance as much as possible. If that is impossible (such as at height), try to be consistent, especially within the left-right pairings.

How do you process that recording?

Deconvolution as described can be performed in various environments. We used Reaper ReaVerb to generate the impulse response .WAV files. You can then load these files into a convolution tool so that you can route sound through the impulse responses. This way, you get a simulation of the space.

Unfortunately, there are not many tools that perform convolution in immersive. There are often limitations; usually, they can only be used with impulse responses from the manufacturer. Moreover, we wanted a simulation of how sound propagates through space depending on the positioning of the source (we simply call it 'crosstalk').

Within the limitations of research, it seemed logical to us at PXL-Music Research to focus on the environment familiar to us and the sound engineer who would be mixing for a digital twin. Therefore, we saw few other options than to create an immersive convolution tool ourselves as a plugin. This plugin would handle the simulation of the room and work in environments with VST, AudioUnit, or AAX support. This covers DAWs, but equally well scripting environments like Max, DSP systems such as those from Fourier Audio, and more.

We have created a proof of concept for a plugin, which we even used live during concerts in our use case at the AB Club. There is also demand from students at PXL-Music to use the plugin for their own research projects. The entire process from concept to proof of concept is too complex to fully explain, as C++ programming language is required prior knowledge. In this regard, we relied heavily on the JUCE software framework , specifically designed for audio software development. JUCE provides an abstraction/framework that allows for platform-independent development. You can focus primarily on signal processing and the user interface, without having to take too much account of all the differences between VST, AudioUnits, AAX, Windows, and Mac.

How do you create an interactive dynamic model?

This section frequently uses the word 'engine', which is confusing because that term is used in both gaming and immersive audio. We have tried to make the distinction as clear as possible.

With the simulation of space, we have one aspect of the digital audio twin. But the end result must be a dynamic real-time audio game engine that responds to the listener's actions.

Although we initially planned to manipulate a DAW into such a dynamic game engine, it soon became clear that both development and use would be very cumbersome. At the same time, there is no audio middleware that can run plugins in VST/AudioUnits/AAX format, let alone in immersive. So, classic game engines were also ruled out.

What we were actually looking for was an immersive live engine like those used for concerts, but where parameters such as listener position and source position could be changed. Our plugin became the dealbreaker again, because immersive engines currently do not support plugins. If we wanted enough flexibility for communication with the visual side, we would essentially have to add an extra processing layer around that immersive engine. We actually found a fairly elegant solution soon after: putting an immersive engine into a Max for Live device.

Max, Spat and Ableton

Max (sometimes called Max/MSP) is a visual programming language, originally developed for interactive music composition by the research institute IRCAM in Paris.

Just like with a modular synthesizer, you connect basic signal processing components using cables. This way, you build complex block diagrams called patchers . You can use Max on its own, or use patchers via Max for Live devices in Ableton Live (in the Suite version). This allows you to develop your own music composition tools without programming knowledge and work with them in Ableton Live.

Max offers many modules for manipulating audio, video, text, and more. Additionally, plugins and concepts such as Open Sound Control, MIDI, FFT, and more are supported. With the help of packages from third-party developers, you can further expand the functionality of your Max as needed, and simultaneously that of your Ableton Live.

One third-party package has been in development since the 1990s: IRCAM Spat ∼ . It shares the same building blocks as the Flux::SPAT Revolution immersive engine (our favorite engine), but in Max, you can combine all these building blocks in your own way to build your own immersive engine. The package is free; with a purchase of Max or Ableton Live Suite, you can get started with it immediately.

Analysis of the digital twin

Now that the tools and vision are clear, we break down the digital twin by function.

In Max, we built a patcher that receives and processes audio from Ableton Live and OSC messages from the game engine. It contains 3 subpatchers:

1. Control patcher: OSC messages (and adjustable settings from the main patcher) arrive here. Based on the data, the 2 other subpatchers are controlled, as well as starting and stopping audio in Ableton Live.

2. Convolution reverb patcher: this contains the impulse responses from 2 spaces - De Kreun and the ballroom of d'Hane Steenhuyse. The control patcher indicates how and when to switch between them. Audio input comes from Ableton Live via Max for Live devices. Audio output goes to the Spat ~ patcher.

3. Spat ~ patcher: a binaural audio engine for use on headphones with a VR headset. In this, the position and orientation of audio sources and the listener in space are set. The convolution reverb is placed around the scene as a 3D audio source. Real-time changes in position and orientation are transmitted via the control patcher so that source localization corresponds to what is seen in the game engine. Audio input comes from Ableton Live via Max for Live devices and the convolution patcher. Audio output goes to the main patcher, which sends it back to Ableton via Max for Live.

Audio is located within Ableton Live. Using the external Max for Live package Audio Routes , we can send multichannel audio from various tracks in Ableton to a single Max for Live patcher. This ensures the patcher has enough input channels for immersive sound. The binaural output from the patcher returns to Ableton and simply outputs through the headphones.

How do you place an artist in the digital twin?

We recorded a performance of Mozart's "Allegro" (K282 III), played by pianist Tom Beghin. He played on a Stein piano, an accurate recreation of a piano played by Mozart.

To place him in the digital twin, there were 2 implications for how we had to deal with spatiality:

Acoustics

The reverb plugin was intended to provide the acoustics—that is, the reflections of the piano sound in the room. Therefore, it was important to have no audible acoustics in the recording, as that would interfere.

We therefore recorded the piano in a dry (heavily acoustically damped) booth at MotorMusic in Mechelen. This was a strange situation for the pianist, because in classical music, the room is an important part of the recording, almost an instrument in itself.

Sound emission

If the piano is in a specific position with a particular orientation, you want it to sound right. In classical recordings, the convention is that the high notes are on the left side of the soundscape and the low notes on the right. This is achieved by placing the microphones to the right of the pianist, halfway across the piano.

But in a digital twin where you can move around the piano as a listener, you want that projection of the sound to be true to life. After all, if you stand behind the pianist or halfway up the piano to the left, the low notes are on the left side of the keyboard and the high notes on the right, the reverse of the convention.

We opted for an immersive OCT-3D microphone setup, supplemented by an AB coupler. We did not export a mix, but rather the individual microphone tracks. These were positioned in Spat ~ to make the sound appearance as realistic as possible. We played the audio back in Ableton Live to break a few more sacred cows.