Open Sound Control as a bridge between the audio and gaming worlds
Updated: Oct 6
Within our use cases, we developed a live show for the bands WITT and Pothamus with immersive sound and visuals. We brought the two domains together in a dynamic, often automated manner. Similarly, our digital twin for d'Hane Steenhuyse and De Kreun combined the principles of a classic game in an engine—as explained in detail in The steps to an immersive digital audio twin — with those of an audio production in a DAW.
OSC clearly emerged as the best way to establish the connection between the two environments and exchange data.
Open Sound Control
Open Sound Control originated from a very simple concept: a way was needed to enable communication between systems (computers, synthesizers, hardware) via a network, inspired by MIDI. The protocol was developed to simply send messages between a single transmitter and one or more receivers. The transmitter sends a simple text message over the network, optionally followed by one or more values/parameters. This can be done via a network cable, but just as easily wirelessly.
Examples are /digitaltwin/room/1/enter or /digitaltwin/listener/position/x 0.25. The formatting of the messages depends on the environment, and agreements between environments are important.
OSC is not a new technology; the first version dates back to 2002, and nothing has changed since 2009. So why are we betting so heavily on it?
OSC ('old tech') vs MIDI ('even older tech')
OSC offers a great deal of freedom and is very accessible. It creates a clear contrast with MIDI, an essential building block for music production. However, MIDI is very strict, limited in its applications, and, above all, unreadable. It pushed the boundaries of the limitations imposed on data transfer at its inception more than 40 years ago to the limit. Since then, MIDI has indeed evolved; we can send data much more easily, but those limitations from 40 years ago remain, such as working with blocks of 8-bit numbers. This is to remain compatible with older devices, but it has consequences.
It is therefore logical that sectors where MIDI is not essential explore alternatives—and they quickly arrive at OSC. It is perhaps the most widely used audio technology outside the audio world, where it is even used in touchscreen software, video editing, brainwave mapping, and more. Closer to this research, we find OSC in game engines, VJ software, immersive engines, head trackers, audio plugins, and more. Immersive audio specifically breaks with the norm by largely dispensing with MIDI and opting solely for OSC. OSC integration in music production may never be as extensive as MIDI support, but the future of immersive audio clearly lies with OSC.
Integration between environments
Open Sound Control has 2 main applications:
Remote control: you want to change the current state of a receiver, in other words, control it. With this, you target one specific receiver, often with messages determined by or agreed upon with the receiver.
Data transfer: the current state of the sender changes and it communicates this to the receiver(s). This is rather abstract; each receiver can respond appropriately without the sender knowing what that entails.
The command /digitaltwin/room/1/enter serves more for control, as you want the receiver to switch to room 1. The command /digitaltwin/listener/position/x 0.25 serves for data transfer; the listener has been moved, and it is up to the receiver to respond to this.
If you have a transmitter and a receiver that both support OSC, you only need to choose a network port through which you want to receive or send OSC messages. There are many options there as well: you can choose to listen only or send only, you can send over multiple ports simultaneously, you can send only to a specific IP address, and so on.
For lists of environments that support OSC, you can visit the Open Sound Control website or the Wikipedia page .
Example: digital twin
The digital twin is unique because we build it in two environments. The visual side was developed in Unreal Engine, and the auditory side in a combination of Max, our own software, and Ableton.
We relied on the basic principles of a game: the game engine generates the visuals and processes in real-time what the listener (or player) does in the virtual space. Via so-called game calls, information is passed via OSC to the audio engine, which also processes that information in real-time and adjusts the sound. In this process, sound essentially follows the image. We used two separate computers, both connected to a network switch via a network cable.
The OSC commands mentioned above are actual messages that we use to match the sound with the image. We have mutually agreed on the format of the messages. Specifically, the following information is communicated via OSC:
The position of the listener in 3 dimensions (x/y/z), each followed by a floating-point number between -1 and 1:
/digitaltwin/listener/pos/x
/digitaltwin/listener/pos/y
/digitaltwin/listener/pos/z
/digitaltwin/listener/pos/z
The listener's orientation (rotation left-right/up-down) + float:
/digitaltwin/listener/rot/yaw
/digitaltwin/listener/rot/pitch
The position of the piano being played, in 3 dimensions (x/y/z):
/digitaltwin/source/pos/x
/digitaltwin/source/pos/y
/digitaltwin/source/pos/z
The orientation of the piano (rotation, or how the instrument is turned).
/digitaltwin/source/rot/yaw
Entering and exiting a specific space.
/digitaltwin/room/0/enter
/digitaltwin/room/0/exit
/digitaltwin/room/1/enter
/digitaltwin/room/1/exit
When the performance starts and stops:
/digitaltwin/transport/start
/digitaltwin/transport/stop
To avoid having to continuously process OSC messages, it was decided to send messages only when a value changes. For example, it makes no sense to constantly transmit the listener's position if they are simply standing still. Messages were also sent exclusively to the IP address of the audio engine computer. In this way, we limit data traffic over the network and everything can be processed in real time.
Example: immersive concerts
WITT
For the WITT show, we used QLab. This is show control software that allows you to automate shows (music, theater, musical, etc.). Specifically, you build lists of cues that you can trigger at a specific point—individually or in groups, from fully manual to an automated timeline. Various actions can be linked to a cue, such as starting timecode, sending OSC and MIDI messages, controlling lighting, playing audio and video, and more.
During the show, it was important that a number of environments were started and/or switched at the same time. QLab was therefore a very deliberate choice. Specifically, the following chain was started for each number:
OSC - to SPAT Revolution: change to the snapshot of the song in the immersive engine. This is important because the positioning of musicians in the immersive mix varied significantly.
MTC - to Pro Tools: start and stop playback via MIDI Timecode. First, the click track starts for musicians (where relevant), then the immersive backing track.
OSC - to Unreal Engine: switch to the effect provided for this track, such as manipulating AI visuals via audio analysis, real-time AI images, ...
OSC - to Unreal Engine: start the visual effect in Unreal Engine.
OSC - to Resolume: connect to the AI-generated visuals for this song. This way, the clip is controllable remotely.
OSC - to Resolume: select the clip in the Resolume timeline, causing it to start.
The order here is very deliberate. The snapshot of the mix must have changed before the sound starts. Musicians were also played a click track from Pro Tools before the start of most songs, so they could come in with the backing track and video just right. The timing of the click track was also precisely timed relative to the end of the previous song so that there would not be too much silence between the applause and the start of the next song. After an exact number of seconds (different per song), the visuals were then started. Thanks to QLab and OSC, we were able to capture that timing during the rehearsal and, so to speak, press Play during the show.
We also clearly notice that precise control of Resolume is possible thanks to the flexibility and freedom of OSC.
VJ Nele Fack had planned visuals that continued running between tracks. We intended to synchronize Resolume with Pro Tools timecode, but then visuals were only available when timecode was running (only during the tracks). Additionally, transitions between the images did not work. The OSC message to select clips in Resolume simulates clicking a clip in the timeline with the mouse. If a video is still playing when clicking, programmed effects such as crossfades will be applied. Once the video is finished, Resolume continues on the timeline, and you get the transition visual between two tracks. If we did not use OSC, it made more sense to control Resolume manually than to synchronize with timecode. There was less guarantee of synchronization than with timecode, but with timecode, every transition would have to be done manually anyway.
Comments