Latency as an obstacle for digital applications

Latency is an important concept for digital audio and video. It is the time delay a signal incurs while traveling from one point to another within digital systems. In analog systems with physical components, there is no truly measurable delay. However, digitally it is unavoidable, as latency is largely a result of the use of buffers, a fundamental building block of computer systems.

Why is this important? In late October 2024, The Eagles began a residency at the MSG Sphere. There, not only does the audience receive sound via the HOLOPLOT speaker system, but monitoring for the musicians also takes place via directional speakers. However, according to band member Joe Walsh, there was too much latency in the sound via the HOLOPLOT system. The delay worked against the sense of timing and made it impossible for the band to play together comfortably. Because the latency could not be reduced, they ultimately played using in-ear monitoring.

Even The Sphere, with its astronomical budget, is not immune to the concept of latency, so we want to clearly explain the concept in this article.

Buffers

What and why?

The brain of a computer is the processor (CPU). It performs the basic calculations on which all software programs in the computer run. The processor takes in data (input data), processes it, and outputs the result of the calculations (output data). A modern processor is enormously complex and performs billions of calculations per second, doing so for multiple programs simultaneously. That seems like a huge amount, but an action that seems simple to us can sometimes require millions of calculations.

It is difficult and dangerous to run a processor in full real-time (without latency). Components such as your hard drive are simply not as fast as your processor or RAM, and you must take this into account. If the processor is overloaded, calculations may be skipped, which will introduce errors. That is why buffers have been introduced: to temporarily write data and retrieve it when needed or when it can be processed. There are all kinds of buffers—not only within the processor—but here we focus primarily on the processor's buffer.

Buffer size and latency

Anyone who has already used a Digital Audio Workstation (DAW) has undoubtedly already set the buffer size , expressed in samples. The value you choose determines how many samples the underlying system will work with at a time: your audio interface software (drivers), the DAW, and your plugins always work with 'blocks' of that chosen size.

Choosing that size is a balance between latency and the processor workload. The smaller the buffer size, the less delay, but also the less time you give the processor to process everything. The processor must have processed the current block before the next one arrives. If this fails, you risk clicking noises and sections where the sound cuts out during recording.

The number of samples processed at a time determines the latency. This is because one must wait until the buffer is filled in order to populate a block. Therefore, there is a fixed waiting time between consecutive blocks, as the number of samples is generated at a fixed rate based on the sample rate. This is the latency, and it is simple to calculate using the following formula:

BUFFER SIZE / SAMPLE RATE = LATENCY .

With a buffer of 256 samples at a sample rate of 48kHz, it takes 256 / 48000 = 0.005333... seconds, or 256 / 48 = 5.3 milliseconds (ms), before it is filled. Therefore, all processing takes place in intervals of 5.3 ms.

You might then wonder whether doubling the sample rate will halve the latency at the same buffer size. That is true, but there are twice as many samples to process within the same time, so there is much more pressure on your processor. With a buffer size of 512 samples at 96kHz, you have the same latency, but your processor still has to process twice as much data within the same time. So that still requires more computing power. When recording, your hard drive also has to write twice as much data, which is much easier nowadays with SSD drives than in the past.

How many buffers?

Each subsequent buffer of the same size adds that same amount of latency. The result from the first buffer can only be expected after 5.3ms, and then the next buffer gets another 5.3ms to process that result. With 2 buffers, this already results in 10.6 ms of latency for 256 samples. Note that it is generally assumed that a delay of 10 ms or more is audible.

When you receive audio and immediately send it back out, such as during recording or live shows, two buffers are used anyway: an input buffer and an output buffer. At 256 ms, we are already at the limit of audible delay. A sufficiently powerful computer to set the buffer size low enough is therefore necessary, because the latency is usually slightly higher since the cables used (such as USB) and software generate a bit of extra latency...

Immersive live shows

With immersive audio, we work with immersive engines. These translate virtual positions that you assign to sound sources in the engine into real positions in your hall or room via the loudspeaker system. There are two types of engines for this:

• Software engines that simply run on your computer, such as the Dolby Atmos Renderer and Flux::SPAT Revolution

• Dedicated hardware devices , such as Adamsson Fletcher Machine , Yamaha AFC, d&b Soundscape, Coda Audio Spacehub , ...

The advantage of hardware devices is that they can minimize latency because the device is specifically designed to perform only the calculations for its operation. This applies to immersive engines, but equally to everything from digital mixing consoles and in-ear monitoring systems to anything involving digital signal processing. Manufacturers will choose a specific component configuration to ensure the device operates optimally with a fixed low latency. You must be aware of this low latency, but you usually cannot adjust it.

With software engines, the engine must compete with the operating system on the computer and other programs that are open or running in the background. This inevitably results in increased latency.

We consistently choose the Flux::SPAT Revolution software engine because we are not tied to a manufacturer's ecosystem and it is a fraction of the cost to purchase. SPAT simply runs on Mac or Windows and therefore integrates easily into a concert hall's systems, which is not always the case with hardware devices due to manufacturer limitations. For our immersive live shows at the AB Club and at Wilde Westen for Unwrap, we therefore had to minimize latency.

Dante and latency

Dante is an Audio-over-IP protocol that allows devices connected to a network switch via standard Ethernet cables to exchange audio via packets (so not the same blocks as before). Because everything is digital, you can create complex connections, duplicate audio without split boxes, and easily enable integrations between audio and video. Dante is the most represented Audio-over-IP protocol in our industry, so we use it ourselves as well.

Latency works slightly differently at Dante. For each device, you can set a latency in milliseconds, between 0.25ms and 10ms. This is not a buffer size but the maximum delay we accept between transmitter and receiver, i.e., the maximum time the receiver will wait for the next packet. With network protocols, latency is viewed more via hops : a connection between two devices (= 1 hop) introduces a delay, and the number of hops to the destination determines the total possible delay.

The latency you set in Dante is not the delay introduced by a device, but rather the maximum time waited between two devices for the next data packet. The highest set latency of the two devices will become the latency of the connection. If your transmitter has 0.25ms latency and the receiver 1.0ms, then the total latency from transmitter to receiver will be 1.0ms – whether necessary or not.

A packet may arrive earlier than the latency of that connection and will simply be forwarded further by Dante. With multiple hops, your packet may have arrived much earlier than the total latency from start to finish. However, if the latency of a hop is set so low that packets arrive too late, the packets are discarded, and the sound may sometimes drop out or you may hear clicking noises. You can check in the Dante Controller software whether any packets have been lost so that you can adjust your latency properly.