Understanding Intelligent Networked Music Performance Experiences: IMPERMANENCE Framework

What Is a Networked Music Performance?

A Networked Music Performance (NMP) takes place when musicians who are geographically separated come together to play, connected through a computer network. Early experiments with NMPs date back to the 1970s, but only in recent years have advances in network communication technologies created the kind of infrastructure needed to successfully realize such performances. In addition, the massive shift to remote interaction during the COVID-19 pandemic sparked renewed interest in making music together across distances.

This chapter introduces the Intelligent networked Music PERforMANce experiENCEs (IMPERMANENCE) framework, a comprehensive model built to give musicians a compelling performance experience while playing remotely. To develop it, we first created the neTworkEd Music PErfoRmANCe rEsearch (TEMPERANCE) framework, which helps identify what musicians truly need in an NMP. The insights gained from TEMPERANCE then guided the design of IMPERMANENCE.

The Rise of Online Music Making

Over the past few decades, continually faster network technologies have cultivated a rich environment for online media. The COVID-19 pandemic made live-streaming and online meetings a daily fixture for millions of people. Music consumption also increased, whether through streaming services, remote concert attendance, or online rehearsals and live performances. Research into NMPs began in the 1970s, but only recently has it matured enough to become a workable option for geographically separated players.

To craft an NMP experience that truly engages musicians, two broad areas need attention. Temporal factors involve how the musicians stay synchronized, while spatial factors involve what they see and hear. A number of dedicated software solutions have been developed over the years, including LOLA, UltraGrid, and JackTrip. These are essentially low-latency audio and video streaming protocols. Minimizing latency matters greatly for a satisfying NMP, but the authors of this work argue that latency alone is not enough. A truly compelling experience requires more than speed.

The IMPERMANENCE framework does not try to replicate a hypothetical real-world setting. Instead, it treats the NMP as its own medium with unique characteristics. To build IMPERMANENCE, the authors first analyzed what musicians need, which led to the TEMPERANCE research framework. From there, a sequence of experiments was designed to clarify how temporal and spatial factors affect musicians’ perceived Quality of Experience (QoE). The results shaped IMPERMANENCE, which draws on signal processing techniques well suited to audio handling, alongside deep learning methods. These latter approaches overcome limitations of classical signal processing, particularly the need for many sensors and the degradation of performance in noisy or reverberant environments.

The TEMPERANCE Framework

The TEMPERANCE framework guides the design and execution of NMP experiments and scenarios. Although flexible enough for various musical genres and remote performance types, it was developed specifically with remote chamber music teaching in mind, under the INTERactive environment for MUSIC learning and practising (INTERMUSIC) project.

The framework defines a performance as what occurs when two or more subjects interact through a medium in a specific environment. A performance exists at the highest conceptual level and can take two main forms: a performed music composition or a taught lesson. It can function as either a rehearsal or a concert, always involving at least two musicians and possibly other people. When subjects share the same room, the performance is local. When they are geographically separated, it is networked — the focus of this work. If more than one subject is located at one remote site, it is called a mixed performance. The medium can be physical (air propagation) or networked (an internet connection).

For any single musician, the room where they play is the real environment, while the representation of the remote room — where the other musicians are — is the remote environment. The framework introduces a set of presence-based constructs that describe the NMP experience and can be applied beyond chamber music settings. To evaluate the performance, data collection is needed, followed by both objective and subjective measurements. Objective measurements numerically assess the performance, while subjective measurements rely mainly on questionnaires that probe the presence constructs and the overall QoE.

Study I: How Musicians Perceive Latency

The first study explored the impact of network latency on performance. Ten volunteers, split into five duets, each performed under six different latency conditions ranging from 28 ms to 134 ms round-trip, presented in a random order for each pair. The musical stimuli came from Béla Bartók's Mikrokosmos piano pieces, which explore relationships between rhythm, melody, and expression.

The performances were analyzed through both objective and subjective means. Objective measurements examined tempo trends (how players changed speed during a piece) and asymmetry (how misaligned the two parts were). Subjective measurements came from questionnaires based on the presence-based constructs. The results revealed no single general trend in how musicians coped with different latency levels; each player behaved differently. Notably, even under high delays, musicians still found ways to adapt or adopt distinct strategies. These findings prompted the researchers to move away from simply minimizing latency — a common goal in existing NMP software — and toward providing tools that actively help musicians work with latency. This led to the adaptive metronome approach used in IMPERMANENCE.

Study II: Audiovisual Immersion

The second study looked at audiovisual immersion in NMP environments. Eight musicians formed four duets, playing Béla Bartók's “44 Duos for 2 Violins” — pedagogical pieces designed to train motor responses, aural skills, rhythm, and structure. Each duet played under two different setups that offered different levels of immersion. The first setup placed a 24-inch screen about 1.76 meters in front of the seated performer, with sound coming from two loudspeakers on either side of the monitor. The second setup aimed to recreate a face-to-face arrangement: a 50-inch screen stood to one side, showing the remote musician as if seated beside the local player. Sound was delivered through open headphones (Sennheiser HD-650) fitted with a custom head-tracker. The head-tracker fed data to a Pure Data patch that provided real-time binaural audio using a set of Head Related Transfer Functions (HRTFs).

A questionnaire measured how both setups affected the musicians. The results carried important design lessons. The simple frontal screen was accepted well enough that the researchers felt no need to invest in special visual hardware for IMPERMANENCE. Auditory perception, however, proved more challenging. While the 3D audio was seen as useful, the headphones were sometimes deemed intrusive. This pushed the team toward researching loudspeaker-based 3D audio rendering instead.

The IMPERMANENCE Framework

Drawing on the results of both TEMPERANCE experiments, a unified framework for NMPs was developed. IMPERMANENCE tackles temporal and spatial factors with the goal of meeting real musicians’ needs.

To offset the disruptive effect of latency, the framework employs an adaptive metronome. A standard metronome ticks at regular intervals, helping musicians maintain tempo during practice. An adaptive metronome adds a beat tracker — a device that detects the tempo from a musical audio signal — and adjusts its own tempo to match the musicians’ performance. This approach sets IMPERMANENCE apart from most other NMP frameworks, which focus on latency reduction. Here, the idea is to help musicians manage the delay rather than fight it, a conclusion supported by Study I, which showed that musicians already develop their own coping strategies.

For spatial factors, the framework addresses both audio and visual domains. Study II showed that a screen adequately handled the visual side, so the main effort goes into creating an interesting auditory environment. The framework includes a method for rendering soundfields using irregular loudspeaker layouts. Because the IMPERMANENCE framework requires sending extra data over the network (multichannel signals, metronome signals, and so on), the team also explored compressing audio via Convolutional Neural Networks (CNNs).

Managing Latency with Adaptive Metronomes

Latency is one of the biggest hurdles in any networked performance. Low-latency solutions exist, but they often need high-end hardware that many musicians do not have. IMPERMANENCE therefore focuses on helping musicians cope with delay rather than eliminating it. Metronomes are lightweight, familiar tools that serve this purpose well. Earlier studies had already used metronomes in NMP contexts. In the IMPERMANENCE framework, the metronome is transformed into something closer to a Virtual Conductor (VC) — a piece of software that gives tempo cues much like a real conductor does in an orchestra. The key innovation is the use of adaptive metronomes, which track the musicians’ tempo during the performance through an off-the-shelf beat tracking algorithm and change their own tempo accordingly. Three adaptive metronome techniques were proposed, differing in how they process tempo information.

The simplest, Single Beat Tracking with Master/Slave Approach (SBT), assumes that musicians use a master-slave compensation strategy. One player acts as the leader, setting the tempo, while the other follows. SBT uses a single adaptive metronome that follows the leader and offers that tempo to the follower, helping the follower keep steady. Tested in real-world conditions at the Conservatorio Giuseppe Verdi di Milano, SBT showed promising results.

More complex approaches track both musicians simultaneously. Crossed Beat Tracking (CBT) assigns two adaptive metronomes: each player hears a metronome set to the other's tempo. Unique Metronome with Virtual Conductor (UMVC) uses two beat trackers but sends a single reference tempo to both players, acting more like a real conductor. These techniques were tested only with amateur musicians, but the preliminary results encouraged further development.

Reproducing Spatial Audio

The way a musician hears the instruments influences the overall NMP experience. However, as Study II revealed, the equipment used for sound reproduction must not feel intrusive. IMPERMANENCE therefore uses an irregular array of loudspeakers, where speakers are spaced at varying distances. Classic soundfield synthesis depends on large, evenly spaced speaker arrays, but NMP practice rooms are often cramped with instruments and cannot easily fit such setups.

An irregular layout is more practical but creates signal-processing challenges: the non-uniform sampling of the soundfield introduces distortions during playback. Finding the correct driving signals for such a configuration is a highly complex, nonlinear problem that resists simple analytical modeling. To meet this challenge, the researchers turned to deep learning, which can automatically extract highly nonlinear functions. To reproduce a desired soundfield, the system must first know where the musicians are located in the room. That knowledge allows correct directional reproduction and even virtual repositioning of players in the remote room. As with other soundfield techniques, source localization often needs many microphones. Here, two localization techniques based on minimal microphone setups were developed.

Localization with Ray Space Transform and Deep Learning

The first localization technique enables 2D localization of a musician using a small number of arbitrarily placed microphones. When the microphones are synchronized, the system computes the Generalized Cross Correlation (GCC) between a reference microphone and the rest. The highest peak in the GCC marks the time delay of the sound arriving at each microphone, which can then be used for localization. But noise and reverberation create extra peaks, making the task harder. The technique presented by the authors uses Convolutional Neural Networks along with the Ray Space Transform (RST) — a compact representation of the soundfield as acoustic rays, captured by multiple Uniform Linear Arrays (ULAs) of microphones. The CNN maps the noisy GCC data from the sparse real microphones onto a simulated RST computed in anechoic conditions using ULAs placed around the whole room. Tests showed that this method achieves accurate source localization even in highly challenging acoustic environments.

The second proposed technique for localization builds on a modified version of the GCC framework. It involves calculating the Frequency-sliding Generalized Cross-correlation (FS-GCC) by applying a sliding window approach. Doing so produces a set of sub-band GCCs that can be assembled into one matrix:

Here, each row holds the GCC for the corresponding frequency band. This approach is useful because, in an anechoic environment—free from noise and reverberation—the FS-GCC matrix has rank one. That property allows low-rank approximations to separate noisy components from the desired signal. Results reported in literature show that this method outperforms the standard GCC on real measurements. It does not rely on deep learning, making it more suited for settings where extensive computational resources are unavailable. Nevertheless, we tested applying deep learning to the FS-GCC method by using a CNN to denoise the noisy input FS-GCC, which yielded better performance.

Soundfield Synthesis Through Irregular Loudspeaker Arrays

To reproduce the soundfield, we use a technique based on deep learning and the Plane Wave Decomposition of the soundfield. Operationally, this follows an approach previously proposed known as Model-based acoustic Rendering (MR). MR reproduces the soundfield accurately when the setup is regular but suffers from reproduction errors in irregular configurations. Our technique employs a CNN that takes the driving signals generated by the MR method and outputs a compensated version of them. Since no ground truth exists for the compensated signals, the network includes a soundfield estimation module that convolves the driving signals with the corresponding point-to-point Green’s function to obtain an estimate. The difference in modulus and phase between the estimated and ground-truth soundfield is used for loss calculation and CNN training. We recently extended this method, following the same approach but extracting the driving signals directly from discrete points in the environment through a CNN, rather than compensating signals from the MR technique.

Speech Reconstruction from CNN Embeddings

Transmitting all necessary data in a NMP can create transmission bottlenecks, especially for multidimensional signals like those in spatial audio. We explored a technique in the literature that could enable audio data compression. This began with a preliminary problem relevant to Explainable Artificial Intelligence (XAI). Audio-focused CNN models typically take time-frequency representations as input, which are then compressed by successive layers to extract high-level features.

We used pre-trained CNNs as feature extractors and built symmetric decoder networks to reconstruct the input time-frequency representation from intermediate layer outputs. Results reveal that reconstructing input from convolutional layers is straightforward, but doing so from fully connected layers is significantly more difficult. This finding motivates further investigation, as well-designed networks could leverage intermediate layers for compression of multidimensional audio signals.