Ubiquitous Music and Interactive Performance: Gesture, Sound, and Cyberspace

Ubiquitous Music and Interactive Performance

What exactly is ubiquitous music, and why does the term matter? According to Pimenta (2009), ubiquitous music systems are "musical computing environments that support multiple users, devices, sound sources and activities in an integrated way." The concept traces back to Mark Weiser, who famously defined ubiquitous computing as "the opposite of Virtual Reality" (Weiser 1996). In his view, ubiquitous computing encourages individuals to interact with computers in everyday life, rather than escaping into a simulated world. Our growing reliance on information-processing machines now seems obvious: we use notebooks, netbooks, mobile phones, music players, car computers, smart appliances, surveillance systems, and medical devices almost without thinking. We exchange data through them — in public or in private, for necessity or for pleasure, alone or connected across the internet.

From ephemeral sound to recorded object

Before recording technology, music existed only in the moment of performance. In classical tradition, a notated score waited for a live interpreter to bring it to life. Over time, notation became more elaborate, capable of capturing increasingly complex musical ideas. A musical score resembles a map rather than the road itself — it requires a performer's interpretation. With the invention of audio recording, however, the immaterial world of organized sound could be crystallized into lasting objects, much like paintings or sculptures.

Analog and digital representations of sound

Digital Signal Processing (DSP) examines how continuous signals — such as sound — can be represented as discrete values over time. Sound reaches our ears through longitudinal waves of compression and expansion traveling through air. These pressure oscillations do not carry matter; they move through it. Inside the ear, these mechanical vibrations become electrical neural signals that the brain interprets as sound, provided the oscillations fall within our auditory range (Fornari 2010).

Humans have long known that sound produces vibration, but storing those vibrations was another matter. The first successful attempt came in 1857, when Édouard-Léon Scott de Martinville invented the phonautograph, which etched acoustic vibrations onto paper rolls. In 1878, Thomas Edison improved on this idea with the phonograph, which cut grooves into a malleable surface such as wax, tin foil, or lead. A needle retracing those grooves could reproduce the original vibrations as sound (Fowler 1967). Digital audio recorders became commercially available only in the 1970s, using Pulse-Code Modulation (PCM), a method invented by Alec Reeves in 1937 for representing sampled analog signals numerically.

If we consider analog audio a first-order representation, digital audio becomes the second order. Electroacoustic music, which flourished in the 1950s, was rooted in the analog domain. Composers recorded sounds on magnetic tape and then literally cut, copied, pasted, and distorted segments. They applied analog effects such as reverb, chorus, echo, and delay, and manipulated speed or direction. All these operations remained within the continuous analog space.

Digital audio breaks the continuum into discrete samples governed by the Nyquist–Shannon sampling theorem (Marks 1991). This theorem states that any continuous periodic signal can be fully reconstructed from samples taken at a rate at least double the frequency of the highest spectral component. Because digital audio is essentially a sequence of numbers, it can be stored, streamed, edited, analyzed, transformed, synthesized, and replicated with ease. Music in digital form has penetrated daily life to an unprecedented degree; listening now happens almost anywhere, anytime.

The immanence of cyberspace

The information revolution did not arrive through massive, centralized supercomputers. Isaac Asimov imagined Multivac — a giant machine capable of answering any question — in his 1956 story "The Last Question." Instead, we have the internet: a global network of countless small computers. In 2008, Google CEO Eric Schmidt estimated that the internet held about five million terabytes of data, of which Google had indexed roughly 200 terabytes (0.004 percent). The cyberspace we routinely access and interact with is vast and still being explored.

Virtual reality has largely yielded to ubiquitous computing. We connect through small devices linked via the internet, accessing huge databases while constantly exchanging tiny pieces of information. Some scholars view this networked society as a form of self‑organization akin to neural pathways, creating what Émile Durkheim (1895) called a collective consciousness. Real-time text messages, chat, and voice over IP no longer surprise us; we can talk or video‑conference with someone on the opposite side of the planet, with instant access to information that would have required immense effort only a generation ago.

Even so, sound remains bound by physical limits. Anyone can notice how slowly sound travels compared to light: a distant worker hammering creates a visible strike well before the noise arrives. Acoustic waves move through air at about 342 meters per second, and the human auditory system detects delays as short as 0.1 seconds (Gelfand 2004). This places a practical constraint on real‑time musical interaction: musicians farther apart than roughly 34 meters will hear noticeable lag. Concert stages are typically smaller than 30 meters — the famous Sala São Paulo in Brazil measures 20.5 meters wide and 11.12 meters deep.

In cyberspace, however, sound information can travel far faster than sound waves through air. Digital audio can be streamed and broadcast over the internet, although compression (like MP3) is often lossy. Even uncompressed digital audio cannot perfectly match an actual sonic source. Yet cyberspace offers something unique: the immanence of distribution across vast distances. Instead of streaming finished audio, we can now transmit real‑time control parameters for virtual musical instruments, allowing remote musicians to interact live.

Transmitting gesture instead of sound

The key idea is to send data about musical gesture — captured in real time by motion sensors — rather than the audio those movements produce. Musical instruments become computer models that emulate acoustic generation processes, and artistic gesture controls their synthesis parameters. These models allow virtual instruments that can be processed locally but controlled remotely, with gestural data traveling through cyberspace in real time. This shift focuses on capturing and transmitting movement through gestural interfaces, virtualizing instruments, and controlling them via remote gesture.

Sound synthesis models can run entirely in software. Several methods reproduce the acoustic behavior of real instruments while offering distinct advantages: low memory requirements (no sampled sounds), fine control over the emulated physics, and the ability to build virtual instruments that would be impossible to construct physically. All this opens extended musical techniques far beyond what acoustic instruments can achieve.

Gesture-driven sound control and ecological modeling

Exploring virtual instruments driven by gesture allows intuitive access to extended techniques for contemporary music. Real-time gestural control of computer sound‑synthesis models enables a group of artists to discover new timbres — for instance, a virtual bell with independent control of each harmonic, or a berimbau whose string can grow to several meters in length.

This approach also encourages ecological modeling in music (Barreiro and Keller 2010), where interactions between agents and musical objects within a virtual environment produce sound events located in time and space. As agents and objects exchange different states, the system continuously adapts to aesthetic needs, guiding agents within their ecological niche. Sound here is the primary medium of interaction — a choice that facilitates acousmatic compositions and interactive performances.

Interactive performances with ubiquitous music

Modern software environments like Pure Data (puredata.info) allow users to connect simple audio‑processing building blocks into a vast array of sonifications. The challenge has shifted from processing power to controllability, making gestural control the core concern. Both linear and nonlinear synthesis models can be driven by sensor‑based gestures.

Example I: tIRAtEIMAS

This installation employs several independent FM synthesis models that generate rich and unexpected textures controlled by hand gestures. By repetition, the performer learns the relationship between gesture and sonic result, allowing intuitive exploration. The setup includes a laptop with webcam, microphone, and a Pure Data patch. The patch uses live video to map eleven spatial regions in front of the performer, each controlling parameters of an FM synthesizer — eleven synthesizers in total. The patch also processes video output, which is projected during the performance, displaying the captured gestures with dynamic visual effects and a moving circle that tracks the performer's hand. The circle moves across the projected regions to indicate which synthesizer is active. The performer controls the patch without touching anything, using two methods: moving hands through the virtual plane of mapped regions, and using percussive onset input. Clapping or snapping captured by the microphone causes all eleven synthesizers to re‑parametrize in response.

Example II: Patch em Preto e Branco

Created by the author, this work arranges a virtual octagon containing nine rectangles — eight on the sides and one in the center. Each rectangle represents a deterministic additive synthesizer producing tonal sounds; each connection represents a stochastic subtractive synthesizer generating percussive sounds. An external sound captured by the laptop microphone has its pitch extracted and used as a parameter for the synthesis models. The installation was initially developed without interactivity, then used as an interactive music performance with the author playing flute alongside the algorithm.

Toward multimedia interactions

Ubiquitous music concepts extend beyond sound alone to create multimedia interactive performances where sonic expression merges with visual elements. The performances described here explore new forms of sonic interaction and artistic notation. In these works, the algorithm written as a Pure Data patch serves as a musical score — the computer model itself — while the artist's improvisation provides the expressive performance.

Fornari, José (2010). “Percepção, Cognição e Afeto Musical”. Capítulo do livro: “Criação Musical e Tecnologias: Teoria e Prática Interdisciplinar”. Editora da Anppom.

Fowler, Charles B. (1967), “The Museum of Music: A History of Mechanical Instruments”, Music Educators Journal (The National Association for Music Education) 54 (2): 45–49.

Gelfand, Stanley A. (2004). “Hearing: An Introduction to Psychological and Physiological Acoustics”. Fourth edition. United States of America: Marcel Dekker.

Marks II, Robert J. (1991). “Introduction to Shannon Sampling and Interpolation Theory”, Ed. Springer-Verlag.

Oppenheim, Alan, et al. (1999). “Discrete-Time Signal Processing”, Prentice Hall.

Pimenta, Marcelo, et al. (2009). “Ubiquitous Music: Concepts and Metaphors”. In: Simpósio Brasileiro de Computação Musical, 12., 2009, Recife. Anais. São Paulo: SBC, 139–150.