Internet music: dream or reality?

The recent explosive growth of the Internet, together with the fact that powerful personal computers are now commonplace, has increased interest in network music. Among all the applications that handle time-dependent data, those involving music impose the most demanding requirements. Traditional network and operating systems do not provide the support necessary at the required quality of service.

This article examines a number of issues concerning music on the Internet and proposes solutions for each of them. It also considers a variety of musical activities and argues that the Internet can be a viable environment even when current technology for transport and operating systems are factored in.

Introduction

In 1995, a group at the University of Illinois developed Vosai — an extension to the Mosaic Web browser that transmits compressed audio over the Internet in real time using adaptive delivery strategies. This pioneering work demonstrated that it was feasible to send complex musical data over the Net. However, it also showed that the complete unpredictability of Internet behavior poses very difficult obstacles for real-time music applications.

At present, the Internet is characterized by long delays in message delivery and by fluctuations in those delays, known as latency and jitter, respectively. These difficulties are amplified when data carries music. Music places far more stringent demands on network performance than do other forms of communication. Today, Internet music is almost entirely one-way transmission with very limited interactivity. This paper examines whether live music performance over the Internet is viable, addressing both artistic and technical concerns.

Musical challenges

Music is a temporal art, and each culture and period has developed protocols to deal with musical time. These protocols have been shaped by the development of instruments, the acoustics of performance spaces, the performer's ability to control timing, and the evolution of the music itself. In the Middle Ages, for example, the acoustic resonance of cathedrals imposed tempo constraints on Gregorian chant, requiring long notes to allow text comprehension. The Baroque and Classical periods offered more favorable acoustics, better instrument technologies, and more sophisticated notation systems, enabling very refined temporal control.

In the last decade, the Internet has become a new space for musical performance, making it necessary to develop new tools and protocols that handle the peculiarities of this environment. These developments will progress only through collaborative work from both the music and science communities.

Technical challenges

Audio data is voluminous and highly dependent on real-time constraints. When a server using a 256 kbps codec connects overseas, the network must consistently sustain at least that average data rate. Equally important—if the data is being played in real time on the client side, jitter must be very small. The stream must arrive at a constant rate without interruptions measured even in fractions of a second.

Network issues can be minimized at the client end by buffering data and delaying playback start—a technique adopted by all streaming-software. While effective for some applications, that approach is unacceptable when a high degree of interactivity is needed. Music performance is an extreme case of latency sensitivity. In ordinary conferencing or collaborative work, delays of one or two seconds are tolerable. But when two musicians play together, a delay of mere hundredths of a second is noticeable, and in some cases a tenth of a second proved "fatal." At the Laboratório de Linguagens Sonoras at PUC/SP, an experiment sent sounds a MAX patch over a network to delay the output of one musician playing a duet. After practice, the musicians compensated for delays up to approximately 80 ms and could still play together. Delays larger than that became intolerable.

The current Internet and operating-system infrastructures are not designed for music interactivity. Yet many well-known techniques developed over several decades may help make the vision of Internet music come true—a vision that this project believes will be realized in the first part of the next millennium.

Related work

During the last decade, much pioneering work has been done on streaming audio over the Internet, but the scenario is rapidly changing due to inexpensive computers now capable of offering serious musical quality. Casey and others have developed design tools and synthesis engines that can model sound, drastically reducing bandwidth. Instead of transmitting raw waveform data (typically measured in hundreds of kilobits or megabits per second), codes transmit numeric waveform models that are synthetically generated at the destination — merely hundreds of bits per second. While solving throughput issues, this approach still leaves jitter and uncertainty poorly addressed—especially since intensive PC-based synthesis may impose extra processing latency.

The Global Visual Music Project is building an architecture for real-time, networked, multi-site visual-music performance. Their environment uses PD (a programming language for audio and graphics) and the GEM extensions. The group originally produced Lemma 2 for percussion and trombone, premiered in a jazz club at the International Music Conference in 1995, the first of an intended series of jam sessions. To minimize jitter and ensure reliability and throughput, all communication still happened over private ISDN connections. That approach makes everyday use cost-prohibitive for most musicians. A more generic solution remains needed.

New groups bring important individual contributions, as there are several intersecting problems to be solved by future research efforts. We discuss each of these issues below.

The infrastructure problems

Most music tasks produce and consume large amounts of data, especially when the stream must be processed on at least two machines and transmitted over a network. Streaming-media software (video on demand or on-line audio) can often tolerate delays of seconds; waiting three seconds for a two-hour movie is barely noticeable. For voice communication or real-time collaboration tools, up to one second delay is acceptable. If the delay exceeds that, conversation flows poorly.

Music-intense applications like distance colaboration differ: they depend on predictable latency, minimal jitter, and consistent throughput. Developers verified that current network and operating system infrastructures were not tailored to these requirements. Protocols underpinning the Internet were designed through the 1980s mainly for data transfer — they favor textual-data exchange and complete file reliability, not large time-dependent data, on-time arrival, and diverse end systems. As earlier noted, music the tightest live tolerances.

Data compression

To move high quality audio, this first necessary resource addresses need for large files systems other professional measures: generic adaptive coding required able further drastically more exact forms require heavy handling — huge raw storage needed at large. Luckily this brings industrial-level concerns only limited to audio — generic the two stereo Tracks sample 16‑bit 44,100 times/second — in total 1,411.2 kbps. At present, many audio engineers now desire better high-rate reproduction; today >– than even single existing PCM stereograms full. Yet loading roughly 635 MB/hour even forces one hard limit.

First means to handle that: data‑compression algorithms. Only audio‑specific technologies reached improved: today common “raw” bzip style slower level.

• Global Systems for Mobile (GSM) — Coding variant at 11.3 kbps, of low H+Fi not used over single code conditions.
• Later advanced via worldwide expert proven.

Alternative representation

Networking advances

Operating system side problems

Operating systems have existing many fully market over one more. For complete ones [improvements over] only good . Considering timely special lines years way gave...

The internet coverage of the NASA JPL Mars Pathfinder mission showed the power of large-scale online event delivery [Golombek et al., 1997]. A network of reflectors spread across five continents delivered live feeds from the NASA Jet Propulsion Laboratory to over one million clients in dozens of countries.

This work also supports broadcasting both live audio and a MIDI-like stream. The only limitations are the available network bandwidth at the client and the reflector sites. Since the system runs on top of TCP/IP and uses the existing internet infrastructure, no quality-of-service guarantees exist regarding delays. We minimize this by buffering some data in the client before displaying it, which introduces additional delays ranging from tenths of a second to a couple of seconds.

We have shown that the internet can already be used for large-scale audio broadcasts and even for conferencing. Those delays are too large for live musical performances. But since they are acceptable for discussion, they would also work for master classes, workshops, and distance music learning.

4.2 Rehearsals and Concerts

Once the next generation of internet tools are deployed and gigabit networks that support quality of service are common, it will be possible to carry out musical performances. This will first be possible on dedicated links and later over the public internet.

As with internet tools, each room participating in a music session will have a processor. This will be responsible for: (1) capturing the music data generated in that room and sending it to participants, listeners, or a reflector; and (2) receiving music data from other sites and playing it locally. Such a processor would support both rehearsals and concerts. A conventional group, like a string quartet, could “meet” several times a week to rehearse even if members live in different countries. Latency-management will ensure the network produces human-tolerable delays. High-quality audio and large screens will provide musicians with a rehearsal environment close to the real one. If needed, a limited number of traditional rehearsals could take place in the days before a concert. However, even the concert could take the form of an event with presence of musicians and audience, all located in different places. In this way, it will become much easier for musicians from different locations to interact with each other and exchange experiences and knowledge.

4.3 Remote Supercomputer Resources

Supercomputers bring an opportunity to experiment with tomorrow’s computing resources, since a supercomputer today has the power of tens of personal computers five to ten years in the future. Musical uses of supercomputers include sound synthesis [Kriese and Tipǫ̧i, 1992] and computationally intensive algorithmic composition tools such as MaxAnnealing [Iazzetta and Kon, 1995]. Supercomputers are, by definition, rare resources with very limited mobility, accessed by very few people. They are shared among many researchers. It is very unlikely that a supercomputer could be taken to a concert hall to perform music locally. However, proper network support would enable bringing the power of a supercomputer into a concert hall. Musicians could connect with local devices that would forward messages to where the supercomputer was available. Upon completing the processing (synthesis or algorithmic composition of parts of a piece), the result would return to the local processor, which would turn the data into music.

4.4 Composition of Internet Music

While final technical solutions are not yet found, we can think of several issues that contemporary composers should keep in mind when composing music for the internet. As stated in section 1.1, the developers of internet music systems must take into account the peculiarities of the specific types of music for which they are intended. What is extremely relevant in one context may not be pertinent in another. At least three factors will guide the design of internet music systems:

1. the existence of a regular pulse: pieces built on regular rhythmic structures (pulse, beats, bars) are much more sensitive to delays and jitter. Certain types of improvisational performances would be less affected by latency shifts.
2. tempo: tolerance to timing deviations is inversely proportional to how fast a piece is played and to the rate at which events occur. Thus, a 100 ms delay may be irrelevant for a slow piece but very disturbing for a very fast one.
3. compositional style: in free improvisation, like some of George Lewis and John Cage’s works, the performer can build musical structures as he or she processes the data from the internet.

On the other hand, in pre-composed music, musicians on both sides of an internet connection must listen to the same audio at the same time, or with very short delay. It is known that a trained musician can compensate for deviations—as when a timpanist plays slightly ahead of the string section in a symphony orchestra to offset the distance sound must travel from the back of the stage to the audience. But the delay must be very small and constant. Since there will always be a delay in audio carried over the internet, the issue to be resolved is this: for a specific kind of music, how small must the delay be to provide a satisfactory interaction between distant players?

By keeping in mind the specific characteristics of this medium, an internet music composer can compensate for the current technical limitations. At the early stage of internet development, good knowledge about technical limitations is fundamental for composing internet music. As technology advances and the challenges described in this paper are addressed and solved, it will become increasingly easier for composers to utilize this new medium.

Conclusions

Within the music community there is increasing interest in the internet in general and in internet music in particular. In spite of very few available tools—comparable in number or quality with tools for musical performance, composition, or education—we have identified the main technical problems, but they can all be solved. Recent progress in systems, quality of service for multimedia, networking, and compression processes ensures that a set of mechanisms can be combined to produce an ideal environment for music over the internet. Composers can expect to access such an environment in the first decade of the next millennium. But one must not wait for them passively, as important aesthetic questions surround the use of the internet in musical practice. These questions are to be posed by the artistic community and answered by researchers in art, science, and technology.