Human-Computer Music Performance: A Vision for Live Interactive Music

Human Computer Music Performance (HCMP) describes the emerging practice of creating computer music systems that can perform live alongside human musicians. The ultimate goal of HCMP is to develop an autonomous artificial performer capable of filling the role of a human player, particularly in popular music settings. Achieving this ambition will require breakthroughs in automated music listening and understanding, new music representations, synchronization techniques, real-time human-computer communication, music generation, sound synthesis, and sound diffusion. As such, HCMP serves as an ideal framework to motivate and integrate advanced music research. Beyond its technical challenges, the field has genuine potential to benefit millions of practicing musicians, from amateurs to professionals. This article presents the vision of HCMP, the core problems that must be solved, and some recent progress toward making this vision a reality.

Most existing work in interactive music systems falls into several categories. The most extensive efforts have involved experimental music, where few traditions or constraints exist. This freedom has allowed creators to focus on interactivity, gestural control, algorithmic composition, and new synthesis techniques, all of which have progressed considerably over the past several decades. Another major area is score following and computer accompaniment. These systems assume the music is predetermined by the composer, making synchronization the central interactive task. Such accompaniment systems typically lack any representation of music theory, structure, or form; they simply play predetermined notes or sounds. Computer accompaniment has been relatively successful because all effort goes into two clear problems: real-time alignment of a performance to a score, and musically adjusting playback to synchronize with another player.

However, real-time alignment is not always useful or applicable. In popular music, detailed scores often do not exist. Lead sheets may notate only chords. Even when a melody is fully notated, performers expect freedom in interpreting rhythms. Aligning to a loosely interpreted score provides very limited information about musical tempo and location. In such cases, a very different approach is needed to synchronize musicians.

"Popular music" is defined here as music that generally maintains a steady tempo, shows clear structures of sections, phrases, harmony, melody, and meter, and usually includes improvisation. Synchronization in this style tends to rely on beats and measures rather than on a score. The written music is likely imprecise, and even when it specifies every note, musicians freely alter chord voicings, strumming patterns, and syncopations. Because it is difficult to synchronize machines to humans, it is common for people to synchronize to machines or to fixed media playback, as occurs in karaoke, street musicians playing with backing tracks, or practicing with Music Minus recordings.

Currently, HCMP remains more of a vision than a widespread practice. The vision calls for autonomous computer systems that can play the role of a skilled musician in live popular music performance. Figure 1 shows a high-level view of a complete HCMP system. Musicians are sensed not only through audio listening but also via a variety of sensors and interfaces. Musical decision making connects real-time performance to music representations that may be very specific or only rough sketches. Music is generated, synthesized, and diffused through loudspeakers into the performance space. Non-audio feedback in the form of displays, tactile feedback, and music notation is also produced to communicate with the human players.

Realizing the HCMP vision requires solving several interconnected problems:

• Synchronization in beat-based music remains largely unexplored from the perspective of implementing musically competent systems. While automatic beat tracking has received study, reliable and robust systems still do not exist. Studies of human synchronization in performance provide relevant findings.
• Communication among musicians takes on special importance in popular music, where the score may only be a sketch and performers generally have freedom to alter the form mid-performance.
• Musicians plan performances using very abstract representations of the music, such as "I'll solo and you come in on the bridge." These plans rely on shared conventions for describing structure and organizing performances.
• Musicians transform their representations of music into live sound. This may involve composing parts, such as writing a bass line from a chord progression. Once notes and phrases are decided, they must be performed musically, whether through synthesis or an acoustic robot. Finally, sounds must be diffused into the performance space to convey the impression of live playing rather than simple playback of a recording.

After a brief discussion of related work, each of these problems is examined in the sections that follow.

Related Work in Interactive Music Performance

As mentioned, most work on interactive computer music addresses experimental contemporary art music or Western art music, ignoring the realities of popular music with its steady tempo, limited use of scores, and demands for very precise timing. In the commercial sphere, Ableton Live offers a powerful interface for beat-based production and control, including some real-time time-stretching and tempo adjustment capabilities, but it is not designed to function as a virtual musician. Robertson and Plumbley's B-Keeper system extends Ableton Live with a real-time beat tracker and user interface, enabling a user to synchronize music to a live drummer. This system implements some HCMP components but does not address other challenges.

Conducting systems are closely related to HCMP, differing mainly in that they assume a dedicated person gives commands to the computer. Conducting provides one interesting way to synchronize computer performers with humans in live settings. Such systems have appeared in public performances, including some controversial presentations where electronics replaced an acoustic orchestra. In many conducting systems, all music is generated by the computer, so cues and synchronization are not critical. These systems have focused on adjusting tempo in real time and performing live time stretching of audio and video in classical music contexts.

Popular music performance with human musicians, on the other hand, raises issues of accurate synchronization and handling cuts, repeats, and other structural changes. Another concern is that conducting systems require a human conductor. Even when an ensemble has a conductor, it is common to add another person solely to operate the computer system. Any technical "solution" that demands the full-time effort of a person must be weighed against simply adding another human musician to the group. For a small ensemble playing popular music, an autonomous virtual musician seems to be a better solution than a conducting system.

Beat Tracking, Tempo, and Synchronization

The irony of working with "steady beat" music is that in live performance, the tempo is never truly steady. Variations of five to ten percent over time periods of about one minute are to be expected, and fairly sudden tempo changes are common. Despite this variability, "steady beat" music remains very steady much of the time, and the predictability of beat positions is essential for synchronization because all parts may be improvised and therefore unpredictable. HCMP systems must identify beats accurately and reliably. This could be achieved with a combination of automatic beat tracking software and gestural sensors such as foot pedals or accelerometers.

Automation is desirable but has two reliability problems. First, automatic beat trackers often make serious mistakes, losing track of the beat entirely. Although the literature sometimes implies beat tracking is largely solved, even state-of-the-art systems fail too frequently for use in live performance. Second, precision remains low. Even when beat trackers function, they may synchronize to audio features like snare drum hits that should be easy to detect, but in reality, music audio often contains events slightly offset from true beat times, leading to inaccuracies. It appears that humans and automatic beat trackers use very different processes to identify beats, and these processes are not always consistent.

Human input through tapping is more robust than beat tracking, but tapping can distract musicians. Performers distracted by performance tasks can tap with inadvertently large skews between tap times and true beat times. As with automatic beat tracking, precision problems affect tapping, especially foot tapping, which is otherwise one of the most reliable and least obtrusive ways of obtaining beat information from live players.

In addition to identifying beats, HCMP systems need to understand how beats relate to overall music structure. An important structural level is the measure or bar. These groups of four beats (typically) serve as points where chord changes and phrase beginnings occur. By determining measure boundaries, HCMP systems can better interpret cues from humans. These cues are often ambiguous at the beat level but refer to the nearest measure boundary. Robustly detecting measure boundaries in real time remains an interesting problem in music analysis.

Human-Computer Communication in Performance

While computer accompaniment systems and beat trackers focus on extracting information from music audio, much interaction in performance happens outside the audio channel. Musicians give visual signals, make eye contact, and use body gestures to communicate music information, including synchronization cues.

Not all these signals are explicit. For example, a trumpet player can usually tell when a fellow player is about to play by noticing when the trumpet is lifted and when the player takes a breath. The musicians may not even consciously be aware of this communication, but it exerts a strong influence that helps them take corrective action if synchronization drifts.

Similarly, HCMP systems need to develop simple and natural communication channels between computer and human musicians. The name HCMP consciously evokes HCl, Human-Computer Interaction, and HCMP research should build on HCl techniques. This area of investigation is rich with possibilities for sensors and computer processing. Exploring real-time interaction techniques for musicians and real-time displays that let computers give cues to humans represents fertile ground for research.

Multiple classes of cues have been identified in an effort to describe musical communications more systematically:

• A Static Score Position Cue communicates the current position in the score to correct synchronization problems.
• An Intention Cue indicates the direction of the performance when multiple options exist—for example, "this is the last time we will repeat this section."
• A Voicing/Arrangement Cue modifies the performance, telling a player to start playing, play louder, or play more notes.

A music-notation-based interface for HCMP has been developed, inspired by earlier work. This notation system is bidirectional: the computer displays its location by highlighting bar lines in the music, confirming to the human that it is in the right place. Conversely, the human can touch or click locations in the score to give cues or indicate where to begin in rehearsal.

Other modes of interaction are also possible. For example, a small touch sensor worn on the finger can give cues while playing an instrument. Hand gesture recognition using a Kinect sensor has explored various free hand cues. Foot tapping at half tempo (cut time) can indicate tempo, and four taps at full tempo can cue the beginning of some music. The possibilities seem endless.

Music Structure, Static Scores, and Dynamic Scores

Music structure and representation have received considerable attention, but HCMP raises some unique questions. Music often involves repetition and hierarchical structure. For instance, a popular song form can be described as "AABA," meaning the first section (eight measures) repeats, followed by a contrasting "B" part, and then the "A" part repeats. In practice, sections are usually varied from one instance to the next. An interesting challenge for automated music processing is to detect this structure from audio or symbolic scores.

Popular music scores exist in many forms, and coordinating these informal, semi-structured objects requires careful design of models, representations, and interfaces. The typical written score uses repeat signs and other constructs to keep notation compact. Often there are exceptions where music is played differently on each repetition. Exceptions may be handled with standard notations like first and second endings, but informal annotations like "Play 1x only" also occur. This static score parallels a static computer program.

When a static score is "executed" or unfolded to create a linear sequence of events, the reality that repetitions differ and occur at different times can be represented directly, albeit with greater redundancy. This dynamic score is analogous to the runtime execution sequence of a program. The mapping between static and dynamic scores is complicated by nondeterminism: a repeat may be marked "ad lib," meaning the number of repetitions is determined at performance time. Thus, the dynamic score cannot be fully determined in advance.

However, dynamic scores are not simply ephemeral traces of a performance. An audio recording corresponds to the dynamic score, as does a MIDI sequence. If a performance consists of humans reading static scores and an HCMP system playing from a MIDI file and an audio file, the static and dynamic representations must be reconciled.

In popular music, scores are often informal, and musicians often make informal plans that do not match the implied plan of the written score. For example, musicians might decide to play intro, verse, chorus, chorus, ending, even if the score shows a second verse. This practice of altering structure is widely accepted. Musicians might even decide to change the key of the second chorus. These informal plans are called "arrangements." An HCMP system must be able to access information from static and dynamic scores and from an arrangement to formulate a performance plan consistent with the intentions of human performers.

It must be easy for human musicians to create and communicate arrangements in seconds or even during performance, because these decisions are often made on the spot. Building simple tools to manage music where these fairly abstract concepts come into play presents another significant challenge.

Another option is to play back prerecorded audio with time‑stretching to align its tempo with live performers [12]. While this approach works well, preparing the audio is time‑consuming and assumes live musicians are available for the recording session. Synthesis methods for many instruments remain unconvincing, and even high‑quality approaches demand such precise control that satisfactory results are difficult to achieve. Progress continues in research systems (e.g., [18, 37]) and commercial systems (e.g., [31, 34, 44]), but HCMP would clearly benefit from further advances in synthesis. Ideally, one could render a score into a compelling performance complete with musical phrasing, stylistically appropriate timing and articulation, dynamics, and vibrato.

frequency‑dependent radiation patterns [3, 9], and our own work on convolution‑based stereo panning and placement [25]. The key challenge for HCMP is to integrate computer performance seamlessly with live acoustic instruments. While the techniques will depend on the context, there is a clear need for better sound reinforcement solutions with implications for all audio interfaces.

A common task for a popular music performer is to work from a “lead sheet” or “chord chart,” which provides the key, harmony, and structure (and sometimes the melody) but few other details. From this minimal information, a drummer can craft an appropriate rhythm that fits the song’s structure, a bass player can create a rhythmic and harmonic foundation, a keyboardist or guitarist can play chords according to the harmony, and other musicians can harmonize the melody or improvise a solo. Writing a new song is often seen as highly creative and difficult, yet creating a bass part or accompanying a singer on piano is routine for a working musician. In fact, most musicians can generate musical parts accurately and in real time while reading a lead sheet. For popular music, many performers are more comfortable improvising from a lead sheet than from conventionally notated music that specifies every note explicitly. Therefore, generating musical parts from a sketch like a lead sheet is a fundamental skill that an HCMP system should implement. This becomes especially important when human performers are not experienced on the instrument the system is intended to play and thus lack the skill or intuition to compose the part.

Programs such as Band‑in‑a‑Box [26] already accomplish the task of creating music from lead sheets, but they provide the user with limited control over the generation; instead, they offer a wide selection of predefined styles. It appears that ongoing research into machine learning, musical analogy, music similarity, and models of musical style could yield more flexible and controllable music generation.

One persistent problem with computer‑generated sound is diffusion into an acoustic space. The one‑dimensional audio signal must be transduced into three‑dimensional sound waves via loudspeakers, each of which imparts its own audible radiation characteristics onto the sound. As computer and digital audio hardware have dropped in price, there has been growing interest in using multiple audio channels to drive speaker arrays that improve and control sound diffusion in two or three dimensions. Examples include linear arrays for controlled wavefronts [4, 6], spherical arrays that emulate sound sources with

followed by the actual beat times, so those corrective taps were ignored. (We subsequently disabled the “spurious” tap rejection.)

Sound generation uses the pitch‑synchronous, overlap‑add (PSOLA) [45] approach to time stretching. Real‑time performance, continuous updates after every tap, latency compensation, and synchronization across 20 channels have required some innovative implementation choices. Most PSOLA systems stretch audio by a fixed factor over a defined time span; they insert or delete whole pitch periods, so the operation is effectively quantized to pitch periods rather than being truly continuous. Since we have 20 separate channels, each carrying a single instrument and its own set of pitch periods, predicting the exact duration of input audio that will be consumed is difficult. This quantization can cause stretched tracks to drift out of sync due to accumulating quantization errors.

To keep all tracks synchronized, we apply a feedback mechanism: the global control system adjusts the overall stretch factor so that the mean audio file position aligns with the live band. Then each track’s stretch factor is adjusted slightly to steer that track’s audio file position toward the global mean.

Sound diffusion uses an array of eight speaker systems positioned across the stage (see Figure 3). Each of the 20 input channels represents one close‑miked string instrument (violin, viola, or cello). Each instrument channel is routed to a single speaker. Instead of a homogenized orchestra sound distributed over many speakers, we get individual instrument sounds radiating from distinct locations and mixing naturally in the room, as they would in an acoustic ensemble.

The results were so convincing that one audio engineer asked which reverb plug‑in we had used—yet the recordings were completely dry, and all sense of “stereo” and reverberation came solely from the diffusion scheme.

Conclusions

From a scientific perspective, HCMP provides a framework to organize, motivate, and coordinate a range of interesting research efforts. It invites us to study music and music performance from many angles, developing new techniques for sensing musical beats, tempo, and structure, as well as new ways for musicians to communicate intentions—especially to computer‑based performers. The fundamental problems in HCMP are general problems of Music Understanding, with broad implications. As this book’s “Multimodal Music Processing” theme suggests, the work is truly multimodal, addressing discrete and symbolic scores, MIDI performance data, music audio, graphical displays, gesture sensors, and other forms of musical communication. The models and analysis methods developed here will find applications in related fields such as music information retrieval, music theory, and music cognition.

Digital sound synthesis has been studied for half a century, with connections to auditory perception, acoustics, digital signal processing, speech synthesis, and mathematics. HCMP challenges us to investigate new techniques for time‑stretching and pitch shifting, while also recognizing the importance of sound diffusion in perceived synthesis quality.

More broadly, HCMP tackles complex, real‑time cooperative tasks. New interfaces are

needed to coordinate computers and humans with minimal explicit or manual control. This could inform other human‑computer interaction scenarios such as driving and piloting, directing disaster relief, or complex mission control, where delegation and coordination are essential. The challenges in HCMP imply an integrated approach that combines machine learning with human factors studies to build reliable interfaces—advancements with many applications beyond music.

Music making is practiced in most American households. The National Association of Music Merchants reported that retail sales in the U.S. were about $8 billion in 2006, including over 5 million instruments sold—and that excludes music education, recordings, and live performance. Hence the potential societal impact of effective new music technology is enormous. HCMP seems an ideal application area where recent progress in music understanding and music information processing can benefit millions of people. Producing results that are both highly practical and useful is not merely a charitable endeavor for researchers; by connecting academic research to create a practice of popular music‑making, the research community stands to gain wider recognition and societal support.

Another motivation for HCMP research is purely artistic. One might criticize the project as an attempt to further simplify and automate an already formulaic popular music industry. Would it not be better, one could ask, to devote resources to experimental music and new art forms? My own hope is to leverage the conventions, opportunities, and sheer numbers in popular music to achieve broad adoption of HCMP. If successful, at least a few artists among millions will undoubtedly invent truly creative uses for HCMP technology that transcend current musical practice. In that sense, HCMP may become an important pathway through which technology shapes the future of music.