Improv: Bringing animation and music together in real time

Improv is a system designed to create real-time, behavior-based animated characters. While other efforts have looked at building autonomous agents that work across networks, they rarely focus on what the author needs. To build rich interactive worlds filled with believable animated actors, creators require the right set of tools. Improv delivers exactly that—tools for building characters that can interact with users and each other in real time, displaying personalities and moods that match the author’s vision.

The character animation side of Improv is split into two main parts. The first is an Animation Engine, which relies on procedural techniques to let authors craft layered, continuous, and non-repetitive motions, along with smooth transitions between them. The second is a Behavior Engine, which gives authors the ability to write sophisticated rules governing how actors communicate, change their behavior, and make decisions. Together, these two engines form an integrated authoring environment for the “minds” and “bodies” of interactive characters.

Recent progress has brought audio and musical features into the Improv system under the name Improv Musique. This work concentrates on two main areas: Interactive Virtual Musicians and Dancers, and Audio Support for Virtual Environments. Its capabilities include actor speech, voice recognition, controllable music, environmental sounds, and input from external devices like video cameras.

Authoring animation within Improv

As an authoring platform, Improv needs to give creative professionals the right tools to build the many pieces of an interactive application. Those tools must feel natural to use, allow for rich and engaging content, and produce behavior at runtime that stays true to the author’s intent. Animated actors should be capable of handling a broad range of user interactions in ways that feel suitable and avoid repetition. This becomes more complex in applications featuring several characters, since all of them must cooperate while remaining faithful to what the author wants them to do. The author ultimately needs control over both the choices an actor makes and how that actor moves.

Architecture

Improv uses a model built around two engines. The Animation Engine works by using descriptions of atomic animated actions—such as “Walk” or “Wave”—to manipulate 3D models. The Behavior Engine handles higher-level capabilities, like deciding to go to a store or start a conversation, and it also determines which animations to trigger. Beyond that, the Behavior Engine maintains an internal representation of the actor, tracking moods, goals, and personality traits. In this design, the Animation Engine represents the actor’s “body” while the Behavior Engine functions as its “mind.”

Animation Engine

The Animation Engine provides tools for generating and mixing realistic gestures and motions on the fly. Actors can move smoothly and naturally from one animation to another in real time. Multiple motions can be layered and blended together to convey different moods and personalities.

An author defines an action simply as a list of joint rotations combined with a range and a time-varying expression for each joint. Most actions are built by modifying a few of these over time using combinations of sine, cosine, and coherent noise (controlled randomness). For instance, sine and cosine signals may be combined within actions to create elliptical rotations. Using coherent noise for limb movements helps authors give the impression of natural motion without needing complex simulation models.

Authors can also import keyframed animation from commercial systems like Alias or SoftImage. Improv internally converts these into actions that specify time-varying values for joint rotations or deformations. From the viewpoint of the rest of the system, imported actions look just like any other action.

Behavior Engine

Because the user is an unpredictable part of the runtime environment, the authors using Improv cannot rely on deterministic scenarios. Every user response presents the actor with choices about what to do next. Since this variability is built in, the audience’s perception of an actor’s personality and mood largely depends on the probabilities the actor uses to pick between different options.

The Behavior Engine supplies several authoring tools for directing an actor’s choices. The most fundamental tool is a simple parallel scripting system where individual scripts—like actions—are grouped into mutually exclusive behaviour sets. However, unlike actions, when a script inside a group is selected, any other script running in that same group stops immediately. In each group, exactly one script runs at any given moment. Typically, an actor will run several scripts simultaneously across different groups. In each script, the most common operation is to pick one item from a list. Those items are usually other scripts or actions for the actor (or sometimes for another actor) to perform.

Authors must anticipate unexpected user responses. That is why it is not enough to provide tools for scripting long linear pathways. Instead, authors must be able to build layers of choices—ranging from broad, slowly evolving plans down to local, rapidly shifting activities—that take into account the constantly updating state of the actor’s surroundings and the unscripted behavior of the human participant.

Individual scripts

A script is structured as a sequence of clauses. At runtime, the system runs these clauses one after another for the chosen script within each group. On each update cycle, either the same clause as the previous cycle runs, or the system moves ahead to the next clause. Authors are given tools to “hold” clauses in response to events or timeouts.

The simplest thing an author can do inside a script clause is to trigger a specific action or script. This is useful when a particular sequence of activities is required. Beyond commands that explicitly trigger actions and scripts, Improv also offers tools for generating the non-deterministic behavior that interactive, non-linear applications need.

With Improv, authors can create decision rules that use information about an actor and its environment. Those rules then determine tendencies toward certain choices over others. The author decides which information is relevant and how it influences the weight assigned to each option. As that information changes, the actor’s preferences shift as well.

Coordinating multiple actors

An author can coordinate a group of actors as if they were a single entity. This is done by allowing actors to trigger each other’s scripts and actions as freely as they trigger their own. If one actor tells a joke, the author may want others to respond—favorably or not—to the punch line. When the joke teller cues the other actors to respond, proper timing is preserved, even if individual actors decide among themselves how exactly to react. In this way, an actor can appear to always know what the others are doing, responding quickly and appropriately in ways that fulfill the author’s goals.

User interaction and multilevel control of actor state

One distinctive feature of Improv is letting the user interact with the system at multiple levels. This means the author can give users the right kind of control for each specific situation. If a user needs very fine-grained control over an actor’s motor skills, the author can grant direct access at the action level. Alternatively, if the user is carrying on a conversation, the author may let the user choose a set of gestures for the actor and then let the actor decide on the specific gestures from moment to moment.

At an even higher level, the author might want the user to direct large groups—like a theatre company or an army. In that case, the user could give directions to the whole group, and each actor would handle the execution independently. Because any layer of an actor’s behavior can be exposed to the user, the author can shift the level of control at any point during the application.

Improv Musique

Over the last year, music, audio, and user-input features have been added to Improv. As a result, the system now offers actor speech with lip syncing, voice recognition, ambient background sound and effects, controllable music sequence playback, singing synthesis, and input from external devices and video.

Implementation

Improv Musique is built out of several components. Features operate on both Macintosh and UNIX computers that communicate through a local area network. Much of the audio system is built using Opcode™ MAX, a visual programming environment for the Macintosh that is used mainly for MIDI applications. In the Musique system, MAX receives and filters data from input devices, plays and processes MIDI and digital audio files, interfaces with the Macintosh voice recognition facilities, and handles network communication with actors.

MAX programs (called “patches”) serve as the central points of input, output, and control for all Musique features. Custom external objects for MAX, written in C, handle digital audio playback, MIDI file playback, voice recognition, and video input. UNIX programs manage network message routing and also perform sound file analysis for lip syncing.

Network communication plays a central role in the integrated audio and visual world of Improv. Messages sent via telnet between actors and the Musique components allow actors to request services like speech playback. They also inform the Musique system about changes in the environment—for example, where the user is located in a virtual space—and receive information about audio and music, such as song tempo or sound volume. Authors can define whatever message types and formats are required to support the various Musique features and direct them to different actors.

Custom MAX objects

Several custom external objects for MAX are central to the Musique system: SeqPlayer, AiffPlayer, Reco, SoundMap, and VideoIn.

SeqPlayer is a Standard MIDI File (SMF) player. It allows a MIDI sequence to be played back beat by beat, making it possible to conduct and control the tempo of a computer band or orchestra roughly the way a human conductor works with a real ensemble. For instance, each beat received by SeqPlayer can output one quarter note’s worth of music from a score file. SeqPlayer also provides bar and beat reporting, jumping, and looping capabilities. MIDI data from SeqPlayer can be processed in many ways before it is sent out, so a musical score can be played back and altered in real time under the control of either a human user or an Improv actor.

AiffPlayer plays Audio Interchange Format File (AIFF) digital audio files. Files play directly from disk, which means sounds of any length can be used. Multiple files can play simultaneously, each with independent volume and panning control. In Improv, AiffPlayer is used for speech file playback, ambient audio, and sound effects.

Reco provides an interface to the Macintosh speaker-independent speech recognition system. It allows the author to define groups of words and phrases that are treated as valid responses. When a user speaks, Reco reports which word or phrase was recognized, or sends a special message for an unrecognized utterance. The system supports filtering to remove optional words and hesitations like “um” and “er.”

SoundMap manages ambient sounds and sound effects for a virtual environment. It stores each sound’s position and attributes within the virtual space and controls playback based on the user’s location. Both MIDI and digital audio playback are supported.

VideoIn captures live input from a video source, typically a camera. It detects motion by comparing each frame against the previous one, reporting motion in zones that the user defines.

Actor speech

Actors can deliver a variety of pre-recorded phrases, which they select in much the same way they choose physical actions. The spoken phrases are recorded and stored as AIFF files on the Macintosh. When an actor picks a phrase, it sends a message to trigger playback via AiffPlayer. The system also tracks the user’s position in the virtual space relative to the speaking actor and adjusts volume and panning accordingly.

To support lip syncing, sound files go through analysis using a Linear Predictive Coding (LPC) program called “lpanal” (from MIT’s Csound package). The output from “lpanal” is further processed to determine where vowels and consonants appear in the audio, or at least when the mouth is open, based on loudness information. This second stage creates and stores an animation code file that maps timing data to facial movements. When the actor speaks a given phrase, the animation code executes simultaneously.

Voice recognition

Actors initialize the Reco object by sending a list of response sets to MAX. Each response set lists expected answers from the user for a particular situation. For instance, when asked a yes-no question, the user might reply “yes,” “no,” “maybe,” or “I don’t know.” These sets are defined with a flexible syntax, making it possible to specify a large number of responses concisely.

When the actor expects a response, it sends a message to activate a specific set. After the user speaks, the Macintosh analyzes the utterance and tries to match it against the active set. Reco then sends a message back to the actor indicating which phrase was recognized—or “unreco” if nothing matched.

Environmental sound

Using a visual interface patch in MAX, a sound designer can place sounds across a map of a virtual environment from an overhead perspective. By adjusting parameters such as range, volume, and panning curves, the designer can tune each sound’s spatial behavior—for example, making it sound more directional or more ambient. SoundMap stores this data and controls playback of the environment accordingly.

During operation, Improv continuously reports the user’s location and orientation in the virtual space. Based on that information, SoundMap turns individual sounds on and off and adjusts their volume and panning to match the user’s position and direction.

User input

Through MAX, the system can accept input from video cameras, serial devices (like magnetic trackers), or MIDI instruments (including keyboards and drum kits). Incoming data is analyzed and filtered within MAX and then used to control music and sound, or to supply information to actors.

For example, motion detection data from VideoIn can tell an actor how “lively” the user is being. A performance on a MIDI instrument can be analyzed for volume and timing cues to guide an actor. A magnetic tracker can help turn a user into a conductor—driving musical performances, as shown below.

Improv in practice

Botanica Virtual

“Walking through the fog, you cross over a bridge and into the bayou. At a fork in the road, you meet Papa Legba, a huddled old man playing the harmonica. He looks up at you and says, ‘At the crossroads, anything can happen.’”

At SIGGRAPH ’96, the team presented The Botanica Virtual, an immersive VR experience where a participant steps into a bayou swamp and meets several characters inspired by Voodoo archetypes. The Improv-driven characters respond to the participant’s actions and also engage in conversation through speech recognition and synthesis.

Dancing Gregor

“Inside the Juke Joint, we hear the sounds of the bar clientele along with music blaring from the jukebox. The bartender turns down the jukebox and the virtual band begins their set. Gregor, a virtual actor, hears the band break into a blues shuffle and begins to dance.”

The virtual band itself is an audio entity directed by a user playing an electronic drum. The user acts as the conductor, setting the band’s tempo and volume. Drum beats stream into MAX, driving the SeqPlayer object, which in turn outputs the MIDI score file. The output is processed so that the velocity of the drum hits controls the band’s loudness.

Gregor “listens” to the band through messages sent via MAX’s telnet object indicating beat, tempo, and volume. He aligns his steps with the beat and chooses his dance style according to tempo and volume. For instance, when the music is slow and loud, Gregor will lean toward dancing in a “limbo” style.

Aria

“The conductor steps up to the podium, picks up the baton and commands the virtual orchestra to play. Gigio, a virtual opera singer, nods to the conductor and surveys the audience. He begins to sing and act out the Aria in the tradition of great tenors past.”

In the Aria installation—also shown at SIGGRAPH ’96—a user conducts both the opera singer and the orchestra using an electronic baton. The baton houses a magnetic tracker to sense its position. Data from the sensor is read into MAX, where it is used to derive the beat from each downward stroke. The stroke amplitude and horizontal position are also captured. Beats from the baton drive the SeqPlayer object, controlling tempo. The amplitude of each baton stroke controls the volume of the orchestra and the vocals, while the baton’s horizontal position controls the vowel sung by the vocalist—who shifts between “a, e, i, o, and u.”

Gigio’s vocal part is synthesized in real time on an SGI machine running MIT’s Csound. Within Csound, the FOF algorithm creates a vocal synthesis that sounds like a male tenor singing vowels.

The MIDI score file output from SeqPlayer contains both the orchestra and vocal parts. The orchestral portion is sent to a MIDI synthesizer, while the vocal data is translated into a Csound command and sent by MAX to the machine running Csound.

Gigio receives messages about the notes and his place in the score. He synchronizes his facial expressions with the vowel being sung and selects his actions based on where he is in the music. For example, he knows exactly when to make his entrance, where the musical climax sits, and when to take his bow.

Credits

Improv is a collaborative project between the New York University Media Research Laboratory and Center for Advanced Technology, together with the Laboratório de Sistemas Integravéis at the University of São Paulo in Brazil. Special thanks go to Ruggero Ruscioni for his guidance on the Aria project. For more information, visit http://cat.nyu.edu/projects.

References

Badler, N., C. Phillips and B. Webber, “Simulating Humans: Computer Graphics, Animation, and Control,” Oxford University Press, 1993.

Blumberg, B. M. and T. A. Galyean, “Multi-Level Direction of Autonomous Creatures for Real-Time Virtual Environments,” Proceedings of SIGGRAPH 95, in Computer Graphics Proceedings, Annual Conference Series, pp. 47–54, ACM SIGGRAPH, New York, 1995.

Bruderlin, A. and T. W. Calvert, “Goal-Directed, Dynamic Animation of Human Walking,” Proceedings of SIGGRAPH 89, in Computer Graphics, 23 (4), pp. 233–242, ACM SIGGRAPH, New York, 1989.

Bruderlin, A. and L. Williams, “Motion Signal Processing,” Proceedings of SIGGRAPH 95, in Computer Graphics Proceedings, Annual Conference Series, pp. 97–104, ACM SIGGRAPH, New York, 1995.

Dannenberg, R.B., “Real-Time Scheduling and Computer Accompaniment,” in Current Directions in Computer Music Research, eds. M.V. Mathews and J.R. Pierce, MIT Press, Cambridge, MA, 1989.

Dannenberg, R.B., “Nitely News,” in program notes from The Second Artificial Intelligence Based Arts Exhibition, Joseph Bates, curator, published in AAAI-9, Seattle, WA, 1994.

Johnson, M., “WavesWorld: A Testbed for Three Dimensional Semi-Autonomous Animated Characters,” PhD Thesis, MIT, 1994.

Matthews, M. V., “The Conductor Program and Mechanical Baton,” in Current Directions in Computer Music Research, eds. M.V. Mathews and J.R. Pierce, MIT Press, Cambridge, MA, 1988.

Minsky, M., Society of Mind, MIT Press, 1986.

Perlin, K. and A. Goldberg, “Improv: A System for Scripting Interactive Actors in Virtual Worlds,” Proceedings of SIGGRAPH 96, in Computer Graphics Proceedings, Annual Conference Series, pp. 205–216, ACM SIGGRAPH, New York, 1996.

Perlin, K., “Real Time Responsive Animation with Personality,” in IEEE Transactions on Visualization and Computer Graphics, 1(1), IEEE, New York, 1995.

Perlin, K., “An Image Synthesizer,” Proceedings of SIGGRAPH 85, in Computer Graphics, 19(3), pp. 287–293, ACM SIGGRAPH, New York, 1985.

Rowe, R., Interactive Music Systems, MIT Press, Cambridge, MA, 1992.

Strassmann, S., “Desktop Theater: Automatic Generation of Expressive Animation,” PhD thesis, MIT Media Lab, June 1991 (online at http://www.strassmann.org/straz/phdthesis.pdf).

Sundberg, J., L. Nord and R. Carlson, eds., Music, Language, Speech, and Brain, MacMillan Press, Mount Vernon, NY, 1991.

Unuma, M., K. Anjyo and R. Takeuchi, “Fourier Principles for Emotion-based Human Figure Animation,” Proceedings of SIGGRAPH 95, in Computer Graphics Proceedings, Annual Conference Series, pp. 91–96, ACM SIGGRAPH, New York, 1995.