Juggling Gestures Analysis for Real-Time Music Control
Juggling gestures analysis for music control
Aymeric Willier, Catherine Marque — UTC UMR 6600, BP 20529, 60205 Compiègne Cedex, France — aymeric.willier@utc.fr
The goal of this research is to give jugglers the ability to control music through their gestures. This stems from a desire to repurpose refined gestures already mastered in one art form for musical purposes. To achieve this, the authors propose a gesture-capture system built around electromyographic signal analysis. Recordings of electromyograms from selected muscles — each playing a distinct part in the juggling motion — are made during a three-ball cascade. The authors then suggest processing those signals to drive musical events using parameters derived from the juggling gesture.
Introduction
Many juggling performances already combine juggling with music. Since juggling manifests primarily through gesture and object motion, it engages mostly our visual sense. When paired with music, it also reaches the ears — but this is typically done by working alongside musicians. A more compelling possibility is to give jugglers direct and expressive control over music using their own gestures: the same motions that move the objects could also guide musical sounds. To realize this, a gesture-capture system is needed that extracts meaningful information about juggling and juggling gestures, then conditions that information for real-time music control. The design must respect juggling constraints such as portability and unobtrusiveness. The solution proposed here is a system based on processing the electromyographic signal.
Gestural control of music
Computer-generated music offers ever-expanding capabilities, and the question of how to control those capabilities has become a rich area of research. Many hardware devices have appeared, each reflecting the personal needs of its builder. Institutions and corporations alike are investing in research on gestural control of sound synthesis and in the design of novel controllers.
Controller design generally splits into two main approaches: instrument-like controllers and alternative controllers. The first category mimics traditional instruments such as keyboards, guitars, drums, or wind instruments. These are the most common controllers; their primary advantage is that they do not demand a dedicated effort to learn. Alternative controllers, meanwhile, fall into two subcategories. The first pursues the development of entirely new musical instruments (e.g., Biomuse, The Hands). The second seeks to repurpose a mastered gesture from another artistic discipline. Dance has supplied many examples of this second approach, such as the digital dance system at DIEM, the Bodysynth, and various experiments cited on the Dance and Technology Zone website.
The approach taken here falls squarely into the latter category. By recycling the expert gestures of juggling — already learned through considerable practice — performers can generate music without needing an additional layer of technical training. This approach indirectly responds to J-C Risset’s observation that musical gesture is not spontaneous but rather a hard-won expert skill: reusing an already mastered gesture from another art form means no extra effort is required beyond what the performer already invested in learning that gesture for its original artistic purpose.
Gestural acquisition
Many motion-capture technologies exist — acoustic, magnetic, optical, mechanical, video, and others — and in the context of designing music controllers, many have been tested. D. Roger offers a thorough review of them, while A. Mulder has proposed a classification into three categories: Inside-In, Inside-Out, and Outside-In. In the Inside-In category, sensors attached to the body track phenomena occurring on or within the body itself. Inside-Out systems track phenomena created outside the body but affected by the body’s motion, with sensors on the body. Outside-In systems reverse that geometry — sensors placed in the environment track the body. Since the aim here is to allow free movement and to focus on juggling gestures rather than ball trajectories, the Inside-In category is the natural choice.
From an application standpoint, M. Wanderley drew a useful distinction between direct and indirect gesture acquisition for music. Most acquisition systems are direct: they measure the physical realization of a gesture in terms of distance, position, angles, and similar quantities. Far fewer systems are indirect, tracking instead the effect of a gesture — for example, analyzing the sound produced by an acoustic instrument to infer how it was played. Hybrid approaches, like T. Machover’s hyper-instruments, combine both. A further kind of indirect acquisition involves capturing the physiological signals that originate the gesture. The present work uses such an approach, extracting information about gesture from the electrical signal associated with muscle activity, known as the electromyogram (EMG). Its main advantages for this setting are that it supports non-invasive, non-obstructive, portable, and lightweight recording systems.
EMG-based gesture capture systems
B. Knapp, H.S. Lusted, and B. Putnam pioneered the use of EMG for music control in the early 1990s with the Biomuse. A. Tanaka continues to use this controller, defining gestures that provide nuanced control over sound synthesis and musical structure via Max software. T. Marrin has also applied EMG in studies of conducting gestures, combining it with motion tracking and additional physiological sensors to analyze expression in conducting. The method described here differs from these earlier efforts: whereas the Biomuse forges special-purpose gestures for fine-grained musical control, and the conductor’s jacket examines gestures made for musical reasons, the present approach captures existing, already-mastered juggling gestures — none of which are produced primarily for musical ends.
The approach unfolds in several stages. First, by observing the relevant gestures, the main muscles involved are identified. Next, the EMG of those muscles is recorded, confirming that the right muscles have been selected. Then, the temporal correspondences between juggling phases and muscle activity are analyzed. Finally, a processing scheme is proposed to turn the chosen EMG into signals suitable for music control. Juggling was chosen partly out of personal interest in this fast-evolving expressive discipline, but also because it conveniently limits — as a first attempt — the observation to upper limbs (although juggling is not confined to them), and because juggling comprises discrete actions whose combination forms a language of gestural expression.
Material and methods
Gesture description and muscle choice
As an initial case, the analysis focuses on the cascade, a basic juggling pattern performed with an odd number of balls. The three-ball cascade is usually the first pattern a juggler learns. It consists of alternating throws from each hand: during the three-ball pattern, one hand throws a ball just as the previous ball from the other hand reaches its highest point.
From the viewpoint of any single ball, the pattern alternates between flight and hand displacement, separated by catches and throws. From the viewpoint of a hand, the motion alternates between carrying the ball and moving without it, also framed by throws and catches.
Figure 1 — The three-ball Cascade
Figure 2 — Muscles observed with SEMG
Key moments of the pattern are the catches and throws, together with the simultaneous motion of balls and hands. A natural first comparison is between the timing of muscle activity and the timing of catches and throws. The first step, then, was to select which muscles to observe. The selection criteria were that a muscle should play a specific role in juggling and be accessible to surface electromyography. Observing the arm during juggling reveals a circumduction of the hand — a combination of external and internal rotation of the arm together with flexion and extension of the elbow. Wrist flexors and extensors are also engaged.
Muscles measurable with surface electromyography include the arm rotators — both internal (Pectoralis Major) and external (Teres Minor) — the elbow flexors and extensors (Biceps Brachii and Triceps Brachii), and the wrist flexors and extensors (Palmaris Longus and Extensor Carpi Radialis Brevis).
Initial observation of SEMG recorded during juggling showed that Biceps and Triceps Brachii are nearly always active. This likely results from the forearm’s supination oscillating around a maintained half-flexed position. Supination activates the BB; the half-flexed posture is maintained by co-activation of BB and TB. The oscillations around that position involve subtle variations in both muscles’ activity that do not show up clearly on the SEMG. Those oscillations are slow and could instead be driven by the deeper Brachialis, which is not accessible with surface electrodes. When the oscillations speed up, variations in BB activity become more evident.
By contrast, the wrist flexor and extensor showed more specific activity occurring in phase with the juggling rhythm. Since extracting that rhythm is one of the primary aims of this work, the study began with these wrist muscles.
A second round of observation revealed that the rotators around the shoulder also exhibit a cyclical activity linked to the juggling rhythm. These will be examined in a later stage. For the initial phase, however, the PL and RB were chosen in the expectation that their activity would correspond to the throws and catches — the two defining moments of the cascade pattern.
EMG Recording
To record surface EMG from the PL and RB of both arms during juggling, a four-channel high-frequency EMG acquisition system from John+Reilhofer was employed. Four pairs of Ag/AgCl electrodes were placed parallel to the muscle fibers, with a constant inter-electrode distance of 2 cm. Skin preparation followed standard methods to reduce resistance below 10 kΩ. A ground electrode was placed over the shoulder joint, away from the target muscles. The system uses a high-frequency radio link between a small conditioner/transmitter box worn by the juggler and a receiver, allowing complete freedom of movement. Leads and electrodes were attached with care to avoid obstructing the juggler. A 1000 Hz low-pass filter is built into the conditioner/transmitter. The four receiver outputs were stored on a magnetic tape recorder, later low-pass filtered at 500 Hz, digitized at 1000 Hz, and processed on a computer using Matlab.
Sensor
To pinpoint the timing of throws and catches precisely and to compare it against the EMG, two additional sensors were employed.
An accelerometer attached to the back of one hand — measuring acceleration perpendicular to the palm — helped detect the arrival of the ball. The resulting signal exhibits a discontinuity at the instant of the catch. However, the accelerometer did not allow a reliable determination of the throw: since hand acceleration is present just before the throw, the ball was expected to depart at the acceleration peak, but this could not be confirmed.
A Hall-effect sensor was therefore added, attached to the palm. One of the three juggling balls was modified to contain a magnet; when this ball enters the hand, the Hall-effect sensor registers the change in the magnetic field. The departure of the same ball corresponds to the end of that field variation. The amplitude of the signal can vary considerably depending on the magnet’s orientation at the moment of the catch. The sensor signal is rectified and low-pass filtered.
Both sensors were recorded alongside the SEMG of PL and RB in just one session, because using two extra sensors simultaneously can disturb the natural juggling motion. Most recordings with other jugglers used only the accelerometer plus EMG.
Description of the task
Jugglers were asked to perform these variations of the three-ball cascade:
- Three-ball cascade at usual speed — their comfortable, habitual pace for this pattern.
- Three-ball cascade at a higher speed — the fastest they could sustain for more than 30 seconds.
- Three-ball cascade at a lower speed — the slowest they could maintain while still keeping the three-ball rhythm.
As an optional addition, five-ball cascades and other patterns — including juggling two or one ball in the rhythm of three, and bouncing juggling — were also recorded (though the bouncing results are not addressed in this article).
Recordings were made with seven professional jugglers. A number of amateur jugglers participated as well, with experience ranging from one to twenty years and training habits from less than one hour per week to more than two hours daily. This diversity allows the study of how training affects the organization of muscular activity for a defined motor task.
Processing
IEMG — The first processing step computes the integrated electromyogram (IEMG), an estimate of the signal’s energy that is well suited to continuous control. The raw data are rectified and then smoothed over a 50 ms window.
Peak detection — Because the IEMG exhibits cyclical peaks — reflecting cyclically bursting SEMG activity — a dedicated peak-detection algorithm was developed. A fixed threshold proved unreliable, as peak amplitudes vary considerably over time. Instead, the threshold adapts dynamically: for each sample the threshold is a fixed percentage of the maximum value observed among the last N samples of the IEMG. N itself is updated whenever a peak is found, based on the number of samples since the previous peak. This method reliably detects IEMG peaks during the cascade, even when the juggling speed changes.
Results
Temporal observation of the signals
Muscle choice — The study began by examining the temporal behavior of the recorded signals to confirm the muscle selection and finalize the experimental protocol. Results from the four muscles — BB, TB, PL, and RB — during a three-ball cascade show that RB and TB are continuously active, while PL and RB produce clear cyclical bursts of activity that align with the juggling rhythm.
Figure 3 — SEMG during the three-ball cascade for PL, RB, BB, and TB.
Figure 4 — Effect of variation of speed on the SEMG recording.
The connection between this cyclical bursting and the juggling rhythm becomes even more apparent when the speed of juggling changes, as seen in Figure 4. The most immediately useful musical parameter to extract would be the juggling rhythm itself, enabling music to stay locked with the juggler’s timing.
Effect of training — Comparing the SEMG of the PL between a professional juggler and an amateur makes the effect of practice evident. In the professional’s case, bursts are shorter, more sharply defined, and interspersed with near-silent rest phases. This finding confirms that developing expertise in a gesture corresponds to an optimization of muscle activation patterns. The SEMG method highlights this optimization clearly. Consequently, only experienced jugglers were used for subsequent EMG processing.
Figure 5 — SEMG of PL recorded on untrained and trained juggler for the three-ball cascade.
Juggling phases identification. — The next step was to identify — on the temporal EMG plots — the two pivotal events in the cascade: throws and catches. This was done by cross-referencing the SEMG with signals from the accelerometer and Hall-effect sensors.
From recordings that simultaneously captured both sensor signals, the catch location could be read off the accelerometer signal at each discontinuity. Meanwhile, the Hall-effect sensor signal indicated when the magnet-containing ball is in the hand. Since only one of the three balls contains the magnet, this disturbance occurs once every three cycles in the accelerometer trace. The alignment of an acceleration discontinuity with the onset of the magnetic perturbation pinpoints the exact moment of the catch. The moment the ball leaves the hand corresponds to the end of the magnetic perturbation, which occurs shortly before the acceleration signal reaches its peak.
By identifying catch and throw events from the sensor discontinuity and the accelerometer maximum, it is possible to pinpoint juggling phases using simultaneous SEMG and accelerometer recordings. In Figure 7, empty-hand phases appear in white and ball-in-hand phases in gray. This allows muscle activity to be correlated with specific throws and catches.
3.2 SEMG processing
A third goal was to extract information from the SEMG useful for music control. We first computed the IEMG, shown for RD and PL of both hands in Figure 8. The IEMG provides a smooth signal that appears well-suited as a continuous controller for sound synthesis. A. Tanaka has noted that the IEMG is especially appropriate for controlling timbral evolution [24]. The temporal organization of activity is also reflected in the IEMG peaks, which correspond to the alternating activity of both arms.
Given that juggling rhythm is one parameter we aim to extract from the SEMG, we developed a peak-detection algorithm. The interval between successive peaks yields the period of juggling movements. The simultaneous peak detection on RB and PL for both arms (Figure 9) further clarifies the temporal organization of muscular activity during the cascade. For each arm, a PL burst of activity is followed by an RB burst. Based on our earlier observation that RB activity peaks at the moment of throw, peak detection on this muscle appears suitable for triggering musical events in time with the throw.
4. Perspectives and Conclusion
The initial portion of our work established which muscles are engaged during juggling. Very few studies have addressed the biomechanics of juggling gestures. We have now identified the muscles involved in both global movements (arm and hand displacement) and those specifically responsible for catching and throwing.
The current processing provides musical control using juggling rhythm, derived from peak detection, and continuous control, based on IEMG. The IEMG is already computed in real time. Our peak-detection algorithm will soon be implemented for real-time use. Localizing two consecutive throws allows us to estimate the hand-motion period for the three-ball cascade.
Future work splits into two directions. The first concerns recording muscles involved in arm rotation, hoping to extract SEMG information more directly related to throw dynamics. The second involves SEMG processing improvements, aiming to accurately detect catch time from the signal. Knowing both throw and catch timings will help determine when the hand is loaded with a ball and the ball’s flight time. C. Shannon proposed a direct relationship between flight time, holding time, and unloaded hand time for a given number of juggled objects [28].
Jugglers vary the rhythm of a pattern, which is perceptible in the hand-motion period, and can also change how they execute a pattern so that, for a fixed period, flight time varies proportionally with holding time [29]. This suggests using both hand-motion period and flight time as parameters for music control through juggling. We plan to refine our peak detection for other juggling patterns, especially arrhythmic ones performed by subjects during optional recording sessions. To further explore gesture dynamics, we will examine the spectral evolution of EMG over time using time-frequency analysis tools, as the time-frequency content of SEMG appears related to movement dynamics [30]. Analyzing specific frequency bands (to be determined) could enable real-time control of music parameters linked to gesture dynamics.