Evolutionary computing for expressive music performance
Of the many open challenges in computer music, few are as demanding as capturing the delicate knowledge that a musician draws on when transforming a written score into an emotionally compelling performance. Researchers at the Music Technology Group of Pompeu Fabra University in Barcelona, including Rafael Ramirez, Amaury Hazan, Jordi Marine, and Xavier Serra, have devised an approach that relies on evolutionary computation. Their method begins by extracting acoustic features from real jazz recordings, producing a symbolic representation of a performer's expressive choices. A specially designed evolutionary algorithm then works through that representation to learn how musicians add nuance to their timing and energy. The resulting model can guide a computer to perform new material with the same expressive quality found in a human saxophonist's playing.
Background and motives
Computer scientists have turned to evolutionary computation more and more frequently in recent years, especially for music-related tasks. Composition remains the most thoroughly explored territory—many systems now evolve melodies, rhythms, and chord progressions as part of creative workflows. Improvisation, too, draws heavily on evolutionary techniques: a genetic algorithm can simulate the way a performer generates fresh musical ideas in real time. Yet despite these advances, the problem of modeling expressive performance with evolutionary approaches has until now received very little attention.
In the past, studies of expressive playing relied on statistical analysis, mathematical models, or analysis-by-synthesis, and each of these routes placed the burden of theorizing directly on the researcher. A person had to devise a rule or an equation that might account for expressive deviations, then test it against actual recordings. The Barcelona group offers an alternative perspective: instead of asking a human theorist to come up with those principles, they instruct a computer to run a sequential-covering genetic algorithm, with the machine discovering underlying regularities by absorbing and analyzing audio data from jazz performances.
Structure of the algorithm
The core procedure joins two algorithmic ideas. The first, sequential-covering, builds a set of rules incrementally: after each new rule is learned, all positive examples that satisfy that rule are removed before moving to the next. The second component—a genetic algorithm—generates each new rule through mutation and crossover. The result is both a window into the expressive principles that a musician applied to a particular piece and a tool that can generate entirely new performances matching those expressive patterns. Three advantages emerge when using an evolutionary scheme, rather than other forms of supervised machine learning, for this specific task:
- The evolving model can be inspected and analyzed along the way.
- Interrupting and steering the model's development feels natural and intuitive.
- Different executions of the algorithm yield different models, an especially useful output because performers often express the same piece in multiple permissible ways.
Related investigative work
Several earlier systems have used evolutionary strategies to evolve chords or melodies. Notable examples include Vox Populi, which evolves harmonic patterns encoded as string bit maps with help from melodic, harmonic, and range-based fitness criteria; along with systems such as Moevius, GenDash, CAMUS, and Genophone. In the world of computer-based improvisation, system proposals use a model meant to reflect a novice jazz musician's experience by evolving melodic structures while chords progress beneath them.
Explicit research into expressive performance using evolution, however, remains rare. The ProMusic project's Grachten applied a genetic algorithm to tune the edit-distance cost functions that annotate timing in a jazz solo. Some colleagues combined generative regression trees with an evolutionary engine to reshape rendered MIDI sequences. Others, like Widmer, have chosen nearly the opposite path, attempting to locate broad rules behind classical playing by training other forms of inductive algorithms on spacious data sets that include Mozart sonata renderings by professional pianists.
Still, most performance-related efforts that rely on automatic learning stop sensibly at the classical repertory. In that world, you cannot freely alter written pitches and expected tempo has a normative range. In jazz performances, note ornamentation and local micro-timing carry significant expressive weight, shaping impulses rarely served by classical or non-learner-centric approaches.
Earlier influential work by Lopez de Mantaras associates SaxEx—a system built on case-based reasoning that retrieves remembered phrases for inexpressive ones when presented with fresh material—so as to insert feeling into computer solos. Yet his architecture resisted scrutiny: a user observing its final outputs could not evaluate the mechanism inside. Later contributions by Ramirez do combine interpretable rule sets alongside generative capabilities for jazz melody models that can explain the expressive modifications the system introduces. The more recent evolutionary performative perspective ties the two lines together—acquiring both a recognizable summary and generative potentialility, not needing at the same time to trust an inherent, baffling, non-ocular process.
Converting audio to symbolic detail
Records of actual human expressiveness, to be turned after the fact into workab le models based on evolutionary searches. Those final intermediate symbolic shape rises from a multiphase pipeline. The work from performing recordings via exact subsequent technique is this:
- Spectral analysis relies on Windows describing analyzing frames on segments sliced from wave streams via careful series: multiplying a processed area sound portion considering previously defined parameter as per frame periodicity starting standard
The extraction of the peaks computed next builds candidate collections toward reading fundamental estimates mapping misprescriptions covering possible lower/higher audio divergence done using so named Two-Way Mismatch method . For any candidate voicing, differential treatment regarding matches: First penal remaps occurs each measure regarding spectral nearest-neighbor absent between sound-defined bin vs exact chart differences assumed test before switching to second mismatch run working from projected forecast space backwards opposing identical second apply schema measures reliability increase: partial taken away marks down existing note often penalizing times octave mistake.
Handling pitfalls
Far from consistent runs: the cepstral background after implementing Two-Way Mismatch’ initialization run discovered noisy interruption appearing extremely.
The improvements adding as correction standard covered situation correction resolved accordingly
Impulse suppression method attached selection
The first refined enabling routine means excluding random spectral clutter interfering fundamental measurements surround threshold referenced peaks were reserved accordingly result differs variety - Noise interfering region shapes dynamic depend size differences of full amplitude profile then depending taking certain base: note Three degrees fade accelerate happens each radial disparity actual objective rightfully achieving appropriate results real feedback arriving sharp assessment provides good runs thereby further real value extracted substantially
Historical dependency
Secondary, designing algorithmic interpretation expands look for previous info sequencing integrating preceding moment estimated done by consider among subsequent intervals and relationships relationship exactly and outcomes likely maintain evaluation gains maintain authentic running window from each step achieved order oriented solid improving projection realism constant output ready and background depend focus achieving from main table thus reliably enable practical completed improved cycle.
Noise removal valve for sound region boundaries filtering dynamic segments resulting further clear segments to examined as provided end trigger
Equip precisely quality operating quiet segment compute floor elimination.
Slicing into notes
Frame indicates note barrier check using components feature analysis followed by initial implement band onset detection rooted around expert knowledge regarding human auditory anticipation zones: Once finished the routine done separate energy track appropriate quickly then crosses syncd back each part aligning measurement outcomes.
We group low parameter and boundary marker then calculate pitch certainty through histogram census:
This all note captured the high accurate step prevents confusion weird but poor estimated trigger into detection next: Cents here defined using frequency relationship relying equation: dividing baseline frequency reference f-ref logarithm conversion assigned.
Learning the rules of feeling
The main algorithm follows dual challenge mode, mining sets of conditions linked to each classification rather regular fitting attempt covering time - we attempt synthesization achieved performance then evaluation offering run from highest operational perspective leading better to match generating human mark with chosen controls eventual entire procedures presented currently serve basis building meaningful music always drawing own traditions, aims and original inside essence evolved space emerges final step original sets itself becomes simulation possibilities. Approaches started reported will progress importantly potentially reveal previously unclear map musical interpretation understand dynamic expressive interplay marks broad breakthrough both creator yet analytical base combination. The timing improvements then, gathered artistic foundations indeed continue extending musical encoding.
The following describes an inductive approach for learning an expressive music performance model from recordings of jazz standards. The goal is to develop a system capable of automatically generating performances that exhibit the expressive qualities typical of human musicians, starting from an inexpressive score description.
Not all expressive transformations can be predicted at the individual note level. Musicians work with abstract structures like musical phrases, making expressive performance a multi-level phenomenon. We therefore aim to build a computational model that integrates both note-level and structure-level information. As an initial step, our analysis is grounded in the implication/realization model proposed by Narmour [41, 42].
### 6.4.1 Musical Analysis
The implication/realization model addresses the perception and cognition of melodies. It asserts that a melodic line continually generates expectations in listeners about how the melody should proceed. These expectations arise from two sources: innate and learned. According to Narmour, we are born with innate knowledge that suggests how a melody should continue, while learned factors—developed through lifelong exposure to music and familiarity with specific styles and melodies—also shape our expectations.
Any two consecutive notes form a melodic interval. If this interval is perceived as incomplete, it becomes an implicative interval: one that implies a subsequent interval with particular characteristics. That is, certain notes are more probable to follow an implicative interval than others. Two key principles identified by Narmour are registral direction and intervallic difference. The principle of registral direction states that small intervals imply continuation in the same registral direction (a small upward interval suggests another upward interval, and similarly for downward), while large intervals imply a change in direction (a large upward interval suggests a downward interval, and vice versa). The principle of intervallic difference states that a small interval (five semitones or fewer) implies a similarly sized interval (within plus or minus two semitones), and a large interval (seven semitones or more) implies a smaller interval. Based on these principles, melodic patterns or groups can be identified that either fulfill or contradict the implication predicted by the principles. Figure 6.3 illustrates prototypical Narmour structures.
A single note often belongs to multiple overlapping structures. Therefore, representing a melody as a sequence of Narmour structures yields a list of overlapping elements. We parse each melody in the training data to automatically generate an implication/realization analysis. Figure 6.4 shows the analysis for a fragment of All of Me.
Fig. 6.3. Prototypical Narmour structures
Fig. 6.4. Narmour analysis of All of Me
### 6.4.2 Training Data
The training data consists of monophonic recordings of four jazz standards (Once I Loved, Like Someone in Love, and Up Jumped Spring), performed by a professional musician at 11 different tempos around the nominal tempo for each piece. The musician determined the nominal tempo as the most natural and comfortable for interpretation, and also identified the fastest and slowest feasible tempos. Recordings were made at regular intervals distributed between these limits. The dataset contains 4,360 performed notes. Each note is annotated with its performed characteristics and a set of attributes describing both properties of the note itself and aspects of its context. Note-specific information includes duration and metrical position within a bar. Contextual information includes performed tempo, neighboring note details, and the Narmour structure in which the note appears (with emphasis on the third position within the Narmour group, as this provides the strongest indicator of expectation).
### 6.4.3 Learning Task
This chapter focuses on note-level expressive transformations—specifically modifications to note duration, onset, and energy. We initially treat each expressive transformation as a classification problem. For note duration transformation, for instance, we classify each note into one of lengthen, shorten, or same. After obtaining a classification mechanism for all training notes, we apply regression to produce a numerical value indicating the transformation amount for each note. Section 6.6.4 details the complete algorithm.
The performance classes of interest are: lengthen, shorten, and same for duration; advance, delay, and same for onset deviation; soft, loud, and same for energy; and ornamentation and none for note alteration. A note is classified as lengthen if its performed duration is 20% or more longer than its nominal (score-based) duration. Shorten is defined analogously. A note qualifies as advance if its performed onset occurs 5% or more of a bar earlier than its nominal onset, and delay if the onset is 5% or more later. A note is loud if it is played louder than its predecessor and above the piece’s average level; soft is defined analogously. These thresholds were selected after experimenting with different ratios to ensure that, for instance, a note marked as lengthen reflected a purposeful decision by the performer rather than imprecision. A note is classified as ornamentation if a note or group of notes not specified in the score has been added to embellish the melody, and none otherwise.
### 6.4.4 Algorithm
We applied a genetic sequential-covering algorithm to the training data. The algorithm incrementally builds a rule set by learning new rules one at a time, removing the positive examples covered by each latest rule before attempting the next rule. Rules are learned using a genetic algorithm that evolves a rule population through mutation and crossover.
For each class of interest (lengthen, shorten, same), rules with the best fitness are collected during evolution. When learning rules for a particular class (e.g., lengthen), examples from the complementary classes are used as negatives. Although the test ran over 40 generations, the fittest rules typically appeared around the twentieth generation.
For note duration, onset, and energy, after obtaining a rule set covering all training examples, we apply linear regression to the examples covered by each rule. This yields a linear equation predicting a numerical value for that rule.
The process produces rules that give numerical predictions rather than nominal class labels. In the case of note alteration (ornamentation), no numerical value is computed; instead, the set of examples covered by the rule is retained. For generation, we use a k-nearest neighbor algorithm to select one example per rule and adapt it to the new melodic context—transposing ornamental notes to fit the melody key and ornamented note pitch. The algorithm is as follows:
GeneticSeqCovAlg(Class, Fitness, Threshold, p, r, m, Examples)
```
Pos = examples belonging to Class
Neg = examples not belonging to Class
Learned_rules = {}
While Pos do
P = generate p hypotheses at random
for each h in P, compute fitness(h)
while max fitness < Threshold do
create new generation Pnew
P = Pnew
for each h in P, compute fitness(h)
Return Newrule from P with highest fitness
Rpos = members of Pos covered by NewRule
compute PredictedValue(Rpos)
NumericNewRule = NewRule with Class replaced by
The outer loop learns one new rule per iteration, removing the covered positive examples before proceeding. The inner loop performs a genetic search for a high-accuracy rule. At each outer iteration, the new rule generalizes the disjunctive hypothesis (the set of instances classified as positive) by adding a new disjunct. The search starts from the most specific hypothesis (the empty disjunction) and continues until all training examples are covered.
NumericNewRule is a rule whose consequent is a linear equation: `X = w0 + w1*a1 + w2*a2 + ... + wk*ak` where `X` is the predicted value expressed as a linear combination of the sample's attributes `a1,...,ak` with predetermined weights `w0,...,wk`.
The weights are computed via linear regression on the set of positive cases `Rpos` covered by that rule. For note alteration (ornamentation), `Value(Rpos)` is simply the set of examples covered by the rule.
The inner loop refines the form of each new rule using a genetic algorithm with parameters `r`, `m`, and `p`, representing the crossover fraction, mutation rate, and population size, respectively. Their exact values are listed in Table 6.1. A new generation is created as follows:
1. Select: probabilistically select `(1 - r) * p` members from `P` to form `Ps`. The probability `Pr(hi)` of selecting hypothesis `hi` is `Pr(hi) = Fitness(hi) / Sum(Fitness(hj))` (1 ≤ j ≤ p). 2. Crossover: probabilistically select `r * p / 2` pairs of hypotheses from `P` based on `Pr(hi)`. For each pair, produce offspring via crossover and add to `Ps`. 3. Mutate: Choose m% of `Ps` members uniformly and apply mutation.
Table 6.1 Parameter values of the genetic algorithm
| Parameter | Identifier | Value | |-----------|------------|-------| | Crossover rate | r | 0.8 | | Mutation rate | m | 0.05 | | Population size | p | 300 |
Hypothesis representation. The hypothesis space for rule preconditions consists of a conjunction of a fixed set of attributes. Each rule is encoded as a bit-string: previous and next note duration (five bits each: much shorter, shorter, same, longer, much longer), previous and next note pitch (five bits each: much lower, lower, same, higher, much higher), metrical strength (five bits: very weak, weak, medium, strong, very strong), tempo (three bits: slow, nominal, fast), and Narmour groups (three bits coding the eight groups from Fig. 6.3). For example, the rule:
"If the previous note duration is much longer and its pitch is the same and it is in a very strong metrical position and the current note appears in Narmour group R, then lengthen the duration of the current note"
is represented by the binary string: `00001 11111 00100 11111 00001 111 110 001`.
The specific meanings assigned to the bits are as follows. Previous and next note durations are much shorter if less than half the current note; shorter if less than the current but more than half; same if equal; much longer and longer are defined analogously. Pitches are much lower if lower by a minor third or more; lower if within a minor third; same if identical; higher and much higher analogously. Metrical position is very strong on the first beat of a bar, strong on the third beat, medium on beats two or four, weak on offbeats, and very weak otherwise. Tempo is slow if the piece is performed at 15% or more below the nominal tempo (the performer’s preferred speed), nominal within 15%, and fast above 15%. For Narmour groups, we encode only one group per note: the group in which the note appears in the third position (or if unavailable, the first or second position, in that order).
Genetic operators. Standard single-point crossover and mutation are applied with constraints. For crossover, cut points must fall on attribute substring boundaries. Mutation points must not create inconsistent strings—only one class can be predicted, so exactly one '1' can appear in the final three-bit substring representing the class.
Fitness function. Each rule’s fitness is based on classification accuracy on the training data. The fitness measure is: `tp^α / (tp + fp)` where tp is the number of true positives, fp the number of false positives, and α a constant controlling the trade-off. With α set to 1, the standard fitness function `tp/(tp+fp)` tends to favor rules covering few true positives with zero false positives over rules covering many true positives with a single false positive. Since we want general rules covering many examples—even with a few false positives—we set α = 1.15, a good balance for our application.
### 6.4.5 Results
Formal evaluation of a model capturing subjective knowledge—as in expressive performance—is always challenging. The most direct evaluation may be to listen to the transformations the model produces. The accompanying DVD includes samples of model-generated transformations (inexpressive score transcriptions alongside their expressive versions as output by our system).
Alternatively, the model can be assessed by comparing its predicted transformations with those actually performed by the musician. Figures 6.5 and 6.6 show note-by-note duration ratios for two models induced from different algorithm runs, compared with the actual recorded duration ratio. Comparable results were obtained for predicted onset deviation and energy variation. The induced models appear to capture the musician’s expressive performance transformations accurately, despite the relatively modest training dataset.
Correlation coefficients for the onset, duration, and energy submodels are 0.80, 0.84, and 0.86, respectively. These figures derive from ten-fold cross-validation on the data. For each fold, performances similar to those in the test set—i.e., the same piece at tempos within 10% of the test set performances—were excluded.
We ran the sequential-covering genetic algorithm 20 times to examine variation in correlation coefficients. No major differences emerged across runs. While different runs produced distinct models, these models share clear performance trends but generate slightly differing expressive outcomes.
Users were allowed to shape the model’s construction by imposing “readability constraints” on rule form. This enables restriction of allowed rule formats (e.g., limiting allowed bit sequences) to improve interpretability. We examined some classification rules induced before replacing the class with the numerical predicted value. Rules of various types were observed: some depend on the note’s own features and performance tempo, while others focus on Narmour analysis alone. Rules incorporating both local note context and Narmour structure were also discovered. Several examples of duration rules are:
`RULE1: 11111 01110 11110 00110 00011 010 010 001` "In nominal tempo, if the next note has similar duration and the current note is at a strong metrical position and belongs to a D Narmour group, lengthen it."
`RULE2: 00111 00111 00011 01101 10101 111 111 100` "If the previous and next notes are longer than or equal to the current note, and the previous note is higher in pitch, shorten the current note."
`RULE3: 01000 11100 01111 01110 00111 111 111 010` "If the previous note is slightly shorter and not much lower in pitch, the next note is not longer and is within a minor third in pitch, and the current note is not on a weak metrical position, keep the duration as is."
Figure 6.6. Correlation between model predictions and actual performed values for Once I Loved at 65 bpm.
`These simple rules exhibit high accuracy.`
Footnote shows confusion in original. The first rule achieves 92% accuracy. [This sentence about confusion during evolutionary optimization is logically incomplete and unclear in meaning. The best attempt: these results indicate the model reliably predicts the musician’s durational choices.]
The third rule captures 90% of the relevant cases. Some rules have musical significance; RULE1, for instance, lengthens a note when the two preceding notes share the same pitch (forming a D Narmour group) and its duration matches the following note. This likely reflects the performer’s intent to emphasize the final note in a same-pitch sequence.
predicts how a note in a given context should be performed—such as stretching or shortening its duration relative to the written value. To build the expressive model, acoustic features were extracted from recordings and transformed into a symbolic performance representation. A sequential-covering genetic algorithm was then applied to the symbolic data along with contextual information. Although the training set was modest, the resulting model captures the performer’s expressive timing transformations with notable accuracy. Several induced classification rules have also proven musically meaningful. Work is currently underway to expand the training data and experiment with different encoded information. Increased data volume, enriched content, and incorporation of background musical knowledge will enable additional models. Future plans involve extending the model to predict not only note-level timing and energy, but also intra-note expressive features like vibrato and instantaneous energy. This will be achieved by characterizing notes through pitch, energy, and timbre attributes, learning to forecast these features based on musical context.
Acknowledgments
This research was funded by the Spanish TIC project ProMusic (TIC2003-07776-C02-01) and the TIN Project ProSeMus (TIN2006-14932-C02-01). We express our gratitude to Emilia Gomez, Esteban Maestre, and Maarten Grachten for their invaluable data processing assistance.