Why Music Structure Annotation Needs a More Rigorous Approach

Why Music Structure Annotation Needs a New Approach

A considerable share of current research on music structure focuses on algorithm improvements—boosting recognition scores or devising new performance metrics. Yet a more fundamental question often gets overlooked: how meaningful are the structure annotations used to evaluate those algorithms? Before worrying about precision, correspondence to initial definitions, or how well an automatic system works, researchers should first consider whether the annotation itself is relevant. This question becomes pressing when comparing annotations of the same tracks—such as those by The Beatles—that come from different test sets created by different teams. The concept of “music structure” has never been clearly pinned down. Surprisingly, people have poured far more energy into automatic estimation and evaluation than into defining what exactly the algorithms are supposed to find.

Building Blocks for Dependable Local Music Annotation

The goal of the present work is to develop a robust definition of music structure annotation. The process begins by laying out a set of rules for reliable local annotations, then reviews existing test sets against those criteria, and finally proposes a multi-dimensional definition. The validity of this approach was tested through a year-long experiment involving three professional musicians acting as computer annotators.

Information extraction vs. imitation. Local music annotation can be split into two categories: information extraction and imitation. Information extraction means mapping a piece of music to extract descriptive aspects—for example, structure annotation, beat annotation, or singing voice annotation. Imitation (or reduction) means finding audio objects that resemble the original piece, which can then be compared to it. This category includes note, chord, and melody annotation.

Four Conditions for Information Extraction

For information extraction to work well on a given corpus, four conditions must hold:

• Definition. An object or descriptor must be clearly defined.
• Certainty. In a given corpus, the object should be recognizable without doubt.
• Concision. The range of available descriptors should be limited.
• Universality. A descriptor should appear with reasonable frequency.

Measuring Certainty: Perceptive Recognition Rate

The condition of certainty corresponds to a quantity called the Perceptive Recognition Rate (PRR). This rate is measured by checking how many times a given object is recognized without doubt in a corpus. The PRR is a crucial factor in annotation. The recognition rate traditionally used to evaluate algorithms is called the Algorithmic Recognition Rate (ARR). If the PRR equals one—the perfect case—then the ARR is justified. If the PRR equals zero—the worst case—then even findings involving the ARR are meaningless. When an object is not easily recognized by ear, references to that object during annotation become inaccurate, and algorithm experiments on that object lose validity. One might argue that ARRs are typically calculated on corpora where the PRR is one, but annotation in any domain produces uncertainty.

A case in point: the chorus. Take the concept of “chorus,” which seems straightforward at first. The traditional definition describes it as a part of a track that includes the lead vocalist, contains the song title in the lyrics, and repeats at least twice during the song. Applying this definition to a test set of 112 songs that are varied, neither especially mainstream nor recent, the PRR drops below 50 percent. For over half of those songs, listeners could not determine whether a chorus existed.

Measuring Concision and Universality

Several metrics help gauge the concision and universality of annotations:

• T: total tracks in the test set
• L: total number of distinct labels used across the test set. Good annotation keeps L small.
• N(l): number of tracks using a specific label l, divided by T. A high N(l) indicates the label concept applies to many tracks (i.e., it is universal). A low N(l) simply suggests the concept is narrow.
• U(l): average use of a specific label l within a track, calculated only for tracks where the label appears at least once. High U(l) means the label fulfills a structural role through internal repetition.
• mS: average count of distinct segments per track. Large values signal many segments, which depends on track duration and music style.
• mL: average count of different labels used in a track. If mL is close to mS, each label typically appears only once.

Review of Existing Music Structure Test Sets

A clear definition of music structure has long been missing, yet several test sets have been proposed. Each is reviewed below using the metrics just described. Note that definition and certainty cannot be evaluated here—the first because it is not provided, the second because it requires presence during annotation. The results are summarized in accompanying tables.

MPEG-7-Audio Test Set

Created by IRCAM in 2001 during development of MPEG-7-Audio, this set included 25 tracks annotated with state and sequence structures. Annotations were cross-checked among MPEG-7-Audio participants. Fifty different labels were used, mixing descriptions of musical role—“intro,” “verse,” “chorus”—with instrumentation—“bass,” “break drum,” “chorus instru,” and so on. The average segment count per track was 17.57, with an average of 7.64 different labels per track. Most labels (like “break 2,” “chorus variante”) appeared only in a single track and usually only once. Exceptions included “break” (N=0.43, mean use 2.16), “chorus” (0.93, 4.38), “intro” (0.86, 1.25), and “verse” (0.93, 3.92). The number of labels here is far too large and their usage too restricted.

The MPEG-7-Audio test set also illustrates a common problem: the music is chosen to fit the annotation system. Many tracks in the “state” part were grunge music, where instrumentation changes sharply between verse and chorus, making annotation very clear. The later “sequence” portion used early Beatles-like music where instrumentation stays constant and structure comes from melodic line variations.

QMUL Test Set

Extending the MPEG-7-Audio state set, the Queen Mary University of London set included 107 pop-rock tracks, many of them Beatles songs, using 107 distinct labels. The average yielded 12.33 segments and 6 labels per track. Most labels were restricted to a single track and single use, with exceptions like “break” (N=0.22, mean use 1.53), “bridge” (0.55, 1.6), “chorus” (0.43, 3.96), “intro” (0.85, 1.27), “outro” (0.38, 1), and “verse” (0.87, 3.30). Again, the label count is too large and usage too limited.

The Beatles Test Set

Developed by Universitat Pompeu Fabra using musicologist Alan W. Pollack's annotations and later revised by Tampere University of Technology, this set described 174 Beatles tracks with 55 labels. It showed low average segments (9.21) and labels (5.23) per track. Even here, most labels were only used once—exceptions included “bridge” (N=0.59, mean use 1.73), “intro” (0.86, 1.08), “outro” (0.82, 1), “refrain” (0.42, 3.41), “verse” (0.86, 3.33), and “verses” (0.28, 1.16). The most common labels refer to musical role: intro, outro, bridge, verse, chorus.

TUT07 Structure Test Set

Tampere University of Technology built the largest test set yet—557 Western popular music pieces across pop, rock, jazz, blues, and schlager genres—annotated by musical role or acoustic similarity using labels like “A,” “B,” and “solo.” Detailed figures could not be provided here, as the annotation key is unavailable.

TU Vienna Test Set

Containing 109 tracks drawn partly from QMUL, RWC, and Beatles sets, this test set introduced the novel idea of allowing multiple simultaneous descriptions of the same segment through a hierarchical XML schema. Because of partial overlap with other databases, specific numbers are not given separately.

RWC Test Set

The RWC set annotated 285 tracks of music, using just 17 labels. The average segment count per track was high at 15.73 (though sub-segments are counted here), and each track used 6.68 different labels on average. Crucially, every label appeared in at least ten tracks: the mean N(l) across labels reached 0.39, with a mean U(l) of 2.16. Labels mostly describe musical role—intro, ending, verse, chorus, bridge—but merge in symbols for acoustic similarity (e.g., “verse-a,” “verse-b”). Despite this mixing, the RWC annotation stands as the best so far, thanks to the limited number of labels and their good coverage. Even so, decisions about musical role vs. acoustic similarity can be inconsistent. Some parts labeled “intro” turn out identical to “verse-a,” underscoring the need to keep viewpoints separate.

Common Problems across Test Sets

Existing music structure annotations share several flaws:

• Excessive label count. Most test sets use many labels that rarely repeat across tracks.
• Blended perspectives. Annotations often confuse musical role (like verse or chorus), acoustic similarity, and instrumentation.
• Boundary inconsistency. Sometimes two similarly different parts are treated as identical; sometimes they are treated as distinct.
• Unclear structural base. It is often unclear whether annotations follow the voice, accompaniment, or something else entirely.
• Sub-mn/iguous boundaries. Segmentation varies within and between tracks.
• Variable granularity. Whether a repeating chord sequence is one A or many smaller “a” subdivisions is decided arbitrarily across a collection.

Toward a Multi-Dimensional Music Structure Annotation

After studying the existing test sets, it becomes clear that many strategies are possible. Any choice of definition can work if an appropriately selected set of music is chosen. For instance, you could describe structures purely in verse/chorus terms if your test set consists entirely of tracks with obvious verses and choruses. The present goal, however, is to find a description of music structure that applies to any style.

The fundamental principle: no mixing viewpoints. Regardless of which description is chosen, blending different perspectives must be avoided. In earlier test sets, labels that combined instrumentation and musical role (“break guitar,” “break piano”) or acoustic similarity and musical role (the part called “introduction” being 100 percent similar to the “verse”) made interpretation noisy and unreliable.

Musical-role-based description. One approach attaches labels according to the role a section plays in a track—“introduction,” “verse,” “chorus,” “bridge,” “ending.” Identical labels thus indicate identical roles. But this approach jumbles several distinct ideas. “Intro” and “outro” depend on time position rather than absolute similarity, and a chorus sometimes occurs at the very start of a song. Moreover, songs often contain multiple variants of the same role, prompting labels like “verse A," "verse B.” Even basic definitions crumble: trying to annotate R'n'B music—among the most popular styles today—yields long stretches that could all be called verses but also hooks, vamps, or simply verses.

Acoustic-similarity-based description. Another approach assigns identical labels to parts that sound alike. Strict identity is rare; more often two parts are, say, 90 percent similar. But similarity depends on the listener’s point-of-view: is timbre more important than harmony or rhythm? Is one instrument weighted over another? Once some metric collapses nuance into a similarity rating, a threshold must be chosen to decide binarily whether parts earn the same label. That choice directly controls the number of labels inside a track.

Instrument-role-based description. Yet another method focuses on instrumentation, but this also conflates distinct information: an electric guitar solo and an acoustic guitar solo could easily be labeled “guitar” despite serving different structural functions.

A proposed solution: multidimensional annotation. The study presented here demonstrates that a more robust system is needed—one that treats musical role, acoustic similarity, and instrumentation as separate, orthogonal dimensions instead of merging them under a single label decision. Each track can be annotated independently along these dimensions using the structured metrics offered above. All possible dimensions (musical role, acoustical relatedness, instrumentation, time position, energy, and so on) form a multidimensional description that carries richer, more interpretable information and is less dependent on the biases of a particular corpus. This description allows both musicologists working on classical studies and practitioners tackling pop catalogs to annotate structurally without forcing square pegs into round holes.

Testing the proposal. The validity of the multidimensional approach was tested by applying it to 300 tracks drawn from various musical genres, using annotation sessions conducted by three experienced musicians over the course of a year. Early analysis shows that separating out descriptive axes allows annotators to recognize most segments clearly and repeatably, which is far less common when annotation mixes types. All six measured metrics—T, L, N(l), U(l), mS, and mL—perform favorably on this proposal test set, particularly in the dimensions of concision and universality. Nonetheless, achieving true Perceptive Recognition Rate reliability still requires annotation protocols where professional musicians and, when possible, crowd feedback confirm segment perception at multiple-level abstractions.

A simpler approach focuses on assigning labels to the instrumental description of a track. This identifies the locations of lead vocals, solo guitar parts, and so forth. While interesting, such a description reveals little about a track’s overall structure. Moreover, specifying an instrument’s identity requires a vast number of labels (e.g., classical guitar, folk-guitar, 12-string-guitar, electric guitar, wah-wah guitar). A more useful strategy describes the “role” a particular instrument plays in the track, such as “Primary Lead” (the clear frontman vocalist or instrument) or “Secondary Lead” (the backing vocalist or sideman). We refer to this as “instrument role” in what follows.

Music Structure Based on the Final Application

One can also base annotation on how the structure will be used. For example, if the goal is to create an audio summary or thumbnail that captures the most memorable part of the track, as in [2], it may be unnecessary to annotate chorus positions; one could instead focus only on the most repeated segments. The drawback is that such an annotation can validate only that specific application and cannot serve other purposes.

Music Structure Based on Perceptual Tests

Another approach relies on perceptual tests to capture average human perception of musical structure, as [4] did for tempo and beat annotation. Beyond the high cost of this method, a further problem is that in the case of “Music Structure,” the labels people use to describe a track’s structure typically differ from person to person.

Proposed Music Structure: Multi-Dimensional Representation

The central idea of the proposed description is to use multiple viewpoints simultaneously, yet independently: “acoustical similarity,” “musical role,” and “instrument role.” This concept reflects how modern music is created through multi-track recording, where a set of main patterns repeats over time with variations.

Superimposed on these patterns are instrumentation layers (such as singing voice) and “musical roles” like introduction, transition, chorus, solo, and ending.

The annotation method is built upon “Constitutive Solid Loops,” which are fundamental blocks whose boundaries and labels derive from a synthetic perception of the various elements. The criteria proposed here encompass familiar structural labels such as “chorus” and “verses” but are far more powerful. Unlike the “verse/chorus” approach, this annotation method can properly describe structure across many styles. One immediate example is its ability to annotate pieces that contain no choruses — a situation far more common than one might instinctively think.

Proposed Method: Multi-Dimensional Music Structure Annotation

4.1 Overall Explanation

The core idea is that a track is composed of:

• A set of Constitutive Solid Loops (CSLoops), each representing a “musical phrase” or “musical exposition” (a chord progression). CSLoops sharing the same ID denote the same “musical phrase,” even though considerable variation can occur between occurrences. Two CSLoops with the same ID can follow each other when the phrase is repeated twice in succession.
• Variations of the CSLoop ID are superimposed. For instance, the same CSLoop appearing in a lighter version (e.g., without drums or bass) is marked “–”; a stronger version (e.g., with an extra second guitar) is marked “++”.
• Significant “instrument roles” are also superimposed, including the presence of “primary leads” (lead singer in popular music, lead instrument in jazz or electronic music), “other leads” (choir, other lead instruments, or melodic samples), or “solo mode” (electric-guitar solo, jazz chorus solo, etc.).
• Each section plays a “musical role” such as intro, outro, transition, obvious chorus, or solo.

The track is thus simultaneously segmented according to these various viewpoints. Any part that is too complex to describe is annotated as “ComplexMode.”

The mandatory decomposition is the CSLoop description. When a CSLoop is an obvious chorus, it is marked as “chorus.” When it is not obviously a chorus, it is not labeled as such but can still be annotated as a repetition of a specific CSLoop occurrence, often including PrimaryLead and OtherLead (Choir) as distinctive features.

For example, Pink Floyd’s “The Dark Side of the Moon” album, which has sold 40 million units (the 6th-best-selling album of all time), contains not a single chorus.

To handle segment subdivision, markers can be placed within a CSLoop segment to indicate possible further divisions. Two marker types, “V1” and “V2,” denote similarity or dissimilarity, respectively, between the parts to the left and right of the marker.

The temporal boundaries of segments and markers are defined as the downbeat closest to the start or end of the respective described object.

4.2 Detailed Description

Table 2 provides the detailed specification and definitions for the proposed “Music Structure” annotation.

Trans: Indicates transitions that are structurally outside the CSLoop.

IO: Denotes intro, outro parts, or exotic sections unrelated to the rest of the song.

CSLoop 1, 2, 3, 4, 5, A, B: Indicates a musical phrase, idea, or subject. The equality rule applies to CSLoops 1–5; two CSLoops with the same ID represent the same thing. This rule does not apply to CSLoop A and B, which are used either when the track contains too many CSLoops for a full annotation or when the annotator cannot decide reliably about equivalence but still wants to mark a section.

– (++): When applied to a CSLoop, it indicates that this occurrence has a much lower (higher) loudness than the rest of the song, or that two of the three references (rhythmic, melodic, harmonic) disappear (are added).

Cplx: Marks a very complex, non-periodic part (e.g., Frank Zappa’s free improvisation sections).

SMode: Indicates a Solo Mode, regardless of which instrument (vocal, guitar, sax, piano) performs the solo; it can be superimposed on a CSLoop to mark the section over which the solo occurs.

PLead1: Denotes the main melodic referent, typically the lead singer or the instrument playing the theme in jazz.

PLead2: Indicates a second (side-man) or third melodic referent, e.g., in a duo or trio.

OLead1, OLead2: Refers to a second melodic referent that is not the main one, such as backing vocals or instruments interacting with the singer’s melody.

Chorus 1, 2: Indicates an “obvious chorus,” i.e., a section that is unmistakably a chorus. Two chorus IDs are possible.

V1 (V2): A marker (unlike the segment descriptions above) that indicates a subdivision within a CSLoop, where the part to the left of V1 is similar (or to the left of V2 is not similar) to the part on the right.

The “Exclusion” column in the table lists label incompatibilities. For instance, a segment cannot be both “–” and “++” at the same time.

4.3 Examples

Figures 1 and 2 present two examples of the proposed method, applied to tracks also part of the test-sets discussed in Part 3.

Figure 1 shows the annotated structure of The Cranberries’ “Zombie.” The annotation is multi-dimensional, with several criteria described simultaneously. The track’s main structure rests on two CSLoops: “CSL1” and “CSL2.” CSL1 serves as the introduction (“IO”) in a lighter form (“–”). This is followed by “CSL2” in a strong version (“++”), then back to “CSL1” in normal form, which here acts as a transition (“trans”). The end of this part features an Other Lead (“OL1”), the guitar melody of “Zombie.” After that, CSL1 is repeated twice with singing voice (“PL1”). In previous test-sets, this section would have been called “verse”; however, designating it as “verse” fails to reveal that it is actually the same part as the transitions (“trans”) and the solo (“SMode”). Next, “CSL2” appears in strong form (“++”) with singing voice (“PL1”) and is clearly a chorus (“Chorus 1”). The remainder of the track follows the same logic until the ending, which uses “CSL1” in light form (“–”) as an outro (“IO”). Note also the comma separations within the CSLoop (“V1,” “V2”) that indicate sub-repetition (“V1”) or subdivision (“V2”). The conciseness of this representation is striking, as is the amount of information it holds.

Figure 2 presents the annotated structure of The Beatles’ “Come Together.” The track consists of four different CSLoops. It begins with “CSL1” in normal form, acting as an introduction (“IO”). The second “CSL1” includes singing voice (“PL1”) and ends with a lighter version (“–”). The next “CSL1” serves as a transition (“trans”). “CSL2” functions as the obvious chorus (“Chorus 1”). Around time 125s, a new part called “CSL3” begins with an Other Lead (OL1) — the guitar — and this Other Lead acts as a solo (“SoloMode” or “SMode”). The track closes with “CSL4,” featuring interleaved “Primary Lead 1” (the singer) and “Other Lead 1” (guitar melody). Again, the description remains remarkably simple given the complexity of the structure.

4.4 Testing Across a Large Variety of Music Genres

The proposed description was tested on a diverse collection of 300 music tracks from multiple genres, including:

• Progressive Rock (Pink Floyd, Queen, Frank Zappa, etc.)
• World Music (Ali Farka Touré, Buena Vista Social Club, Stan Getz / Gilberto Gil, etc.)
• Electronic Music (The Chemical Brothers, Squarepusher, etc.)
• Rap Music (50 Cent, Outkast, etc.)
• Mainstream Music (Michael Jackson, The Beatles, Eric Clapton, Nirvana, Cranberries, Bauhaus, The Cure, etc.)

4.5 Information Extraction Conditions Applied to the Proposed Multi-Dimensional Music Structure Annotation

Given that the “Definition” condition is fulfilled, we measure the other conditions — “Certainty” (PRR), “Concision,” and “Universality” (T, L, N(l), U(l), mS, mL values) — using the annotations of these 300 tracks.

PRR: Observations over a three-month period indicate that this multi-dimensional annotation method yields results that are reliable over time and, importantly, independent of the annotator. The method achieves a much higher inter-annotator agreement than existing methods (tested on the same tracks), demonstrating a high PRR.

L: 21 different labels are used (19 when omitting the comma subdivisions V1 and V2). As with the RWC dataset, the total number of labels is small.

mS: On average, a track is divided into 38.93 segments (22.80 when omitting comma subdivisions V1 and V2). This mS value is very high, because we consider the appearance of each new superimposed label within another label, such as PLead1 appearing in the middle of a CSLoop, as creating a new segment. Considering only CSLoop

segmentation would decrease mS considerably. This property makes the description scalable.

mL: On average, a track uses 9.80 different labels (8.11 when omitting comma subdivisions). This high value stems from the multi-dimensionality of the description. Because several viewpoints are used simultaneously, multiple labels co-exist at the same time (e.g., “CSLoop1” with “–” and “Plead1”), raising the number of distinct labels within a track.

N(l) and U(l): Detailed results for N(l) and U(l) appear in the last two columns of Table 2. The mean (across labels) N(l) is 0.47 (0.39 for RWC), which is very high. The mean U(l) is 3.21 (2.16 for RWC). These figures demonstrate that the concepts used are quite universal across many tracks (high N(l)) and each plays a structural role within an individual track (high U(l)). That these values exceed those for RWC, and given that our test-set spans a far wider range of genres, is very promising for this approach. Only “CplxMode” and “Chorus2” appear in few tracks, which aligns with their intended functions (“too complex to be annotated” and “two different chorus”).

Annotated examples from the test-set are accessible at: http://recherche.ircam.fr/equipes/analyse-synthese/peeters/pub/2009_LSAS/.

4.6 Use of the Proposed Multi-Dimensional Music Structure Annotation in M.I.R.

Initial research on Music Structure annotation aimed to create a test-set for evaluating music structure estimation algorithms. Since typical algorithms estimate a mono-dimensional structure, we developed a methodology for reducing the multi-dimensional annotation to a single dimension. A rule set based on weighting the various dimensions allows one to decide whether a CSLoop is “constitutive” of the track’s structure. The other criteria (PrimaryLead, OtherLead, “–”, “++”, etc.) are then treated as additional descriptions of the constitutive CSLoops and are used to establish equivalence between them and identify repetitions over time. Of the 300 tracks in the test-set, only 200 could be reduced to a mono-dimensional structure; the remaining 100 did not meet the requirement of having repetitive parts (whether acoustically, by musical role, or by instrumentation). Reducing them would have resulted in a very low PRR.

To address this problem of non-reducible tracks, multi-dimensional structure estimation algorithms will need to be developed.

Beyond evaluation, the multi-dimensional annotations provide rich information about how music tracks are constructed. They reveal temporal relationships between various dimensions — for instance, the use of “++” over a CSLoop before the entry of a PrimaryLead

— or capture genre-specific stereotypes, such as a “chorus” based on the same CSLoop as a “verse.”

Conclusion and Future Work

In this work, we proposed a set of conditions to define robust concepts for local music annotation. We applied these conditions to create a robust “Music Structure” annotation system, centered on a multi-dimensional description that simultaneously uses several superimposed viewpoints: “musical role,” “acoustical similarity,” and “instrument role.” The annotation method was tested on a collection of 300 tracks from diverse genres. All four measures — Definition, Concision, Universality, and Certainty — surpassed results obtained from previous test-sets. Notably, the proposed method enables much higher inter-annotator agreement.

Future work will focus on developing a quantitative measure for the Perceptual Recognition Rate (PRR) used in the experiment. This could be obtained by adopting the performance measures (insertion, deletion, equivalence between labels) commonly used for evaluating M.I.R. algorithms, but applied between annotations from different annotators.

Future efforts will also apply the same approach to other standard local music annotation tasks, such as singing voice, chord, or melody description.

Acknowledgments

This work was partially supported by the “Quaero” Programme, funded by OSEO, the French state agency for innovation. This work could not have been realized without the excellent contributions of Jean-Francois Rousse and Maxence Riffault. We would also like to thank the three anonymous reviewers for their fruitful comments, which helped improve this paper.

References

1. S. Abdallah, K. Nolan, M. Sandler, M. Casey, and C. Rhodes. Theory and evaluation of a bayesian music structure extractor. In Proc. of ISMIR, pages 420–425, London, UK, 2005.
2. M. Cooper and J. Foote. Automatic music summarization via similarity analysis. In Proc. of ISMIR, pages 81–85, Paris, France, 2002.
3. M. Goto. Rwc (real world computing) music database, 2005.
4. D. Moelants and M. McKinney. Tempo perception and musical content: What makes a piece slow, fast, or temporally ambiguous? In International Conference on Music Perception and Cognition, Evanston, IL, 2004.
5. MPEG-7. Information technology – multimedia content description interface – part 4: Audio, 2002.
6. B. Ong and P. Herrera. Semantic segmentation of music audio contents. In Proc. of ICMC, pages 61–64, Barcelona, Spain, 2005.
7. J. Paulus and A. Klapuri. Labelling the structural parts of a music piece with markov models. In Proc. of CMMR, Copenhagen, Denmark, 2008.
8. E. Peiszer, T. Lidy, and A. Rauber. Automatic audio segmentation: Segment boundary and structure detection in popular music. In Proc. of LSAS, Paris, France, 2008.
9. A. Pollack. ”notes on ...” series. the official rec.music.beatles, 1989–2001.