How Interactive Digital Music Is Reshaping Listener Engagement

The relationship between listeners and commercial music has long been dynamic. From the days of vinyl, audiences could scratch, manipulate tempo, and blend songs together. Today, mobile devices offer even greater processing power, opening up new possibilities for interactive music experiences.

This article explores a novel approach to interactive digital music, focusing on the emerging album app format. The research, developed through the album app of the artist Daisy and The Dark, extends existing playback paradigms by letting listeners explore alternative takes, genre-specific renditions, and direct interaction with mix stems. The audio stays true to the artist's studio vision — user-influenced but machine-controlled.

Understanding interactive versus dynamic music

Interactive music, as defined here, changes through deliberate user intention, while dynamic music shifts primarily in response to an external variable — for instance, a video game scenario. Two types of interactivity emerge: produced and algorithmic.

Produced interactivity means the artist creates a range of prepared musical options. Think of a rock track where you could, at any point, switch from electric guitars to acoustic ones, turn drums into hand percussion, or highlight strings. Multiple vocal takes or different instrumental parts could also be offered. The artist controls these boundaries, maintaining artistic integrity while opening up choice.

Algorithmic interactivity relies on mathematical or statistical processes to vary playback, with user influence shaping the result. You might select genres or styles, but the system responds differently than in produced interactivity. For instance, a series of gestures could be automated into a unique sequence, or pre-set real-time processing modes could be triggered on demand.

The broader project landscape

Creating tools that let musicians, producers, and record labels develop their own interactive music is a major concern. A content management system with a simple interface would allow for consistent population of the format. Standard mastering practices, like matching loudness across an album, become complex when songs can change dynamically. Collecting data about user response also adds insight. These topics go beyond this particular discussion.

Lessons from game audio

Computer games have featured dynamic, often adaptive, music since the 1980s. LucasArts' iMuse system, for example, triggered MIDI sequences based on in-game decision points, steering composition through a branching tree of possibilities. This evolved into horizontal re-sequencing (reordering pre-composed blocks) and vertical re-orchestration (adding or removing concurrent music layers as gameplay progressed).

Berndt and colleagues, studying game audio in 2012, pointed out that static or completely random music both fall short. Their survey suggested consumers were ready for greater variation. A 2014 study by Gasselseder then provided scientific evidence that dynamic music significantly increased player immersion.

The album app format

Album apps combine audio with artistic content — artwork, video, lyrics, games, social feeds and crucially, interactive features. They offer a secure format less prone to tampering or piracy than simpler music files. This strengthens piracy defiance. The concept could extend to smart TVs and video game consoles, and increasingly, streaming services replicate some of these functions. The potential for convergence between album apps and streaming platforms is considerable.

Notable interactive releases

Several previous projects shaped the landscape. Rush offered combined music/media apps as early as 2010. Bjork's 2011 Biophilia remains the most recognized album app, integrating song-specific interactive sound and visual features. Peter Gabriel's Music Tiles app, while not an album, enables audience mixing as a game. Jorge Drexler's n app specifically composed songs for interactive tablets. Gwilym Gold's Tender Metal used algorithms to generate a different synthesis every listening, without user control. The Jammit series since 2012 achieved easy muting, volume control of individual stems, and slowdown, combined with notation. Pitbull's Planet Pit acts primarily as a promotional tool despite many interactive multimedia additions. DJ Vadim's Immersive Albums from 2013 let viewers substitute audio during video, swapping vocalists or drum parts through interactive clickable media.

In 2014, Reactify licensed artist originals for CTRL, a remixing app allowing track mapping into a DJ-led mix used by Swizz Beatz. That year also saw potentially the first chart-eligible app developed by Shakhovskoy and Toulson for Francois and The Atlas Mountain's Piano Ombre album through Script LTD software. Bernhoft added complete song stem distribution through master mix updates, allowing continuous multitracking of loops.

Multi-track file formats and their development

Several multi-track audio formats emerged historically. iKlax packages nine distinguishable audio subsystems (potentially ten on mobile). It enables per-stem muting and exporting, programmed just enough to stabilize playback on numerous track delivery interfaces. Audizen further produced a Music 2.0 variant that folded into development of an MPEG Interactive Audio Application (AAA envelope, limiting baseline interop). MXP4 opened for wider adaptation around video overlay notifying live biography feeds intended for subscription in 2009. Open offerings like MOGG initially facilitated prototype management work but few took royalty placements intact. BRONZE composition served output consumption solely through computing logic from a MIDI shell primarily managed via internal distribution between track assets. Most recently, Native Instruments designed the MPEG-a enhanced Stems package over 2015 traversal to let deejays trigger audio transport bridging the software/hardware perimeter as required. Well maintained tool translation formalist transition packages permit creative environment scaling virtually wherever allowed exclusive conditional permission.

Research methodology and objectives

This project set out to discover how cutting alternative digital distribution mechanics could—

• offer engagement paradigms refreshing collaborative commercial acquisition
• generate deeper emotional trust loops between talent alignment

Design & technology implementation chain

To address either broad demonstration one iteration underwent bi-month prototype. Agiler circle constructed until reasonable engine parameter satisfied representation requirement compliant author series satisfied independent rule subtheme environment assembly. Vocal prototyping had succeeded across a diversity pilot ranging programming abstraction from Max/PFF to relative eventual pure C Objective runtime (current integration stabilized from experience of teammate submission arrangement and potentiality through a XU mark version shared distribution dynamic operating fluid mapping library routine aligning form tested stability overhead estimate.

At function core in post it defines triple lateral anchor enabling subsequent higher product evaluation. Front screens slide manipulation motion into rule-block coordinate handling: common starting switch points interact motion buttons either control adjustment informs playback rendering adjustment code mapping from sliding rendering co-record validation memory trigger adjustment patterns adjust flow update accurately reacting seamlessly to screen motion drawn from inside user action condition state modifying graphical position parameter.

Engine playback & how many files timeline

Micro interstitials allowing runtime running operating typical triple layers representation concurrent. Rule construction enforced quality matched material from synchronization loop giving adjustment automatic linear reduction making minor template sure concurrent responsive minimal check amount from cross architecture tested timeline support case revision hardware maintained status across origin push guarantee listening possibility parameter safe effect simultaneous rendering:

• play six ready file track parent object but possible maximal rendering gives listener up to 36 layers arrangement overall arrangement flexibility ceiling work ensuring soft coverage conversion satisfied adjust safety margin keeping
• stock six family to align instrument coupling naming variant - could be instruments not defined preset to actual final combination texture offer entirely customize broad access music solution within album.

The SLAudioEngine and SLtimer objects together form the core of the app’s bespoke audio engine, built in Objective-C using Apple’s Xcode environment. The SLAudioEngine instantiates 36 AUFilePlayers that read audio from file URLs specified in the songName.plist file. These players feed into a 36-input AUMixer, which enables centralized control of audio playback, while the SLtimer manages synchronization and scheduling.

Each stem within the SLAudioEngine has its own fade engine for creating transitions—timed fades to a predefined volume level. Transition profiles are stored in the songName.plist preset, assembled through the custom-built CMS. Users select from a palette of pre-defined profiles. Every transition is defined by a mix level in dB; signals below –99 dBFS are treated as silent and flagged accordingly. Fades can follow either a linear or an equal-power curve, progressing through 100 steps within a user-definable time window. Fades may begin on any step and stop on any later step.

The algorithm currently allows the CMS user to define up to 36 transitions. When triggered by an end-user gesture, each fade engine checks whether the current mixer channel volume must be adjusted to match the transition’s defined level. If the target volume exceeds the current level, a fade-in occurs across the specified step range; conversely, a fade-out takes place if the target is lower.

The transition time, set in real-time from the app GUI, can be varied by the user. The CMS also supports quantization, locking the triggering moment to the song’s tempo within a calibrated range, ensuring musically appropriate timing. Transitions may also be triggered by intelligent computer control—either randomly or in response to algorithmic events.

4.2. Interactive User Interfaces

Several interactive GUIs were prototyped, and four were selected for the final application: stem faders, stem switches, manual mix crossfades, and intelligent mix crossfades.

The simplest interfaces let users fade stems in/out and switch them on/off. Figures 4a and 4b show the stem faders GUI for the song “Red Planet.” Each vertical slider controls a single stem’s playback volume—drums, bass, synthesizers, cellos, strings, and vocals in this case.

A user can create a custom stem mix, but also mute a single instrument to sing or play along, or even solo a single stem—for example, isolating the bass guitar—to practice mimicking and eventually replacing that part in the full track.

The stem switches GUI for “Waltzing” lets users manipulate three distinct versions of the same song. With the six central stem buttons active (Figure 4c), the main radio mix plays, using stems for two piano lines, cello, synthesizer, electric guitar, and vocals. The left column activates an electronic remix (drums, bass, piano, synthesizer, electric guitar, vocals), and the right column an orchestral version (different cello, violin, and vocal lines). Users can select one of the three pre-produced versions or, when combined with the stem fader positions (Figure 4c–4d), build their own mix from stems across all three versions. The fader positions affect all combinations of the stem switches.

The manual mix crossfades GUI was developed for both “Ghost” and “Red Planet” songs, each with a different geometry (Figure 5a–5b). “Ghost” uses a triangular crossfade mixer incorporating three song versions. Placing the small white cursor (the mix control) at the triangle’s top plays the main radio mix; the bottom corners activate acoustic and electronic versions. Users move the control to blend between two or all versions. Tapping the mix control toggles vocals on/off, allowing an instrumental version, though stem faders also influence the result.

The “Red Planet” GUI (Figure 5b) works similarly but with a circular design. The main radio mix plays when the control is at the center; acoustic, electronic, dub, and choral versions sit at the 0°, 90°, 180°, and 270° positions on the circle perimeter.

The Circus interface serves as an intelligent mix crossfade GUI, arranged as a 5×5 matrix of cells, each representing a different mix (Figure 5c). Users engage new mixes by tapping or dragging across the matrix. Initially the mix control sits at the center for the song’s main radio mix. Each corner offers an alternative—acoustic and electronic remixes—while cells near those corners provide subtly different variations. When a new mix is selected, a dedicated crossfade algorithm smoothly transitions the audio. The Circus GUI also features a unique variPlay mode activated by any of four buttons below the matrix (Figure 5d). With variPlay active, a quadrant highlights, and an intelligent algorithm automatically selects new mixes at various points in the song, ensuring each listen is unique. Four variPlay modes are available, each presenting a different quadrant.

4.3. Additional Album App Features

The app includes other rich media features aimed at enhancing the overall user experience. The main home screen (Figure 6a) lets users select features by dragging menu items to the center, triggering an animated launch to the chosen page. The music album page (Figure 6b) provides access to the EP’s four tracks; selecting a song immediately opens its associated song-player interface.

From within any song-player, swiping left or right brings up lyrics (Figure 6c) and a narrative describing the song’s origin (Figure 6d). The band section, reached from the home screen, offers Daisy and The Dark’s full biography, artist images, performer credits, and production credits. A gallery section displays album artwork and live performance photographs, while an extras section links to social-network features.

4.4. Data Collection

The app integrates analytics via the proprietary Flurry protocols, capturing data on user engagement with various features—menu selections, song interactions, duration spent on each screen, play counts for all songs, and the frequency of interaction with different GUIs and playback options. Sufficient user adoption would enable quantitative analysis of which sections receive the most attention.

A built-in research questionnaire collects both qualitative and quantitative user experience data. Questions ask users to rate each interactive album app feature, provide an overall impression, and indicate whether they would purchase commercial music in this format in the future.

4.5. Content Management System

The CMS enables content providers—music producers or record labels—to integrate their audio into a compiled app. Users can specify stem combinations, trim volumes, set fade-time parameters, and define the app’s behavior in response to button presses. Transitions between song sections can be auditioned, and variPlay modes configured. Crucial to this workflow is instant feedback that replicates the final app’s behavior, allowing iterative fine-tuning and experimentation.

The Content Manager was built in Max/MSP, chosen for its versatility in controlling audio, capturing multi-touch data, and formatting output into a JSON file readable by the app. Figure 7 (left) shows the CMS GUI; a pop-up sub-window (right) represents an iPhone user’s viewpoint.

5. Discussion and Conclusions

The app holds data for 96 audio stems. To keep the total download size minimal, 256 kbps MP3 files were used rather than lossless pulse-code-modulation (PCM) audio. When downloaded from the iPhone App Store, the app is 790 MB—slightly smaller than Björk’s Biophillia app at 849 MB.

Most commercial iOS apps rely on AVPlayers from the AVFoundation library, but these only offer non-synchronous playback and could not support the 36-track parallel architecture required here. Hence a bespoke low-level Audio Unit engine was designed with custom players, and the SLAudioEngine proved far more CPU-efficient. The multi-track formats reviewed in Section 2.4 were ruled out due to insufficient controllability in this highly parallel system.

Future listening tests could quantitatively assess sound-quality loss from using data-compressed audio. However, development-team testing indicated that mixing MP3 stem files into a stereo monitoring stream perceptibly outperformed the equivalent process with PCM stems—mixing uncompressed stems then encoding the result to MP3. This is unsurprising given how MP3 data reduction behaves, as it works more transparently on unmixed instrument files. Loudness levels across all files were carefully calibrated for a smooth listening experience; several interesting loudness observations will be reported in future publications.

The app offers a variety of interactive music GUIs. The Stem Fader mode parallels certain apps discussed in Section 2.3 yet remains valuable within this diverse suite. Each GUI behaves differently and may appeal to different users; this aspect will be evaluated in future research. A thorough analysis of Flurry-collected user data will identify which GUIs see the most engagement, supported by focus-group workshops that collect direct user feedback and responses to the built-in questionnaire. Complete user-experience evaluation will appear in later publications.

As shown in Section 2.3, album app releases remain scarce, and establishing a common format now could provide long-term benefits before too many disparate variants proliferate. Although interactive music is in its early stages and will likely evolve through many paradigms—some more successful than others—this research presents a functional model that could grow further. Commercial adoption of a unified format would improve profit margins as releases become more numerous, since development costs would already be amortized, and lead times to release would shorten dramatically.

Opportunities also exist to develop more advanced graphical animations that respond to the sonic experience, leveraging visual programming techniques from the mobile games industry. Interestingly, game developers have found crossfading between disparate music pieces challenging. As Gungormusler notes: “one major drawback of the approach is the lack of smooth transitions between the different selections of musical pieces, which is said to be left as future work” [21, p. 3]. The audio mixing architecture and associated algorithms from this research could therefore be adapted for future game audio systems.

6. References

[1] K. Collins, Game Sound: An Introduction to the History, Theory, and Practice of Video Game Music and Sound Design. MIT Press, 2008.

[2] M. Z. Land and P. N. Mcconnell, “Method and apparatus for dynamically composing music and sound effects using a computer entertainment system,” US5315057 (A), 24-May-1994.

[3] K. B. McAlpine, M. Bett, and J. Scanlan, “Approaches to Creating Real-Time Adaptive Music in Interactive Entertainment: A Musical Perspective,” presented at the Audio Engineering Society Conference: 35th International Conference: Audio for Games, 2009.

[4] A. Berndt, R. Dachselt, and R. Groh, “A Survey of Variation Techniques for Repetitive Games Music,” in Proceedings of the 7th Audio Mostly Conference: A Conference on Interaction with Sound, New York, NY, USA, 2012, pp. 61–67.

[5] H.-P. Gasselseder, “Dynamic Music and Immersion in the Action-adventure an Empirical Investigation,” in Proceedings of the 9th Audio Mostly: A Conference on Interaction With Sound, New York, NY, USA, 2014, pp. 28:1–28:8.

[6] Jammit, “Jammit | Jam with your Favorite Band.” [Online]. Available: http://www.jammit.com/. [Accessed: 06-Aug-2015].

[7] “Reactify,” Reactify. [Online]. Available: http://reactifymusic.com/portfolio/. [Accessed: 05-Aug-2015].

[8] J. Shakhovskoy and R. Toulson, “Future Music Formats: evaluating the ‘album app,’” Journal on the Art of Record Production, no. 10, Jul. 2015.

[9] D. Greeves, “Can The App Save The Album?,” Sound On Sound, Oct-2014. [Online]. Available: http://www.soundonsound.com/sos/oct14/articles/apps.htm. [Accessed: 06-Aug-2015].

[10] T. Redhead, “Composing and Recording for Fluid Digital Music Forms,” JARP, no. 10.

[11] C. Taglialatela, “MPEG-A Interactive Music Application Format (IM AF) Encoder,” Seconda Universita’ Degli Studi di Napoli, Naples, Italy, 2012.

[12] “iKlax multitrack music file format - Get the SDK for encoding and decoding.” [Online]. Available: http://www.iklax.com/. [Accessed: 05-Aug-2015].

[13] Music Minus One, “Music Minus One - musicminusone.com.” [Online]. Available: https://www.musicminusone.com/. [Accessed: 06-Aug-2015].

[14] MPEG, “Interactive Music Application Format.” [Online]. Available: http://mpeg.chiariglione.org/standards/mpeg-a/interactive-music-applicationformat. [Accessed: 05-Aug-2015].

[15] “MOGG Files - Multitrack Digital Audio Format,” MOGG Files - Multitrack Digital Audio Format. [Online]. Available: https://moggfiles.wordpress.com/. [Accessed: 05-Aug-2015].

[16] “Music Software Song Galaxy Online Shop,” Song Galaxy. [Online]. Available: http://songgalaxy.com//software.php. [Accessed: 05-Aug-2015].

[17] “Gwilym Gold.” [Online]. Available: http://gwilymgold.com/. [Accessed: 12-Feb-2014].

[18] M. Grierson, “BronzeFormat,” 2012. [Online]. Available: http://bronzeformat.com/. [Accessed: 12-Feb-2014].

[19] Stems-Music, “Stems is for DJs,” Stems, 2015. [Online]. Available: http://www.stems-music.com/stems-is-for-djs/. [Accessed: 05-Aug-2015].

[20] magicspark, “Flurry Analytics,” Flurry, 28-Mar-2014. [Online]. Available: http://www.flurry.com/solutions/analytics. [Accessed: 10-Aug-2015].

[21] A. Gungormusler, N. Paterson-Paulberg, and M. Haahr, “barelyMusician: An Adaptive Music Engine for Video Games,” presented at the Audio Engineering Society Conference: 56th International Conference: Audio for Games, 2015.