Game Technology in the Music Classroom: A Platform for Music and Sound Design

Sound and music play a vital role in video games, just as they do in film and animation. Yet their incorporation into mainstream music education has been slow to gain traction. Several factors explain this lag: educators often lack familiarity and expertise with game audio, the practice is sometimes dismissed as populist and unworthy of serious study, and schools rarely have access to appropriate tools and resources.

Creating sound and music for video games can become a meaningful part of music classroom activities. Design patterns in music and audio translate naturally into game development contexts, offering students a practical, creative framework for learning.

Game engine technology—especially design tools that open up those engines to non-programmers—is no longer the exclusive domain of industry professionals. Software like Unity makes previously specialized knowledge accessible to amateur developers. Simpler platforms such as GameSalad offer highly scaffolded environments for game design and straightforward sound asset integration. Meanwhile, accessible media programming environments like Scratch allow students to explore advanced concepts such as generative music and interactive audio with relative ease.

These tools create new learning contexts centered on designing computational artifacts. From a music education perspective, this means fresh opportunities for students to craft musical and diegetic audio components for game worlds. Using video game design (and modification—or "modding") as an educational activity has a well-established track record in disciplines such as computer science and graphic design. Music educators are finding similar reasons to engage with game technology: student motivation, vocational preparation, and interdisciplinary integration.

This chapter does not focus on playing video games or gamifying activities, even if those games target musical skills like aural awareness or rhythmic timing. Instead, it examines how creative music tasks can be enriched by embedding them within the design process of video games themselves.

Three areas of video game audio receive particular attention: sound design, non-linear music, and creative coding. Their diversity underscores the breadth of possibilities in game music and sound. For each area, a corresponding music education case study illustrates what these concepts look like in practice.

Sound Design

Sound design for video games involves deciding where sound will enhance gameplay, selecting or creating those sounds, and integrating them into the game experience. Audio requirements may include spoken dialogue, sound effects, music, and atmospheric backgrounds. Both creative and technical challenges arise, and each offers fertile ground for learning.

In video games, sounds function as media assets: individual audio files that must work together as an integrated whole. Thinking of a sound world as a cohesive assembly of sonic objects has an intriguing intellectual pedigree. Pierre Schaeffer described sound objects as entities with sonic, musical, and meaningful characteristics. Al Bregman portrayed audio scenes as compositions of acoustic components and perceptual streams. Adam Harper characterized music as systems of variables or changing attributes.

Sounds serve to reinforce game events, add depth, and build emotion. Their role is important, yet—as with music for film and other audiovisual projects—sound design is often left until the last minute. Finnish sound designer Joonas Turner noted: "Sound still seems to be the underdog, even though it is one third of the overall immersion and feel of the game." More optimistically, he added: "Lately I have been noticing that people have been getting more into sounds through the current rise of indie games."

At its simplest, sound design involves creating effects through outdoor field recordings, in-studio Foley, sound synthesis, or selecting and editing existing sounds. Sound effect libraries prove especially helpful when realistic sounds are needed but impractical to record—car tyres skidding, for instance. Crafting original sound effects is more demanding and calls for recording, editing, and synthesis skills. Basic tools like Audacity are freely available for schools, and considerable work can be done even with a tablet and low-cost apps.

Given the complexity of building a game from scratch, music education contexts usually have students create sounds for an existing game, replacing its original audio. Most game development environments include demo games suitable for this purpose. Modding games as a pedagogical activity is relatively widespread. According to researchers El-Nasr and Smith, "modders are able to focus on learning fundamental design skills because game engines and their tools eliminate much of the overhead associated with building convincing products." The process of adding or replacing a sound typically involves importing the audio file to an asset register and implementing it to align with the relevant game event or object.

Sounds can exhibit behaviours within a game: repeating, varying in loudness, length, pitch, panning location, and effects levels such as reverb. These controls differ among game engine environments; some also allow additional scripting through a coding interface. Paying close attention to sound behaviours sparks classroom discussions about how sounds indicate actions in the world, how they occupy spatial locations, and what dynamic changes reveal about a sound's source or environment.

There are solid educational reasons for using music technologies to integrate sound and visual media—reasons that include scaffolding, contextualising, and the cultural cachet these technical associations bring. Jonathan Savage reflected on a composer named Alex who, despite a frustrating school music experience disconnected from his interests, went on to become a successful sound designer. Savage concluded: "Alex's work as a sound designer offers an exploration of exciting new notions of artistic practice that integrate rich mixes of subject learning within ICT. This could help us lead music education towards a holistic model of artistic practice mediated through the effective use of ICT rather than traditional or pre-existing musical practice merely done with ICT." One can only imagine the head start a student like Alex would have if sound design were proactively introduced as part of every music curriculum.

Case Study 1 — W Elementary School in Yongin City

This first case study looks at using digital sound design for visual media as a strategy to increase student engagement with music and sonic awareness. The program was reported by Eunjin Kim and involved sixteen elementary students in South Korea. It did not specifically use game development technologies, but it reflected the broader objective of employing visual and interactive media to support music learning.

An after-school program brought together fifth and sixth graders from a small rural school in Gyeonggi Province. Roughly half the children had instrument-playing experience, but none had formal composition training. The program began following the release of a new Korean national curriculum that championed self-motivation and active participation—in contrast to the music appreciation model traditionally followed in Korean classrooms. Kim described conventional Korean music education as "teacher-centered, text-based, and knowledge-delivered which, in turn, emphasized 'listening to music' and 'rote memory of musical knowledge'." The experimental program, by contrast, emphasized a "technology-mediated teaching and learning approach" using software including ALSong, Tunearound, and Movie Maker.

Activities focused on sound and music creation for applied contexts. Kim wrote: "These activities were selected because we deemed them most suited to raising the interest of students of the Internet generation." Lessons included creating cell phone ring tones, sound effects for video, background music for stories, and advertising jingles. Kim described the process in detail:

"Students were encouraged to organize themselves into teams to carry out the class project. After viewing the image shown by the teacher, the students collected and analyzed music materials such as sound effects and sounds of instruments from the available music software. Following this, students were asked to think about the image and the music materials they had chosen to create their music. Once they had created their musical impression of the image, they presented their work to the class. The teacher then provided feedback."

Kim monitored participant responses through pre- and post-program mind maps, interviews, and questionnaires. Students reported enjoying the activities because, unlike previous music education experiences, they were allowed to compose music, use the computer, have practical experience, and create something instantly. After the program, participants exhibited more acute listening awareness of their surroundings and showed enthusiasm for using music technologies in everyday life. They could see the diverse real-life applications of those technologies. These results echo those of earlier studies, including Savage's, that employed media-rich contexts to expand music education beyond traditional boundaries into the broader sonic world.

Non-linear Music

Video games are essentially interactive, non-linear storytelling platforms. While game designers establish a basic narrative structure, the moment-to-moment details—and, in some cases, long-term form—are unpredictable, driven by player actions. This presents a unique challenge: how to deliver an integrated, temporally coherent musical experience when the emotional trajectory and scheduling of events remain unknown in advance. A range of techniques and tools now allow for adaptive, dynamic music that suits the story and responds to user choices.

Karen Collins distinguished two types of non-linear audio in games: interactive and adaptive. Interactive audio involves sounds triggered directly by the player's actions; adaptive audio events are cued according to the ongoing game state, which is constantly evolving. Adaptive music changes as the game state changes—for instance, based on the number of enemies, duration of play, current score, game level, virtual location, and so forth.

Simple approaches to handling unpredictable game duration include infinite repetition of musical material, pausing music after a certain time, and allowing users to choose their own background tracks. These methods work in some contexts, but as player expectations rise, so do the sophistication of compositional strategies and audio system design.

Several authors advocate a matrix-based approach to managing non-linear music playback. In essence, the musical score is divided into component parts that the game engine reassembles on the fly to match the game state. Divisions happen both horizontally—splitting musical parts or tracks—and vertically, segmenting the music into sequential sections. Each part draws from a pool of clips. The horizontal timeline positions indicate bar placement for playback. This matrix of segments forms the basis for algorithmic rearrangements, including repetitions, guided by rules set by the composer or game designer. Those writing adaptive music must anticipate likely combinations and ensure various arrangements work. Typically, parts are composed to allow shifts in dramatic intensity as the narrative ebbs and flows.

Although game engines now offer some music playback capabilities, specialized middleware—such as FMod, Miles, and Wwise—is more commonly used to manage game audio. These systems allow an audio designer to specify playback decisions through relatively intuitive interfaces: randomly selecting alternate clips for variety, adjusting loop counts, triggering assets based on game parameters varying real-time effects, and so on. In music production and computer music circles, tools like Max, Ableton, and Logic regularly handle interactive music systems, but they cannot be integrated natively into a game engine. They work for simulating non-linear processes rather than driving in-game audio behaviour. An exception is Pure Data, an open-source software that can be embedded into a game, most often using the libpd library.

Some games use generative music, where the score is algorithmically written during play. Spore (Electronic Arts, 2008), whose music Brian Eno composed, stands as a well-known example and uses the libpd library. Although software systems like Noatikl 2 support generative composition, building generative systems typically demands dedicated development and integration into the game engine—a topic discussed later in this chapter.

In school settings, music students encounter progressively complex challenges as their compositional and technical skills develop. Creating music with combinatorial fragments can range from simply muting or unmuting parts to real-time reharmonisation over probabilistic progressions. This graded complexity offers a clear pathway for building student skills and understanding of game music concepts. Equally important is selecting suitable games for which to craft music and choosing the right tools for managing audio integration.

Above all, whatever clever techniques and tools become available, the overriding goal is a sound world that reinforces the game narrative, character, and pace. Composer David Kanaga suggests that musicians should approach this challenge by "reading games as scores"—that is, apprehending structure, pace, and internal rhythm, and remaining true to those time-structures when planning the music and designing playback mechanics.

Case Study 2 — Berklee College

Creative Coding

The term “creative coding” (or creative computing) is debated in meaning. In computing circles, it often refers to applying computation in novel ways (Zhang & Yang 2013). For this discussion, it means using computer programming for creative arts activities — assumed to include video game development, sound design, and music composition. In education, a growing movement empowers children by teaching them to program computers, gaining control over them (Rushkoff 2010). As one creative coding instructor put it: “When kids realize that by learning to code, they can control computers and make them do their will, they sort of pause for a moment in shock. Then they smile” (Fredrickson 2014, n.p.). Giving students the skills to control their computers for responsive music—such as interactive game play—can be equally empowering.

Game engines frequently support coding to extend functionality or simulate external interactions by users or game elements. These programming systems often use interpreted (scripting) languages, making code-driven interaction dynamic. For example, the Unity game engine can be scripted in C# or Javascript, and the Wwise audio middleware can use Lua. The differences between these languages are subtle; for adding functionality to sound or music processes, any language works. Interpreted languages are dynamic but lack speed, so scripting handles parameter control and function triggering rather than audio signal processing. Several game projects used Pure Data (Pd), a visual programming language. Commercial music software also adds scripting and visual capabilities — Ableton Live uses Max for Live, and Logic X has the Scripter MIDI plugin. When using Pd or Max for Live, users must know these operate independently of the game engine or middleware, requiring communication between software systems to synchronize activities.

Scripting languages in game development are general-purpose; they automate any aspect of the game, not just music and audio. In particular, game logic and character behavior—often called artificial intelligence or game AI—are frequently driven by scripts.

Scripting languages can create algorithmic music generated on the fly during gameplay, providing significant musical variability in response to changing conditions. Musical processes (algorithms) are coded in a language and execute during gameplay to produce part or all of the score. With increased computing power in modern game hardware, algorithmic music is expanding. A notable use in a major game title is the score for Spore (Electronic Arts, 2008), which used note-based algorithmic music. Composer Brian Eno collaborated with Electronic Arts programmers in the Pure Data environment, finding that generative modules based on music perception and functional harmony worked best (McLeran 2009). Non-linear and algorithmic processes form a continuum of dynamic music organization operating at various granularity levels.

Another example is Escape Point (Prechtl et al. 2014), a non-commercial Unity game with extensive algorithmic music. Game logic was coded in C# in Unity; musical logic used Max, controlling sample playback via MIDI and communicating with Unity via UDP. Developers coordinated the game’s emotional narrative with the music’s expressive character by monitoring danger—based on enemy proximity—and passing this value to the music engine to vary musical features accordingly.

Educators aiming to incorporate creative coding in game music and sound projects must select suitable software environments matching students’ capabilities. Several game-oriented programming environments designed for education exist, though their audio support varies. Examples include Alice, Scratch, Stencyl, Gamefroot, and GameSalad — all worth considering in educational contexts.

Case Study 3 – Scratch and Music

Scratch is a media-rich programming environment for young people and inexperienced programmers (Resnick et al. 2009). The current web-based version is widely available. The Scratch website enables creating, sharing, discussing, and remixing others’ projects. Code is written by assembling visual code blocks, which reinforce code structure with shape and color while minimizing syntax errors by reducing text typing. A code example is shown in Figure 3.

Scratch is a pedagogical tool rather than a professional production environment. Developed at the MIT Media Lab, it builds on constrained programming ‘microworlds’ for exploring ideas in code (Papert 1980). Its creators state: “Our primary goal is not to prepare people for careers as professional programmers, but rather to nurture the development of a new generation of creative, systematic thinkers who are comfortable using programming to express their ideas” (Resnick et al. 2009:n.p.).

Enabling interactive game creation is a core Scratch goal. New projects start with a virtual canvas and a cat sprite that can be animated. Editing categories include sprites (characters), scripts (code), costumes (graphics), and sound (audio). Though music and sound are prominent in simple Scratch animations and interactives, its value here lies as an environment where game design, sound design, and adaptive music can all be tackled by users with minimal prior coding. Such environments allow music students to transcend being simply sound makers responsible for audio; they become game makers overseeing design, graphics, and interaction holistically.

While Scratch’s audio abilities are modest, an enthusiastic educator community has fully leveraged the computing-and-music combination. Leading this is the Performamatics project at the University of Massachusetts Lowell, funded by the US National Science Foundation. A key aspect is combining computational thinking with music-making in Scratch, representing musical processes as algorithms in code (Greher & Heines 2014). Though many interactive music-and-media activities arise from this program and the earlier Sound Thinking undergraduate course, the focus is on physical computing and instrument building. Another music project source is Scratch team member Eric Rosenbaum (aka ericr in the Scratch community), plus examples more directly linking music and sound to video games.

The Me Bee Flower Demo game by Alex Ruthmann (2013) is a Scratch project with dynamically controlled recorded sound effects and algorithmically generated background music (see Figure 3). The game uses three sprites (bees, flowers, and the player “me”), each with associated sound effects and behavioral code for walking, flying, stinging, picking flowers, etc. The stage includes code tracking game status, continuously playing background music, foley sounds, and winning/losing themes.

Figure 3. Me Bee Flower Demo canvas (left) and generative music code fragment (right).

Environments like Scratch and projects like Me Bee Flower demonstrate that tools and processes exist to help music students engage with all aspects of video games, including dynamic sound effects control and non-linear music. Although code may seem an unorthodox music representation, it may be no harder to grasp than other notation systems for students with little formal training. Parallel to traditional music training, representing music as code in a video game context enables systematic idea expression and creative design engagement. Professional tools like Unity offer similar interaction, but their sophistication can impede access for inexperienced students (and teachers). Scratch exemplifies a growing number of accessible platforms supporting creative coding that music educators are increasingly embracing.