Web-Based Sound Design

Historically, sound design and technology have had a symbiotic relationship predicated on communication and an ability to convey information: Think of the dings as an elevator reaches its destination, or a phone ringing to alert people to a call. That relationship has produced more sophisticated, emotionally rich results as technology has progressed, from the deconstructed synth sounds dotting the stereo soundscape of Francis Ford Coppola’s 1979 film Apocalypse Now and the Brian Eno-composed tranquil start-up tones for Windows 95 to the instantly recognizable, percolating iPhone ringtone.

Still, defining good sound design can be an elusive endeavor. Because the aim is often to create an immersive, unobtrusive experience, the measures of success include hard-to-quantify things such as the emotions provoked. “Nothing’s tangible with any of this,” agrees Adam Pleiman, president and creative director of Sound Images, a Cincinnati-based sound branding and music agency. “It’s either heard, or it’s a feeling or emotion that you’re trying to solidify through the use of sound, or make something real or not real. And if the job’s well done it’s absolutely thankless, because people are like, ‘Well, that was cool,’ and they don’t think of it.”

Web-based sound design is even more of a slippery field. Best case, this audio is integrated into the experience so well that it doesn’t even register with people’s consciousness. At worst, sounds are intrusive and unexpected, an unpleasant experience that jars the senses. “Websites should ask permission from the user before taking over their speakers,” says Mitch Wells, creator of Websynths.com, a browser-based synthesizer with robust capabilities, as he points out how web-based sound design is distinct from other sound design fields. “Everything most websites do should be able to work well without sound, with the sound simply adding to the experience. Video games and films, on the other hand, can assume the user can hear them, so they can make sound more pivotal to the storytelling.”

Still, web-based sound design is an emerging discipline informed by both rigorous research and gleeful bouts of experimentation. As a result, it’s somewhat of an amorphous field, one shaped not only by technological advances but also a combination of user creativity and cultural trends. Online addicts might rarely think much about how the sounds they hear were created, or what purpose they serve, but there are plenty of scholars, developers, designers and musicians giving serious thought to the transformative power of web-based sound design.

What is web-based sound design?

Web-based sound design isn’t interchangeable with the sound design applied to online videos, and it’s only one part of the overall browsing experience, a complementary facet bundled into aspects of both interaction design and user experience. Consequently, web-based sound design can serve multiple purposes depending on the audience and aim of the particular website.

Sites can leverage sound for informational purposes: Inside Abbey Road, an interactive look at the legendary London recording studio, uses audio snippets, interactive studio gear and other effects and voiceovers to educate visitors about composition and arrangements as well as what it’s like to record music there. Sites such as Noisli and Coffitivity are more utilitarian, providing customizable soundtracks of ambient background noise to help increase productivity and focus. RBMA’s own Mixing Board encourages users to make remixes of songs in order to foment creativity and produce varying emotional responses, while the Hover States blog is a curated showcase of cutting-edge, alternative digital design meant to inspire.

Then there’s the beloved Google Doodles, known to pair lively animation and interactive gameplay with imaginative sound or music. The latter elements are frequently the handiwork of Google Senior UX Designer Manuel Clément, a classically trained musician who currently works as part of the company’s Cardboard Virtual Reality team. Besides an interaction design background, Clément brings a knowledge of music theory and love of songwriting and composition to his Doodle work: He plays guitar, piano and keyboards, creates music for video games, events and film with a friend under the name ION and loves tinkering with vintage and modern gear.

But it’s also safe to say Clément’s musical background has permeated his work life. For example, while on the team building Google’s self-driving car, he created the vehicle’s suite of internal sounds from scratch. “I’ve used my musical training for all my career,” Clément says. “All the product design I’ve done – if it’s user interaction, if it’s building a prototype, if it’s coming up with ideas for design, whether it’s visual, emotion or sound – I’m using the same part of my brain. It is all about, ‘What is the resulting tone? What is the emotion I’m feeling when I’m hearing Beethoven’s 7th Symphony when it starts?’ I think there’s a resulting tone, a resulting chord, coming out of every experience of everyday life. When you’re creating user experiences on virtual user spaces or on real products, you’re triggering emotional responses in [people].”

Clément’s Doodle-geared sound design reflects this passionate mindset. In early March of this year, Google celebrated the 105th birthday of theremin pioneer Clara Rockmore with a vibrant interactive Doodle that taught people the instrument by asking them to mimic three increasingly difficult note sequences. After these lessons, players could freestyle-play the digital theremin and customize its sound by clicking on a series of buttons to change the key, scale, octave and waveform.

The Doodle’s attention to detail was superb, from its vintage sepia tones and Art Deco-inspired vibe to the subtle sound dressing: Short chimes sounded after users passed (or failed) a lesson, while the retro-looking knobs and buttons used to adjust the theremin settings made soft clicks and clacks as they were twisted or depressed. These two categories of effects were all Clément: For the latter element, he hit and slapped surfaces around his house and used unorthodox sources, including a toaster oven’s knobs, to create the tones.

He’s applied a similar combination of rigor and whimsy to sounds he’s created for other Google Doodles. For a Zamboni-celebrating interactive game made to look and feel like 8-bit Nintendo gameplay, Clément used Reason to create a soundtrack that recalls the system’s charmingly simple electronic sounds. In contrast, for a Doodle game marking the 66th anniversary of the Roswell crash, he went around his house and sampled sounds – pots and pans thrown together, noises from a pet chicken, even his own snoring – to augment the game. In conversation, he comes alive when talking about this composition; the enthusiasm he has for creating these effects and music is infectious.

On some level, this kind of sound design differs from Clément’s more practical projects – “It’s more playful,” he says, and “There’s no safety issue in using sounds for the Doodle” – although it’s no less serious or thoughtful. “There is user annoyance, right? When I design for [Doodles] as well, I brainstorm with [the team] as far as how loud certain sounds should be. It is very much like designing sounds for games outside of Google, I imagine. So it’s a little bit less scientific. At the same time, it is not like playing around with random sounds. We discuss the context of the Doodle.”

History of Web Sound Design

That deliberate, strategic approach differentiates modern web-based sound design from the early days of the discipline, when a fascination with bells and whistles trumped usability or functionality. “[Web-based sound design has] historically been a very difficult thing to get right,” says James Chambers, co-founder and director of the London animation studio Animade. “There was a lot of overuse in the days of Flash – I remember sites where sound was used for every interaction and rollover. It feels like people have reacted to that overuse and it’s much, much more restrained now, which is for the better.”

Clément remembers those days of sound overload well. In fact, he dropped out of school in twelfth grade in the late ’90s, during what he calls the “gold rush” of Flash, to make those kinds of sites, doing everything from programming the websites and designing interactions to composing music and sounds. “In my experience, basically everybody wanted to have really splashy, flashy Flash websites,” he recalls. “I made a lot of them when I was 18. Basically the idea was, OK, the website starts with an intro and there’s a button called ‘Skip Intro.’ A lot of people that will read this article will laugh at this and smile because it’s this awful thing now, but those were the days.” Mitch Wells also recalls making a sound-enhanced Flash banner for the Discovery Channel TV series Walking With Dinosaurs in 1999, but it was more subtle, playing just three different dinosaur sounds when users touched three separate spots.

While this thirst for web audio seems quaint now, the fervor makes sense for the time. Browsers didn’t have the capability to stream embedded sounds until the mid- to late-’90s, which was when the Quicktime and Live Audio plug-ins emerged, as well as the dominant Real Audio format, and the popular embeddable MIDI format was basic and restricted by sound card limitations. (Other contemporary attempts at hauling web sound design into the future, such as Thomas Dolby’s forward-thinking Rich Music Format, weren’t quite as successful.) Slow internet connections also hampered web sound design, as dial-up modems weren’t exactly ideal for elaborate online audio projects.

In the wake of Flash overload came “the rise of the importance of usability in websites, which means put the user first,” Clément notes. “Audio had a role, but it was more of a user-initiated, opt-in experience at that point, where you might go to a website and just like you do on YouTube, it’s silent until you go and decide, ‘I want to play this music video.’” That opt-in approach has generally stayed intact, even as web-based sound design has been transformed by innovation – first with the introduction of HTML5 in 2008, and then with the 2011 unveiling of the Web Audio API. The latter development has finally eliminated the need for audio plug-ins or third-party software, and enables browsers to play sound directly and seamlessly.

“The Web Audio API is completely changing the game,” Wells says. “[It’s] allowing anyone handy with Javascript to do most anything with sound that they can dream up. There is a big unexplored world of possibilities, and the Web Audio API has made it easier now to invent within than ever before.” It’s certainly been integral to the creation of Wells’ own Websynths.com, which offers over 130 types of pre-loaded tones and sounds, and incredible (and incredibly complex) customization and filters.

What’s Next?

Although Chambers is heartened by the recent release of some “very impressive” web-based software and sequencers – a sign to him that the Web Audio API “is maturing into something stable enough to use for serious applications, which is great to see” – he cautions that web sound design is still a work in progress.

“One of the greatest challenges currently is to create a consistent experience for all users, regardless of the device they are using,” he says, while also noting that the Web Audio API’s performance has been inconsistent across browsers in the past. “Because this is so difficult, I believe sound design is often not as well considered as other aspects of a web-based experience. Sound is still not part of the fabric of the web in the way it could be. Internet culture is still very much a visual one.”

Dr. Kevin Walker, the head of the Information Experience Design (IED) program at London’s Royal College of Art, echoes the latter sentiment. “The web is a very visual environment,” he says, “and not enough people pay attention to sound and the very usefulness and power of sound – partly because we’re living in a house with other people, or we have headphones on or we don’t, so we turn off the sound, or we’re sitting in an office environment with other people and take the sound off. So it’s overly visual, and lots of people are pushing back against that and thinking about the sort of sonic experience of surfing the web.”

Walker cites the rise of online sonification, where data is visualized or interpreted by sound rather than numbers, as one result of that pushback. One of the more mesmerizing examples of web-based sonification (and visualization) is the “Listen to Wikipedia” project, a website that plays a different, distinct tone when entries are modified by users. The cumulative effect was accurately described by Vice's Motherboard as “[Brian Eno’s] Music For Airports but a bit more aimless, more human.”

The Royal College of Art is pushing the dialogue forward by offering a sound design pathway within the IED program. This pathway exists squarely at the intersection of art and design, and is both practice-based and rooted in theory. “The sonic experience of browsing the web is way, way behind the visual experience, but people are starting to pay attention in interesting ways,” Walker says. “The way that we approach that is through experimental design, grounded in cognitive, social and cultural research. My [IED] program is sort of in-between fine art and experimental design. We’re not strictly commercial, doing interface design or user experience or something like that, but we’re not entirely on the fine arts side, either. We’re in-between, and that’s sort of a good place for us.”

The connection between education and web-based sound design is also at the heart of Google’s recently launched, experiment-rich Chrome Music Lab. The landing page features interactive games meant to explain various musical characteristics – including rhythm, chords and arpeggios – that also highlight the capabilities of Web Audio, as well as the user-focused getUserMedia API and the interactive music framework Tone.js. The latter is responsible for such gems as Jazz Computer, a song that blooms when someone scrolls down a webpage bursting with dizzying graphics.

More Web Audio enhancements – such as browser-based multichannel playback – have also encouraged the kind of sonic flexibility seen mainly in video games, which has pleased Sound Images’ Pleiman. “I think back to early computers that could only play a CD or sounds from the app that’s open,” he says. “To know that those ones and zeroes are getting crunched by your computer and not being fed to you from somewhere else where you have to wait on it, that’s really exciting.”

Of course, the elephant in the room that Walker recognizes is the fact that there are younger consumers choosing mobile phones or tablets and forgoing desktop browsers or laptops. “I’d be interested to know how people are exploring web-based sound design in different contexts,” Chambers says. “A mobile experience is much more private than a desktop one.” At the very least, the experience of listening to sound on a non-laptop device introduces new challenges through which to puzzle, from subpar fidelity to calibrating sound to account for both close headphone listening and airings in noisy public places.

Even with this ominous portent, web-based sound design isn’t an antiquated concept or on its way to becoming obsolete. As abstract as the discipline can be, its endgame is remarkably similar to the goals of other, more established sound design fields, which underscores that its premise and future are more stable than we might realize.

“[Sound design] is enriching the experience,” Pleiman says. “It’s making what you’re doing, what you’re seeing and how you’re interacting with what you’re interacting with that much more meaningful. In a movie theater, you’re watching – it used to be stills that are going through a projector that have become motion. And without sound, they would literally just be a high-speed slideshow. What sound does is it makes it real, and it tells you what to feel.”

“Those same tenets apply to a rich web experience, to a rich digital experience. Video games got it right and have gotten to that point, and this is literally an extension of that: ‘What do you want your people who are interacting with you to feel like?’ Music and sound, they don’t ask the viewer what to feel: they tell them. So tell ’em the right thing, and you’ll find their experience to be fuller.”