Audio Feedback and Calm Computing

I've been reading and learning about non-speech audio feedback and how it might be used in NUI. A particularly good resource on the subject is a book that was being written by Bill Buxton and others in the early 90's, but was never finished called Auditory Interfaces: The Use of Non-speech Audio at the Interface - the unfinished book is on-line and free to read.

There is a lot of ways to slice and dice the topic non-speech audio feedback, but one way of looking at it is in terms of signals and data representation.

Non-speech audio feedback as signals would include the various chimes and beeps your computer makes. These are called Audio Icons or Earcons. One of the things I'm interested in is what is called "source" sounds or sounds that provide clues as to the function or operation that is causing the sounds. Source sounds are analogs drawn from real-world sounds. One of the best examples, in my opinion, is the sound that Apple Mail makes on a Mac or iPhone when sending email. When Apple Mail successfully sends an email you hear a small swoosh sound like a bottle rocket taking off. It's not loud or intrusive but it is very distinctive and it immediately communicates to the user that an email has been sent. Another example of a source sound is the sound of a door closing or opening when one of your "buddies" on your favorite instant messenger client logs off or on.

Using sound as data representation is also very interesting. What, for example, would the stock market sound like if its ebbs and flows were converted into musical notes? Using sound to represent the shape and contour of data may sound a bit weird, but there are lots of really useful applications.

I personally like the idea of using non-speech audio feedback for what Mark Weiser called "Calm Technology". Calm technology, or calm computing, is some type of feedback about the running state of a system which is manifest on the "periphery" of our consciousness.

The example of Calm Technology presented in Mark Weiser and John Seely Brown's seminal article on the topic, was a long string of plastic hanging from the ceiling in the corner of the office. The end of the plastic string at the ceiling was attached to a spinning motor whose speed was determined by the amount of data flowing over the company Ethernet cables. As the motor turned, the string would naturally whirl in circles either very slowly, when the Ethernet was not busy, or quickly like a tornado when the Ethernet had a lot of data streaming through it. Most of the time workers in the office wouldn't pay much attention to the plastic string twirling in the corner only taking notice when it started spinning quickly. This would be an example of a visual Calm Technology. The use of non-speech audio feedback for Calm Computing is really interesting.

Have you ever had a problem with your car that you first noticed as a strange sound? Although we are not aware of it, we tend monitor the running state of our cars by the sounds they make. The sound of the engine, the sound of the tires on the road, the sound of the air flowing through the vents. When our car starts to make a new sound, something its never done before, we take notice. What if an ambient sound, something subtle and even enjoyable, could be assigned to the running of your computer?

Personal computers are running dozens of processes all the time. Most of those processes are for good, but sometimes bad process (e.g. viruses) are also running. How does the average person know if a virus has invaded their computer. There is generally no way to tell except to run a virus checker and (based on recent experience) virus checkers don't always detect a virus. But what if, like some new sound made by your car, you could tell that something was different about your computer by the sounds it makes while running? That could be very useful.

Imagine that every process running on your computer had a unique croak, chirp or trill - the sounds of frogs, crickets, and cicadas of a small pond at dusk. If every process had a unique croak, chirp, or trill - a sound that is the same every time the process is run - our computers would have a kind of natural ambient pond-like sound when it ran. At first we would take notice but after a short time the sound would settle into the periphery of our awareness so that we would only take notice when a new, and unexpected sound, was introduced. If we just installed some new software a new sound would register when the software was installed and become a part of the natural and healthy ambient audio rhythm of the computer. If, however, some new process - one we did not intentionally install - was introduced such as a virus, the new pond-sound (i.e. croak, chirp or trill) would be out of place and stand out. We might take notice and wonder, what new process is running?

As an experiment, play the YouTube video below but turn down the sound so its very quiet and minimize your browser so you are not looking at the video, only listening to it. After a few seconds start doing something else on your computer like reading some other article and just let the sounds of the pond sink into the background. It's a nice calming collection of sounds and you can quickly tune it out and focus on other matters. If a new sound was suddenly introduced, however, you would probably take notice. The sounds in this video are constant, nothing really new is introduced, but try to imagine a new kind of croak, chirp or trill suddenly making an appearance.