Magic as a Metaphor for NUI Design: Part 1

Windows, Icons, Menus, and Pointers (WIMP), the de facto Human-Computer Interface (HCI) of today, is just starting to give way to other disciplines in HCI. These include Natural User Interfaces, Tangible User Interfaces, and Speech Recognition - all of which have many things in common including the concept that they give ordinary gestures, speech, and objects behavior that is super-real.

Natural User interface (NUI), Tangible User Interfaces (TUI), and Speech Recognition (SR) have been treated as different concepts but as the term NUI becomes mainstream it seems to be subsuming TUI and SR. Soon NUI will stand for any human-computer interaction that emphasizes gestures, touch, objects-are-information, or speech.

What is fascinating to me is how similar NUI technologies are to magic and how magic can be used as a design metaphor for NUI. How is magic a design metaphor for NUI? It's in the method in which we elicit effects and accomplish tasks. Magic is supernatural, beyond nature. NUI is technological but it's also super-real.

The term super-real is used to describe the idea that NUI design borrows from the observable qualities of our interactions with real-world objects (i.e. physics), but not precisely. NUI favors usability over precisely modeling the physical world. Good NUI design makes visuals, and even devices, respond and act like natural objects to a point, but then modifies those visuals to provide afforances and user feedback that is most convenient for the user. For example flicking an object across a multi-touch table shouldn't cause the object to fly off the visible screen on to the floor. It should bounce off the edge to land an inch or so away from that edge regardless of how hard its flicked. That's not natural, but it's not exactly magic either. NUI doesn't precisely map to the natural world and, in fact, we don't want it too. We want NUI to be super-real; to go beyond what is natural and to me that sets up the possibility of using magic as a metaphor for designing (not explaining) NUI.

As Arther C. Clarke said:

Any sufficiently advanced technology is indistinguishable from magic.

The kind of magic I'm talking about is not folk magic or the art of illusion, but the modern fantasy fiction concept of magic. Magic in fantasy fiction has captured our imaginations and has shaped the Western concept of magic over the past 100 years. Our understanding of magic in fantasy fiction comes from original works by H. P. Lovecraft and J.R.R. Tolkien, and all the derivatives of those works such as the role playing game Dungeons and Dragons, video games like World of Watercraft, and wildly popular fantasy books like the Harry Potter series.

Where magic has spells and charms, wands, crystal balls, magic mirrors, and books of magic; NUI has speech and gesture recognition (spells), remotes with accelerometers (wands), multitouch surfaces (crystal balls and magic mirrors), how-to books (books of magic), and so on.

In the next installment to this series of articles I'll talk about how Speech and Gestures can be modeled on the interactions a practitioner of Magic (fictional) would have while casting spells.
Series Links

1. Magic as a Metaphor for NUI Design: Part 1
2. Magic as a Metaphor for NUI Design: Part 2, Casting Spells
3. Magic as a Metaphor for NUI Design: Part 3, Enchanted Artifacts

Disclaimers:
1: This post is subject to constant editing.
2: I don't believe that magic, ghosts, or anything else that is supernatural is real. I'm just using the fantasy concept of magic as a metaphor for designing NUI.

Update June 19, 2009

This set of posts on magic and design were inspired by a conversation (via email) that I had with Tim High back in February of this year. In an email Tim said the following, which has had a major impact on my thinking when it comes to software development and design:

"You might find this amusing: like all computer geeks, I've spent my share of time drooling over Lord of the Rings and other fantasy books imagining I could cast spells by sneezing, but of course realizing that that's as close as I'll ever get to real magic. But in the last few years, I've changed my mind - magic is whatever you can do in the real world, especially with the new technology that's popping up everywhere. I think we're all studying at Hogwarts these days. Just like at Hogwarts, everyone is a magician (as opposed to other fantasies, where magicians are a mysterious breed). But in Harry Potter's world, your power is related to how hard you study and how creative you can be.

This really hit home for me when I got my iPod Touch. Suddenly, I realized that I had the modern equivalent of a magic wand in my hands. I learned the basic spells immediately: play music, look at photos of my friends and family... when I downloaded the first free "flashlight" app, I thought it was ridiculous... their "flashlight" was just an all white screen. But I take a taxi home every day at night, and with my "wand" I was able with a single flick to cast a "light spell" and read in the cab on the way home. I use it to look for things under the bed, look at the car engine, whatever. Then, as long as you're on the internet, you have a magic spell that can whip up information on just about anything (Google and Wikipedia). Various forms of telepathy: cell phone, email, IM... and on and on. It's amazing how much this and other 21st century technologies have perfect analogies to Harry Potter magic. As far as I'm concerned, there's no difference."

It's important to recognize the people that inspire us, the email by Tim inspired me to write this series of articles. This extract from Tim's email is posted with his permission.