Why You Won't Find NUI in the Wikipedia

Although the term "Natural User Interface" (NUI) has come into its own when describing a different paradigm for human-computer interaction (different from command-line and GUI), getting a basic entry in Wikipedia on the subject has proven to be a real challenge.

Multitouch is a subset of NUI and so the proper definition of the topic on Wikipedia is important to me personally. Since I learned of the term (i.e. NUI), I've thought it odd that the Wikipedia entry on the subject remains in dispute. Bill Gates has even used the term for goodness sakes! Twice!

The problem isn't that good people can't agree on its definition (I'm sure that's possible after a few hundred edits); the problem is that the Wikipedia editors keep removing entries defining NUI. Why? The Wikipeida editors are pointing to the removal code G11: Unambiguous Advertising and Promotion. It seems that the folks who coined the term are to blame, the sinister NUI Group!

Yep, because there is a company named NUI Inc., which created the the open source community called NUI Group (I've always found this confusing myself) the attempts to define NUI on Wikipedia is seen as a transparent attempt to promote the commercial company which uses the term in their name. Or is it the open source community?

What's interesting is that this same criteria has not been applied to GUI. For example, there is a company called GUIdesign.com and GUI - Designers and even a personal/professional web site called GUIGuy.com, but there doesn't seem to be a dispute about the entry for GUI on Wikipedia. This smells like an evil plot!

All kidding aside, it would be nice if we could get a NUI entry into the Wikipedia. The NUI Group has its own wiki with what I would call a non-laymen definition of the term - its ambiguous and rambling which is always the way practitioners define their own topic area.

I thought about writing an entry in Wikipedia on NUI myself, but the thought of putting all that work into it just to have an editor decide that my entry is a G11, makes the idea less appealing. Still, perhaps there are others who can make the time and have the inclination to fight back and get NUI a permanent page on Wikipedia.

Eventually, the term NUI will have a page on Wikipedia, but in the mean time your average Joe Sixpack is going to be hard pressed to find a concise definition of the subject.

Update Later the same day

Well I just posted this blog entry this morning and already Paul Dawson has taken up the challenge by seeding the NUI entry in Wikipedia with an excellent layman's definition. I hope that others will take the time to add to Paul's work and flush out the Wikipedia entry. Thank you, Paul!

(Note to NUI Inc.: Please resist the urge to post links to your company or to claim coinage of the term for now. We want this one to stick and can give you credit after its been flushed out and remains active for a couple of months. Thanks!)

The Irony of Minority Report

The movie Minority Report has become an iconic example of NUI for the masses. The irony is: While Minority Report has brought a lot healthy mass media attention to NUI, its generally seen by practitioners as being a horrible example of a NUI itself.

Jonathan Brill has published a really fun and interesting deconstruction of Tom Cruise's human-computer interface. My favorite part of the blog is is when Jonathan is talking about how Tom Cruise is standing there waving his arms around.

"While typing and mousing might cause mild fatigue, making your upper arms defy gravity for hours or even minutes on end is straight up painful. In fact, this position is so painful that CIA used this "Stress Position" to break Guantanamo Detainees."

In the end, Minority Report will be seen as having a positive impact on NUI because it has elevated awareness of the paradigm, so we should not be too critical. It's just something we have to deal with, or as Nathan Moody puts it:

"All of our project pretty much start off the same way, in which we say, 'Yes. We've seen Minority Report.' - after we get through that unpleasantry we go through the regular design process .... "

Two Videos You Must See

Thanks to my friend Jonathan Brill for his post about videos from UX Week 08 where he points to two videos you really have to see if you are, as he puts it, serious about multitouch and NUI.

The first video is by Darren David and Nathan Moody of Stimulant. It provides some excellent advice about designing multitouch systems.

More importantly is the video of Dennis Wixon, the Usability Research Director on Surface. He provides an excellent analysis of what NUI is, and why its important. At the end he goes on to talk about what comes after NUI, which is XUI or Organic User Interface. It's chalk full of incredibly useful insights and its a video I plan to watch a few more times because its so important.

If you've followed my work prior to multitouch and want to understand why I abandoned 14 years of hard work for a new career in experience design and development, Dennis Wixon's video answers that question. It's because I want to be a part of the next major shift in computing, NUI.

The State of the iPhone is Strong - Very Strong

TechCrunch published an excellent editorial about why the iPhone will continue to dominate the smartphone market as THE mobile device to have.

As I mentioned in a previous post, its the iPhone ecosystem that makes the difference not just good design. I think the TechCrunch author believes as I do that the ecosystem established around the iPhone is nearly insurmountable obstycle to real competition against the iPhone. As the article puts it: 21 million phone sold, 35 thousand applicaitons and, very soon, 1 billion downloads.

Now to be honest my iPhone, which is the first generation, is falling a part. I can't use it without speaker phone, ear bugs, or my Jawbone bluetooth headset because ear-speaker is broken. My iPhone has frozen-up on me a dozen times, I've lost data, and the reception is horrible. That said, I wouldn't want any other phone (except maybe the iPhone 3 coming this June). I'm not loyal to Apple I just can't live without my iPhone.

Punchcut: Considerations for Designing Touch UI

Punchcut, a design company, has released a short video called "5 considerations When Designing Touch Interfaces"

Here are the five considerations as presented in the video:

1. Design for Immediate Access: When everything is touchable a streamlined interface help user navigate quickly.

2. Tap is the new click: Gestures must be intuitive. Elements must respond as user expects.

3. Clear mental models create a window of perception that adds dimension to the touch experience.

4. Design for real hand sizes: Fingers come in all sizes so ensure that design is for real people.

5. Touch feedback is key: Users are mobile and often distracted. Clear feedback reinforces what is important.

The end of the video goes on to say (and I'm not sure if this is part of Consideration 5 or just a post script):

"Touch is merely one dimension that brings a human side to technology. An insightful experience will create deeper connections to the people and the content we care about most."

So what do I make of all this? Sadly, not much. While I'm sure that Punchcut is a very talented design company they seem to be demonstrating the art of stating the obvious.

Update Later that night.

I received a comment on this story from Joe Pemberton of Punchuct who said...

"Richard, your readers might enjoy the full article, rather than just the visualization of these considerations. It provides a bit more meat on the bones. Read the full write-up: Design Considerations for Touch UI"

The article Joe refers to is really well written and succinct. I also think it's spot on and very useful. While the video is visually interesting its message is pretty weak. The article fills in the details that are missing and it's well worth a read. Nice work to the author Jared Benson!

Update April 23rd, 2009

Josh Blake deconstructs the video itself and shows that there is a lot more to the presentation than I thought. It's actually not just flash and bling, the video is supposed to demonstrate different gestures. Anyway, Josh does a nice deep dive on the topic if you are interested.

Microsoft Surface in Health Care

Using Surface applications as an extension of a larger system, as opposed to being used in isolation, is central to the work I'm doing at CSG.

The idea is that surface is one of many "channels" through which clients, customers, patients or whatever, can access and manipulate their own personal information as well as collaborate with subject matter experts.

An excellent example of this kind of application is the integration of HealthVault and Amalga with Surface. HealthVault and Amalga are patient medical records systems designed to be accessible by both doctors at clinics and patients from home.

This video shows how a doctor can use the surface for a consultation with a patient and how the patient can review their own medical results and data, including information provided by the doctor during the consultation, later at home. It's a very powerful use of Surface in the larger context of a multi-channel solution and one has enormous potential across industries. Thanks to TheHealthCareBlog.com for this story.

Why Mobile Windows 6.5 Won't Matter

I'm not a Microsoft lover or hater - I own and develop applications for both the Microsoft Surface and (very soon) the iPhone - but hearing that Windows Mobile, which I believe to be a really great smartphone platform, is going to support multitouch with capacitive touchscreens just doesn't excite me.

Why? Because Apple has already blown the doors off of multitouch on the mobile and no one is going to come close for a long time to come. Not because Apple will continue to make a superior product, but because they have established an enormous ecosystem that is, in my opinion, simply unstoppable.

Take a look at the App Store which is quickly approaching 1 billion downloads. 1 billion! There are more than 25 thousand applications to choose from. That's redicioulus. How will any platform ever catch up? I don't think they will for a long time to come. It's not the hardware that matters its what you can do with it.

The iPhone is not going to continue to dominate the touch-mobile simply because its pretty: its going to dominate the touch-mobile space and become a big player in the World mobile arena because Apple was smart enough to make it easy for people to publish software applications and, more importantly, for people to download them directly to their phones.

RIM and Microsoft and Nokia can come out with their own versions of the App Store, but it will be too little to late. You had your chance guys and you caved into the mobile carriers and blew it.

Multitouch in the Classroom

Back in October of last year I blogged about the SMART table, which is a multitouch table focused on the education market. I don't know much about this device, but I was delighted to find the blog of a teacher who is going to share his experiences with it in the class room.

In his first blog on the subject, Tom Barret, raises some great questions about the sustainability of the device with a limited number of applications in a classroom of technically smart kids with a limited attention span. In his second blog post he talks about receiving it and not being able to hook up the power because the Canadian company that makes it, forgot to send the specialized keys that open on the control panel.

Tom Barret is a good writer and even if you are not all that interested in education his blog is fun to read. It will be interesting to hear how the kids in his class react to the table, but more importantly whether or not the table is an effective learning tool not only in the first week or two, but after a couple of months.

Epson X-Desk: Another Multitouch Table

Engadget is reporting that Epson, the folks that make printers and projectors, have released a multitouch table similar to Microsoft Surface. I love these big, horizontal multiouch surfaces so I was excited to read the article which includes a video and a German guy who says "Und" a lot.

The thing I like most about this table is the fact that they have integrated speech recognition into the platform. Speech recognition and multitouch make for a very powerful combination - much more powerful than speech or multitouch alone. The thing I don't like about it is that its tag recognition systems appears to be limited to 16k unique IDs. That's not enough to make it practical for the really huge installations we are working on at CSG. For example, if you want to give all your customers a unique tag and you have several million of them then a 16k tag isn't going to cut it.

Apparently there is no published price and there doesn't seem to be any information about the type of operating system or SDK used to program these 52" tables. I'll see if I can get some more info from the company and report back later.

Update April 24, 2009

Here is another video of the X-Desk complements of C|Net UK.

Update April 26, 2009

Josh Blake wrote an interesting post criticizing X-Desk and assuring the Microsoft Surface team that they "should not be afraid of X-Desk at all". I haven't seen the X-Desk in person but I do know that Surface is well-made and that it will be hard for anyone to beat it as a multitouch table designed for commercial installations.

Windows 7 RC on May 5th

Apparently Microsoft leaked information about the Release Candidate (RC) of Windows 7 again and this time we have a date, May 5th 2009. There was the leak in February about the final release being around end of the year, and another leak about the RC at the end of March.

Normally, I could care less about a beta (that's what an RC is really; an advanced beta version) product from anyone, but when it comes to multitouch and Windows I pay attention. Why? Because as I've said before Windows 7 is going to take multitouch mainstream in the same way that iPhone has, only bigger. I don't mean to say that everyone will be using Windows 7 as of January 2010 or that even the majority of the world will be using it by January 2012. No, adoption may be very slow, even if the product is very good. It's a bad economy. Vista was less than hugely successful. Full adoption of Windows 7 may never happen. Perhaps Windows 8 will do better.

But the level of success enjoyed by Windows 7 isn't really important from an industry perspective. What is important is that multitouch will become a first class citizen for the nearly ubiquitus Windows operating system. The same line of operating systems used by nearly everyone (90% - 95%) of the World. That's mainstream. It doesn't get any more mainstream than that.

With mainstream adoption will come lots of opportunity for people to do really cool things with multitouch and probably some really lame things as well. The technology industry will make some big mistakes. Multitouch will suffer a lash-back around 2013 as people realize that it doesn't make everything better. But after expectations for multitouch fall and rise again to meet actual capabilities, multitouch will be everywhere and it will be the kind of magic technology we all take for granted.

By 2015 this blog won't exist because talking about multitouch in a blog in 2015 will be like having a blog today that's dedicated to the discussion of the keyboard and the computer mouse. It would be kind of dull and silly. All of that because Microsoft chose to make multitouch a part of Windows 7 in 2009. Think about it.

What Industry Will Apple Try to Dominate Next? Video.

When it comes to electronic entertainment you pretty much have the four big industries: music, movies, television, and video games. Apple was very successful with iTunes and the iPod making them a dominate player in the music business. iTunes has expanded to movies and television to some degree but for the most part they are not calling the shots in that industry yet. In video games, iPhone has become a surprisingly successful player selling or giving away several thousand titles from their App Store and as Trip Hawkins of Electronic Arts says the iPhone and iTouch is “freaking out Sony and Nintendo” the 800lbs gorillas in the gaming industry.

Almost overnight Apple became a dominate player in the music business. The next night they completely changed the mobile phone business. The following night they became a major player in the video gaming industry. The question is: What’s next?

You’ve probably read about the 10” screens that Apple has ordered and the huge order for more memory from the same company that supplies them with iPhone and iTouch memory. Some have speculated that memory is for a 32 gig iPhone, but I think it’s intended for the same device as the 10” screen.

It’s my theory – and that’s all it is – that the 10” screen will be a new video platform. Where the iPod came to dominate music players and the iPhone has become a seemingly unstoppable force in the mobile phone industry, the 10” mystery device will be Apple’s foray into a portable video game and video movie players.

Think of how much fun playing video games would be on a 10” touch screen and how powerful the the App Store is for delivering a diverse and unlimited number of games from the unclean masses. The video gaming industry, thanks to iPhone and iTouch, is becoming an open playing field no longer limited to the select companies that can afford to spend hundreds of thousands, nay, millions developing games. For an entry fee of $75.00 (the cost of a iPhone developer license) anyone can develop a new game and submit it to the App Store. And according to statistics released by Apple recently, most of the new games are approved and distributed.

But the 10” mystery device is not just for gaming. No. It’s also for downloadable movies and television. I’ve watched quite a few shows on my iPhone: the experience is convenient because I can take my iPhone anywhere, but it would sure be a lot better on a 10” screen. Why haven’t we been able to download movies and TV shows directly to our iPhones and iPods? What’s the hold up? Sure you can download them to iTunes and then transfer them to your handhelds, but what’s the fun of that? I want to download movies and TV shows directly to my 10” portable mystery device. The 10” mystery device will allow you to download your favorite movies and TV shows directly, rather than having to use your home or laptop computer as an intermediary.

I’m pretty sure Apple is not making a “netbook”, in fact, Apple would never do something like that because netbooks are - in my opinion – a joke. Netbooks are too big to stick in your pocket and too small for productivity applications (have you ever tried to write a document on a 12" screen? It sucks) . No. It’s not a netbook it’s a multitouch tablet. And I tell you another thing: the 10” tablet is not intended for productivity applications any more than the iPhone was meant for word processing. The 10” tablet is going to be for playing multitouch videos games and watching movies and, of course, for surfing the World Wide Web. But they won’t call it a tablet any more than they will call their notebook computers (i.e. MacBooks) notebooks or their smartphone (i.e. iPhone) a smartphone or their music player (i.e. iPod) a “MP3 Player”. No, Apple will call it the iVideo, iTouch or iSlate or something else, but a multitouch tablet computer it will be. The new device will support web browsing as well – why do you think the Safari web browser supports multitouch? Multitouch support in the Safari web browser is not just for the iPhone or desktop computers with touch screens to come, it's for this new 10” multitouch tablet.

But the real money is in video and that’s where Apple will cash in just as they’ve done with iTunes and the App Store. Oh, yeah and if you are a Hulu.com fan like I am then your bound to be disappointed because the 10” tablet is not going to support Flash or Silverlight. No, it will support QuickTime and perhaps a new proprietary video player specifically designed for Apple’s new on-line movie service. Watch out Sony, Miramax, Hollywood, and Touchstone Pictures: Here comes Apple.

Update 20 min later

Really weird but I just read a blog post by John Lianoglou at Uncarved which says some of the same things I did in the above post, misses some points (like the whole Video game and movie focus), and identifies some things I think are spot on (i.e. the vertical port for normal desktop usage). It's a great read. I especially like the part where he says, "We all know that whatever Apple does ultimately announce will end up making the product I’ve described utterly pedestrian; that’s just what they do." I feel the same way about my predictions, John.

Visual Planet's Transparent, Holigraphic, Multitouch Displays

Visual Planet is a really cool display company that makes and sells multitouch displays. They have some awesome technology that allows you to have a transparent sheet of glass up to a 100" that provides crisp images in the foreground projected by a holographic lazer which can be more easily hidden than a normal projector.

They also provide a capacitive touch technology for both their transparent displays as well as LCD screens that allows multitouch. Visual Planet has used its technologies for store front windows, stand-alone kiosks, and even trasparent walls with projections and multitouch capabilities. Check out their videos for a truely mind blowing experience.

Windows 7 Multitouch API Changes

Thanks to Alon of the Sele Group for a heads up on changes to the Win32 API changes in Windows 7 for multitouch. Apparently three multitouch events were consolidated into one with some other changes that will break Windows 7 multitouch code written to prior to the Release Candidate (RC).

It's a remeinder that until Windows 7 is locked down and shipped coding to it has risks. Hopefully the API will be forzen soon and the WPF implemenation will be published (although that too will most likely go through changes before final release).

Dasher: A new text input system for multitouch?

The standard typewriter as we know it today has been around for about 100 years and in that time, as an alternative to pencils and pens, it has served us well and will continue to do so for another fifty year or more.

But typewriters are strongly dependent on tactile feedback and raised surfaces something that multitouch surfaces today don't support, so I'm always on the look out for alternative mechanisms of text entry for multitouch systems. The problem is there just are not that many promising alternatives. Swype comes to mind as one of the better alternatives.

Well thanks to pureelite I've found another alternative, Dasher. Dasher uses navigation as a means of text entry and it seems to work pretty well but has some obvious drawbacks. It requires visual attention; you can't navigate-by-feel with Dasher like you can type-by-feel on standard keyboard. Also, its a bit buggy (the Java version I tried out) and could use an extreme makeover ascetically. Still Dasher shows promise and should be dismissed to quickly.

The hard truth is that none of the alternatives to touch typing matches its speed or broad acceptance, so standard keyboards will be the norm for a long time to come. I actually really like the FrogPad which I own (complements of FrogPad's CEO, Linda Marroquin) and while you can type as fast on a FrogPad as a standard keyboard (for most people anyway) general acceptance is a challenge. Still, I recommend that HCI folks look into FrogPad as its design is really excellent.

So, for now at least, the standard keyboard is still the best we can do and a mechanical one, as opposed to a soft keyboard, is a superior experience. Perhaps we should just get used to the idea that when it comes to text entry, its hard to beat the keyboard for speed and efficiency.

Android 1.5 Early Look: No Multitouch SDK

The Google Android developer site announced an "early look" a the next version of the Java-based smart phone SDK. What is conspicuously missing in a multitouch SDK.

You may remember that buried deep in the Android OS, Luke Hutchinson discovered support for multitouch.

When people realized that multitouch was actually possible with Android, but was not officially supported, another story broke claiming that Google had not implemented multitouch at the request of Apple.

Since then the hack to add multitouch to Android has been published so all Android developers can take advantage of it (assuming they are programming to hardware with a multitouch screen).

As an example of just how powerful multitouch can be on Android, Endgadget broke the story about Elan Microelectronics video demonstration of their own multitouch hand held technology, eFinger, Sunday which is broadly reported as being implemented on Android. ( Remember Elan is suing apple for infringement of their multitouch patent.) I was especially impressed with the soft keyboard which combines the pop-up key animations seen in the Apple iPhone with swiping gestures. It makes me wonder if Elan Microelectronics has a cross licensing deal with Swype - Swype claims IP rights over swiping keyboards. But that's another story.

So with all this out in the open and demonstrations of just how cool multitouch on Android really is, why wasn't multitouch support included in the preview of Android 1.5? Perhaps they didn't have time to add it, or maybe they are still honoring a request not implement it by Apple. Either way its a big omission considering the fact that touch screens on smart phones has become THE must have feature.

(thanks to Phandroid for the tip off about the Android 1.5 early look announcement)

Multitouch Barcelona

Multitouch Barcelona has some cool videos of a giant multitouch installation they did for the Red Bull Music Academy 08. I love the giant space invaders game and the physics demonstrations. Nice work guys!

Multitouch Barcelona describes itself as "a recently born interaction design group that explores natural communication between people and technology. We design experiences that merge real and digital into a creative environment where people are invited to touch, play, move, feel as they do in the real world."

The Multitouch Tug of War

Well if you haven't heard about it by now, Elan Microelectronics has filed a lawsuit against Apple which says that the iPhone, iPod and so on violate a patent that Elan was awarded for multitouch more than 10 years ago.

The Elan patent seems pretty straightforward. Elan already succeeded in forcing Synaptics, the leading touchpad OEM, to the bargaining table for a cross licensing deal based on the same patent they are using to sue Apple.

Will Apple cave and do a cross licensing deal with Elan? What about Microsoft, Palm, Nokia, and every other vendor that wants to implement multitouch? Who owns multitouch? Is it Elan? Is it Apple? Is there too much prior art? These questions are probably going to be answered as major players fight over the intellectual property rights of the biggest HCI paradigm shift since the transition from command line to GUI.

Binding Knobs and Dials to Multitouch

Multitouch, by nature, is an imprecise method of interacting with data. Your fingers on a touch surface are not capable of extreem precision and for the most part that's not an issue. But sometimes you need to do very precise work such as is the case with mechanical or electrical design.

To deal with scenarios where precision is needed, its possible to bind tangible tools with software actions on a multitouch screen. For example, it will be come common practice to use a stylus in combination with multitouch devices for productivity applications. A draftsmen or designer might have several different stylus, or a configurable stylus, for different line thicknesses or types of strokes.

There will also be a need to bind buttons, knobs, and dials to multitouch surfaces in order to get more precise control over the content being edited or manipulated. While Microsoft Surface allows you to do this today with tagged objects, the precision is not that great. In an article published on TechFlash, Microsoft engineers talk about a research project that dynamically binds software operations to hardware controls. It's being presented at CHI 2009 in Boston this week.

In my opinion, this has enormous potential for productivity applications as well as precise controllers and monitors used in industry. Multitouch by itself has limited applications but when combined with tangible controllers it creates a very expressive human-computer interaction.

IllusionLabs Multitouch Table Games

Illusion Labs, which has been successful at creating iPhone games, ported their skating and other games to a multitouch table which they demonstrate in this YouTube video.

Apparently the table itself is a one-off built just for demonstration purposes. [correction: According to Seth Sanders the table is provided by TouchTech.]

The question is: do video games work on the multitouch table form factor? They seem to work pretty well in this video demonstration, but you have to wonder if the player's arms and back won't get wornout reaching across the table for long periods of time. This type of thing would seem to make more sense in a drafting-board style form factor but even then stress on the user would probably proove intollerable after a while.

Build Your Own Multitouch Table for $350.00

MaximumPC just published a pretty good article detailing how to build your own multitouch table for $350.00. With a price point of 12k, the Surface computer is a bit out of reach for most people but this is much more affordable. Of course you have to be good with tools and at least not dangerous with electronics but more power to those who are and those who try.

Along with this article you should also check out the NUI Group which is a gathering place for people building their own multitouch tables. You'll find a lot more multitouch table designs and how-to articles on that site.

Augmented Reality with a Touch of Multitouch

The New York Times published an article at the end of March called, "Thinking of Going Blond? Consult the Kiosk First". It's about a new type of retail kiosk concept developed by Intel and Frog Design that allows shoppers not only to self checkout, but - in the case of other concepts also in development - to see what they look like with different apparel and even different hair color.

Actually, this is more about augmented reality than multitouch but the device includes a multitouch horizontal surface as well as a vertical screen (how could they not include touch?).

What's really cool about this article, for me anyway, is that its right smack in the middle of what we are doing at CSG. We are using Surface units in retail and other industries to drive stronger sales and to "close the deal" with the customer at the moment of strongest desire. CSG is working on self-identification, a big part of the kiosk idea addressed in this article, multitouch (of course), and we've even begun to experiment with augmented reality as well as a number of other technologies.

Touch The Future Now

Not much to say about this except that its kind of a funny video - I wish cleaning up was this easy.

Here is another video of the same ilk that I thought was funny.

Microsoft Researches Universal Gestures

Microsoft researchers are presenting a paper this week on universal hand gestures in computing at CHI 2009. Apparently they did a study where they asked test subjects what gesture they would use to indicate a specific computer operation, like "I need help" ( Most people drew a question mark on the screen). Actually, according to the article, many of the gestures were in-the-air signals with hands and fingers.

I think this is important research because there are few established universal gestures (e.g. touch, pinch, swipe) and we are surely going to see many different kinds of gestures (some good, and more bad) used by multitouch device manufactures.

One tidbit from the article I enjoyed reading, because its speaks to a pet peeve of mine, is that people generally don't distinguish between one, two, or multiple fingers in manipulations. So for example, designing gestures around a one finger swipe as different from a three fingers swipe (you can thank Apple for that gem) is a bad idea.

That said there are some in the air gestures that have significance depending on the number of fingers, but these are not at all universal as they have strong cultural bindings. For example, the "thumbs up" means "everything is well" in America but it also means "Frak You" in South America, West Africa and parts of Europe.

DiamondTouch: Multitouch and Multi-user

One of the biggest problems with multitouch systems - usually encountered with larger table top and wall mounted systems - is distinguishing between the touch of different people.

The classic example of this problem, one that I encountered myself, are paint applications. In a collaborative multitouch situation, because the multitouch device cannot distinguish between different users, all users have to use the same color at the same time if working on a communal canvas. User A can't be working with red while user B is working with blue if they want to annotate or color the same area at the same time.

The DiamondTouch addresses this classic issue by making the users a physical part of the capacitive touch detection system. When a user touches the table the current (very minimal) is actually routed from the antenna array under the table though the users body to the computer. To make this work each user has to have their own receiver which might be a wrist band or a conductive pad on their chair. Since each user has a unique receiver the multitouch system can determine who is touching the screen where and so each user can operate on the same canvas with different output. I don't know if that makes much sense as its hard to explain but the web site of the company that licenses DiamondTouch provides more explanations.

DiamondTouch was originally developed by researchers at Mitsubishi Electronic Research Laboratories (MERL) starting in 2000 and was just spun off under a new company, Circle12. The device can be used with standard windows via a mouse emulator and no programming, but Circle12 also provides their own DiamondTouch SDK which supports development in C++, Java, or ActiveX as well as a Flash-based SDK called DTFlash.

The thing I don't like about DiamondTouch, and its something Circle12 is working on, is the fact that the device is front projected. That said, I really like the ability to distinguish between different users touching the same content area. I think this is a big hurdle to collaborative multitouch applications. I also like the resolution of the DiamondTouch which can be as high as 2752 x 2064 or video while allowing users to manipulate points as small at 0.3mm in diameter - no other multitouch system I know offers that level of granularity.

Monson-Haefe's Law of Multitouch

I've decided to declare the First Law of Multitouch Design and name it after my self.

Monson-Haefel's Law

"The value of multitouch equals the net value added to each user's interaction conducted through that multitouch device, valued from the perspective of each user, and summed for all"

Inspired by "Bechman's Law".

Windows 7 WPF Touch API

While WPF developers are waiting for WPF 4.0 to come out with multitouch support, there is a stop gap solution of sorts. MSDN has an open source library that maps touch events to stylus events (e.g. StylusDown --> TouchDownHandler ) and includes a gesture engine with gesture events like pinch, pan, zoom, rotate, and two finger tap.

This library allows you to write WPF applications that can process touch input. I suspect, although I don't know, that this library is just a temporary solution that will become absorbed or abandoned when WPF 4.0 is finally released. That said, a stop-gap solution is better than no solution at this point.

Dr. Dobbs Multitouch Design Article

Dr. Dobbs has published an article called "Designing for Multi-Touch, Multi-User and Gesture-Based Systems" Joe Eden of Infragistics.

It's a nice introduction to the the design concerns faced by multi-touch developers for Microsoft Surface and other tabletop multitouch systems.

One term I hadn't heard of before is Phicons, which are "physical icons that can be used to show possible actions". For example, a tagged wafer or controller that you pick up and drop on the screen to control it. Of course, everyone doing Surface development is already using tagged objects this way, but its nice to have a term, Phicon, to use when discussing them.

Eyes-Fee Multitouch

One of the complaints about multitouch is that its not a technology accessible to the vision impaired, not to mention the danger of trying to use a multitouch phone while driving - not something that I would recommend.

The Eyes-Free Project at Google Code has a solution that was developed by a blind engineer at Google for touch screen Android phones (I guess now that the touchscreen Andorid cat is out the bag than can start innovating around that feature).

Anyway, the Eyes-Free project developed an Android touch screen shell that allows you to dial phone numbers, look up contacts, get the date and time, and so on without having to look at your touch phone. It basically uses vibration and general swiping direction to drill down to specific numbers, letters or menu options. It's kind of hard to explain so I recommend watching the videos.

There's also a "eyes-free" iPhone applicaiton, A Special Phone, which can dial up to six phone numbers by shaking the phone. One shake dials your first contact, two shakes your second and so on. It's a pretty cool idea not only for the blind, or while driving, but also for very little kids to dial 911 or Momy in case of an emergency.

So does "eyes-free" controlers for touch screens make sense? "Eyes-free touch-screens" sounds like an oxymorone but actually it makes sense. Touch screen phones are infinetly reconfigurable.
The buttons and controlers can be anywhere you like and you can decide the meaning of gestures per application. In the case of the Eyes-Free Google project you can define your own swipes and drill down menu for custom applications. That's not something you could do with anything but a softwscreen. So while it sounds weird, flat glass touch screens can actually be an excellent solution for eyes-free applications.

3D Glasses with Microsoft Surface

Phonomblue introduced a Microsoft Surface application that projects 3D which can be enjoyed with those snazzy 3D glasses. There are a bunch of videos posted by them on Vimo. I would have written this off but after seeing Caroline a few weeks ago it seems more intriguing.

Hollywood, in their infinit lack of wisdom, is once again betting big on 3D. The truth is while it works for a few movies most people don't want to watch a movie with 3D cardboard glasses on.

For surface I see the application as having pretty limited but still interesting applications. Information visualization and games come to mind. It's also an interesting gimmick and crowd attractor.

SLAP Widgets for Multitouch Tables

Researchers from the The Media Computing Group at RWTH Aachen University, introduced a new concept, called SLAP Widgets, that combines soft controls with tangible controls. It's a nice application of tangible computing and something that might work very well with Microsoft Surface.

SLAP (Silicone Illuminated Active Peripherals) are input devices formed of transparent silicone including buttons, dials, sliders, and a keyboard. The SLAP widgets are placed on the multitouch surface and, using tag recognition, labels and indicators are projected on the table surface just below the physical peripherals. It's a very cool idea, one that is similar in some ways to Microsoft SecondLight, which can project above the screen into tangible objects, but SLAP seems simpler to implement.

I especially like the SLAP keyboard which addresses the difficult issue of using keyboards on multitouch surfaces without haptic feedback. There is a really nice video and you can learn more on their site. Nice work guys! (thanks to Jonathan Brill for pointing this out on his blog).

Transparent Touch Pad

CTIA 09, the the biggest and the most important mobile phone conference there is, will have plenty of multitouch mobile phone announcements this year.

One phone that is particularly novel is the LG-DG900 which not only sports a nice sized multitouch screen but also a transparent touch key pad. The LG-DG900 has been kept under lock and key and no one outside of LG has had a chance to even hold it, but you can see some great pictures of the device on akihabaranews.com as well as a video Endgadget.

Ikao Touch Ceiling!

Ikao, a company formed from an MIT research project, has just announced the first multitouch ceiling device. Using "above-screen" projection technology with inferred sensors the touch ceiling provides a multitouch experience not unlike Microsoft Surface, only in the form factor of a ceiling rather than a table.

Dr. Fran Ikao, CEO of the new start up, is shown demonstrating the Touch Ceiling by Ikao using a specially designed stylus. Apparently, the stylus is necessary if you cannot reach the ceiling but a full multitouch experience is possible if the ceiling is lowered to about 5' - 7' feet (depending on the user's height), or with the use of specially fabricated extenders similar to the stilts used by drywallers.

Dr. Ikao said that the ceiling is an "ideal multitouch surface" because it's never obstructed by any other objects and is always available regardless of where the user is located.

Updated April 7, 2009

Just discovered this hilarious face-gesture joke announcement from Opera software.

Updated (later that night)

It seems that Ikao isn't the only one innovating in multitouch. Check out the very cool adaptation of Microsoft Surface as multitouch stove top.

In addition, the Microsoft Exchange Team plans to package each licensed copy of Microsoft Exchange with a Microsoft Surface for easy of administration. It's great to see Microsoft bundling Surface with its other products.

Surface in Sin City

When I first heard that the Rio was going to have a Surface, I thought "Wow, how cool is that!" I figured that Surface could be a very cool gambling device, but I found out that you have to jump through some serious hoops in Vegas to get a machine certified for gambling.

When I saw the Flirt video I thought it was pretty stupid, but than again I haven't been dating for a long time (happily married for 15 years) so perhaps I was just out of touch.

It's now my understanding from several different people who are in the know, that the Surface units at hotels and other hospitality settings are not returning on their investment. This makes sense, after all, how long can a novelty really pay off? In order to survive in the hospitality industry, or anywhere else for that matter, the Surface has to provide a return on investment. It has to encourage people to buy more stuff, or help them get more stuff done, or see stuff more clearly. Just being an attraction is not enough.

A shout out to Julie Meridian who recently penned this blog post about the good, the bad, and the ugly aspects of the Surface device and applications at the Rio in Las Vegas. The title of this blog is ripped off from Julie's post. Thanks, Julie! It's a great read! (thanks to Jonathan Brill's blog for the pointer).

Is Multitouch on Linux Dead?

A couple of years ago there was some talk about multitouch on Linux, but more recently the topic seems to have died completely.

Back in 2007, after Microsoft announced Surface, there was a lot said about Linux MPX as the open source alternative to Surface, but that seems to have gone nowhere since the developer of MPX moved on to other things.

There was also some pretty cool multitouch videos posted to Youtube.com a couple of years back, but since there's been little or nothing. What's going on with Linux and multitouch? Please provide some recent pointers if you know of any.