NUI's Text Entry Dilemma

Over the course of this past year I've written numerous blog entries that highlight one of the biggest problems with multi-touch surfaces, text entry. Most recently there has been a lot of talk about a new patent for which Apple has applied. It's a touch screen that can provide physical tactile feedback. The solution is explained best in this post at My Digital Life. Does the articulating framework discussed in the Apple patient solve the text entry dilemma? I'm not convinced it does but it certainly has potential. Will it be a part of the rumored iSlate? I seriously doubt it but only time will tell.

So what is NUI's Text Entry Dilemma? In a nutshell entering text using a multi-touch screen, be it on a small device like the iPhone or on larger devices like the Microsoft Surface, is difficult because the user has to watch their fingers as they type on a "soft" keyboard. By soft keyboard I mean a graphical representation of a keyboard on a flat touch screen. In addition, alternative input solutions such as pen-based input and natural speech just don't work all that well.

The truth is there are very few solution which seem to work in practice. The soft keyboard on an iPhone works fairly well for short messages or web searches, but its slow compared to thumbing on a mobile phone keypad. The use of soft keyboards on medium sized devices such as tablet computers and large devices such as Microsoft Surface is a non-starter. Just try to compose an email on either a Windows 7 tablet or a Microsoft Surface using a soft keyboard and you'll soon agree it just doesn't work.

At the beginning of this year I wrote an blog entry which tried to explain the importance of tactile and force feedback in touch typing. What I was essentially saying is that you can't touch type with a flat smooth surface like a touch Screen because you can't feel the keys.

In October I spoke about multi-touch keyboards - what I call big track pads - and multi-touch mice as non-starters. Despite the hupla about the multi-touch mice Microsoft was experimenting with as well as a multi-touch keyboard Apple was patenting, I found both solutions to be un-workable. The truth is that you have to be able to see a direct correlation between your fingers positions on the multi-touch surface and the display in order to be productive. The only solution that I know of that provides this is the 10/GUI concept. Of course I might be totally wrong about this. Our more artistic friends who have been using things like the Wacom Bamboo might disagree with me on this one. They've demonstrated over the years that using a flat surface and stylus can work well even if the interaction is indirect.

This year I also pointed to a couple of multi-touch alternatives to traditional hunt and peck including Dasher and Swipe, but again these solutions require the user to watch their fingers as they type which doesn't work all that well in a productivity applications. A couple solutions that augmented the multi-touch screens with physical keyboards were also discussed including the SLAP Widgets and research by Bjoern Hartman which seem promising. The solution here is applicable to large table-top surfaces and could be extended, in principle, to small and medium sized devices. For example, mobile phones that provide both touch screens and physical keypads or Tablets that allow you to flip the screen to expose the keyboard in a traditional laptop configuration. The question is: are these really the only solutions that work? They don't seem to advance the NUI landscape very much. Perhaps a pure NUI interface for productivity application simply isn't viable.

Finally I discussed the possibility of replacing the keyboard with pen input or natural speech input. Both solutions have enormous potential, but as I discovered with real-world pen-based input and natural speech solutions the promise far out strips the delivery. Neither pen-based hand writing recognition or natural speech recognition work very well at all.

So after a year of thinking, reading, and writing about text entry in Natural User Interfaces I've come to the conclusion that the use of a physical keypad or keyboard which is somehow integrated with or tethered to a multi-touch screen is the only solution that seems work for anything but trivial text entry. This is a bit disappointing but it seems to be a reality given the current technical capabilities of alternatives including soft keyboards, pen-based input, and natural speech. Perhaps Apple's articulated framework is the solution, but the problem I see with that is force feedback. I can't tell by reading the patient if the articulated frameworks provides force feedback but without it I fear the solution is not going to work all that well.