Using Keyboards and Mice in Conjunction with Multi-Touch Tables

A reader pointed me to a video (see below) referenced by Johnny Chung Lee (a wonderful democratizer of NUI) showing how keyboards and mice might be combined with multi-touch tables in productivity applications.

The concept and video was created by Bjoern Hartman for UIST '09 (ACM Symposium on User Interface Software and Technology).

I've said a number of times (1, 2, 3, 4) that keyboard entry on a multi-touch surface just doesn't work. I've wondered how we can combine keyboards with large multi-touch surfaces for productivity applications but never really had any good ideas.

What this demo shows is that the keyboard and even a mouse can be made a tangible aspect of the multi-touch surface. By Tangible I means that the table recognizes the keyboard and mouse and based on how its manipulated, allows those devices to provide input. The way its used in this video is just awesome.

The keyboard sits on the table and can be assigned for data entry to any content by proximity. Dragging content so that it's adjacent to the keyboard allows the keyboard to provide data entry to that specific piece of content. The mouse interaction was also interesting. Place the mouse next to the keyboard and they can work together on that same piece of content. Better yet, use the mouse to manipulate objects on the multi-touch surface that are out of reach using a guide-line.

In terms of increasing the potential for productivity applications, I see this video and the work of Bjoern Hartman is extremely important; it will certainty influence my thinking about NUI productivity applications. Nice work, Bjoern!