Pranav Mistry gesticulates like he’s taking a picture.
Mistry actually is, except there’s no sign of any camera in his hands.
Fixing his gaze on a scene he wants to capture, he gestures as though clicking a “virtual” camera’s shutter with his forefinger.
He makes another hand sign to project pictures he’s captured on to a nearby wall — and the pictures do materialize there.
Then with a few other deft finger movements, he selects some images, deletes others, and e-mails his favourite pics to friends — all without once touching a camera, a smart phone, or any other computing device.
Lest you’re wondering, Mistry is no sleight-of-hand illusionist doing a David Copperfield on his audience.
He’s a researcher at Massachusetts Institute of Technology’s Media Laboratory where brilliant ideas are as commonplace as coffee, and innovation is the name of the game.
But Mistry’s work at Boston-based MIT’s Media Lab is no game.
Related stories
Coming soon — devices that diagnose and repair themselves
Your R&D department is … everywhere
Some of his inventions are currently being tested out in disparate environments worldwide with a view to full-scale commercial deployment.
The MIT researcher was at the Mixx Canada 2010 conference in Toronto last week demonstrating to a captive audience some of his projects that successfully tear down barriers between the virtual and physical worlds.
“Today information in the digital world is mediated to us through a computing device — a PC, a net book, a smart phone and so on,” Mistry explained.
“Some manufacturers have taken this a step further, adding electronics inside physical objects to make them intelligent. So you can have a smart coffee cup that changes colour based on the temperature of the liquid inside.”
Even in such scenarios, he said, the information is still encompassed within a device, and accessible only to those who have the device and know how to operate it.
“But that’s not how things work in the real world.”
Deliver us from all devices
A desire to “liberate” intangible information from the confines of tangible devices drives most of Mistry’s research work at MIT Media Lab. “The idea is to extract information and then paint it on any physical object or surface — a wall, a book, a newspaper.”
Essentially, he said, we’re witnessing the advent of the “anywhere interface”.
The “virtual camera” project Mistry demonstrated at the Mixx Canada conference exemplifies this concept.
Users don’t need to physically handle a camera or a computer. They use hand gestures to snap pictures and then another set of gestures to view, organize, and share those images.
What’s more the user interface isn’t an LCD or computer monitor, but literally any surface.
The underlying technology for this and other projects is relatively inexpensive and easy to set up, Mistry said. For instance, key equipment for the “virtual camera” project includes a very tiny, especially rigged smart camera, and a projector the size of a loonie — both of which can be hung around one’s neck like a pendant.
The camera, through a special software program written by Mistry, tracks the user’s hand movements.
In the first version of the project, Mistry and his team used colour markers to facilitate the tracking process.
“This head-mounted version worked, but it looked and felt awkward.”
The pendant is a huge improvement, Mistry said. He and his team are now working on a clip version of the device, which will be even simpler to wear and virtually inconspicuous.
And there are several other applications of this interface-agnostic technology. Some of these Mistry demoed at the Mixx Canada event.
They include:
- Projecting a map on to any surface with a gesture and then using other finger signs to re-size the map, making it bigger or smaller
- Checking e-mail or getting the latest weather data on any interface
- Browsing through a book in a store and having the Amazon rating appear at the back of the book’s jacket
- Checking the weather forecast in your daily paper, and having the latest updates from weather.com projected on the forecast page
- If you’re an airline traveler, accessing dynamic updates about a flight delay or gate change on your boarding pass
Ideas for Mistry’s projects often arise from real life conundrums.
For instance, he said, arguments between him and his sister about what to watch on TV sparked an idea for a key project, now being prototyped for commercial application.
“My sis and I have TV viewing preferences that aren’t typical,” Mistry noted. “I like Cartoon Network, and she wants to watch the sports channel.”
To resolve this stalemate, Mistry created a technology that enables up to five people watch different content on the same screen, at the same time by donning a pair of glasses and headphones.
“We’re applying this, on an experimental basis, at a Japanese airport, where persons from different parts of the world would wear glasses that enable them to see all the signage in their own language.”
The same technology is currently being tested out in a movie theatre. “We’re getting a group to watch a movie together, but each person in the group will be viewing a different segment of the same movie at the same theatre.”
Google map with a difference
Mapping is yet another area where Mistry has sought to break down barriers between the physical and digital worlds.
“Today mapping apps are fairly advanced, but they all use words as search terms,” he noted. “We took a different approach, creating a Google map of the physical world, where real life objects serve as the search criteria, instead of keywords.”
Mistry explained how it works in practice.
A person could come to a venue – such as an airport – and put an object on the map. “If the object is his boarding pass, the map would give him directions to his gate; if it’s a cigarette box, the map displays smoking zones at the airport; if he places a coffee cup it indicates the nearest Starbucks, and so on.”
He said cameras pick up information about objects placed on top of the “map” while software algorithms do the rest of the work. “We’ve also created an MIT version of this map, where we place our student card on the map and it shows us where our next class is, how long it will take us to get there, and so on.”
From stickies to quickies
The same principle is at work in another application that uses post-it notes as the interface, and tremendously enhances the productivity of the information worker.
“I wrote a software program that understands movement of a pen on paper,” Mistry said. “Let’s say a person writes a message [on a post-it note] to John. Purpose-built software recognizes the note as a message to John, and automatically retrieves John’s phone number from the computer’s phone book, and sends John an SMS.”
If the note is about a meeting, the software sends a meeting request to John and blocks off that time in my To Do list in Outlook or any other productivity application.
