Tech

A theme-park animatronic trick could allow people act more naturally in videoconferences.

Shader lamps is a technique that projects an animated face that looks three-dimensional onto a dummy's blank face. Now the trick has been exploited to project a person's features onto a animatronic double somewhere else.

Before the dummy's blank polystyrene face can be brought to life, the real person has to have still photographs taken from the front and side to create a 3D model of their head. This model allows the output from a single camera to be distorted to make an image that looks right when projected onto the dummy.

The user wears a headband that is tracked by a camera so that the remote dummy can swivel and tilt its head to match their movements. An audio feed with a slight delay built in means that the person's words are synchronised with their movements, and the person being projected can see the scene around their remote double thanks to a panoramic camera in the dummy's head.

Second-class citizen

The video above shows the animatronic shader-lamps avatar demonstrated by a comedian at a recent conference. The system has a number of advantages over conventional screen-based video conferencing, says Greg Welch, the computer scientist at the University of North Carolina at Chapel Hill leading the project with his colleague Henry Fuchs.

"In existing 2D videoconferencing systems, the remote person is kind of a second-class citizen: they're in this box sitting in one place, they look different," says Welch.

"And then there are technical problems," adds Welch. The camera position in conventional videoconferencing makes it hard for viewers to judge where an on-screen interlocutor is looking, which is particularly problematic, he says: "It often looks like you're not paying attention."

Those problems are minimised in his system, he says, because the remote person's eye and head movements are accurately replayed in real time.

Self-assertion

The most sophisticated commercially available teleconferencing systems can cost upwards of $1 million, says Wijnand IJsselsteijn, a member of a European consortium working on a separate 3D teleconferencing system. "But they still suffer problems in conveying eye contact and gaze direction naturally."

Using projection-augmented dummies is a novel way to address that, he says, and should also make it easier for a person not in the room to have an equal part in a conversation.

"When you're the remote person, it's really hard to get the attention of the room," IJsselsteijn explains. "But when you can move something in that room you have a much more physical presence." Simply turning your head would allow you to assert yourself, because the movement would be mirrored by the avatar, he says. "Irrespective of how realistic it is, that's a new way to gain attention."

Mobile avatar

A range of improvements are planned to the prototype: for example, using multiple projectors to cover the sides as well as the front of the dummy's. A mobile version is also planned: "One of the inspirations for this system was a conversation with a prominent physician who asked if we could make it possible for him to visit remote patients as a tangible avatar," explains Fuchs.

It could also be a boon to patients, he adds. "There are people all over the world who are unable for medical reasons to leave their house," says Fuchs. A mobile version of this system could provide a "prosthetic presence" they could use to venture out and interact with other people, he says.

The animatronic shader-lamps avatar was presented at the International Symposium on Mixed and Augmented Reality in Orlando, Florida, last week.

If you would like to reuse any content from New Scientist, either in print or online, please contact the syndication department first for permission. New Scientist does not own rights to photos, but there are a variety of licensing options available for use of articles and graphics we own the copyright to.