What the Movie Avatar Can Teach Augmented Reality

The biggest news about the movie Avatar has been the 3D experience and the way its blown the doors off the previous records. The movie has garnered huge success because it pushed the boundaries of technology and told an interesting story.

I loved the movie and the way 3D helped give more perspective to the enviroment. My own Star Trek loving mother didn’t even realize the Na’vi were CGI. She thought they were people in blue suits (really… I’m not joking.) And though storytelling will become important to later advanced augmented reality applications, it’s not what I wanted to point out.

James Cameron is part art-dude and part tech-geek. He waited for years for the technology to ripen enough to do the movie the way he wanted. One of the innovations that he created for the movie was the Fusion camera for the live-action sequences. Normally, scenes are filmed before a green screen and then the CGI is added afterwards. The actors play a game of make-believe and the director has to guess at how the enviroment will unfold around them. CGI movies tend view flatly because the emotions are added later by the special effects guys and not the actors on the scene. Cameron has changed all that.

The Fusion camera system is an augmented reality viewport into the CGI world. When Cameron was filming the actors, he was able to direct them and see the results. When he looks through his camera, he can see them interacting with the world Pandora as the nine foot Na’vi and help them tell the story. The camera itself wasn’t even a real camera in the sense that it filmed the action. The camera allowed Cameron to see the action being recorded by multiple sensors and cameras.  Once the action was recorded, he could go back and reshoot the action from a different perspective, even with the actors gone.

Facial expression was another hurdle they had to jump to make the movie work. So they added little cameras hanging on people’s heads to capture their range of facial expressions and then tweaked algorithms to get them to react correctly.  Even now we can pull off this trick.

Together these systems are similar to an immersive augmented reality world. While we don’t have the HMDs, complete camera access and processing power to pull off the world of Pandora now, time and continued improvement will make lesser versions possible.

If you look at the Fusion camera system, the camera is essentially the HMD display, albeit a large and bulky one. Multiple cameras, RFIDs and tracking markers help the computer understand the world, and complex and powerful computers put all the pieces together. I can only imagine that this system could be turned into a mind-blowing game in an empty warehouse with the proper HMDs.

Essentially, the movie Avatar teaches us that augmented reality has sky-high practical possibilities. All the components of his Fusion system can be ported to the commercial world (not now, but in three or four years) and used to make complex and believable environments overlaid our own world.

In the future, you too can be a nine-foot tall blue Na’vi and you won’t even have to have your soul sucked through a fiber-optic tree.