Bruce Sterling is wrong about Augmented Reality.

First off, I want to express that I have the utmost respect for Bruce Sterling as both a writer and a visionary.  As a longtime fan of cyberpunk and science fiction in general, Bruce has been a part of the pantheon of authors I regularly visit.  And as a writer, I study his stories so I can improve my practice of the craft.  Just last week I was reading his story “Our Neural Chernobyl” in the teaching anthology Paragons and was blown away by his ability to tell a thrilling story devoid of characters and plot and subsisting entirely of theme.  Bruce is a true master of the craft of writing.

But on the subject of augmented reality being a subject of literary endeavours, I believe, and will attempt to prove in this article, that Bruce Sterling is wrong.

Last week, in an always entertaining interview from Tish Shute and Ori Inbar on UgoTrade, Bruce Sterling stated that: “I’m not sure it makes a lot of sense to write fiction nowadays “about AR,” because it’s no longer a fictional topic. It’s become like writing fiction “about cinema.” You can write good fiction about someone who works in cinema, but not fiction about cinema itself. AR is not sci-fi “Augmented Reality” any more, it’s become a real-world phenomenon, a new industry of real augmentation..”

As I understand it, his basic argument is that because augmented reality is a real technology now and not just a fictional – though probable – idea, that it is not a worthy subject for writing fiction about.  And when we say “about,” we can probably assume that Bruce doesn’t just mean that AR cannot be a part of a story.  More so that the story cannot be an exploration of the idea of augmented reality, because the idea already exists as a tangible product that one purchases and uses and therefore there is nothing to explore when anyone can visit.

Or said another way, science fiction is about exploring the possibilities presented by technology in relation to the human condition.  If that possibility has become an actuality, then what is left to explore?

So there’s where I will begin my counter-argument.  With the assertion that augmented reality has become, excuse the redundancy, a reality.  What we’re seeing currently in the marketplace and even in the research domains are nascent attempts at creating the experience of augmented reality.

Often when we’re speaking of the true potentials of augmented reality, we’re using words like immersive and ubiquitous.  The artifices are data shades and eye-screens in the form of contacts.  The data stream is superfluous and overwhelming.  We have none of these things currently.

What we are seeing in the marketplace and in the research labs are gimmicks and toys and games.  We have many examples of AR that stir the imagination.  I, for one, have been promoting these applications of AR for many years now.  But is AR a technology that has truly changed the way we work and play and live our lives?   Not at all.  The yellow line in an NFL game is the most common way that AR has touched the masses.

And once a technology becomes reality, why should science fiction cease to care about it?  I speak not about technology as a prop or the furniture of the story (to use a term by George RR Martin), but to write about the story as it revolves around the technology, changing people’s lives by their use of it and thus showing the technology through the lens of human behavior.

I also believe that AR is still a relevant focus for science fiction because AR is not a simple tool, it is a whole new medium for the transfer and dissemination of information.

The nearest and probably most obvious comparison is virtual reality.  VR at its best was a destination and a substituite for the real world.  VR has always been a difficult sell to the masses and a narrow niche best populated by the niche groups that could take advantage of its morphology with the best example being the furries in Second Life.

AR differs from VR in the same way that the national highway system differs from a late-night Denny’s.  AR, as a new medium of information transfer, will change our daily lives, while VR was a place to escape from reality.

I’m not suggesting here, that Bruce was arguing that AR and VR are the same.  But I think illustrating the differences helps explain why AR is a new medium rather than just a new technology that will change people’s lives in varied and profound ways, so it cannot be dismissed as a topic for fiction just because I can swat invisible mosquitoes on my iPhone.  We’ve barely scratched the surface (or multi-touched it.)

The magic of humanity’s rise through the evolutionary brackets has been from its superior ability to communicate information.  The greatest of game changing technologies have always been the ones related to medium: spoken language, written language, printing press, radio, television, the Internet and now augmented reality.

The wonder of AR is that we’re taking these high powered computers we carry in our pockets and the gargantuan mountains of information stored on clouds and hard drives and attaching them to the moments and locations we need them.

As a fifteen year employee of Toyota, I can assure you the key ingredient to the Toyota Production System that has changed industries worldwide, is the ability to get the right information to the right people at the right time.  It all comes down to that.

And that’s a major reason why I believe that AR is still ripe as a topic for science fiction.  As a medium for information, it can be used as more than just a travelling on-the-spot wikipedia.  Augmented reality invites both the viewing, but the creation and collecting of new information.

AR is also paradoxically about the efficiency of human action in relation to usable data and the avoidance of reality in the form of pictures and graphics.  AR is more than just data to find the best routes to the local Starbucks.  It’s also seeing your local street corner as a property in a city wide game of Monopoly.

The best mediums, like television and the Internet, both instruct and destroy.  Information can be additive and in the same breath addictive.  If only Philip K. Dick were alive, he would have a lot to say about AR (and in some ways, he already has.)

Lastly, I say that AR can be a topic of science fiction as a writer–though I am no where near the stature of Bruce Sterling, nor have even the slightest twinkle of his legacy.  Instead, I offer my own experiences as proof and that I am interested in AR so much that I’ve written three books, have another three in the planning, and am producing an anthology this summer–all about AR.  I’m putting my money where my mouth is, so to speak.

And Bruce might respond that the ability to do something is not the same as that it should be done.

Which is a valid point.  How can one decide if AR is still worthy to be written about?  Is it based on awards?  Vernor Vinge won the Hugo for Rainbows End, the only book about AR that I’m aware of besides my own.  But that was four years ago in 2007.  Have any other books about AR even been written in that time?

A couple of years ago, I collected with Bruce’s help, the totality of writings about AR at the time.  I’m sure we missed some stories and books, but most of them used AR in only a tangential way–more of a neat toy or a furniture, rather than exploring how AR changed people’s lives.

Then should sales be the deciding factor on if AR should be written about?  If readers aren’t interested in purchasing it, then its time has come and gone.  But once again, we’re presented with too little data.  Even myself, at this time, have only one indie published book about the subject and too little sales data to mean anything.

And maybe that’s why it cannot be said that AR as a topic for science fiction has already played out.  We don’t know yet as there isn’t enough data.  As a logical thinker and a visionary, I think Bruce would respect that.  In fact, Bruce states in this article that: “any good futurist is a historian.”  We have so little history on AR as an impact to human life.

I also suggest that we shouldn’t be in a practice of talking about things that shouldn’t done (minus the obvious ethical and moral cases.)  Who’s to say that every nook and cranny of any science-fiction topic has been thoroughly explored? Even well-known cave systems have uncovered new sections long after the “experts” have marked it off the list.

So I say that augmented reality should be a topic for science fiction writing for years to come, and not just as an about, but also as a backdrop for other stories.  The medium is just too rich and varied to be ignored.

Last Note – I considered whether or not I should post this for a few days, deciding if I had written it for conflict’s sake, or if I truly had a constructive argument.  My conclusion, as you can obviously see, was to post.  Conflict for the sake of conflict is pointless and serves only the look-at-me type of blog poster sometimes found on the Internet today.  Constructive conflict that instructs the arguer and hopefully the reader, and maybe even convinces the counter-arguer to reconsider their point, is worth the conflict.  And since I felt I learned from writing this piece, in the end I should share it.  And besides, writing posts that no one reads, even if they are a constructive argument, is like arguing alone and in the dark.  Sure, you’re arguing, but in the end, you’re still alone and in the dark.

Perhaps I should title this post, The Myth of the Muse, because that’s really what I’m going to talk about.   As a writer, a blogger, or as an engineer, I am on occasion asked how I come up with my ideas.  The questioner usually prefaces my answer by stating that it is probably: my intelligence, my weird mind, a divine muse, or just plain luck.

Fortunately, it’s none of the above.

For if the world relied on such randomness to expand the realms of man then why would the majority of the world’s greatest inventions have only happened in the last century?  The effluence of creativity says more about the collective efforts of those that came before and the prevalence of information that is the grist that we dine upon.

Or said by one of the great minds of the human race, Albert Einstein: “The secret of creativity is knowing how to hide your sources.”

I would hate to have been a writer in the distant past when muses were in vogue.  To rely upon the vary airs themselves to provide my ideas would be a frustrating wait.

Our brains are a wonder, capable of combining randomness into new things.  The greatest combination ever: The Reese Peanut Butter Cup is made up of two entirely different things smashed together in a convenient cup. But if the inventor of the RPBC had not known what chocolate or peanut butter was, then the delicious candy treat would have never been made.

Such is the same with creativity.  One must fill up the brain with newness, so interesting and unexpected connections can be made.

At Toyota, we sometimes practice the Ohno Circle.  Taiichi Ohno, one of the early great innovators of the Toyota Production System, had an exercise where he would have a subordinate stand in a circle on the production line for the whole shift.  The subordinate was required to open their mind and try to see the line, thinking about what was happening, making connections of understanding they may not have had before.

When presented with a new challenge on the plant floor, I will often wander to the area and watch production, with a blank as mind as possible so to not prejudice my sight with knowledge.  I build my writing ideas in much the same manner, reading from a wide variety of sources, both fiction and non-fiction to fill up my brain.  Often I file away the fact or interesting tidbit of knowledge without knowing how it will fit in later.  Sometimes the thought is more formed, which can be dangerous in itself as once I lock it into a story, or project, or activity; it’s hard to hammer it back into its original form to be used somewhere else.

That is one reason I am all too happy to labor away on my blog and at Games Alfresco talking about augmented reality, the robot apocalypse, or whatever random bit of flotsam and jetsam I found on the internet.  Spending time learning and expanding my view of the world provides ample ideas for my day job at Toyota or my writing one, and I would never want to give that up.  The task itself of writing the articles are the reward.

So the answer to my creativity, and I assume the majority of the human race, is that it takes work.  One must seek out and challenge oneself with new information, trying not to codify the knowledge too soundly, so it cannot be reused in other interesting, or awesomely wonderful ways.  In the end, once one’s cup is full, then it becomes possible to dribble out a few beads and stand back to rejoice in the new creation.

ISMAR09 Sympsium & Expo is right around the corner and I’m pleased to announce that I’ll be not only attending ISMAR09, but I will be presenting during the Manufacturing section on Monday and joining the panel discussion as a representative of Toyota.  I’ll be reviewing the possibilities of augmented reality in an industrial setting, including a project that I’m working on with Metaio.  Hopefully I can enlighten participants on the constraints and challenges of implementing augmented reality on the factory floor. 

I’m announcing this on my blog, so if anyone has any requests for topics during the discussion, I can be sure to bring them up.  I was only able to confirm attendance last week, so there’s still plenty of time to update my presentation. 

Not only am I excited about presenting, but I’m more excited about getting to meet a lot of the people I’ve interacted with throughout the AR world.  I’ll be there from Sunday until Tuesday afternoon, so if you’re going to be there then, I’d love to meet up, chat and maybe, share a drink.

To reach me before or to contact me during the conferance, email me at tom.carpenter (@) tema.toyota.com or twitter to thomaskcarpente.  (Yes, the name is missing the ‘r’ since my name was too big for twitter.)  I’ll be tweeting updates during the conferance (as well, I use it regularly for posts and other augmented reality related news.)

For some reason this augmented reality video from a couple of years ago is making the rounds again on various blogs and even Gizmondo is reporting it as fresh news.  Since its resurfaced, as a fellow car engineer for Toyota, I’ve decided to address some of the challenges to making a system like this really work.

1. AR Glasses

The first and obvious challenge is that the glasses aren’t yet a reality though they will be this year or next.   While a mechanic doesn’t care about style, the glasses still have to function well.  How fast of a refresh rate will the glasses need to not create “head lag” as the mechanic looks around in the engine.  The controls for the unit will also matter, since they’ll need to stop, start and rewind. 

2. Orientation

Without a paper marker, the glasses will need to orient the graphics onto physical locations on the car.  How well will they respond to dirty motors, poor lighting or the mechanic getting in the way of the camera?

3. Spatial Issues

When doing maintenance on parts that are easily accessible, augmented reality works well, but what about when the water pump under the wheel well needs replacement?  How well will augmented reality project the proper maintenance steps deep into the car engine space? 

4. Filling the Database

In the early nineties, our plant purchased a “wonderful” maintenance system that would house all of our PMs, keep track of all parts and what machines they went to and make sure we kept our machines in tip-top shape.  The flaw in this wonderful idea was that someone had to actually put all this data into the system and maintain its integrity as we upgraded machines and moved equipment around.  For high-frequency work, these AR maintenance glasses would be easy to develop, but who is going to make the other two hundred infrequent displays needed?

Overall, the promise of using augmented reality in industrial situations is quite large.  Given an improvement in the technology, I could find dozens of applications in my own plant.  Visualizing difficult tasks for team members using 3D data has huge potential, but it has to work right and be cheap enough to make it to the plant floor.  While I think AR will make its way into the production of cars (in fact, I’m working with Metaio on one such application) many issues need to be solved to make it a widespread usage.

One of the most promising uses of augmented reality is in the field of education.  While some might think that using AR for education is just a gimmick designed to trick kids into learning, there is a scientific basis for why using visual information is important. 

Communication occurs through our senses.  We have five senses, but we don’t use them equally.  This is how we use them:

Visual – 75%

Hearing – 13%

Touch – 6%

Smell – 3%

Taste – 3%

As you see the sense we use the most, overwhelmingly so, is vision.  We get most of our information through the eyes. 

At Toyota, my employer, we put this into practice by requiring all papers to be condensed into one page.  We use lots of graphs, diagrams and as little words as possible.  We teach people to communicate through visualization, a prized skill in Toyota, because you can quickly convey a lot of information in a short time.  Even multi-million dollar projects are funded from one-page presentations given in less than five minutes.

This holds true for education.  So much is lost when a teacher stands at the board and lectures.   Little of the information gets through to the subjects, resulting in wasted time and uninterested students.  A recent post from Immersive Tech reminded me of the power of visualization.  While the video is from last November, it helps illustrate my words (yet again proving the point), so I’m reposting it.

I’m impressed by the interactivity of the molecular structure lessons, illustrating beautifully the complex nature of atoms with simple sweeps of the hand.  Seeing is believing and by seeing the lesson at a desk or on the projector, the students can stay engaged in the lessons. 

There’s a reason the phrase, “Do what I say and not what I do,”  exists.

We’re recognize the power of our actions outweighs the power of our words.

While I’ve only recently joined the AR conversation, Ori at Games Alfresco, and Rouli at Augmented Times, have been promoting AR for the last year.  Rouli announced this week they will be combining forces to help create a central clearinghouse for AR information.  And I for one welcome our new AR overlords. 

Seriously, I think this is a great idea.  A lot of AR content exists in various blogs, and I think the conversation between them could be enhanced by a site bringing them together.  Bravo guys, I’ll be happy to contribute. 

Lastly, Jan over at the Augmented Reality Blog, brought up that he would be giving us a little series about augmented reality in a production process.  I’ve had the pleasure of speaking to Jan about his company’s products for some projects at Toyota, and I’m looking forward to hearing more about his other ideas.

Today I had a nice phone conferance with Thomas and Jan from Metaio.  Jan is the author of the Augmented Reality Blog.  Jan had contacted me after reading my writeup about the use of AR in manufacturing based on a Microsoft video.  Thomas is one of their designers and has experience with automotive manufacturing.  And I’m looking for applications of AR to improve production at my Toyota casting plant. 

The basics of our discussion was how to apply AR concepts to a manufacturing enviroment.  The advantage I see for AR in a production enviroment is it brings complex levels of information to a real time enviroment.  Many of the challenges of running a factory is getting the right information to the right people to make the right decisions.  Every day people must make complex decisions, often with limited information, to perform their job.  We apply many systems to these operations to ensure we’re producing a quality product at a good price, but often times we waste efforts because everyone doesn’t have the information they need at the moment they need it. 

The combination of real time information in a production enviroment can reduce waste and improve quality if done correctly.  One example I can illustrate this example is with die repair.  We use steel dies for creating our castings and the dimensions of these dies need to be held tight tolerances of less than 0.20mm.  This means we must constantly repair our dies because of wear from the aluminum (aluminum is corrosive to steel). 

When we have to repair our die, we first have to weld it, then we machine it back to the required dimensions and then we check the quality.  If we make a mistake, we have to go back to the beginning and start over.  A common mistake is not putting enough weld material down, so when the weld is machined back to standard, gaps are created from the missing weld.  This picture illustrates the example:

Using AR we can superimpose the correct dimension of the final shape over the worn die, so the welder knows if he’s put enough weld material on the die.  This gets the right information to the right person at the right time.  Before we had the right information, but we didn’t get it to the right person until much later in the process, so effort was wasted in between. 

This is one small example of using AR in a production enviroment.  And though we only had a brief discussion today about potential applications, my mind has been a-whirl about other possibilities, so I’m looking forward to working with Thomas and Jan to explore the potentials of this fledgling technology.

Tim Ferriss on his popular Four Hour Work Week blog, had an interested guest post about the practical use of philosophy in business from Ryan Holiday.  Ryan focuses on the branch of philosophy called Stoicism.  I’m not going to further dissect his excellent analysis and application of the philosophy, but I would like to challenge his assertion that Stoicism is the only philosophy that has practical applications (I would also point out that Stoicism is similar to the Eastern philosophy of Taoism, best explained in The Tao of Pooh.) 

I would like to discuss the philosophies of Socrates in relation to Toyota. 

It’s no secret that one of the keys to Toyota’s success is its Toyota Production System.  In the United States, the Toyota Production System (TPS) tends to be dumbed down into lean manufacturing.  Lean manufacturing is akin to taking a martial arts class so you can beat up the guy down the street because he looked at your sister.  It gets you quick satisfaction, but doesn’t help grow you as a person. 

The TPS I want to talk about is rooted in The Toyota Way, which is a guiding philosophy of the company.  The aspects of the The Toyota Way I would like to talk about are the concepts of 5-Whys and the open mind.  The concept of 5-Why is that when presented with a problem, keeping asking the question “Why” until the true root cause is found.  Often times, the superior will ask his subordinate these questions when the superior feels that the subordinate hasn’t driven down to the real root cause.  This method is essentially, the Socratic Method. 

It is an important tool of any business because often times we’re busy solving the symptom of the problem and not the true root cause.  As an entrepreneur it can be even more important, because you don’t want to be bogged down making every decision for your employees.  By asking them questions, and letting them arrive at the correct conclusion (with proper feedback and positive encouragement), you can solve the problem and empower them.  As Tim states numerous times in 4HWW, you want to eliminate yourself from the process. 

The second, and most important philosophy that is similar between Toyota and Socrates, is the concept of the open mind or in the words of Socrates “I know that I know nothing“.  This concept most manifests itself during formal individual or group problem solving activities.  Leaders of a Jishuken activity (translates to “self-study”) are chosen by their willingness to have an open mind. 

The reason for emphasis placed on the open mind is a student who comes in with a predisposed mind will not succeed because they will be trapped by the limits of their knowledge.  One first has to admit that you know nothing (in a relative sense, we’re not asking to forget everything!) before you can truly approach the problem.  The open mind is sometimes developed through exercises such as an Ohno Circle or impossible tasks that force the student to look at things in new ways. 

I have found that the more knowledgeable members of Toyota are the most willing to say that they “do not know.”  I have heard Plant Managers and Managing Officers admit their not knowing.  They did not feel the need to “seem to know”, but were perfectly calm with their ignorance.  Then in doing so, they would ask many questions that would both inform the answerer and themselves. 

One of the highest levels of TPS thinking comes from the OMDD group within Toyota.  Students are put through rigorous training for two years and then are expected to return to their group and level up the thinking of those around them.  The most important trait that an OMDD candidate is expected to have is an open mind. 

While I do not know if the Toyoda family had read Socrates in their development of The Toyota Way, I am certain that if they didn’t; they arrived by the same willingness to examine everything with a critical eye, most importantly, themselves. 

“I know you won’t believe me, but the highest form of Human Excellence is to question oneself and others” — Socrates

I’m almost embarrassed I haven’t seen this earlier, since I work for Toyota, but better late than never.  Not much to it, but it’s nice to see AR popping up in my neck of the woods.  The best part comes when they explode the car view to show all the parts.