The Dangers of Computer Vision.

(I thought about calling this article “The Passive and Insidious Collection of Data with Cameras” or “Should We Fear Facial Recognition?” but I think the current title gets the point across.)

Computer algrorithms have reached the point that facial recognition has become easy enough to become common place.  They can even recognize emotions and other bodily conditions.

This Brazilian police department is to use AR glasses to help them distiguish criminals.  Usage such as this seems innocuous and frankly, is an application that most of us would cheer.  Who doesn’t like the idea that our police officers could pick out the terrorist in the crowd just by scanning the faces.

And by now, most have heard of Google Goggles, the program that allows a computer to recognize everyday objects.  Right now it does better with artwork or specific objects, but they’re working towards a complete knowledge of any item or person.

Does this trend of more advanced computer vision put our privacies at risk?

Imagine a world in which everytime you go into a Starbucks to get your double-mocha latte, a camera hooked to a computer owned by Starbucks recognizes you and notes your time of arrival, your facial expressions, your clothing (which indicates your income status), who you talk to, what products your eyes linger over, do you sing to the music playing in the store and a host of other details.

Would you be worried about this collection of your personal data?  In some respects, the data is there to make my purchasing experience better.  Using advanced data mining techniques,  Store #452 knows that the collection of 54% female and 46% male with a median of $74,329 per year salary likes light jazz to be played and prefers more muffins than donuts to be displayed.

Most people probably wouldn’t notice, but your Starbucks experience at Store #452 would become slightly better.  Just enough to take the edge off your early morning commute.

What if instead, the local police department, through the use of their traffic light cameras, has determined that you speed down Hallmark Hill every afternoon at 4:50pm?  So their crime analyzer program suggests that Officer DeLoop wait at the bottom of Hallmark Hill and catch you speeding.  Technically, you were breaking the rules, but maybe it was only because the hill is steep and it’s hard to ride the break down the whole slope.  The other 99.8% of the time you’re a good drive and obey the traffic laws.  How would that ticket make you feel if you knew how they’d caught you?

Or what if a national network of cameras is used to catch terrorists?  The computers sift and sort through millions of faces to find matches with known terrorists from satellite training camp videos.  The computer thinks that you, based on similarities in heights, gait and facial characterists, have an 86% chance of being a Belgian terrorist.  After a few weeks of interrogation and background check, your name is cleared, but what about the cost to your life?

Even after the event is over, how likely would it be that all traces of your “terrorist” connection would be cleansed from all government records.  Even now the government is nortoriously bad about removing incorrect names from the No Fly List.

As computers can do more and more passive data collection, how do we protect ourselves from complete and total personal invasion?  How should data be allowed to be collected?  Should we know where cameras have been placed?  Can anyone collect data?  Can be individualized and who has rights to it?

Cameras

Cameras first and foremost should be obvious.  Both where they are and when they are on and collecting data.  It’s bad enough that websites collect our P’s and Q’s without our permission and sell them.

For commercial use, cameras should also have a range and that range should only be on the store property and signs should be well posted that computer vision is in action and what time of data is being collected and how it’s being used.  That way, we can decide if we are going to come in contact with their camera and computer vision system.

In the public arena, camera placement is trickier.  Governments will want them placed everywhere and without an obvious marker showing their location.  To them, catching us and collecting data unaware is the point.

Aggragate data versus Individualized Data

There are two general ways data can be collected.  The first is by aggregating the data, keeping it untethered to any individual.  Trends and tipping points can be understood, but no individual is pulled out for scrunity.

An example would be using crowd counters to help place police officers in a big city.  Cameras could tell HQ where crowds were proliferating and HQ, based on the data, could send officers to be ready in case of unplanned for crime.

Businesses like Walmart might track general habits of its customers, noting products they pick up and put down, much in the same way a website tracks click-throughs.

Individual data is the more insidious method.  Instead of optimizing the store based on the cross-section of customers that pass through its doors, individual data would allow businesses to know who was in the store and what their preferances were and how to micro-target them with in-store ads and coupons.

The worst part is that a business could then sell that data to another business, much in the same way they collect and resell website data.  In fact, it would be likely that speciality companies would step in and do the actual camera placement, computer vision and sifting and sorting and then feedback the data to the store, while selling the rest on the open data market.

For the government, lines become very blurry, very quickly, and the imparative buzzword is security.  If the One Percent Doctrine can make premptive war a priority, why not keep a tigher watch on our higher risk citizens.  All in the name of keeping the children safe.

The telecommunications business has been in bed with the government for quite some time, letting them syphon off information and access based on the needs of our security.  The government wouldn’t even have to get permission from the telecommunications industry with a good national web of cameras.  They would be pulling directly from the source.  Us.

If this seems implausible, just ask Britons who are already being watched by the largest closed circuit TV system in the world.  Currently, it’s estimated that the government has one CC TV for every fourteen of its citizens.  That’s even more than China.  They’d like to install them in private homes, too.

What can we do about it?

Right now, it’s hard to say how the whole process will play out.  While we know that cameras will have the possibility of performing this kind of insidious data collection, it’s not in wide use yet.  But I believe, as technology like augmented reality spread and cameras become smaller and more mobile (this TV show is putting cameras on police officers), this will become a concern, so it’s better for us to become educated to the dangers and voice our opinions early and often.

But what opinions should we voice?  To this question, I offer a few points of reason.

The Data Collecting Credo

• Data can be collected in the aggregate without permission
• Individual data cannot be collected or sold without permission
• Cameras must be obvious and indicate when they are collecting
• Data used should be for the benefit of both parties
• Commercial or government data collection ends at the private property line
• Cameras cannot collect data outside of private or commercial property lines

As always, leave a comment if you have something to say.

The continued advance of computer intelligence, riding the rails of Moore’s Law, have made events like IBM’s Watson win over two champions on Jeopardy inevitable.  While thoughts of the Singularity, a future in which we cannot predict because computers will out-innovate humans, are fascinating to consider, I prefer a human centered future.  And augmented reality has a large place in a human centered world.

While the underpinnings of this chess trainer demonstration with its Hiro-markers are circa 2009, the ideas still play out.  Instead of concerns that computers like Watson will replace humans, I believe that Watson will help us take advantage of our natural abilities.  Instead of wasting time memorizing trivia, we can focus on the combining of dissimilar ideas into new synergies.

So I say, bring on Watson.  Just with a side of AR, please.

When writing The Digital Sea, invisibility was one of the cool effects I thought was possible with ubiquitous augmented reality.  I didn’t expect to see tangible examples so soon.  Granted, without AR glasses, all the effects are static on the screen and only eye candy.  But what glorious eye candy Fukatsu-san makes.  The predator alien would be proud.

The second video from yummyfuture shows us how to make ghost images of furniture (or whatever you’d like to do.)

I think I could watch new Kinect video’s all day.

The Kinect has become the all-in-one sensor bar of choice for modders everywhere.  The ability for the Kinect to do facial and gesture recognition, sensing 3D under any ambient light condition, combined with a rapidly expanding hacker tools, has made it indispensable for true AR.

While the Kinect Hacks site has been documenting the every aspect of the accelerating progress, I’m just interested in the applications for augmented reality.  This one caught my eye today as the implications reminded me of one of my favorite dystopia films – The Running Man.  The resolutions and seamless adjustments are decades away, but it’s fun to imagine anyway.

A little real-time camera trickery and some augmented reality, and wa-la…invisible cube.

A camera fixed on the concrete cube sculpture recognizes the presence of human faces within its scope. With a randomized choice it will focus on one of the bystanders and adjust its movement to his; tracking the eye movements of the viewer, a software computes the corresponding angle of view projecting onto the cube the very section of the space the sculpture is blocking from the viewers eye; thus making the cube appear transparent.
The video sculpture, Durchsehen, Exp. 01 (Augmented Perspective) overwrites the common notion of perspective and plays with the significance of perspective in an art historical perspective; the work of art evades the gaze of the viewer or rather: the two are equated. The gaze of the observer coincides with the object of observance in a piece that also draws a line to former strategies of dealing with vision and depiction: the renaissance praxis of “painting on glass”.
Through the real-time projection on the cube a 3dimensional depiction of 2dimensionality occurs; the catoptric turns dioptric. The framing plane of the conventional video image becomes fragmented as work and reality intertwine in an augmented perspective.

Learn more about it from the creators Daniel Franke and Markus Kison.

A few months ago Staffan Dryselius made a splash on Team Hack-a-Day with his DIY data glasses.  Since then he’s been working with a team to improve his design and would like to form a larger partnership with anyone interested in working on or owning a pair of AR glasses.  Having a working HMD for augmented reality would help the technology gain wider use.  Currently, we’re stuck with magic lens or web cam AR if we want to play with our favorite technology, though both have come a long way since early 2009.

The group is calling the glasses Maxcware (website not fully functional yet, but contact Staffan below if you want to join.)  If you’re not familiar with the reference in the name, I’ll give you a hint.  The name is from a science-fiction novel from this decade and if you haven’t read it, I highly recommend it (and it’s in the AR reading list.)  If you’d like to contribute to the group, contact him at staffan (at) maxcware (dot) com.

So to learn more about the project, I sat down to interview the man behind the vision (pun intended), Staffan, and since we’re talking about a visual medium here, I’m going to show you the glasses before we get to the interview.

Tom:

Interest in commercial HMDs has increased with spread of
smartphones.  Why did you decide to tackle this problem that the glasses
makers have failed to deliver on?

Staffan:

I had more or less despaired about any non-heinous, high-resolution
see-through HMDs emerging in my lifetime when Vuzix showed off their
new Wraps at CES 2009. When all they finally delivered to the market
turned out to be an opaque lump of plastic, I had finally had it. I
started to suspect that the public would continue to be spoonfed
incremental yesterware more or less forever. No single maker would
have the guts to make their inventory unsellable by launching anything
really nice and new unless forced at gunpoint. I would never get the
glasses I wanted unless I made them myself.

I first got excited about HMDs some time around 1995. There was lots
of interesting research done at the time, and also quite a few
companies advertising products “soon to appear in a store near you”. I
think it was around -96 something that Sony actually launched their
Glasstron model, and there was also the “Olympus Eyetrek” soon
afterwards. I however decided to wait a bit, great things seemed to be
just around the corner. Especially one company, Digilens had an
awesome idea for optical see-through AR-type displays using switchable
Braggs gratings in 98-99… I was very excited at that one especially
(the company has by the way resurfaced as SBG Labs with yet another
vaporware design).

Then the dot.bomb exploded, and everything digital died. “Virtual
Reality” became “Definitive Nonexistence”. The headsets by Sony and
Olympus were phased out, and the Digilens homepage died shortly after
they decided to do fibernet switching chips rather than HMDs… That
was more or less the situation for many years, and I was very
disappointed and soon promised myself to try and forget all about
HMD:s until I saw an advertisement for something really good I could
actually buy in a shop.

Since then, I have read several science fiction books featuring HMDs,
seen the developments in smartphone AR emerge and again felt
frustrated about the non-existence of  useful HMDs.

Enter 2009 and CES. Vuzix were showing their new Wraps. Wow! At last!
I couldn’t wait for the release date for their fabulous new optical
see-through displays! The disappointment was what made me do it. Even
though I didn’t really know how, I had to give it a shot.

Tom:

From the picture, the screens appear to be non see-through.  Is there any
possibility of making them see-through so true augmented reality can be
accomplished?

Staffan:

Yes. And that is the plan too, of course. The first step is to add a
camera to the glasses to feed the display ambient video blended with
digital content. It is much neater not having to point the camera of
the phone itself around to use AR applications in the glasses, as must
be done today. As soon as possible we will also want to add
accelerometers and magnetometers to the glasses.

The top half of the glasses will continue to be completely clear.
There is no need to expand the physical screens any further, only the
virtual screen estate. Those two are quite separate entities, but it
is only when keeping the optics sufficiently close to the eyes that
this becomes really obvious. It’s like peeping through a keyhole: Keep
your eye close enough and the aperture lets you see the whole room.

From the beginning I saw the “see-around” (or rather “see above”)
design combined with “really near eye” optics as just a pragmatic way
to make something useful with available technology. However, a very
nice aspect of the “really near eye”-design is that the physical
movements of the eyes can actually become useful instead of being just
another engineering obstacle. It is especially useful that the eyelids
work as natural shutters, switching to the view that is most
appropriate for the moment. When looking straight ahead or upwards,
the lower eyelids completely block the screen so that light from the
displays doesn’t disturb the natural vision. When looking down, the
upper eyelids block lots of the ambient light that may otherwise
bleach the screen.

Apart from a convenient way to keep alive when traversing a street, it
also means that camera see-through becomes practical. At first, the
mere thought of camera see-through made me shudder. Although that
solution can more or less immediately be used together with Layar and
all the rest of the applications for smartphones, both limited field
of view and latency are fierce problems to combat without a
possibility to momentarily switch to complete see-through. The latency
may not seem too bad at first, but try and navigate while walking at
any speed using only the viewfinder of a video camera. Fixing a camera
to the glasses is far worse and reacts to every jerk of the head. To
keep the screen from bobbing about, you have to take it real slow… If
motion sickness is not enough to make a person reconsider, then the
inevitable robotic choreography should inspire second thoughts about
testing the concept in public.

Tom:
How do you envision the use of these glasses?  Hooked up to an iPhone or
Android (or whatever smartphone you use) to project the screen realtime?  Or
some other usage?

Staffan:

All that is needed is connectivity and some basic sensors. They’re all
there in today’s smartphones, so yes, the glasses will hook up to
them. As many different makes as possible and as easily as possible.
As for uses… Wow! Where to start?

…Humanity is a little like the first amphibians. We’re popping our
heads above the surface of the primordial soup right now. There is a
completely new digital world in the making out there. We are just not
very well adapted to take part in it, and the interfaces we use today
are laughably inadequate for interaction. With AR glasses we may at
least get up from our asses and shut the door on the cubicle. Reading
company spreadsheets can be done just as well on the way to the beach.
The best ideas may come to our mind when we are in the supermarket,
only today we forget before we’re back at the computer. No more so. A
digital post-it or email is quickly edited in the corner of the eye.

But work and “productivity” is boring… Instead Google should be there
with us when we see a new butterfly in the park. Getting lost in the
city in the age of GPS? –That’s laughable! Directions should be where
they belong, as AR overlay. No more getting scammed in a shop. The
barcodes should trigger balloons with user tests and best prices on
the go. Blogger? -Updating the skateblog should be done when we are
actually up and rolling, complete with action footage and biometrics.
Why make do with just the normal senses? Nightvision? –No problems.
X-ray vision may come in handy while sharking by the pool, just pop
out the IR-filter if you are so inclined. Bad-hair-day? –Put on a
digital wig and a happy face. Bored? –Just connect to a robocam in a
Tokyo bar. Going to a meatspace party? Bring your avatar buddy along…

Starting to sound outlandish yet? SciFi? I say all this is very close
at hand, and we just need to light the match to set the digitality
ablaze. It is long overdue…

Tom:
You mention on the hackaday post that the image is doubled on the two
screens.  Have you figured out how to split the image to get the true 1280
width?

Staffan:

More or less. We will probably want to device a completely new
graphics card instead of doing too many hacks on the original MyVue
PCB, but it is doable even on that one. I’m just afraid that we may be
wasting valuable time going down that alley too far. I think it will
be better to put something together that doesn’t require un-human
soldering skills to work. I want to put together a more manageable kit
instead so that as many people as possible can get involved. The Kopin
displays are however well documented, and there is no magic involved
in interfacing with them. I  have a friend working on it, but don’t
want to push it. In the Hackerspace groups I also mention, we are
getting better organized. There is now a webspace up and running (for
our internal purposes as of yet), and we are putting together a
“to-do”-list allocating work-packages for the different members. The
front-page of Hackaday gave some new contacts too, and I’m having
serious pangs from my conscience for not handling them yet! I’ve been
lazing away with the family doing things like sailing and the like…

Tom:

How much would it cost if someone wanted to make their own pair?

Staffan:

A pair of MyVu glasses cost about $150 on eBay. Add some Fimo putty
and a pair of oversize-sunglasses (the kind that fits over regular
glasses) plus a couple of days work (depending on skill), and you have
a crude but passable pair.

If you want something better, you may download the meshes for the pair
I have and order better frames from an online prototype maker. I don’t
have the figures for how much that would be, but that is a quite
expensive alternative. Better then to wait until I can fill an order
with a Chinese factory. A box with a 100 pairs will cost about $100 a
pair.

There is then the video card, better battery and case… No figures there yet.

Tom:

Why are you going about this as an open source project?

Staffan:

Further, those who do understand say I’m either mad to disclose
everything on the net and to loose an excellent business opportunity,
or call me names for destroying the patentability for others. I
usually retort that the industry hasn’t moved at all for a decade, and
why do they believe I would fare any better? I also try to line out
the difficulties with classical innovation processes. I’ve been there,
on both sides of the fence. As inventor and as executive in a joint
industry-governmental innovation system. I know how bad it can be.So
much time and creative drive can be lost in anger over incompetence,
greed and dishonesty that you simply don’t want to think at all about
your project.

I believe that instead of getting entangled in patenting processes, VC
negotiations, hunting for (competent!) technical consultants,
marketing and manufacturing partners, it would be far better to copy
some applicable concepts from open source software development. With
the glasses, I want to perform an experiment. I would so much want to
put together a really nerdy team of developers that are driven by the
fun of problem solving and a feeling of contributing to a community
rather than for direct economic benefit. Not that there may not be a
chance to make some cash one day for everyone involved, only the money
should not be the driving force.

Here is a good clip to illustrate what I mean:
http://www.youtube.com/watch?v=u6XAPnuFjJc

Naturally, hardware is much more difficult to develop in a community
than software. However, I believe the time may be ripe for a test of
the concept. If any project can succeed, I think AR glasses is it. The
time should definitely be ripe for technologically inclined people to
want those for themselves. I know I definitely do. The basic concept
is also in place, and both PCB:s and plastic/mechanical components are
quite cheap to manufacture these days, even in singular quantities.
Many of the potential combined early adopters and developers will be
able to make their own glasses and feed the loop.

I can see an emerging ecosystem where different participants can
specialize and even start to make some money from selling
non-complicated sub-systems. There is also potential for spin-off
projects and services that will benefit from AR-glasses. There are
many angles to this experiment…

** End of Interview **

Whew.  I agree, Staffan, the time is ripe for a technologically savvy group to tackle the AR glasses problem.  And given the importance of this little piece of hardware to the overall AR ecosystem, I think it’s worth our time and hopefully worth your time to join this project if you have something to give in the way of knowledge, expertise or time.

So stop by Maxcware or contact him at staffan (at) maxcware (dot) com if you’d like to contribute.  Or at the very least, sound off your encouragement at Games Alfresco.

We’re a long way from fooling the man on the street, but augmented reality may soon play trickery on your TV screens.  This video montage combining scenes from movies like The Running Man and Wag the Dog; and bits of real-time AR from the last few years makes for a convincing argument about the future of this new medium.

I’m partial to such obscuring of reality as it speaks to my science fiction interests.  Whenever I see a video like this is makes me think of one of my favorite authors, Philip K. Dick.  And while this argument is probably twenty years too soon, the ubiquitous use of AR may eventually enchant the populous with its devious wares.  Though you may scoff at such influences, think long and hard about how current technologies and techniques subject the masses to keep the absence of reason as their masters.

But like I said, that argument is twenty-years too soon, or twenty-years too late, if you consider the wealth of propaganda techniques which by and large are more insidious in their use.  Of course, the real danger is not that sophisticated techniques will be developed to hide or alter the truth, its that they will know everything about you so that they can tailor their obfuscation to maximum effect.  But I digress.

A workable HMD augmented reality unit is a major step forward for the technology.  Once a see-through model is available, we can break out of the “magic lens” smartphone.  However, a see-through HMD is still a couple of years away and until then we’ve got to make due with what’s currently available. 

Craig Kapp, a full time graduate student at the Interactive Telecommunications Program (ITP) at NYU, has put together a working HMD unit with the Vuzix VR920 model and the ARCam. 

When I contacted Craig this is what he had to say:

Thanks for the message. I am using the Vuzix VR920 head mounted display in conjunction with their brand new “CamAR” attachment – it’s a snap-on webcam that fits snugly onto the front of the goggle. Here’s some more info on my blog about the project.

I’m working on a larger project that involves these goggles + a voice detection package to create a sonically controllable AR environment. Feel free to check back in a week or two for an update!

While a unit like this would never be usable by the masses, it could allow researchers and hard-core AR enthusiasts to create early full view games and apps.  I’m looking forward to hearing how Craig’s project turns out.  Maybe some of the other AR gear heads could lend a hand with making the project more immersive or porting it to a smartphone. 

On the heels of my post yesterday about the Outernet Guidance Initiative and the perils of data harvesting, I find this bit of flotsam foaming out of the datasphere.  Don’t be fooled by the lack of presentation fireworks or the bleary-eyed speaker reading his lines early in the morning without his cup of joe. 

The MobilHD system from Sensor Cast carpet-bombs an area with cameras for complete coverage.  The data is downloaded into a server that indexes the images into a dizzying array of statistics.  If you’re on the paranoid side of the data gathering scale, then this little piece of hardware will certainly set your sphincter to pucker. 

The specs on the system in question are gathering data in a generalized manner without attaching your personal ID, but with facial recognition software available it wouldn’t be hard to marry the two and create a gold mine of data harvesting. 

How much longer until we’re signing digital waiver forms when we enter the mall?  And Cory Doctorow’s gait-recognition hi-jinx for his protag from the book Little Brother don’t sound too far off.