Programmer Joe

SeAR July 2010: Augmented Reality the Next Next Big Thing from Joe Ludwig on Vimeo.

This is my talk from last Wednesday’s Seattle Augmented Reality Meetup. I will upload the discussion that followed as a separate video later today. Comments and feedback are welcome… just comment below or over on Vimeo.

The Problem

I am bad with faces. I mean really bad with faces. My brain just doesn’t seem to be very good at mapping what someone looks like with their name.  This often makes things difficult for me at networking events and conferences.

The Solution

A passive mobile application that scans the environment around the user for faces. When it detects a face that it recognizes the application speaks the person’s name to the user via their bluetooth earpiece. Ideally this solution would also involve a discrete camera that could operate without being obvious to the people it is operating on. The point is to serve as a passive aid to memory while not changing the behavior of the people you are interacting with.

The Competition

There are a couple concept applications out there along this line including Recognizr and Comverse Social AR. Both of these applications have the same problem, which is that you have to hold up your phone to take a photo of the person you want to identify, then wait for the result to come back.  That is intrusive enough that a simple “I’m sorry, remind me what your name is…” would be a better option.

The Pieces

1. Facial recognition. There are many providers of facial detection and recognition APIs, so it should be possible to license this piece. Unfortunately most of the providers don’t seem to be very good at licensing their SDK to people. I get the idea that these are all very small companies that spun out of someone’s PhD research.
   1. PittPatt never replied to my email.
   2. Luxand put me on their marketing email list, but never sent me an evaluation key.
   3. Betaface actually gave me a chance to evaluate their SDK. It works quite well. I wasn’t a fan of their licensing terms, but you might have different needs than I did.
   4. Ayonix got back to me right away but never provided the promised evaluation link.
   5. I don’t remember if I contacted Seeing Machines or not.
2. Bluetooth camera – I bought an OptiEye. It works pretty well. If you ask them nicely they will send you the protocol documentation. The specs claim a four hour battery life, which is plenty for most networking events.
3. Text to speech – I haven’t done any research here.  Many applications do it, though, so I would imagine SDKs are available.  If nothing else the user could record the names and the software could just play back the recordings.
4. Mobile computer – Both Android and iPhone allow communication over RFCOMM, which is what the OptiEye uses. Existing devices are also too weak in the CPU department to do much visual processing on the phone, but they could stream video or individual frames up to a server for further processing.

What do you think?  Dream product?  Interesting project? Terrible idea?

At TED 2010 Blaise Aguera y Arcas from Microsoft demoed live integration of video into the existing structure-from-motion dataset in Photosynth. Though his demo showed a video feed moving around a scene the same data could just as easily be turned around to find the precise position of the camera in real-time. That capability is a key part of building a head-mounted augmented reality system.

Two weeks later Google announced that they are incorporating user photos into Google Street View. This requires essentially the same data as Photosynth. Google has the added advantage that they can combine it with the Street View images and LIDAR data they are already collecting. Though they haven’t demonstrated real-time capability with this data they certainly have all the pieces they need to make this happen.

Access to the data required to perform pose recognition with cameras is a novelty at the moment, but if mobile augmented reality takes off in a big way it will become a key component of that system.  In my opinion this component is too important to be left in the hands of one company. A much more desirable situation would be to have an OpenStreetMap-type project to accumulate and curate a freely available dataset to provide structure from motion and pose recognition for use in mobile augmented reality and whatever other uses someone can dream up.

OpenStreetMap is a project that sprung up to provide access to data that was free from the costs and restrictions that come with commercial data. It uses a Creative Commons license to make the data free for use by anyone for most any purpose. Although OpenStreetMap came about in response to the restrictions on commercial data sources, the same approach could be taken for 3D structure and image data even though commercial sources for that data do not yet exist. If OpenStreetMap had existed when car navigation systems became feasible in the late nineties it is likely that many commercial products could have been developed on open data at far lower cost and in much more variety.

All such a project needs is a small number of dedicated people to get it started. Download a copy of Bundler (an open source structure from motion library based on the same research that spawned Photosynth) and seek out publicly available photograph libraries. Then talk a cloud computing provider into sponsoring the project by hosting the data and build things up from there. The project won’t have many users for a few years, but as the accuracy and coverage of the dataset grows the set of applications based on this open data will grow too. Somebody just has to get the ball rolling.

This rounds out my trilogy of year end posts. Here is what I think will happen in the coming year. I would love to hear your thoughts on these predictions:

1. Star Trek Online will be the only significant MMO launch in 2010. It will do well enough to make Atari and Cryptic plenty of money, but will not do nearly as well as World of Warcraft, so many people will consider it a failure. (Those people are dumb.)
2. At least three of the major unreleased MMO projects will be cancelled. I have a guess about which two are most likely, but I’ll keep that to myself.  Qualifying projects include:
   • Guild Wars 2
   • Whatever Carbine is working on
   • That console MMO Turbine hasn’t said much about
   • Whatever Trion is working on
   • The Sci-Fi channel tie-in MMO that Trion has said they’re working on
   • Whatever Zenimax is working on that may or may not be Fallout
   • Whatever 38 Studios is working on
   • The Agency
   • Whatever Gazillion’s Gargantuan studio is working on
   • Star Wars: The Old Republic
   • The second MMO that CCP is working on down in Atlanta where all those White Wolf people are.  Hmm. What could it be?
   • Whatever Red 5 is working on
   • DC Online
   • That other MMO I know NCSoft is working on that is completely under the radar
   • Whatever Slipgate Ironworks is working on
   • The second MMO Blizzard has in the works
   • APB
   • Jumpgate: Evolution
3. Project Natal and the Playstation Motion Controller will both come out.  Natal will do fairly well. Both controllers will allow some new kinds of games, but we won’t see any compelling examples of those games until 2011.
4. Unemployment will peak and then start to fall.
5. The compass+GPS augmented reality products will begin to shift to general location-awareness and away from their Augmented Reality roots. They will de-emphasize magic lens and start to emphasize aggregation of nearby content.
6. No consumer-level see-through displays will come out in 2010. Significant progress toward them will be made, but nothing will be released.
7. Neither Google nor Apple will release any kind of AR-focused hardware
8. The use of “wave it in front of your webcam” type AR in advertising will peak with an AR-enhanced ad in the Superbowl.  The backlash will begin. By the end of the year the advertising world will have moved on.
9. Apple will release its tablet and a new iPhone (faster and more storage) but won’t release anything that is specifically an AR product.
10. Apple will address the pain caused by its app-store approval process, at least in part. I have no idea what their specific solution will be, but they aren’t going to let their developer community grow to hate them.
11. Android will continue to pick up steam. By the end of the year Android will boast 50,000 applications.
12. People will spend the entire year trying to find something really useful to do with Google Wave. They won’t succeed in 2010.
13. Google will make Wave interoperate with email. This will make it useful as an email client if nothing else.

Ok, that’s the last of this kind of post for at least a year.  If only I could get back to posting regular stuff again.

Around this time of year for the past few years I have written a blog post listing what I expected to occur during the coming year. Since this new year marks the start of a new decade, I thought I would start a new tradition and write a post on my expectations for the coming decade. 2020 is a long way away, so I’m sure most of this will miss the mark. Hopefully at least 48 year old me will be amused by what 38 year old me had to say.

Please note that just because something is on this list does not mean that it’s something I want to happen, only that it’s something I think will happen. Anything that’s missing from this list is probably just something I didn’t think of.

I would love to hear your thoughts on any or all of these.  Please comment below.

General Technology Trends:

1. Moore’s Law will continue to operate for the entire decade. That means a given form-factor of computing device will be approximately 100x the power of the same form-factor today.
2. Mobile computing will dominate. Everyone who owns a laptop or desktop today will have a mobile device that is about 10x the power of their current computer.  We may still call these “phones”, but placing voice calls will only be one tiny part of what they do. This device will replace most users’ desktop and laptop computers.
3. Digital Distribution will be king. Only a tiny fraction of the media that’s currently consumed digitally (TV, movies, music, and software) will be purchased on a hunk of plastic. Both the subscription model (aka Rhapsody or cable television) or the purchase model (aka iTunes or DVDs) will have at least 20% market share, but one of those two models will be gradually taking over. Advertising supported media will be just as big of a deal as it now, but the user will have much more control over how they consume that media (think Hulu rather than broadcast television.) Books are on the same trajectory, but in 2020 the majority of books will still be sold on dead trees.
4. Speach recognition will gain a lot of ground as the primary way we enter text into a computer. Offices are one place where this trend won’t have advanced very far mostly because of the noise involved.

Game Industry Trends:

1. Total revenues from video games of all kinds (including mobile and social games) will exceed revenue from movies and television (independantly, not added together.) Games will finally learn to exploit merchandising and secondary markets as vigorously as movies do.
2. In 2020 no one will be selling a dedicated gaming console. All computing devices in production in ten years will be about consuming other kinds of media just as much as they are about playing games.
3. Desktop PC gaming will be all but dead, with the majority of triple-A games coming out for multi-media consoles or mobile devices.
4. Gaming that involves exercise will be the primary way that the majority of people get their exercise.
5. Location-aware games will be common.

Augmented reality:

1. A growing minority of people in the developed world will wear heads up displays almost all the time. These displays will be capable of information overlays, but will mostly be about contextual information that is not overlaid on the world. These products will be on the verge of hitting the mainstream, but won’t quite be mainstream yet.
2. Development of these displays will be by small companies (perhaps companies that are around now) but those companies will be acquired by massive consumer electronics multinationals before wearable displays hit the mainstream.
3. Recognition of people and text in images (and video) will be nearly perfect, at least in reasonable lighting conditions.
4. Gestural interfaces will be commonplace. Many hard-core computer users will be sad at how clumsy they are compared to keyboard and mouse.

The fate of specific companies:

1. Google will be huge and influential. Their influence will likely peak in the 2010s, but it will difficult to see that from the ground. Google will have had some sort of anti-monopoly action taken against them.
2. Microsoft will fail to transition to the new mobile-centric world and will be in decline. They will still be a very powerful multi-billion-dollar company, but will not own the end-user to nearly the extent they do now.
3. A company that exists today will be the dominant social network.  that could be Facebook, Twitter, or YouTube, but it probably won’t be MySpace.
4. Apple will be huge and influential. They won’t ever be as dominant as Microsoft was in the 90s, but they will be very successful. Steve Jobs will still be running the company.

US Politics:

1. Gay marriage will be legal in most states.
2. Marijuana use will be legal in California and a few other states.
3. We won’t have elected a woman president. (My wife came up with this one, but I agree with her.)
4. The problems of illegal immigration will not be solved.
5. The problems of providing health-care to everyone that needs it will not be solved.
6. Privacy in an age of always-on location-aware devices will be a huge topic of debate.
7. Silicon Valley will remain the world’s premier startup region.
8. The US will still have troops in both Iraq and Afganistan. These will be like the troops we still have in Germany and South Korea, and will not be in combat often, if ever.

International Politics:

1. Carbon emissions will be at approximately their peak in 2020.
2. Oil production will also be peaking around 2020.
3. Most other countries will be ahead of the US in terms of switching to renewable energy.
4. Most of the rest of the world will have consumer-friendly privacy regulations in place. Those countries will scratch their heads at the debate raging in the US.

Things that will not happen:

1. We will not have flying cars, jet-packs, or most of the other things promised by Sci-Fi in the 50s.
2. There will not be peace in the middle east.
3. Africa will still be the poorest continent.
4. Brain-computer interfaces will still not work very well. No one will be uploading themselves into a computer.
5. We won’t have a human equivalent AI.
6. We won’t know how to reliably unfreeze people.
7. World War Three won’t have happened.