Augmented Reality: An Interview With Joseph Rampolla

One of the nice benefits of publishing this blog is that it has allowed me to talk to a variety of fascinating people who I normally might not have met.  Joe Rampolla is one of those people.  Joe and I were having a discussion about digital forensics when at the end of the conversation he asked me if I had ever heard of augmented reality. I had to admit that I knew nothing about it. Joe proceeded to amaze me with his level of knowledge on the subject and I was intrigued with the potential for both good and evil that this technology brings with it. I asked him if he would be interested in doing an interview for the blog and he readily accepted. As Joe will illustrate for you during the course of this interview, augmented reality has the potential to fundamentally change how we interact with each other and the world around us.

Professional Biography of Joseph Rampolla

Joseph Rampolla has been a law enforcement officer for sixteen years. In 1994 he received a Masters of Arts degree in Criminal Justice from John Jay College in New York City. Joseph holds a Bachelor of Arts degree in Law & Society from Ramapo College of New Jersey. He became a police officer in 1995 and currently holds the rank of Lieutenant for the Park Ridge Police Department. He has supervised numerous criminal investigations within the department and oversees the Detective Bureau. In 2003 he was assigned to a regional computer crimes task force. He has successfully completed training offered by county, state and federal agencies as well as leading technology companies with a focus in the areas of computer forensics, Internet child exploitation, cyber-bullying, cyber counter-terrorism, human trafficking, and Peer-to-Peer file sharing investigations. He is a member of the HTCIA, HTCC, and IACIS where he has earned the classification of Certified Forensic Computer Examiner (CFCE) and AccessData Certified Examiner (ACE). Joseph enjoys teaching the topics of cyber crimes, augmented reality / virtual worlds, cyber-bullying and advanced undercover Internet Relay Chat (IRC) investigations. He has taught International law enforcement at Microsoft in Redmond WA, in the Ontario Canadian Providence, and taught cybercrime topics to all levels of law enforcement for the National Internet Crimes Against Children Task Force. Joseph was the co-creator of the Internet Safety DVD Series – Point of No Return which featured the cyber-bullying video “Sticks and Stones” and the predator video “The Web.”

AFoD: You and I had an interesting discussion recently where you educated me on the topic of augmented reality. I found our conversation so fascinating that I had to get you to do an interview for the blog. Let's start with the basic question. What is augmented reality?

JR: Augmented Reality is taking digital or computer generated images and overlaying them over a real-time environment.  The best way people would relate to Augmented Reality, better known as AR, is by thinking of a fighter pilot.  We have all seen the view or Heads Up Display (HUD) of a pilots view that shows a digital overlay that shows an artificial horizon, the digital altitude, digital speed, and a host of pilot dashboard information seen looking out the cockpit window.  Another example would be the artificial first down marker that helps football TV viewers know how far the offensive team needs to go to get a first down.  We know that the yellow line is not really on the field, but once the digital overlay is placed into the live environment, it assists millions of viewers in a real time environment to enhance our viewing experience.  These examples are primitive compared to new exploding uses of a technology that has been around for quite a while.  The high quality of cameras, huge data and bandwidth pipes, along with the powerful computing power of smart phones has created the perfect storm.

AFoD: Why is augmented reality something that technical investigators need to be concerned about in regards to smart phones?

JR: There are a number of reasons why technical investigators need to be concerned about smart phones.  We are moving to a dominated mobile platform period (which will continue to increase in time).  All of your social networking and communication platforms will have a mobile presence.   The following types of investigations to name a few: Child pornography, cyberbullying / sexting, harassment, stalking, corporate espionage, digital piracy, terrorism, gang recruitment - will all move to a ubiquitous mobile platform.  The amount of mobile apps coming out each day is staggering.  The Android and iPhone smartphones are increasing in use which forces technical investigators to shift their focus to mobile forensics and concentrate on the mobile application programming interface (API).  A digital forensic shift in value is moving from the home PCs to mobile smartphones.  These smartphones give the user all the access they need to check email, text, check and post to social networking sites which will reduce the manhours of home PC use in my opinion.  This premise means more digital evidence nuggets will be on smartphones then on home PCs.  If technical investigators do not get ramped up on mobile forensic trends, they will find themselves reading by candlelight and writing with an ink and quill.  I find myself more reliant upon my smartphone then my home PC.  It is more critical for me to backup data and apps on my Droid then thinking about my mobileless home clunker.

AFoD: Can you give us some examples of how augmented reality would manifest itself in a way that would be relevant to an investigation? Just how are the bad guys using this technology?

JR: Because this technology is finally gaining traction, the better question will be "Just how WILL the bad guys use this technology."  The technology needs to become mainstreamed first and be ubiquitous before truly being utilized by perverts, crooks, criminals and terrorists.
This technology is poised to explode into mainstream society but has not done so yet.  It is currently being used in marketing and advertising realms at this moment but that should change very soon.  Companies like Viewdle and Polar Rose (which was recently acquired by Apple) are beta testing Augmented Reality and facial recognition.  As Augmented Reality facial recognition technology blends with social networking mediums, we will see issues of stalking, identity theft, harassment and other criminal uses.   The porn industry is investing large sums of money into augmented reality which will naturally pave the way for child pornography uses.  Wherever society finds pornography, child pornography is not too far behind.   Currently the porn industry has AR markers that can be held up to a web cam and show the viewer being surrounded by porn video clips.  This gives the viewer the experience of feeling like they are in the pornography and to give the illusion that they are part of the experience as opposed to being a remote viewer of the "action."  Imagine a scenario where a virtual avatar or character could sit at your kitchen table if you were wearing augmented reality glasses.  The glasses would show a digital depiction of that avatar sitting in front of you which could conspire on how to commit a criminal act while the person who is controlling the avatar could be safely out of reach of the US government.  This raises serious concerns for US National Security and US Law Enforcement.  Virtual criminal packages could be left in public areas and could only be detected by someone that is part of that Augmented Reality / virtual network.   The iPhone has an app called Tagdis.  You can write virtual graffiti on a public building or police station.  This virtual graffiti can only been seen with the use of the smartphone app.  Criminals, drug dealers, or other miscreants could leave virtual markings or clues for other criminals in virtual space and a person unaware of that digital space would pass by that location with no knowledge of the virtual message.
Digital investigators need to be aware that a new digital space will emerge with important evidence that will be related to future crimes and societal digital markings.  Digital investigators will need to focus on the remnants of evidence that will be left on smartphones and other electronic devices.  AR will change how we and society sees things just as we forget that invisible signals and beams are flying through our atmosphere and environment right under our naked eye.

AFoD: This is amazing stuff, Joe.  Before we dive into this further, can you recommend any resources on the web that might offer a visual illustration of what you are talking about?

JR:

Facial Recognition:

http://www.youtube.com/watch?v=0QBLKBYrgvk

http://www.youtube.com/watch?v=x0FasRTTk4k

Turn People into experts with Augmented Reality:

http://www.youtube.com/watch?v=P9KPJlA5yds

Augmented Reality HUD display - Vehicles

http://www.youtube.com/watch?v=REXer_yW6S8

Topps Baseball Cards

http://www.youtube.com/watch?v=I7jm-AsY0lU

Augmented Reality Pornography

http://www.youtube.com/watch?v=5GXuS1N1SSM

Planefinder AR

http://www.youtube.com/watch?v=b64xvlOvdlM&feature=related

AFoD: We're really just at the starting point of something very big with all of this, aren't we? I'm struck by how this could be used for some very positive things as well as for some deeply evil acts.  Do you have any concerns about this technology from a cyberbullying standpoint? For example, I can see how instead of just writing something nasty about some poor girl on a bathroom wall, you could now write those things on her virtually through this technology including how she can be located or contacted.

JR: You raise some very good points.  There is definitely the ability to use this technology for cyberbullying or dissing people through public virtual postings.  I got a better one for you.  People start to advertise their virtual ads over the virtual space in times square.  I put a big virtual banner for Guess Jeans in virtual space over the JCPenny real billboard in Times Square.  Am I violating any advertising cybersquatting laws? (maybe cybersquatting is the wrong word on this one) Who knows.  I am sure the lawyers will come out of the woodwork on this one.

When it comes to cyberbullying I guess the issue will be could you anonymously put stuff in a virtual realm without being traced.  I guess this depends on what open virtual networks are created and how can a user anonymize their posting without a trail leading back to the person that posted it.  Will throw away smartphones be accessible to the public to pull this off?  Will nefarious people post virtual misinformation in the virtual realm to create confusion, spread lies / rumors.  It will be interesting to see where things move with this technology and how the business model takes it.  Advertising agencies are the leader in this area at the moment and the porn industry is investing a lot of money too.  I have more questions than answers unfortunately.

AFoD: As you look forward to the future, how do you think this technology will impact our culture from a social standpoint as well as how we conduct business?

JR: Augmented Reality will be a game changer.  It will have a huge impact on a culture in many different ways.  For starters, it will be embedded in social networking sites.  Social proximity networks will thrive with augmented reality because geolocation will change the way our society and culture sees technology.  People will utilize the scanning of people with smartphones and tie in to a person’s social networking profile.  Kids and adults will scan people in their live environment to identify each other and will reveal who we are and give glimpses as to our posts, our blogs, our profiles, and where we go in our daily lives.  People will voluntarily allow open networks to thrive and share real time information.  The scariness begins when closed networks begin to form.  Closed networks of criminal organizations, predator networks and secret societies will share information about law enforcement, adolescents, and other nefarious groups that share common criminal goals.  Stalking, identity theft, terrorism networks and beyond will put this amazing technology for bad purposes will eventually rear its ugly head.  Markerters, advertisers, and commercial applications will boom.  It will become integrated with our thirst for realtime information and aid in what items we buy, the groups we want to be a part of, and even groups that we are unaware of that want to know more about us.  This creates a slippery slope where many users are unaware of other users real intentions of why they want this information about us.  Think of the child predator groups that will form to trade information of unsuspected children who post their geolocations without a true understanding that they may be in harms way. Augmented Reality applications tied into devices such as the AR Parrot (remote helicopter) will create drone devices that can spy in hotel windows or spy on innocent victims.  There does not always have to be nefarious purpose to utilize this technology but eventually lawmakers will have to reconsider stronger privacy laws to protect society.  The long term effects will become apparent as the technology goes mainstream.  Society will never be the same as AR unfolds before our eyes.  Society will enjoy it but also can be repulsed as AR crimes began to take hold in our communities and our society.

AFoD: If you could gather all of the people who are going to be planning, developing, and marketing augmented reality applications into one room, what would you tell them? It sounds like a proper Uncle Ben speech is in order at a minimum.

JR: If I could gather all the stakeholders regarding AR into one room my message would be clear and simple.  The Wright Brothers took their first powered flight for 12 seconds in 1903.  Little did they know that their innovative efforts would pave the wave for Stealth Bombers and planes used as missiles in the 9/11 attacks.  I have had the pleasure to speak to many brilliant AR creators and developers who are benevolent and kind and want to have a positive impact with technology on our society.  I ask them to wear an "evil cap" also when developing new innovations and to think about the downside to this amazing technology.  Most developers would have no reason to think like a bad guy since it does achieve their final goal or complete their project.  Think of how criminals and miscreants can create harm and danger when developing AR.  Let’s not make it so easy for criminal elements to capitalize on this technology and get input from law enforcement, psychologists, child protective specialists, and other important disciplines while advancing technology.  I think it is important for all sectors of society to work with AR developers to create safe applications while limiting the knowledge of those with bad intentions.  It’s like having the formula for a potential atomic bomb and posting it on every blog in the world.  We must do everything in our capability not to release that dangerous blueprint to the wrong set of eyes.  Some could argue that it will get out anyway so why bother.  I would rather know that AR innovators took every precaution to safeguard this technology while utilizing this incredible innovation that will change our society like never before.

AFoD: What should lawmakers being doing at this point to address issues pertaining to augmented reality?

JR: Obviously lawmakers and the Criminal Justice System need to be up to speed on AR.  The next 12 months will be a good indicator on how this technology rolls out and advances.  As I mentioned previously, social proximity networks will become more popular as you can geographically see other "live & realtime" users within your immediate region.  I checked in with my FourSquare account the other day at Newark Airport and saw 63 other users check in around me (at that moment).  I could see their pictures, their posting, and a host of information they allowed me to view.  I could have easily stalked some of the FourSquare users since I knew what they looked like and what terminal they were sitting at as they awaited their flight.  As I viewed the other places they checked in and analyzed their friends, I could clearly profile their lives.  Some were young college students, others were struggling musicians, and some were successful white collar workers.  I saw their likes and dislikes and could easily approach them and capitalize on their unknowing "book of knowledge" they projected into cyberspace.  I couldn't help to think of the shrieking sound of "Danger Will Robinson...Danger."

AFoD: What else would you like people to know about augmented reality?

JR: Augmented Reality is a wonderful technology that could do so many good things.  I don't want to seem like an alarmist but having our heads in the sand is not productive either.  I want to work with developers in finding ways to use this technology to track bad guys and help law enforcement protect society.  Think of a wide scale emergency where Augmented Reality apps can lead a person to safety or alert them to dangers areas is an important arena.  Let’s imagine that there was a Hazmat spill in an urban setting.   Users with AR equipped smartphones could be alerted to the dangerous toxic cloud and use the AR app to evacuate the area.  Emergency crews could use AR to find victims or routes that will assist their rescue attempts while limiting their exposure to harmful gases.  AR can be used for an enormous amount of positive uses.  I don't want people to miss that point as I get out my message.  I want to work with developers and law enforcement officials to find positive ways to enhance our profession and safeguard our citizens.   I sometimes get blank stares from developers because when they hear that I am in law enforcement they don't understand why I am concerned or even involved with AR.  The AR concept is so new, many do not look down the road to understand its full capabilities.  I hope to raise awareness through lectures, presentations, and an upcoming book on Augmented Reality and the double edge sword it brings to the technological evolution.