Thanks to the runaway success of Pokémon Go last year, the experience of augmented reality (AR) is familiar to a mainstream audience. AR got an even bigger boost this year with the release of Apple’s ARKit, which brought a more advanced implementation of augmented reality into everyday iPhones (6 and above, though the X has additional features). Like Google’s ARCore, not to mention Microsoft Hololens, the Apple ARKit actually uses computer vision to help devices truly perceive a space in real time, making for more robust implementations of AR. The end result is that developers are knocking themselves over to create the next killer AR app, and the AR market is expected to reach $83 billion by 2021, according to Digi-Capital. Such activity and outlook are not lost on brands, which, not surprisingly, are among the most innovative pioneers in this growing space.
To understand a bit more why this latest generation of AR is so appealing, it helps explore why it’s different. The first generation of Pokémon Go, while groundbreaking in its ability to evangelize the AR experience, was basic in that it relied upon GPS and cell phone triangulation to place the animated pocket monsters into specific geographic locations. So, yes, it ‘augmented’ the world in a smartphone viewfinder by showing Pikachu at, say, the corner deli, but that yellowish rodent-like creature was just randomly inserted in and around the deli – not specifically sitting at the counter or shopping the aisles in life size. This approach also led to issues of battery drain, disproportionate sizing of Pokémon in relation to their environs, and other glitches that drove some users to play with the AR function off.
The latest generation of AR as exemplified by ARKit, ARCore, Hololens, Facebook Camera Effects, among other technologies, takes it further with what is generally known as Simultaneous Localization and Mapping (SLAM), which adds a mix of cameras, gyroscopes, accelerometers, and computer vision (such as object recognition) to essentially help generate a 3D map of a space so that the virtual objects, animations, and other augmentations can be accurately and contextually placed into an environment in real time. In other words, Pikachu might be able to browse for items in the store aisles or stand behind the cash register and ring you up. This is one step further than even some of the recent computer vision-based AR children’s’ book and game apps or interactive temporary tattoos, which all relied on image recognition of still images to kick off AR animations.
The focus on physical spaces has naturally led brands in the home furnishing arena to dive deep into AR on the iPhone. New apps from Pottery Barn and Ikea, for example, use AR to virtually place life-size furniture inside your home; just use your phone camera to scan the space where you want to add furniture, then choose from any virtual item in the respective store’s catalog and place it anywhere in the room. You’ll know immediately if it’s too big or bulky or if the colors clash because the item is accurately sized and rendered based on your initial scan. Not surprisingly, Amazon is also adding AR to its retail experience. The new AR View functionality in the Amazon app broadens the types of products available for virtual showrooming to include kitchen appliances, electronics, toys, and more. The online shopping behemoth is certainly paving the way for a broader range of retailers to innovate on the AR front, offering a mix of customer service retail aid and next-gen branding alike.
In the entertainment realm, AR is a natural fit. Syfy, for example, just released Happy AR, a companion app to its new cop comedy-drama show, Happy. The app features an interactive version of the animated unicorn-like character from the show that sizes itself accurately to whatever space you scan, assuming you have iPhone 6 or newer. In other words, get closer to Happy, voiced by Patton Oswalt, and he’ll get bigger. Ask him to ‘helicopter,’ and he’ll rise to just below the ceiling. Push the ‘Poop’ button and some pellets come out that end up on your floor (but then, thankfully, disappear). Meanwhile, the upcoming iOS game, Walking Dead: Our World, will use the ARKit to provide a more real-time contextual experience to a location-based Pokémon Go-like premise, but with the AMC hit show’s zombies instead.
Besides retail and entertainment, automotive brands are also experimenting with AR, not only on ARKit, but also Hololens. In late November, BMW became the first brand to use Snapchat’s new Augmented Trial Lens , which places the new X2 contextually in different colors in the space where the user scans. In addition, the automaker and Snapchat released an exclusive Face Lens that superimposed the X2’s Galvanic Gold paint onto users’ faces, a feature requiring next-gen face-tracking capability to cover and wear on the face realistically in video. BMW also created AR content for Microsoft Hololens, a headset platform that is popular with automakers.
At the recent LA Auto Show, MINI unveiled a new experience for Microsoft Hololens, which lets users choose different features such as roof racks and wheel rims and then superimpose them realistically on specific models such as the 2018 MINI Cooper S Countryman (the software has been trained specifically on these cars so it can recognize them in real time, no matter where they are situated). MINI plans to roll out these headsets to dealerships to aid car buyers when choosing options. Meanwhile, in conjunction with the 50th anniversary of the Civic, Honda teamed up with the Spinifex Group to create the Honda Lens Experience, a Hololens program that walks the user through the history and different features of the Honda Civic, superimposing interactive timelines, animations, and more in different locations around the car depending on where the user goes. The Hololens has two cameras – a stereoscopic approach like two human eyes — and an array of mostly inertial and motion sensors, which allows it to accurately map and then place holograms into the space in front of the user. Since it’s a headset, it’s more immersive than a smartphone screen. The Hololens is also able to recognize gestures, making interactive elements more realistic; users playing through the above headset experiences click on virtual buttons with their real hands to control what they see.
Microsoft Hololens may have more cameras than most smartphones – the ARKit software enables a clever workaround whereby it captures stereoscopic shots in quick succession when you move your phone back and forth to scan a space — but the iPhone X has one additional killer feature that is begging for experimentation by developers and brands: the TrueDepth camera. Like a Microsoft Kinect but pointed at the user’s face, the TrueDepth camera projects more than 30,000 dots onto a user’s face to map it in different positions and lighting, while the actual camera looks for all kinds of specific distinguishing details on the skin to further make it less hackable with, say, a picture. This same technology is used to power the Animojis, an iPhone X-exclusive feature that sticks a fully moving animated mask onto a user’s face. Animojis adapt realistically to a user’s facial expressions and movements, sort of like a green screen but for faces. Surprisingly, this hasn’t been exploited much by brands or games in any significant way yet, though arguably that is what Snapchat and BMW pulled off with the Gold Face Filter. Imagine what a sports team or other brand could do with AR face paint options for fans and customers, respectively
Future iterations of AR in smartphones will likely bake in additional cameras and more powerful processors, along with redundancies such as cloud databases of pre-rendered SLAM maps that would be accessed on the fly, enabling phones to more quickly ID a space, manage processing resources better, and save on battery usage, a strategy not unlike what is being developed for the autonomous car space, where quick knowledge of a vehicle’s surroundings is a down-to-the-millisecond matter of life or death.
What’s more, new headsets from Magic Leap and more consumer-friendly, wearable eyeglass versions of the Microsoft Hololens, and rumored headsets from Apple promise more advanced and mass-market forms of AR in the future. As always, cutting-edge brands will be at the forefront of innovation there because their businesses will depend on it.