How Augmented Reality Is Transforming Work

At a GE Renewable Energy factory in Pensacola, FL, a technician is assembling the wiring of an electrical cabinet that goes into the hub of a wind turbine—a complex process that involves matching hundreds of wires into their corresponding sockets. The task is traditionally performed with heavy reliance on an instruction manual that contains the insertion location of each wire. But for the past 40 minutes, the technician has been meticulously going about his business with an almost unsettling precision, without pausing to glance at the thick manual sitting closed in a corner of the workshop.

The technician hasn't memorized the insane amount of instructions contained in the manual. Instead, he relies on Google Glass, a device worn like a pair of glasses that uses augmented reality (AR) technology to project step-by-step instructions for the task in his field of vision.

AR, the younger cousin of virtual reality (VR), overlays graphics and information onto real-world imagery. AR grew in popularity with the launch of mobile gaming sensation Pokemon Go and goofy Snapchat filters. But major tech companies—including Google, Facebook, and Apple—are now providing platforms and tools for developing AR applications.

Beyond gaming and entertainment, the bigger promise for AR, which is estimated to become a $49 billion market by 2021, is for the professional hands-on workforce: Access to information and assistance on the go can make a huge difference in speed and efficiency.

General Electric is one of several companies quietly testing AR technology as a method for improving productivity and reducing errors. In the case of the wind turbine electrical cabinet, a first-time use of AR glasses by the operator has resulted in a 34 percent speed gain. This trend is finding its way into other large companies across different industries, driving noticeable improvements in manufacturing, warehouse management, equipment maintenance, design, and more.

The Rebirth of AR Glasses

Google Glass made its debut for a small group in 2013 with the Explorer Edition, but it failed to gain traction due to price, the lack of a clear function, buggy performance, and overall creepiness. People shunned it, establishments banned it, its users became known as "Glassholes," and by 2015, Google Glass for consumers was shelved.

But the same technology found a new home in hands-on workplaces such as factories, warehouses, and hospitals, where it earned the name "assisted reality."

"A variety of companies that work with us and work with other members in the ecosystem saw this as a potential game changer," says Jay Kim, Chief Strategy Officer at Upskill, a prominent industrial AR solution provider. "There were real problems we were solving, whereas in the consumer context, these devices were just nice to have."

Giving workers frictionless access to information is an obvious use case for smart glasses. And companies such as Upskill enable organizations to integrate AR technologies into their workstreams. GE, one of Upskill's main clients, has been using the company's AR applications in a number of its sectors, including renewable energy and aviation.

As they work, employees using smart glasses can get access to instructions and detailed content about the task at hand without interrupting their work. They interact with the gear through voice commands or by swiping and tapping the side of the glass. The devices let them capture information such as footage or pictures from their work environment and send it for storage in the company's backend servers.

"In an increasingly competitive global economic landscape, enterprise buyers look at every edge they can attain to maintain their competitive advantage over others," Kim says. Upskill now provides service to a number of high-profile clients across various industries, including Boeing, Shell, and Hershey. "With AR, we've been able to drive very powerful outcomes with a number of our customers across every facet of what the hands-on workforce does, in the factory, in the manufacturing environment, out in the field and in warehouses."

Mixed Reality Is Coming

Other companies are enhancing workforce performance through mixed reality (MR), a more advanced form of augmented reality that stands somewhere between traditional AR and VR. As opposed to standard AR, which overlays graphical objects on top of real-world imagery, mixed reality has depth and creates the impression that those objects are embedded in real space. For instance, in an MR experience, a virtual object might be partially or completely obscured if a real world object stands in its path.

The technology is still in its early stages. Headsets are bulkier, cover the entire vision of the user, and have a limited field of view. And beyond the high-end headsets, most MR devices require wearers to be tethered to a computer, which makes their use a bit limited in mobile work settings.

However, the immersive experience of MR has some promising use cases, and many companies and investors are betting on its future. Magic Leap, an MR headset startup, has raised over $1 billion in funding without even releasing its initial product. Established companies such as Microsoft and Epson have also made their move into the space.

Aerospace and defense giant Lockheed Martin is using mixed reality in the building and designing of physical prototypes such as the Orion spacecraft and the NextSTEP space habitat, two projects it's conducting in partnership with NASA.

"You're putting an astronaut in the physical seat or shell of the Orion, and you want that astronaut to see what the inside is going to look like, but you don't have a physical mockup of that," says Darin Bolthouse, manager of Lockheed's Collaborative Human Immersive Laboratory (CHILP). "We build the outer shell or basic structure of that system as a full-scale physical mockup, and then somebody can put on an augmented reality device and start to see additional engineering details."

Using a head-mounted display such as the Microsoft HoloLens, a true MR device, a user can see the control panels, wiring, and other parts of the finalized model. "Before augmented reality, you would have to take time to build those additional details into the mockup, using printed texture maps or additional physical mockups," Bolthouse says. Now they can directly project the CAD drawings of the model into the MR headset.

In 2015, Microsoft and Autodesk, the leader in CAD software, partnered to provide tools for visualizing and sharing 3D designs with MR technology. There are a number of areas where such tools can make a big difference, including construction, architecture, and industrial engineering. The technology can help bring designers, engineers, architects, workers, and even clients on the same page by helping them visualize the project as it would appear in its actual end environment, instead of poring over 2D maps or looking around 3D models in CAD software.

"Mixed reality is the next big technological frontier within the broader context of AR," says Kim. "Where the technology is going to evolve is that, as the content sources become ready to be consumed in an immersive fashion, we can certainly see companies start to adopt more and more of the technology... That's going to be the next kind of evolution into a world where AR is everywhere."

A Growing Market

According to a Forrester Research study, an estimated 14.4 million US workers will be wearing smart glasses in the workplace by 2025. Earlier this year, Google Glass returned with an Enterprise Edition that fixed many of the technical flaws of the initial product. It can now be tacked on safety glasses, making it suitable for more work environments.

But Google is not the only player in the space; Vuzix, Intel, and Iristick also have AR smart glasses for work. And an increasing number of client companies are jumping on the AR bandwagon. Aerospace giant Boeing, another client of Upskill, uses AR in the construction of wire harnesses, a manual process that is both sensitive and painstaking and involves assembling thousands upon thousands of wires for each aircraft.

The company replaced phonebook-size manuals and laptops with a specialized AR app and smart glasses. The application takes the user through the steps to complete the order. The user can interact with the application through voice commands and query for assembly roadmaps for each wire. This seamless experience has enabled the aircraft manufacturer to reduce production time by 25 percent.

"Equally as important as that productivity gain was their error rate that was effectively being driven to zero," Kim says. "Not only were they making things faster, they were also making sure that every product that was coming off that assembly line was built right. Those two factors combined translate to millions of dollars of savings when projected upon their entire operations."

The application also enables users to replay previously recorded footage of the assembly for guidance, or to stream their point-of-view video to an expert for remote assistance.

"The remote assistance is a simple but very important use case of smart glasses in professional work," says Peter Verstraeten, CEO of Proceedix, a Belgium-based solution provider for cloud and wearable applications. "It's like a hands-free Skype that aligns the line of sight of the field engineer and control room experts."

AGCO, a major manufacturer of agricultural equipment, enlisted the services of Proceedix a couple of years ago to incorporate smart glass technology across its factories and workshops. After working out the kinks, the company fully integrated AR into its workflow. Among the tasks that smart glasses accomplish for on-site AGCO's workforce is getting remote help. Field service reps can send photos or livestream video of machinery to the AGCO tech support through their smart glass application and get assistance on fixing problems. As Verstraeten explains, the application of smart glass and AR is helping companies save time and traveling costs for their highly trained experts, one of their scarcest resources.

Coca Cola, an Upskill client, has integrated smart glasses in its bottling facilities to obviate the need to fly in experts from Germany, where their equipment suppliers are located, for tasks such as maintenance and changeover. "By providing remote assistance in a hands-free fashion, they were able to go and get a second pair of eyes on the job, assist the operator, and reduce downtime along their manufacturing processes," Kim says.

"There are tremendous advantages," says Bolthouse, the engineer from Lockheed Martin, a company where manufacturing tasks often take place in clean rooms that require special suits and entrance procedures. Having access to hands-free remote assistance from experts without having them show up on the shop floor can save a lot of time and energy.

Readying the Workforce for the Future

Advances in artificial intelligence and machine learning have caused a major disruption across the employment landscape. While we're not yet speaking of the total obsolescence of human labor, the skill requirements for jobs in various domains are shifting and increasing incrementally, creating a widening dearth of qualified candidates across various domains.

For instance, the US manufacturing sector is faced with a growing shortage of industrial workers. According to a 2015 Deloitte study, over 3.5 million manufacturing jobs will need to be filled in the next decade. But because of the lack of skilled workers, 2 million of those jobs will remain unfilled. Many experts believe the solution is a combination of man and machine. In this regard, assisted reality can be a game changer.

"We believe that while technology isn't necessarily [a] panacea [for] the skills gap in and of itself, it certainly has a role in being able to reduce the barrier of entry to a number of different skill positions, so people can more effectively move from one task to another or from one job to the other by being able to get live guidance at the task," says Kim.

Use cases as simple as displaying information can help workers adapt to tasks that previously required them to partially or fully memorize instructions to become proficient. Beyond that, other benefits are inherent to seeing information in the context of the real world.

"It's just going to make the ability to understand what to do easier, because you're not going to have to be interpreting more complex engineering drawings or other information," says Bolthouse. "By presenting information in a more intuitive and natural way, workers will no longer need to acquire some of the skill sets they had to have in the past."

Training the workforce is another area in which augmented reality can provide positive assistance. "Headsets like the HoloLens or Epson Moverio can help build interactive training and simulation scenarios, allowing the wearer to learn about what to do in reality by putting a digital layer directly in the field of view," says Verstraeten, the CEO of Proceedix. But he also points out the cost of the content production as a limiting factor. "It can only be justified by scenarios that are helping many users, many times, or where the cost of failure is huge," he explains.

Lockheed is using AR for educational purposes in its solar array production facility in Sunnyvale, CA, where it creates the solar wings for its satellites. Using iPads and AR apps, workers view virtual models as they learn the steps of each assembly, and they can compare the real part with the AR model as they go through their work.

AR and Other Technologies

The real benefits of AR in improving the versatility of the workforce come into play when it's combined with other emerging technologies, bringing digital workflows, measurability, and transparency to an entire manufacturing process.

GE Aviation started using AR along with IoT technology to reduce errors and help workers improve their skills at performing sensitive tasks. In its Cincinnati facility, the company's mechanics are using smart glasses along with IoT-enabled torque wrenches in adjusting B-nuts in engine fluid lines and hoses. As they move through standard procedures and come to a step where they need to apply the torque wrench, the AR application reads the torque value from the device in real time and projects it on the smart glass display.

"Not only are you now using AR to get assistance on how to do the work, you're completing the loop by sending data back to the system, which gives you a certain level of compliance and gives you greater insights into how exactly your product was built and maintained," says Kim.

As technologies such as deep learning and computer vision become more advanced and pervasive across industries, AR devices will someday be able to analyze and understand what users are doing and help them in accomplishing tasks.

"If you have a CAD model representation of an assembly, the devices will eventually be able to validate that you did it correctly, or recognize and point out if it doesn't match the reference material," says Bolthouse. "The software and hardware combination doesn't do that in a robust way right now, but that's the intention of a lot of these things."

Challenges

Despite the impressive advances in augmented reality, hurdles remain. The price of AR gear has dropped considerably but is still high; for example, the Google Glass Enterprise Edition has a $1,500 price tag. For untethered mixed-reality headsets such as the Microsoft HoloLens, prices hover at around $3,000. Though in most cases, the return on investment is tremendous, the entry costs for equipping a considerable number of workers with smart glasses or MR headsets is a barrier that not all companies can handle.

Current devices are also not ready to provide a fully immersive experience yet. "AR applications have to be derived from CAD drawings that are built on top of 8-core workstations with massive graphics cards," says Kim, a requirement that makes it near-impossible to deliver that content natively on smart glasses or headsets, which have small amounts of RAM and processing capabilities.

"You'll run into the limits of [computation]," Kim says. "If a company is trying to go and drive augmented reality applications that are more immersive and provide overlays on top of real objects, then the cost to go and build those applications is significant."

Lockheed's Bolthouse names limited field of view, nausea, heaviness, and buggy hand-gesture recognition as some of the technical shortcoming of current hardware. "It's like we're still in the brick-cellphone era of wearable AR devices," he says. "Though the value of how these devices are going to be applied is extremely obvious, and we're anticipating their use, I think before widespread adoption, you have to solve a lot of these technical issues."

For the moment, assisted reality—delivering information found in large companies' existing databases—is where AR can be adopted at scale, a usage that more and more companies are employing. "It's a smaller screen experience and not immersive, but they contain all the information that people are accustomed to seeing day in and day out as they're doing their work," Kim says. "We think that's the first step to the wide-scale adoption of AR down the road."

As the original Google Glass demonstrated, we might not yet be ready to see people wearing AR glasses and headsets in streets, shops, and (especially) public restrooms. But the hands-on workforce is eager to embrace it. And as it evolves, AR will probably find much more acceptance in the consumer space. After all, a couple of decades ago, few people imagined we would all be toting pocket-size phone-computers. Today, it's hard to imagine life without them.

This article originally appeared on PCMag.com.