Upskill Is the Coolest AR Startup You Haven't Heard of Yet
In mid-2010, the US military was looking to develop new head-up display technology for soldiers in the field. A company called Battlefield Telecommunications Systems (BTS) was tasked as the government's software provider for biometric force predictions, meaning real-time data displayed to a soldier's headset to indicate whether the person in front of them is a friend or foe.
Beyond the obvious Black Mirror premise, three execs at BTS saw the potential to take the augmented reality (AR) technology at work and apply it far beyond the battlefield. They left to found what is now Upskill.
More than seven years later, Upskill's software underpins Google Glass Enterprise and several of the most popular enterprise AR devices on the market, customized for a wide range of business scenarios. The startup now creates software for companies like Boeing, Coca Cola, and GE, boasting more than $28 million in venture capital (VC) funding. That includes money from Boeing and GE Ventures, which have now bought into Upskill's future.
Skylight, Upskill's industrial AR platform, is deployed in production across an array of fieldwork, heavy manufacturing, and warehousing environments. The technology augments workers' experiences with custom voice and gesture-driven apps, and provides HD audio and video feeds for a first-person view into everything from field service training to constructing jet engines.
I spoke to Upskill's co-founders—CEO Brian Ballard, CTO Jeff Jenkins, and CFO Chris Hoyt—as well as GE Ventures to break down how Upskill's hardware-agnostic AR software works, where the company is going, and the evolution of augmented reality on an enterprise scale.
Company DossierName: UpskillFounded: 2010Founders: CEO Brian Ballard, CTO Jeff Jenkins, CFO Chris HoytHQ: Herndon, VAWhat They Do: Industrial Augmented Reality SoftwareWhat That Means: Building custom AR experiences compatible across AR headsets and integrated into enterprise IT infrastructureHardware Supported: Google Glass Enterprise, ODG R7, Vuzix M300, RealWear HMT-1Business Model: Custom Development and Software-as-a-ServiceStatus: Live; customers include Boeing, Coca Cola, GE, Rio Tinto, Jabil, Merck, Shell, and Johnson & JohnsonFunding: Over $28 million. Backers include GE Ventures, Boeing, Salesforce Ventures, NEA, Work-BenchNext Steps: Feature and platform enhancement; expanding its customer base; raising further funding
From Military Research to Factory Floors
Upskill's founders all came from vastly different backgrounds, but CEO Brian Ballard came in with deep experience in military tech. Ballard was a Field Operations Officer for the Department of Defense for more than a decade developing cutting-edge defense technologies. He then spent several years as a contractor on intelligence and reconnaissance tech before coming on as director of product development for BTS.
After Ballard, Jenkins, and Hoyt left BTS to spin out APX Labs (which was rebranded as Upskill at the start of 2017), Hoyt explained that, in the beginning, APX was able to benefit from jumping right onto some government contracts.
"The US government was willing to invest in new technology and do it contractually so that APX at the time could develop our technology and intellectual property while serving the needs of the government," said Hoyt. "We were able to grow and mature without having to take outside capital."
Upskill Co-Founders (From Left): Jeff Jenkins, Chris Hoyt, Brian Ballard
By 2013, they saw the opportunity was much bigger than military tech. APX began expanding its augmented reality tech to cater to the entire hands-on workforce. The software evolved to suit specialized industrial use cases, giving workers heads-up work or training instructions. Headsets could also receive real-time notifications showing changes in data and events going on around them that the user on a factory floor or in the field may not be able to see directly. Most importantly for the businesses deploying AR headsets at scale, Upskill is hardware-agnostic and syncs with enterprise IT systems to integrate all the data these wearable devices collect.
"So much of the challenge is getting data from one place to another," said Jenkins. "We've tackled it by making our platform as flexible as possible. From the technology side, that's a tall order. A key focus on our roadmap is pushing the envelope in terms of how easy integration can be to make adoption of these next generation devices easier."
Ballard said Upskill currently services four major industries: aerospace and defense customers like Boeing, Lockheed Martin; heavy manufacturing (GE, carmakers); logistics scenarios like shipping and warehousing; and field service scenarios such as Coca Cola technicians using smart glasses to service their bottling plants. Ballard broke down some of the pilot projects.
"In the Coke scenario, a remote technician is helping debug an error on the bottling line. Think about a changeover process where a certain product line is ending its production run. For example, the line is making Sprite today and Coke Zero tomorrow," said Ballard. "Every part of the changeover has to be done properly, but all of this is now digitized and in your field of view through the headset."
Upskill also works with GE across scenarios including engine maintenance, supply chain and warehousing, and wind turbine assembly. Ralph Taylor-Smith, Managing Director at GE Ventures, focuses on advanced manufacturing and has helped oversee both GE's pilot projects with Upskill and its funding investments in the startup.
Taylor-Smith explained how GE is using AR technology in a number of different ways, using analogies like the precise turns of a torque wrench. Upskill's Skylight platform collects and stores all the real-time data from the worker's glasses, including time-stamped video and granular calibration data, to reduce manufacturing errors and increase productivity in the field.
"In some settings where you need to turn a screw a particular way to a specific level, you actually have to torque the screw. What Upskill has been able to deliver is a unique application using augmented reality and smart glasses where effectively you can build in a virtual torque wrench. So when you torque the screw while wearing the glasses, it processes information about whether you're properly tightening or sealing the joint. You have all the metrics you need projected in your field of view," said Taylor-Smith.
"One key app [GE has] replicated several times is a field service engineer in the field with Google Glass or other hardware on using Skylight software, and everything they see is projected back to headquarters in real time," he continued. "A lot of these field service and assembly applications will have a major impact. Most people don't think about something like a bolt or screw tightening and associate that with augmented reality, but those are the kinds of unique applications you can get utilizing Upskill's software."
Inside the Skylight Platform
Upskill popped onto the radar of many last year when the startup was announced as one of the partner sellers for Google Glass Enterprise. In fact, Ballard explained that Upskill has been working with Google for more than four years. He talked about why the first iteration of Glass faltered commercially, and why the enterprise is the right fit for the revamped version of the pioneering AR headset.
"Glass is a great product, but the second generation product on the market means it's benefited from learning from its predecessor. Glass Explorer Edition had pretty heavy adoption and a lot of feedback from consumer and corporate customers," said Ballard. "In the beginning they were a consumer company with consumer pricing and we were campaigning their product teams to say, 'have you considered this piece of the market?' Over time it turned into a collaborative relationship as they re-launched for the enterprise. We're a sounding board for Google and vice-versa. A lot of the discussion in the beginning was about what was tangible, but now we're sharing data and simulating performance in natural use cases down to battery life and pixel densities."
Boeing, GE, and others have been piloting Glass Enterprise since long before the official launch last year. Upskill also supports enterprise AR headsets from vendors including ODG, RealWear, and Vuzix, but Ballard said the Skylight platform can support any hardware that fits into Upskill's "hands-free users first" strategy, encompassing what he called "assisted reality" devices like Glass Enterprise.
We're still in the early generations of head-mounted AR experiences, and what makes Upskill's technology viable in the long run is its Skylight platform running underneath the hardware and wearable devices. Ballard walked me through the platform, which includes the Skylight Application Builder, Skylight Connect, Skylight Live, and the Skylight software development kit (SDK).
The key elements of the platform are the application builder and the integration tool, drag-and-drop low-code development interfaces for quickly building and connecting AR apps. The idea here is that an engineer or technician on the floor who knows exactly the application they need can build it quickly without any coding knowledge.
"It's a graphical interface deployed without writing a line of code. On the left side is a library with functions and views, in the center is the overall workflow, and below is a preview of the screen you're building across different types of hardware. So you can switch live between RealWear, Vuzix, etc," explained Ballard. "From the web interface you can chards the function of the card-based screens on the fly."
Skylight Connect is the integration module. In a similar low-code interface, the user can connect devices with authorized databases and web services connected to the company's infrastructure. From an IT perspective, this is the real money-saver as it reduces the complexity of maintenance and support from a database perspective and when a worker creates a new app, it makes it easy to simply plug it in and start receiving data.
"The Application Builder and Connect address a major need in the market. On average, the enterprise users in these scenarios would need to know enterprise code development, database connectivity, and understand the wearable user experience and things like Android development. That's a very broad skill set," said Ballard. "We've taken every piece of that and put it in a low-code interface. You want the process engineer who knows what should be done to be building exactly the app they need."
Skylight also includes a management interface, which lets shift managers go into the Assignments tab to set up apps and scenarios. The manager could assign a particular engineering demo, for instance, to a specific Google Glass headset and immediately push the app down to the user.
"This helps with the iteration process," said Ballard. "If you're really ready to scale the adoption of augmented reality, it has to be right the first time when you push it out to the workforce. Whether it's the aspect ratio or the color or contrast or pixel density on the screen."
Finally there's Skylight Live, the newest capabilities in the platform. Upskill completed its first acquisition last year of video collaboration company Pristine and its flagship product EyeSight. Now integrated into the platform as Skylight Live, the tools give each headset a suite of "see-what-I-see" features including remote camera zoom and capture, and annotation on shared documents and snapshots.
In the Application Builder, all the user needs to do is drop a photo or video capture button or a call for help option into the workflow to integrate the experience.
"Skylight Live is a streaming HD video feed from an expert back to the glasses. Frequently you've got an example where an expert might not be able to hop on a plane and instantly fix something. If a local technician has found the problem, the expert can walk them through how to fix it," said Ballard. "You can use the snapshot button that's immediately sent back to see what the technician would've seen. This feature pays for itself in the first year. It's a phone call versus a plane ticket out to an oil rig."
As for how the technology will evolve from here, Jenkins said Upskill will continue to focus on both new user-facing features and more seamless backend integrations into work processes. In the longer term, he said the company looks to expand its use cases and hopefully start support a wider range of both augmented and mixed reality devices.
Ultimately, Ballard said Upskill may support full-blown mixed reality headsets like Microsoft HoloLens and the Windows Mixed Reality ecosystem should the enterprise need arise as the hardware becomes more economical, but for Upskill it's about building around industrial AR use cases with tangible value.
"There are three areas of the stack: the hardware on the market, improving the adaptability of the interface with better spatial and visual interaction, and then enabling more back-end integrations across our user base," said Ballard."
Tacking a Huge Market
The enterprise augmented reality space is highly competitive, and Upskill is far from the only company building AR software for industrial scenarios. Upskill is one of more than a dozen Glass Enterprise partners, and is also competing other startups and with larger tech companies like PTC that sell integrated AR software platforms for industrial and Internet of Things (IoT) use cases.
It's a crowded market, but a lucrative one. Juniper Research projects that the AR market will hit $5.7 billion in revenue by 2021, with the majority of that coming from the enterprise. Upskill has plenty of competition but boasts strong funding, an impressive customer base, and a sound business model. Hoyt explained that 80-90 percent of the company's revenue comes from annual software licensing sales, plus the service component of consulting with a business on exactly the apps and AR solution they need. Across four funding rounds, Upskill has raised more than $28.6 million from traditional VCs like NEA and Work-Bench as well as the venture arms of large corporations like Boeing and GE.
GE Ventures' Ralph Taylor-Smith said GE has seen significant gains in productivity from implementing Upskill's technology. What intrigued them more from an investment perspective, however, is the startup's hardware-agnostic platform and the ability for engineers to build their own apps and replicate them across lines of business.
"Whether we're talking about Google Glass or Microsoft HoloLens are all those various platforms to augment reality, the potential here is that you can implement them all with one software platform," said Taylor-Smith. "That level of flexibility means Upskill's Skylight technology can be used with any hardware platform now, and those that emerge in the future."
GE's next project with Upskill is to deploy AR apps in its renewable energy business, providing complex digital instructions to workers out in the field. Taylor-Smith said one of the key benefits of a platform like Upskill's is that you can easily adapt the technology for different applications without losing any productivity or efficiency.
"A great application you could see being utilized is let's say a major hurricane where the electric grids are severely impacted," said Taylor-Smith. "The company sends out a whole bunch of people to check on the power grids and bring them back up, and they're all wearing augmented reality glasses and communicating with headquarters in real time. You could see the same technology being used in the oil and gas industry and beyond."
Head-mounted augmented reality will reach maturity in the enterprise far before it reaches any sort of critical mass with consumers, who will be kept plenty busy with AR-enabled smartphones like the iPhone X . Google figured that out when they pivoted Glass to the enterprise, and headsets are already starting to evolve beyond the prototype level as the technology inches slowly forward. For those businesses looking to adopt and implement new AR experiences as fast as they emerge, it starts with the software underneath to make it all work.
This article originally appeared on PCMag.com.