Applying Augmented Reality for Industrial Manufacturing
By Jay Gorajia, Siemens Digital Industries Software
The future of manufacturing will include elements of Augmented Reality (AR). Manufacturers are looking for innovative ways to solve today’s problems, and AR may be the key. Pokemon GO™ and Ikea Place applications continue to drive awareness for AR, and technology companies are developing solutions to solve key productivity, quality and efficiency challenges using AR. Innovative companies such as Ikea, Mitsubishi, Toyota and Lego, and 10 percent of the Fortune 500 companies have already begun exploring augmented reality applications1. In addition, Gartner predicts that by 2020, 20 percent of large enterprises will evaluate and adopt augmented reality, virtual reality and mixed reality solutions as part of their digital transformation strategy2.
Technology companies continue to develop solutions to solve key productivity, quality and efficiency challenges in manufacturing using AR. In parallel, manufacturers are looking for innovative ways to solve today’s problems, and AR may be the key. AR has been around for some time, however, the crossroad of visual processing power, data processing capabilities, and compute power has suddenly made any mixed reality solution now viable.
The Reality-Virtuality Continuum
The notion of a reality-virtuality continuum was first introduced professor Paul Milgram, a professor of engineering at the University of Toronto, more than two decades ago. A paper published in 1994 describes the concepts of Mixed Reality in the context of the Reality-Virtuality Continuum1 whereby the space between the Real Environment and the Virtual Environment is the Mixed Reality environment, as described in Figure 1.
Figure 1: Reality-Virtuality (RV) Continuum
In the manufacturing context, the far right (Virtual Environment) describes what is often referred to as the Digital Twin of a factory, product and production, whereas the far left (Real Environment) describe the actual factory floor with workers operating machines based on information on screens or other instruments.
Augmented Reality in Manufacturing Use Cases
There are several use cases in which augmented reality has high impact on manufacturing. These high impact use cases for manufacturing include: assembly instructions and validation of complex assembly, asset maintenance instructions, expert remote instructions and support, programming and commissioning validation, and real time quality processing and visualization. Most use cases in manufacturing today seem to focus on integrating AR at the actual production line where it is most feasible to link the actual machinery, robots and parts, and the 3D graphic models.
Some of these use cases include: helping a shop floor worker identify the next part to assemble in the assembly sequence, validating a robotic program using an AR generated part instead of an actual part, and overlaying the 3D model on top of the actual part for quality inspection purposes (“as-planned” vs. “as-built” scenarios). When reviewing use cases throughout the product lifecycle (Figure 1), it is clear that the use cases that are relevant for AR technology are the ones further down the cycle, where the digital twin (the virtual model of the product), can interact with the real production line and related products, as shown in Figure 2.
Figure 2 – Digital twin demo on an airplane nose (One Aviation) from the Hannover Messe trade fair show, 2018, Hannover, Germany.
Model-based work instructions, leveraging the design and assembly process data within managed data structures, could easily clarify an assembly process for the shop floor worker by overlaying the actual design data 3D model on top of the actual, current assembly structure, thus shortening the process cycle time and reducing human error. Another significant benefit for this AR model-based work instruction solution resides within the automation which includes model-based work instructions. Using the CAD model, the process sequence comprises all byproducts of work done in the earlier stages of the product lifecycle, which can be leveraged. Such a solution should obviate the need to manually create and update documents as part of the work instruction authoring process (Figure 3). It also speeds up onboarding processes as it makes the work instruction product more intuitive.
Figure 3 – Siemens’ holistic product design through manufacturing value chain.
Another relevant use case that addresses production assembly planning validation includes an AR part precisely mounted on a fixture at the real production line to validate a welding operation program written directly to the robot controller. This saves time and money on manufacturing a real prototype, which, in addition to its costs and manufacturing time, also might not comply with common standards.
Training new employees on the shop floor on assembly procedures is an AR use case which could accelerate new employees’ learning curve and save valuable time for experienced employees that currently must mentor the new ones. For quality issues and validation, AR can also be used to validate parts manufacturing and assembly structures in an “as-planned” vs. “as-built scenarios, where an employee compares the physical product against the digital twin for quality purposes.
A scalable, robust AR solution must provide an accurate and reliable visual experience, which requires correct positioning, tracking, and visual processing. Tracking is how the augmented reality algorithm recognizes real-world objects so it can accurately place virtual objects onto it. Most of the common commercial solutions that perform tracking today, such as ARCore by Google, ARKit by Apple, and Vuforia, use algorithms that inspect visual features in the camera feed to determine the exact position of the camera. It uses contours and outlines in the incoming image stream of the actual object to determine the exact position to place the 3D model. These solutions allow flexible and simple setup, but they are sensitive to environmental factors such as light variances and line of sight. They also require the existence of clear, highly detailed features of the scanned object to return an accurate image.
A robust solution must consider variances of location, find objects “somewhere” on a large table, and identify correctly the edges and location of the specific instruction object, while contending with some of the challenges described in Figure 4.
Figure 4: AR tracking challenges in industrial environments.
This article describes the design and implementation of a robust industrial augmented reality assembly instruction and validation solution in which various choices of tracking methodology is reviewed to address challenges of environmental factors, and the prerequisite need to have scanned objects in the environment. In addition, the solution “validates” that a manual operation was performed, which is key to ensuring a robust solution. Simply providing instructions is not enough for critical manufacturing needs. This solution is currently being tested in an actual production line at the Siemens Energy factory in Berlin, Germany (Figure 5).
Figure 5 – An AR-enhanced work instruction solution for an assembly station at the Siemens gas and power factory, a gas-insulated switchgear manufacturing line in Berlin, Germany.
References:
[1] “Building the Business Case for Augmented Reality.” Building the Business Case for Augmented Reality, Cognizant, 19 June 2018.
[2] Griffen, Bryan. “Augmented Reality and the Smart Factory.” Augmented Reality and the Smart Factory, Www.manufacturing.net, 12 Apr. 2019. www.manufacturing.net/article/2019/04/augmented-reality-and-smart-factory.
