Towards Data and Model Interoperability for Industrial Extended Reality in Manufacturing

Industrial extended reality (XR) can be defined as the use of augmented reality (AR), mixed reality (MR), and/or virtual reality (VR) within industrial applications. Industrial XR can provide a human-centered mechanism for the delivery of data in austere, complex, and unpredictable environments. The Department of Defense (DoD) has already leveraged advanced visualization systems, such as XR, for a number of industrial and industry-similar applications, including (i) warfighter training in complex scenarios [1], (ii) manufacturing and sustainment of weapon systems [2], and (iii) geospatial overlay on the battlefield [3].

Figure 1 presents the perspective of Milgram et al. on the “reality-virtuality (RV) continuum’” [4] relating the real (or physical) environment and objects to their virtual (or digital) counterparts. This continuum provides a framework for defining the scopes of VR, AR, and MR. VR is a simulated experience that relates user behavior (such as eye movement and hand gestures) to a defined virtual world, eliciting a sense of immersion. Contrary to VR, AR anchors the user experience within a real environment. AR scenes often spatially locate digital data and/or objects relative to salient physical features such that these data “live” in the real world. MR is any point on the RV continuum that blends the physical anchoring of AR’s real environment and the immersion of VR’s virtual environment to unlock more intuitive human engagement and interaction.

One primary challenge for the use of industrial XR is the lack of interoperability. For each of the XR categories, developers must coordinate real-world data with virtual data, each possessing (or lacking) their own coordinate systems, modeling paradigms, and governing physics. Any errors in coordination between virtual and real data ultimately affect embodiment and presence, i.e., the user’s sense of natural engagement in an XR scene. Issues with interoperability in manufacturing and sustainment scenarios can occur at multiple levels, which can be summarized as follows:

• Data or entity: Manufacturing data comprise a huge variety of types, with additional variability originating from the breadth of specific use cases. Such data may be nominal, ordinal, continuous, and/or discrete, and consist of any arbitrary composition (e.g., scalars, vectors, meshes, images, etc.). Data can also span a variety of physical descriptions of the world, for example spatial versus temporal entities. As a result of such differences, a fundamental lack of interoperability must be overcome by highly flexible storage schema, data registration, mapping, or other data-related techniques.

• Model or representation: The manufacturing community is a leading adopter of standards for domain-specific model definitions [5]. However, most manufacturing standards are developed in silo with quite narrow perspectives in mind. As a result, coordinating models and representations, even ones that are developed by internationally recognized standards development organizations (SDOs), within a single XR scene remains challenging, albeit even more challenging when dealing with proprietary formats.

• Workflow: Meaningful industrial XR scenes often merge highly specialized models that are native to domain-specific workflows, such as computational fluid dynamics and collision detection for robotic planning. Such software tools are critical tools in the manufacturing workflow. However, there is a significant lack of robust communication and exchange capabilities across these various workflow platforms, partly due to a lack of expressiveness in the available application programming interfaces (APIs). The lack of such robust APIs hinders interoperability for real-world use cases.

Early XR adopters have identified the grand challenge of realizing interoperability for industrial data and systems [6]. Most efforts have been targeted at developing standards per industrial use case, such as standardizing machine controller data readouts, e.g., MTConnect [7], and three-dimensional (3D) data presentation, e.g., ASME Y14.41 [8]. While there has been a number of recent efforts to improve the interoperability of industrial XR technologies, progress has been hindered by an innate separation between the domain-specific models, e.g., manufacturing execution data, material specifications, and product manufacturing information (PMI), with XR (often-standard) components, e.g., multiscale spatial representations and data formats optimized for run-time presentation, e.g., glTF 2.0 [9].

In this paper, we elaborate on promising research directions and opportunities around which the manufacturing and visualization academic community can rally. To establish such research directions, we (1) conducted a meta-review on well-established state-of-the-art reviews that have already presented in-depth surveys on application areas for industrial XR, such as maintenance, assembly, and inspection and (2) mapped those findings to publicly published priorities from across the DoD. Note that our presented perspectives are underpinned by our focus on manufacturing and sustainment. Other application spaces for industrial XR are important yet might not necessarily fall under our primary focus.

Several prior studies have documented the technical gaps, trends, and opportunities associated with implementing industrial XR. To provide context, we summarized seven well-recognized review papers in the area of industrial AR [10–16]. Note that we focus on AR, since AR provides the most value to in-field DoD applications, such as manufacturing and sustainment use cases. That being said, remote training through VR remains a focal point for the DoD at large; however, in our opinion, VR-based training is a mature technology offered as a service by a number of specialized companies.

Table 1 summarizes key points delivered by the papers, ordered from highest to lowest consensus. Amid a steep uptake of early adopters, Nee et al. [10] presented an early review on benefits and trends of AR and VR applications in design and manufacturing. Fraga-Lamas et al. [11] conducted a more focused review of AR introduction into the shipyard, focusing on low-level attributes of successful implementations. Palmarani et al. [12] conducted a use-case centric review for XR installations for maintenance. Bottani et al. [13] conducted an analysis of 10 years (i.e., 2006–2017) of publications related to industrial AR and found that most papers are related to the areas of assembly, maintenance, and training/learning. De Souza Cardozo et al. [14] conducted a systematic review for AR and MR across industrial applications and found that the main challenges and constraints relate to the following (in order of frequency): hardware; projection quality, accuracy, and interaction; tracking methods; users’ health and acceptance; and development complexity. Egger et al. [15] conducted a holistic review on the use of XR within “intelligent manufacturing” spanning across all domains and use cases. Baroroh et al. [16] established a recent review on AR and MR implementations associated with smart manufacturing (i.e., Industry 4.0) and focused on the human-in-the-loop elements of the XR experience. High-level commonalities across the reviews can be reviewed in Table 1.

The IEEE Standard Computer Dictionary defines interoperability as “the ability of two or more systems or components to exchange information and to use the information that has been exchanged’” [17]. Most work aimed at improving interoperability targets specific applications. For example, interoperability across digital systems for production and sustainment environments has been a significant area of research for decades [18,19]. One common approach to improve interoperability across manufacturing systems is through standards [5,20]. However, a limitation of standards is that they usually require specification scoped to particular use cases. For example, standards that specify design representations do not directly correlate with controller readouts from the machines that manufacture physical instances of those designs. As a result, mappings must be manually encoded to enable a true digital thread [21]. Since these mappings are normally created for specific applications and system configurations, they are brittle to changes in equipment or modifications of the manufacturing process.

Achieving more interoperable and integrated software-based pipelines is especially challenging when two or more different application areas are involved, such as the integration of gaming-based models with engineering-based information [22]. For industrial XR, the goals of the underlying data formats and models fundamentally differ between the design and XR communities. For example, design and manufacturing values mathematically defined geometries leaning towards expressiveness, i.e., encoding high-quality geometric data to ensure watertight definitions. On the other hand, XR model expressions, such as glTF [9] and FBX [23], value efficiency and light-weighting. As a result, integrating annotations (and other critical authoring information) is left at the hands of those that develop XR scenes. This means developers are often scrambling to collect (often incomplete) data from disparate sources to construct scenes by “hand-prepping” authoring annotations.

One key element of XR scenes is maintaining a reference frame to establish a common coordinate system. When utilizing XR in manufacturing, registering digital assets into a common reference frame presents additional challenges since other physical assets may retain their own coordinate system, such as robotic arms and computer numerical control (CNC) machines. Figure 2 provides an example of an Air Force funded project that overcame this challenge for a demonstration [24]. In this example, an operator (Fig. 2(a)) interacts with a 6-DOF robotic arm to perform a repair process on a sample. Figure 2(a) illustrates the real-world coordinate system that must be considered to maintain safe collaboration between the human and robot. Figure 2(b) shows the view of the operator with the overlay planning interactions through the Microsoft Hololens. The coordinate system of the real-world must be registered with the digital objects that the operator views in the headset to realize efficient interaction. Figure 2(c) provides a screenshot of the digital simulation of the robot’s process plan that must be coordinated with the human interactions from the physical world. Though the coordination across multiple spatial and temporal scales is possible, often the integration is done on a one-off basis with limited applicability to other use cases.

Other data-related interoperability challenges arise from misalignment in goals between XR and manufacturing data. Currently, XR hardware is limited by the amount of data that can be presented to the user at any one time. As a result, 3D models must be decimated to support more efficient interaction and practical visualization. In manufacturing, the precision of 3D presentation is paramount for most use cases. This issue can be the root cause of errors in the presentation of annotations and other critical data, e.g., the bounding box of the repair region for which the robot in Fig. 2 must act upon. A negotiation between the two perspectives must take place, whether that means the XR installation must introduce additional computing power offline or encode an additional step in the virtual space to check the placement of planning instructions (i.e., Fig. 2(c)). Note that the tradeoff between model precision and XR device performance is driven by human perception and cognition [25]. Kruijff et al. [25] classified the causes of perceptual issues into five categories: environment, capturing, augmentation, display device, and user. Quantifying user-acceptable thresholds for model precision and device performance is very much use-case dependent. Ultimately, well-structured user studies must be designed and executed to set requirements for effective XR scenes [26].

To realize interoperability in design and manufacturing systems, SDOs have released standards that address specific exchange and curation requirements per use case. For example, the STandard for the Exchange of Product Data (STEP), or the ISO 10303 series, is one of the most widely used representations (or data models) for exchanging product definitions, including geometries, kinematics, and product manufacturing information (PMI) [27]. Though the affordances provided by STEP can cover most exchange challenges, computer-aided design (CAD) software packages lag in implementing the full capabilities provided therein. As a result, information loss often occurs at the stage of model packaging for exchange.

Figure 3 presents a depiction of a workflow that is affected by such model exchange challenges. In this example, we show a pipeline for developing animation-driven industrial XR scenes. Note that the complexity of such a pipeline can be greatly reduced by remaining in a single PLM platform; however, in most DoD scenarios, multiple organizations are involved and remaining in a single platform with full access to all data structures cannot be ensured. In the scenario conveyed in Fig. 3, a set of developers interact with disparate software tools to generate assembly (.STP), animation (.DAE), and XR scenes. Often, information is lost between each translation and exchange step introducing additional human labor. Also, the loss of information in this forward feeding process eliminates the possibility of updating upstream processes if something changes. This example is meant to convey that current exchange models for industrial XR are not robust enough to ensure content adaptability.

To date, the overwhelming majority of workflows and use case demonstrations in engineering with XR utilize professional gaming engines, namely The Unreal Engine [28] and Unity [29], for content creation. Engineers have used these powerful frameworks to create, develop, and deploy a wide range of applications. While offering a turnkey entry point for XR visualizations on a wide variety of platforms, there is a cost associated with data formatting and conversions into files more easily consumable by the engines. In some cases, these conversions may prove trivial, but for other, more exotic data types these could result in a significant part of the application’s development time. Furthermore, the learning curve for even the simplest form of content development may prove to be too large of a barrier for a typical engineer with a limited to nonexistent game-development background looking to explore the use of XR for their application. As such, we see a need in the engineering community for the continued development of (semi-)generic yet accessible entry points into content creation geared towards engineers (as opposed to game developers). These new access points could come in the form of on-boarding applications with inputs for data formats commonly used within a given engineering discipline or, conversely, the increased support of XR ports from within existing software tools used inside the given discipline.

As previously mentioned, the community is aware of the significant interoperability challenge for industrial XR, having been noted [30] through industrial consortium activities, academic workshops, and research contributions. Specific efforts include (1) open source efforts for mapping PMI from authoritative engineering design data, i.e., STP, to lightweight AR-amenable representations [31], (2) leveraging known techniques for encoding industrial animations in a faster, more updateable manner [32], and (3) developing open-source connectors to enable quick viewing of engineering simulations on XR [33]. Figure 4 provides a high-level perspective of AFRL’s Aerospace Analysis and Design in Virtual and Augmented Reality toolKit (AArDVARK), an in-house solution for porting simulation design quickly and robustly to XR environments. Figure 4(a) conveys that AArDVARK is able to convert a variety of design and engineering model types into AR-amenable representations for intuitive presentation. Figure 4(b) illustrates the web-based architecture of AArDVARK, specifically designed to minimize the software and hardware infrastructure necessary to step into an XR experience.

All three efforts are supported with follow-on efforts by AFRL and anticipate significant buy-in from the other DoD services and partners. In short, additional work in this space is required to highlight platform-agnostic benefits for industrial XR installations. As standard data, model, and workflow structures continue to proliferate across industry, pressure on the large computer engineering service providers will continue to grow. Eventually, better conformance to global standards by the software providers can make significant impacts on the industrial XR scene development process.

For the remainder of the paper, we highlight some key research directions and opportunities for working towards more interoperable industrial XR systems.

To identify trends and potential technical roadmaps for industrial XR, we systematically recorded many future directions proposed by the review papers conveyed in Table 1. We then manually (and subjectively) tagged each potential opportunity to uncover potential commonalities. We classified each research opportunity as either near-term, mid-term, and far-term. Table 2 presents 40 research and development (R&D) directions conveyed across the review papers studied. These R&D efforts are not necessarily XR-driven opportunities; take “5G Infrastructure” for example. However, technological improvements in adjacent fields, e.g., optics, machine vision, and sensors, will enhance XR experiences. We identified 17 near-term, 13 mid-term, and 10 far-term opportunities.

Leveraging our colleagues’ recommendations, we built out technology vignettes, mirroring programmatic visions that link the suggested R&D objectives as building blocks. The outcome of this exercise is conveyed in Figs. 5 and 6. Figure 5 illustrates potential research paths to improve data-driven processes for (and/or with) XR. Figure 6 captures potential research paths for improving human factors and ergonomics for XR projects. Below, we elaborate on far-term research goals that can help realize interoperability for industrial XR, indicated as the eventual end state of bold lined paths in Figs. 5 and 6. Those goals include the following:

• human collaboration with computational intelligence,

• AR for smart manufacturing prognostics,

• integrating AR into a broader data infrastructure, and

• adaptive AR interfaces.

Human collaboration with computational intelligence is about leveraging AR so that human operators can easily interface with the automated systems present in a manufacturing environment. AR content that is adaptable to individual human operators and to the many systems present in a manufacturing environment sets up easier interfacing systems leveraging computational intelligence in real-time. Adaptability can only be achieved by standardized data protocols that provide coordination between virtual data/content existing in AR and the real data of manufacturing systems. Hence, to achieve such a goal, we recommend additional work on leveraging existing and new standardized data protocols and representations to develop approaches to encode XR content more amenable to chain-to-chain updates [32].

Standardized data protocols provide coordination between and across digital environments and the real data of manufacturing systems. Implementing standardized data protocols within manufacturing activities, such as execution and control, facilitate the introduction of more advanced decision-making, such as the incorporation of data-driven modeling. Leveraging data-driven processes can provide the opportunity for greater industrial efficiency, e.g., improving prognostics and health management (PHM) with predictive maintenance where machine learning is used to define maintenance epochs. Adding AR into predictive maintenance introduces a data presentation module on the floor and can avoid unnecessary time sinks.

Standardized data protocols provide coordination between and across digital environments and the real data of manufacturing systems. Content authoring systems are the tools used to create AR scenes, including domain-specific annotations such as GD&T callouts and other PMI. A robust content authoring system that is built in and around standard data representations can facilitate a future state of industrial AR systems that are flexible, adaptable, and more update-able. Standard communication protocols facilitate more innovation through the discovery of underutilized data streams. That being said, we recommend additional work with user-centric experimentation for a deeper understanding of what’s the right data to collect, modify, and visualize per use case.

Per Fig. 6, we can summarize the future opportunities relevant to interoperability and human factors as adaptive AR interfaces.

AR haptics provides a mode of physical interactions for the user to the interface, including hand gestures, voice commands, and better peripheral proxies. Some AR experiences in design or training would be improved if haptics were utilized to provide physical feedback. This is the goal of having task-customized user-friendly interfaces. The next step would be to have interfaces that are not only customized to a task but adaptive to the current environment. One example of potential future work is the development of physics-based simulations coupled with live AR interactions to provide feedback to the user through various sensory modes. For such ideas, it is critical to keep the application space in mind to avoid safety hazards in dynamic and austere environments, such as DoD sustainment depots.

Compared to research lab settings, conditions within industrial environments are unpredictable and subideal, e.g., inconsistent lighting and visibility. These characteristics provide unique challenges for the marker tracking required in much of AR. Ultimately, the insight developed from overcoming lighting challenges will allow for advanced tracking techniques such as CAD feature extraction tracking to be used in an industrial setting. An AR system could use this improvement to provide interfacing options the user that adapt to the displayed part and feature in front of them. Note that feature-based tracking exists to some degree yet is computationally expensive and unreliable. Tradeoff studies between feature identification modes are warranted, specifically under the conditions presented in a manufacturing setting.

Interoperability continues to hinder progress for industrial XR systems. One challenge for improvement is the lack of “ownership” of the problem itself. Commonizing data structures for industrial XR does not nicely fit into a single swimlane of activities. Often, the work itself is far too engineering-focused to be considered as a prioritized objective of an academic laboratory. At the same time, the incentive/value structure is lacking for software providers to fit the necessary effort into their development roadmaps. That being said, we believe that as industrial XR continues to proliferate, the cost burden of development time to overcome data exchange challenges could incentivize significant change.

As a result, industrial XR scene developers are the stakeholders that face the most pain. Much like the adoption and dissemination of smart manufacturing (or Industry 4.0) best practices, full realization of interoperable industrial XR requires internationally recognized standards that offer practical solutions without hindering innovation. We recommend additional testing and development of ecosystems that leverage standards across application domains. Standards groups of particular merit include the Open Geospatial Consortium and the Khronos Group. Historically, the associated communities of these SDOs, i.e., the Geospatial Information Science Industry and the Gaming/Entertainment Industry, have not worked together as much as the alignment across technical interests would suggest. Recently, the launch of the Metaverse Standards Forum [34] seems promising, as it provides an informal mechanism for cross-domain collaborations.

Additional development in the integration and linking of such standard data representations will uncover more scaleable solutions for industrial XR. We hope for additional engagements that incorporate more domain-specific challenges, such as those faced by exchanging data from secure engineering systems to the edge. Designing the right balance between cloud-based (i.e., more centralized) and edge-based (i.e., less centralized) decision-making is an active research area [35]. Reducing up-front (or at the edge) work enhances system scalability and reduces overall cost of computation. Satyanarayanan et al. [36] presented the GigaSight framework, an edge architecture that leverages cloudlets to perform computer vision analytics in near real time. Extension of such work in a secure environment would be a significant leap for DoD applications. Similar momentum can be seen in communities interested in delivering complex machine learning algorithms on small edge devices, such as the tinyML consortium.²

If the relevant research disciplines appropriately rally, we envision a frictionless XR ecosystem wherein data are passed in real-time with no loss of information or fidelity. To achieve this vision, two routes are possible: (1) the deployment of a single platform “to rule them all” or (2) the wide adoption of neutral and open data formats. The latter seems more tenable and sustainable, as evidenced by the wide adoption of open XR-relevant standards, such as OpenXR,³ and CityGML.⁴ Additionally, we believe that the aforementioned manufacturing standards should also sit at the heart of discussions moving forward. Per Fig. 5, Standardized Data Protocols is the R&D opportunity that presents the best potential return on investment as it could represent the start of a path to each of the six captured far-term opportunities:

• Human Collaboration with Computational Intelligence,

• Real Time Data Utilization from Intelligent Manufacturing Information Systems,

• AR for Smart Manufacturing Prognostics,

• Human Machine Interaction Safety Through Path Planning Algorithms,

• Integrating AR into a Broader Data Infrastructure, and

• AR Data Collection for Process Improvement.

To improve adoption of standards, further development on robust mappings across standard data formats is required. Such mappings could be done through structured queries [37], inference making through ontologies [38], and ad-hoc semantic-based language interpretation [39], to name a few directions.

At the Air Force Research Laboratory, we intend to press forward in illuminating fundamental challenges related to industrial XR coupled with agile robotic systems. We anticipate that achieving more interoperable industrial XR systems will be a key goal and focus of our work.

We recognize those that have contributed in the standards community, dedicating time, and effort to improve interoperability. Standards are a key aspect of a plan forward to realize interoperable, scale-able, and agile industrial XR.

Distribution A. Approved for public release: Distribution unlimited under AFRL-2023-0707 determined on 08 Feb 2023.

There are no conflicts of interest.

The authors attest that all data for this study are included in the paper.

AFRL =

Air Force Research Laboratory

AP =

application protocol

API =

application programming interface

AR =

augmented reality

AArDVARK =

Aerospace Analysis and Design in Virtual and Augmented Reality toolKit

CAD =

computer-aided design

CNC =

computer numerical control

DoD =

Department of Defense

DOF =

degree of freedom

FBX =

FilmBox 3D File Format

glTF =

Graphics Language Transmission Format

GD&T =

geometric dimensioning and tolerancing

HMD =

head mounted display

ISO =

International Organization for Standardization

MR =

mixed reality

OGC =

Open Geospatial Consortium

PHM =

prognostics and health management

PMI =

product manufacturing information

RDF =

resource description framework

RV =

reality-virtuality

SDO =

standards development organization

STEP =

STandard for the Exchange of Product Model Data

VR =

virtual reality

XR =

extended reality

3D =

three-dimensional