TECHNOLOGY AREAS: Sensors; Information Systems; Human Systems OBJECTIVE: This topic seeks to develop Extended Reality (XR)-based software that can support aircraft maintenance training modernization and operational use by enabling active training and task assistance as well as automated training content generation. DESCRIPTION: It is essential that the Air Force to modernize its air maintenance training procedures to meet the demands of future operations and deter our adversaries. XR technology has the potential to provide a new platform for our maintainers to continuously develop and augment their skills. Current solutions suffer from rigid pre-recorded training content that is not appropriate for the high mix low volume nature of aircraft maintenance and repair. However, there are critical S&T challenges that must be solved to fully leverage this technology beyond what is currently being done. Specifically, active training and task assistance as well as automated generation of training content and digital twins must be enabled to fully realize the potential of XR-based for aircraft sustainment. The primary S&T challenge of this project is to create a near real-time, XR-based, markerless motion tracking solution that approaches 5mm accuracy when dealing with aircraft component sized parts. This will enable the primary objectives of active training and task assistance, as well as automated generation of XR training content for high mix cases. The active training concept goes beyond showing users what should be done and monitors that tasks are being done correctly. This is key for aircraft maintenance as there are many tedious tasks prone to accidental human error even for experienced users. Similarly, with regard to generating XR training content we must move towards a model where an XR programming expert is not required to write code to capture the many possible scenarios for which training content is needed. Thus, experts on the wide variety of high mix maintenance tasks currently being done can simply watch themselves performing the tasks with an XR device to generate training content for new users. Doing the tracking through the XR device is key to the envisioned applications. Aircraft repair depots are large complex, dynamic environments. It is not feasible to install external 3D scanning and tracking hardware for every possible use case. Further, such tools typically suffer from complicated calibration procedures that leads to poor adoption in the depots. Demand signals from the organic industrial base are clear that an inside-out XR-based solution with no additional infrastructure or time-consuming calibration procedures is necessary. PHASE I: This topic is intended for XR-based aircraft maintenance training technology proven ready to move directly into a Phase II. Therefore, a Phase I award is not required. The offeror is required to provide detail and documentation in the Direct to Phase II proposal which demonstrates accomplishment of a Phase I-like effort, including a feasibility study. This includes determining, insofar as possible, the scientific and technical merit and feasibility of ideas appearing to have commercial potential. Specifically, given that this effort requires XR-based inside-out markerless tracking of complex objects, the desired Phase I-like effort should have established a capability to spatially scan, generate 3D from 2D reconstructions, and then recognize specific static but potentially complex objects via a commercial, off-the-shelf augmented reality (AR) device's onboard sensors. Ideally the Phase-I-like effort would also establish the ability to localize an AR device with respect to a digital version of a detected static object. Reported results showing both the level of localization accuracy, benchmark objects used, and complexity/size of object limitations should be provided. It must have validated the product-market fit between the proposed solution and a potential AF stakeholder. The offeror should have defined a clear, immediately actionable plan with the proposed solution and the AF customer. The feasibility study should have; Identified the prime potential AF end user(s) for the non-Defense commercial offering to solve the AF need, i.e., how it has been modified; Described integration cost and feasibility with current mission-specific products; Described if/how the demonstration can be used by other DoD or Governmental customers. PHASE II: Eligibility for D2P2 is predicated on the offeror having performed a Phase I-like effort predominantly separate from the SBIR Programs. Under the phase II effort, the offeror shall sufficiently develop the technology in order to conduct a small number of relevant demonstrations. Specifically, the offeror will (1) Augment existing XR-based markerless spatial scanning and localization for static objects with markerless model-based object tracking.; (2) Integrate AI and selective scanning to enable motion tracking to approach real-time updates for potentially complex objects; (3) Demonstrate the technology in a crawl before you walk approach by initially tracking one object with a stationary XR device, then track 1 object with a moving XR device, then track 2 objects with a moving XR device, and finally track many objects moving; (4) Leverage XR-based tracking tools to implement an active training use case; (5) Leverage XR-based tracking tools to implement an automated training content generation use case; (6) Test the new capabilities in an operating environment. The offeror must identify technology hurdles they are expected to encounter during the development program, as well as potential solutions to mitigate risk to the program. PHASE III DUAL USE APPLICATIONS: The contractor will increase the Technology Readiness Level of the XR software tools developed under the Phase II effort and validate performance and achievement of objectives in pilot production at a USAF air logistics center. The contractor will pursue commercialization of the various technologies developed in Phase II for transitioning the technology to various defense aerospace Original Equipment Manufacturers, their supply chain, and the Air Force and broader DoD organic industrial base. Direct access with end users and government customers will be provided with opportunities to receive Phase III awards for providing the government additional research & development, or direct procurement of products and services developed in coordination with the program. REFERENCES: 1. Arena, Fabio, et al. "An overview of augmented reality." Computers 11.2 (2022): 28; 2. Bottani, Eleonora, and Giuseppe Vignali. "Augmented reality technology in the manufacturing industry: A review of the last decade." Iise Transactions 51.3 (2019): 284-310; 3. Aharchi, M., and M. Ait Kbir. "A review on 3D reconstruction techniques from 2D images." Innovations in Smart Cities Applications Edition 3: The Proceedings of the 4th International Conference on Smart City Applications 4. Springer International Publishing, 2020; 4. Han, Pengfei, and Gang Zhao. "A review of edge-based 3D tracking of rigid objects." Virtual Reality & Intelligent Hardware 1.6 (2019): 580-596; KEYWORDS: extended reality, augmented reality, mixed reality, spatial tracking, 3d from 2d reconstruction, spatial scanning
