Grantee Research Project Results

Final Report: Mapping Indoor Environments Using Real-time Artificial Intelligence and Localization (M RAL)

EPA Contract Number: 68HERC24C0019
Title: Mapping Indoor Environments Using Real-time Artificial Intelligence and Localization (M RAL)
Investigators: Gaona, Tyler
Small Business: VISIMO, LLC
EPA Contact: Richards, April
Phase: I
Project Period: December 1, 2023 through May 30, 2024
Project Amount: $99,799
RFA: Small Business Innovation Research (SBIR) - Phase I (2024) RFA Text |  Recipients Lists
Research Category: Small Business Innovation Research (SBIR)

Description:

VISIMO’s Mapping Indoor Environments Using Real-Time Artificial Intelligence (AI) and Localization (MURAL) seeks to provide the Environmental Protection Agency (EPA) On-Scene Coordinators (OSCs), Regional Response Teams (RRTs), and Special Teams, first responders, and private-sector stakeholders an efficient and rapid AI-enabled decision support tool for real-time map generation and user localization. This solution will enable first responders to rapidly share information about the internal structure of a building in a disaster scenario as well as disseminate locations of team members within the internal structure. Additionally, MURAL will be capable of generating a database of indoor mapped environments as well as the paths that users followed through those environments, providing valuable feedback for users and their supervisor in response roles. This will support emergency response procedures by decreasing the time needed to conduct iterative after-action reviews and discussions to illustrate an unknown environment, increasing a team leader or supervisor’s awareness of where a team member is located and ability to share the intelligence with others to ensure common understanding. Current tools for indoor mapping and localization are limited to ideal conditions, such as pre-mapped locations or the availability of a wireless connection, and often require costly and delicate external hardware. MURAL was designed to address and overcome these limitations. During Phase I, VISIMO demonstrated the feasibility of collecting camera and Inertial Measurement Unit (IMU) data from a user’s smartphone to display user localization and a 3D schematic map without relying on external hardware sensors, pre-mapping, or wireless connectivity.

Summary/Accomplishments (Outputs/Outcomes):

Phase I research included three major phases: (1) develop a localization component; (2) implement map generation algorithm, and (3) design application architecture. During the localization development, VISIMO first investigated how to collect IMU data from a user’s smartphone. The team then explored different methods to receive the IMU data in an interface where users would be able to access the data from any Operating System (OS) (i.e., iOS or Android software).  The phase concluded after successfully implementing a web application that could ingest IMU and camera data and integration of that application with a Simultaneous Localization and Mapping (SLAM) algorithm capable of specifying a user’s location within a building.

While researching feasible localization components and models, VISIMO experimented with various open-source mapping algorithms to support MURAL’s map generation capabilities. By understanding the algorithm’s input requirements, the team was able to refine the PWA to generate the proper data output format for the mapping algorithm to produce a 3D schematic map. VISIMO calibrated the camera sensors to enable MURAL to gather camera data points and display the user’s environment. The team conducted testing to optimize the data collection process and refine the map generation process. After validating the map generation process, VISIMO developed a new algorithm for transforming point-cloud data from mapping algorithms into a clear map of a building’s walls.

Finally, VISIMO developed a Phase II application architecture to integrate the components into a feasible proof-of-concept application. The team evaluated the feasibility of MURAL’s features and functionalities to ensure that the integrated algorithms and components will work in tandem and produced user localization and 3D map in real time without requiring any external hardware or relying on Bluetooth or Wi-Fi connectivity.

VISIMO successfully built a Progressive Web Application (PWA) that was able to receive smartphone camera data from both iOS and android devices and track the user’s location around a building using a camera-only SLAM algorithm. VISIMO was able to implement this algorithm to run in real time; VISIMO also determined that calibration data was needed for precise determination of location coordinates, which can therefore be developed in future work. For a SLAM algorithm using both camera and IMU data, while the PWA could request both data streams at an appropriate sampling rate, restrictions from the smartphone’s browser prevented the simultaneous gathering of camera and IMU data streams without corruption, as well as a reduced rate of IMU data. To overcome this, VISIMO identified two possible architectures for future prototype development: either 1) a platform-independent application built using Web Assembly, or 2) separate native applications for iOS and Android devices. For IMU-only localization, VISIMO developed a novel transformer-based architecture, but unfortunately this model did not outperform existing models. VISIMO determined that, while published methods require a second specialized device, modern smartphones contain software libraries that can accomplish this functionality on a single device. Finally, VISIMO developed a novel method for turning SLAM output point clouds into easily readable maps of structures. VISIMO was able to successfully generate maps of unknown environments that had a 79% success rate of detecting walls and displaying a 3D schematic map. Each model component was tested without connecting to Bluetooth or Wi-Fi capabilities to ensure that the algorithm was able to produce the requisite data for map generation and user localization without network support. By containing the model’s algorithms within the user’s device, the model can independently receive the necessary visual and positioning data in the proper format without relying on external networking capabilities, or IMU sensor and camera plug-ins.

Conclusions:

VISIMO has successfully determined the feasibility and foundations for developing a complete prototype of MURAL in Phase II work. VISIMO proved that a localization component based solely on smartphone camera data can run in real time with a native or Web Assembly application and proper calibration. This data can be fed to a wall identification algorithm to display a 3D schematic map for a Common Operating Picture (COP), enabling team members to share a common understanding of an unknown environment. Users are not required to incorporate external hardware or pre-mapping procedures to collect camera and IMU data from their device to support MURAL’s localization component. MURAL does not rely on Bluetooth or Wi-Fi connection to assist with displaying the user position in real time. Further development is needed to optimize the localization model to ensure more accurate tracking of user movement, particularly in situations where an IMU-only algorithm is preferable. These results prove the tool’s feasibility and its ability to increase emergency response procedures and support decision making processes by providing accurate information for further analysis.

During the Phase I period of performance, VISIMO developed a commercialization plan to guide its discovery interview approach and validate end-user and customer assumptions. VISIMO interviewed ten end users with various backgrounds and experiences, including an EPA On-Scene Coordinator and Environmental Scientist, law enforcement officers, and Urban Search and Rescue (USAR) members. Through these interviews, VISIMO researched the unique challenges the EPA and first responders face during disaster scenarios. While existing tools can either track user movements or map unknown locations, they rely on external hardware and sensors or require uninterrupted wireless connectivity for real-world use. In addition, emergency responders primarily rely on verbal and radio communications to share information about a member’s location within an unknown environment, becoming vulnerable if their sole communication node fails to maintain connection. VISIMO’s tool resolves these issues as MURAL only requires a device’s camera and IMU sensor to collect and process data for user localization and display the user position within a 3D map. This will support future emergency response as it decreases the time required for teams to discuss internal structures based on memory during high-stress situations and allows team leaders and supervisors to track user movement within an unknown building, increasing overall situation awareness. 

While this tool was initially designed to support the EPA’s need for COP generation, it can be adapted to assist other end-use domains (e.g., law enforcement organizations, criminal investigation divisions, and insurance companies). This tool will increase a user’s understanding of an unknown environment and facilitate tracking user movement, ultimately assisting in greater information collection clarity without relying on users to memorize internal structure details or communicate solely via verbal communications to track user locations.