NVIDIA's OpenUSD and Halos Framework Enhance Robotaxi Safety and Physical AI Systems

Editor’s note: This post is part of Into the Omniverse, a series focused on how developers, 3D practitioners, and enterprises can transform their workflows using the latest advancements in OpenUSD and NVIDIA Omniverse.

With the evolution of Physical AI, real-world applications are becoming more prevalent, particularly in the realms of intelligent robotics and autonomous vehicles (AVs), including robotaxis. These systems are required to adeptly sense, reason, and act in dynamic environments. To scale these technologies safely and effectively, developers are turning to workflows that integrate real-world data with high-fidelity simulations and reliable AI models, all built on the foundation of the OpenUSD framework.

The recent release of the OpenUSD Core Specification 1.0 introduces standard data types, file formats, and composition behaviors. This standardization is essential for creating interoperable USD pipelines that can accommodate the complexities of autonomous systems.

Using OpenUSD as a base, NVIDIA Omniverse libraries leverage NVIDIA RTX for rendering and physics simulation. This allows for the creation of digital twins and SimReady assets that closely mirror real-world conditions, crucial for synthetic data generation and testing.

Building the Foundation for Safe Physical AI

Open Standards and SimReady Assets

The OpenUSD Core Specification 1.0 establishes data models and behaviors necessary for SimReady assets. This foundation makes it possible for developers to create simulation pipelines that are functional and interoperable for AI applicability across robotics and industrial processes.

SimReady 3D assets derived from OpenUSD can be utilized directly in NVIDIA Isaac Sim. Key features like USDPhysics colliders and rigid body dynamics facilitate realistic testing of robots in virtual environments that simulate actual operations closely.

Open-Source Learning

The Learn OpenUSD curriculum, now open source on GitHub, allows contributors to adapt educational content for diverse audiences and use cases, giving educators the resources needed to train teams on OpenUSD simulation workflows effectively.

Generative Worlds as Safety Multiplier

Techniques like Gaussian splatting, which utilizes editable 3D components, are speeding up the creation of safe robotics testing environments. At SIGGRAPH Asia, NVIDIA Research unveiled Play4D, a streaming pipeline that employs 4D Gaussian splatting, enhancing the realism of dynamic scenes.

Collaborating with World Labs, researchers can use the Marble generative world model in conjunction with NVIDIA Isaac Sim and Omniverse NuRec for rapid generation of photorealistic, physics-ready 3D environments based on text prompts, reducing weeks of work to mere hours. These environments become training grounds for physical AI, providing an extensive range of scenarios for rigorous practice.

Lightwheel Helps Teams Scale Robot Training with SimReady Assets

Building on OpenUSD capabilities, Lightwheel creates comprehensive SimReady asset libraries that simplify the assembly of high-fidelity digital twins. These assets feature precise geometries and validated physical properties, ready for incorporation into NVIDIA Isaac Sim and Isaac Lab for effective robot training.

End-to-End Autonomous Vehicle Safety

Recent advancements are amplifying safety protocols for autonomous vehicles through new research and tools that enhance testing rigors. Notably, NVIDIA researchers in collaboration with Harvard and Stanford have introduced the Sim2Val framework, which statistically integrates real-world data with simulated test results. This significantly diminishes the demand for extensive real-world testing, demonstrating how robotaxis can operate safely even under rare circumstances.

NVIDIA has also released the open-source NuRec Fixer model, capable of rectifying artifacts in neural data reconstructions, thereby improving the quality of SimReady assets.

AV Ecosystem Leaders Putting Physical AI Safety to Work

Several industry leaders, including Bosch, Nuro, and Wayve are engaging with the NVIDIA Halos AI Systems Inspection Lab. This initiative is designed to streamline the deployment of robotaxi fleets safely. Onsemi has achieved a milestone by becoming the first company to pass the inspection focusing on AI systems for AVs.

The open-source CARLA simulator harnesses both NVIDIA NuRec and Cosmos Transfer for generating diverse simulations, while Voxel51’s FiftyOne engine is instrumental in curating and evaluating extensive multimodal datasets across the AV pipeline.

Additionally, the Mcity research team at the University of Michigan is enhancing its digital twin AV test site using Omniverse technologies. This initiative aims to integrate state-of-the-art physics-based models for sensors, optimizing both real and simulated data for comprehensive testing.

Get Plugged Into the World of OpenUSD and Physical AI Safety

For those interested in enhancing their understanding of OpenUSD, NVIDIA Halos, and physical AI safety, consider accessing the following resources:

• Watch the on-demand NVIDIA GTC session titled “Reconstructing Reality: Simulating Indoor and Outdoor Environments for Physical AI.”
• Visit the NVIDIA Halos AI Systems Inspection Lab webpage for more information.
• Follow the NVIDIA DRIVE LinkedIn newsletter: “NVIDIA Safety in the Loop.”
• Read the corporate blog post about how AI is advancing Level 4 autonomy: How AI Is Unlocking Level 4 Autonomy.
• Get started with the Learn OpenUSD curriculum, which is now available as an open-source resource.

Stay updated by subscribing to NVIDIA news, becoming part of the community, and following NVIDIA Omniverse on Instagram, LinkedIn, Medium, and X.