At SIGGRAPH, NVIDIA announced updates to the NVIDIA Omniverse libraries and Cosmos world foundation models (WFMs). Powered by OpenUSD, developers can access new libraries, models, and development tools for building physically accurate virtual environments and AI agents that understand the real world.

Simulating robots that understand and interact quickly and reliably with the real world is now easier with the latest releases from NVIDIA, including:

New Omniverse libraries advance world composition applications

OpenUSD serves as a foundational technology for physical AI workflows, empowering developers to integrate 3D data for highly detailed simulations and accelerating development across robotics, autonomous vehicles, and industrial systems. 

The OpenUSD Exchange SDK 2.0 includes new modules for UsdPhysics authoring and a layered asset structure​, facilitating direct integration of physics into robot models. The SDK is an open-source framework for streamlining 3D data workflows, enabling developers to integrate their 3D data for highly detailed simulations and accelerate robotics development. It is available now on GitHub and PyPi.

In physical AI, bringing geometric data into a simulated environment is only the beginning. To train models effectively, materials must be more than visually realistic; they need to be simulation-ready. NVIDIA released a SimReady materials library. This open-source collection of thousands of substrate materials for physical AI, authored in USD with MaterialX/OpenPBR feature:

UsdShade-based materials leverage OpenPBR v1.1 to provide high-quality, physically-based rendering (PBR).Semantic labeling for non-visual data in LIDAR and RADAR (like thermal, ultrasonic). 

All materials in the library are needed for accurate sensor simulation and realistic physical interactions. Because it supports both visual and non-visual rendering, developers can import into simulation environments such as NVIDIA Isaac Sim and add realistic materials to their simulations, for faster iteration and higher fidelity in physical AI pipelines..

Building an interoperable data pipeline from MuJoCo to USD

NVIDIA and Google DeepMind are empowering robotics developers through enhanced data interoperability between MuJoCo (MJCF) and USD formats, enabling more accurate and consistent robot data generation.

A new mujoco-usd-converter, built using OpenUSD Exchange SDK 2.0 and the MjcPhysics USD schemas from Google DeepMind, provides interoperable conversion of MJCF data to the USD format. Early release of MJCF to USD data conversion is also available on GitHub and PyPi.

To start using the converter, install the Python wheel into a virtual environment using your favorite package manager:

pip install mujoco-usd-convertermujoco_usd_converter /path/to/robot.xml /path/to/usd_robot

See mujoco_usd_converter --help for CLI arguments.

Alternatively, the same converter functionality can be accessed from the Python module directly, which is useful when further transforming the USD data after conversion.

import mujoco_usd_converterimport usdex.corefrom pxr import Sdf, Usdconverter = mujoco_usd_converter.Converter()asset: Sdf.AssetPath = converter.convert("/path/to/robot.xml", "/path/to/usd_robot")stage: Usd.Stage = Usd.Stage.Open(asset.path)# modify further using Usd or usdex.core functionalityusdex.core.saveStage(stage, comment="modified after conversion")

Google DeepMind has introduced a new experimental release. It enables USD data imports into MuJoCo through the new MjcPhysics USD Schemas and a new SdfFileFormatPlugin that brings MJCF into the USD ecosystem. Native support for simulation exporting as animation layers is also on the way. Release details can be found in Google DeepMind documentation and GitHub.

Real-world scene reconstruction for physical AI

NVIDIA Omniverse NuRec libraries combine NVIDIA RTX ray tracing with 3D Gaussian splatting to turn real-world sensor data into interactive OpenUSD simulation environments. This results in high-fidelity world reconstruction at incredible scale and efficiency. NuRec rendering is integrated in the Omniverse Kit SDK, Isaac Sim, and CARLA, a leading open-source simulator used by over 150,000 developers. 

The following is a sample tutorial on how to capture real-world data, train a reconstruction, and load the results into Isaac Sim:

Capture a scene with 100 photos from all angles and good lightingReconstruct the scene with COLMAP and 3DGUTDeploy the scene by exporting in USD, normalizing, and importing to Isaac Sim.

Read the detailed steps on how to import a real-world scene into Isaac Sim.

Isaac Sim 5.0 and NVIDIA Isaac Lab 2.2 

Isaac Sim 5.0 and Isaac Lab 2.2 are advancing open-source robot simulation and learning frameworks, making simulation setup faster, more consistent, and closer to real-world behavior. Available on GitHub, the early developer preview of Isaac Sim 5.0 includes OpenUSD-based robot and OmniSensor schemas that standardize robot and sensor definitions in USD. 

The PhysX extension now supports a new joint friction model defined through an OpenUSD schema. Developed with Hexagon’s Robotics division and maxon, it enables simulated robot movements to more closely match real-life behavior.

Industry leaders like Amazon Lab126, Boston Dynamics, Figure AI, Haply Robotics, Hexagon, Lightwheel, RAI Institute, Resim.ai, and Skild AI are adopting NVIDIA Isaac libraries and AI models—such as Isaac Sim and Isaac Lab—to accelerate their AI robotics development.

Get started with robot simulation

Start building today with the latest NVIDIA Omniverse Libraries and tools, including NuRec, Omniverse Kit SDK 108, SimReady materials library, NVIDIA MuJoCo-to-USD data converter (early release on GitHub and PyPi), USD Exchange SDK 2.0, and early developer previews of Isaac Sim 5.0 and Isaac Lab 2.2.

Check out how these OpenUSD all-stars are accelerating their physical AI workflows. Interested in becoming an all-star? Take the next step in your career with the OpenUSD Development certification—an industry-recognized, professional-level exam that validates your ability to build, maintain, and optimize 3D content pipelines using OpenUSD.

Stay up to date by subscribing to NVIDIA news and following NVIDIA Omniverse on Discord and YouTube. Watch the NVIDIA Research special address at SIGGRAPH.