NRE — NVIDIA Omniverse NuRec (Neural Reconstruction Engine)

Purpose

Drive the public NVIDIA Omniverse NuRec / Neural Reconstruction Engine containers (nvcr.io/nvidia/nre/nre, nvcr.io/nvidia/nre/nre-tools) to train a 3DGUT/3DGRT Gaussian reconstruction from an NCore V4 camera+LiDAR clip, render novel views (locally or via gRPC), generate aux data, export PLY/depth/mesh/ego-mask/tracks, package Asset Harvester output into a USDZ, and evaluate rendering metrics.

This skill carries the host-side toolkit around the NRE CLI: NGC credential resolution, cached-image notes, local render recipes, MP4 encoding, warm serve-grpc boot/teardown scripts, a thin Python gRPC client for repeated RGB renders, bundled rig JSONs, pre-baked custom-rig trajectories, and bash / Hydra / OSMO workflow templates.

When to Use / When NOT to Use

Use this skill when the user has an NCore V4 clip (or a USDZ + NRE artifact pair) on a Linux x86_64 host with an NVIDIA GPU and an NGC API key, and wants to train, render, generate aux data, export artifacts, insert/remove actors, run the gRPC server, or evaluate metrics. Concrete triggers:

• Train a multi-camera + LiDAR AV clip into a renderable USDZ

scene with 3DGUT (or 3DGRT ray-traced) Gaussians.

• Generate NuRec auxiliary data (seg, depth, ego mask, DINOv2,

LiDAR-seg visibility) using nre-tools.

• Render frames locally (no server) along the training rig or a

custom rig + offsets.

• Render novel views via the sensorsim gRPC API (CARLA, Isaac

Sim, AlpaSim, custom simulator), optionally with Difix artifact-removal.

• Render LiDAR sweeps via render-grpc --lidar.
• Export PLY / ego masks / depth / Poisson mesh / ground mesh /

point clouds / cuboid tracks / NCore tracks / custom rig trajectories.

• Insert / remove / replace 3D actors with export-external-assets

+ render-grpc --edit-assets.

• Render the gated HF dataset

nvidia/PhysicalAI-Autonomous-Vehicles-NuRec.

• Upgrade an old USDZ once (upgrade-artifact).
• Inspect / evaluate (export-parsed-config, gaussian-statistics,

eval-rendering-metrics, compute-metrics, eval-ground-mesh).

• Browse a USDZ or PLY in the in-container viewer.

Do NOT use this skill when:

• The user still needs to convert raw sensor data into NCore V4

(use the ncore skill first; NRE consumes NCore-formatted shards).

• The user wants per-object 3D asset extraction from sparse views

(use asset-harvester; NRE only consumes AH outputs via export-external-assets).

• The user only needs to clean up already-rendered frames using

the standalone Cosmos-based Fixer (use nurec-fixer). NRE's inline --enable-difix flag is still on this skill's surface, but the standalone harmonizer pipeline is owned by nurec-fixer.

Inputs

• dataset_dir — host directory holding the NCore shards

(<NAME>.zarr.itar, <NAME>.json, and any pre-generated <NAME>.aux.*.zarr auxiliary shards). Required.

• dataset_name — basename of the NCore dataset (the part

before .zarr.itar). Required.

• output_dir — host directory NRE will fill with checkpoints,

parsed config, metrics, videos, and USDZ artifacts. Required.

• cameraids / lidarids — sensor IDs from the NCore JSON to

include. Default: all sensors per recipe.

• config_name — Hydra config path resolved inside the

container. Pick by source dataset: - Waymo Open Dataset → configs/apps/AV/Waymo/3dgutdynamic.yaml (and its mcmc / roadsemantic / static siblings). These are Waymo-only — they bake in the Waymo sensor rig and conventions. - NVIDIA Physical AI Autonomous Vehicles (PAI) → /apps/prod/Hyperion-8.1/car2sim6cam.yaml (the Hyperion-8.1 car2sim 6-camera recipe used by the Maglev PAI pipeline). Typically referenced via the small overlay shipped at [references/configs/pai.yaml](references/configs/pai.yaml), which extends car2sim6cam.yaml with PAI's lidartop360fov ID, six-camera validation set, and lidar intensity supervision; mount it as {nreconfigdir}/externaloverrides.yaml and pass --config-name=external_overrides. - PandaSet / NVIDIA AV (NV) / Tesla / Alpasim → see the matching configs/apps/AV/{PandaSet,NV,Tesla}/… or configs/apps/Alpasim/… recipes in references/configuration.md. - Do not use the Waymo recipes for PAI clips — the sensor rig, validation cameras, and lidar IDs differ.

• mode — train, val, or trainval. Default: trainval.
• NGCAPIKEY — required env var. Generate at

<https://org.ngc.nvidia.com/setup/api-keys>;.

Instructions

1. Validate prerequisites. Have the agent execute

scripts/validatesetup.py via its standard script runner (runscript("scripts/validatesetup.py"), or python scripts/validatesetup.py [--strict]). It checks Docker, NVIDIA Container Toolkit, GPU/driver, and NGCAPIKEY. Resolve any FAIL line before pulling the image.

1. Authenticate Docker to NGC + pull the public containers.

See [references/install.md](references/install.md). Pull both nvcr.io/nvidia/nre/nre:latest and nvcr.io/nvidia/nre/nre-tools:latest.

1. Confirm input layout. The dataset directory must contain

<NAME>.zarr.itar, <NAME>.json, and any <NAME>.aux.*.zarr shards. If the NCore data is fresh, generate auxiliary data first — see references/aux-data.md.

1. Train / validate the reconstruction. Run the train recipe

in [references/cookbook.md](references/cookbook.md) with the chosen Hydra --config-name, mode, dataset.path, and outdir. For multi-GPU append trainer.worldsize=<N> trainer.num_nodes=<M> (see Workflow D). Set checkpoint.artifact.enabled=true if you intend to render or serve the result.

1. Export downstream artifacts. Use export sub-commands

(export-gaussian-plys, export-mesh, export-ground-mesh, export-ego-mask, export-depth, export-sequence-tracks, export-ncore-tracks, …) — full surface in references/cli-reference.md.

1. Render novel views — pick the backend.

- Local CLI — nre render --artifact-path <usdz> writes frames on disk along the training trajectory, with optional rig offsets or --custom-rig-trajectory. No gRPC server. See references/local-render.md. - Warm RGB service — boot serve-grpc once with scripts/sessionwarmserver.sh, extract protobuf stubs, and use references/NRERenderClient/scripts/thinclient.py for repeated single-camera or batchrenderrgb calls. - Remote CLI / simulator integration — serve-grpc + render-grpc (or your own client via nre.grpc.protos). Required for LiDAR rendering, simulator loops, Difix, or --edit-assets. See references/grpc-api.md and references/physical-ai-render.md.

1. Edit actors (optional). Run export-external-assets to

repackage Asset-Harvester output into a new USDZ, then pass the produced edit-assets.json to render-grpc --edit-assets (with serve-grpc --enable-editing-actors). See references/asset-editing.md.

1. Validate the result. Confirm

<output_dir>/<RUN-ID>/usd-out/last.usdz opens, metrics.yaml reports a reasonable test/psnr, and the generated MP4s render. For more thorough metrics use Workflow I (eval) in [references/workflows.md](references/workflows.md). Tear down any gRPC server (Ctrl-C or docker rm -f).

For any NRE task expected to run 5 minutes or longer (training, OSMO jobs, multi-clip renders), follow references/long-running-tasks.md: delegate to a subagent / background job and report compact status at least every 5 minutes.

Examples

Example 1 — End-to-end NCore → USDZ → render

Walk Workflow A in [references/workflows.md](references/workflows.md): validate host → generate aux data → train (cookbook recipe) → export → local render or serve-grpc. Concrete commands live in the referenced files; this index does not duplicate them.

Example 2 — Skip training, render the gated Physical AI dataset

Walk Workflow B: download nvidia/PhysicalAI-Autonomous-Vehicles-NuRec from HuggingFace, then jump to serve-grpc + a Python client. Coordinate-frame conversion code is in references/physical-ai-render.md.

Example 3 — Insert Asset-Harvester actors into a USDZ

Walk Workflow C: run asset-harvester, then export-external-assets, edit edit-assets.json, and call serve-grpc --enable-editing-actors + render-grpc --edit-assets. Schema lives in references/asset-editing.md.

Example 4 — Warm-server thin-client for repeated RGB renders

Walk the warm-server quick start at the bottom of [references/workflows.md](references/workflows.md). Boot scripts/sessionwarmserver.sh, render with thinclient.py, tear down with scripts/sessionteardown.sh.

Backend Selection

Pick the smallest backend that exposes the requested feature:

• Local Docker, single command. Use nre render, render-grpc,

or an export sub-command directly. Simplest for one-off renders, LiDAR sweeps, actor edits, rolling shutter, in-container video export, or exact --replicate-training-views behavior. See references/local-render.md, references/nre-image-notes.md, and references/mp4-encoding.md.

• Local Docker, warm serve-grpc + thin host client. Use for

render-heavy RGB sessions where repeated Docker/Python/CUDA cold-start dominates latency, or where multiple cameras should be rendered through one batchrenderrgb RPC. See references/NRERenderClient/README.md and scripts/sessionwarmserver.sh / scripts/sessionteardown.sh.

• OSMO / cluster workflows. Use the templates under

references/example-workflows/osmo/ for multi-clip fan-out, isolation from the local machine, or training jobs that should not run on the user's workstation. Follow references/ngc-and-registry.md for registry credentials and references/long-running-tasks.md for polling discipline.

Output Format

Structured deliverables placed under ${outputdir}/${RUNID}/ by the NRE container (no JSON state file required from the agent):

• config/parsed.yaml — Hydra-resolved training config.
• checkpoints/last.ckpt (plus periodic snapshots).
• val/metrics.yaml — per-frame PSNR / SSIM / LPIPS under

test/*.

• val/.mp4, val/<frame>/.png — depth, opacity, segmentation,

RGB visualisations.

• usd-out/last.usdz — USDZ containing the trained reconstruction,

datainfo.json, rigtrajectories.json, sequencetracks.json, parsedconfig.yaml, checkpoint.ckpt, optional mesh.ply, and map.xodr. Render with nre render, serve-grpc + render-grpc, the in-container viewer, or hand to a downstream simulator (CARLA, AlpaSim, Isaac Sim).

• .ply / egomask/ / depth/* / sequencetracks.json /

ncoretracks.json / mesh.ply / groundmesh.ply — produced by the matching export sub-command.

Scripts

References

Read these on demand; keep SKILL.md as the routing layer.

• [references/install.md](references/install.md) — `docker

login nvcr.io, image pull, full prerequisite matrix, and safe secret-handling for NGCAPIKEY / HF_TOKEN`.

• [references/cookbook.md](references/cookbook.md) — most-used

docker run invocations: train + validate, re-validate with shift, local render at quarter or native res, serve-grpc boot, LiDAR sweep, in-container --help.

• [references/workflows.md](references/workflows.md) — workflows

A – I end-to-end, plus the warm-server thin-client quick start.

• [references/troubleshooting.md](references/troubleshooting.md)

— extended error matrix (OOM, wandb blocking, --artifact-glob mismatches, deprecated flags, gRPC LiDAR size, etc.).

• [references/teardown.md](references/teardown.md) — disk

cleanup, post-teardown verification, ownership-recovery.

• references/cli-reference.md — full sub-command surface of the

NRE container (training, validation, render, serve-grpc, render-grpc, render-novel-trajectory, every export-*, upgrade-config / upgrade-artifact, gaussian-statistics, eval-rendering-metrics, compute-metrics, viewer, ply_viewer, profile-dataloader, run-script, the nre-tools aux-data + AH entry points).

• references/configuration.md — Hydra recipe map for Waymo / NV

/ PandaSet / Tesla / Alpasim, plus override matrix.

• references/aux-data.md — nre-tools auxiliary-data CLI.
• references/local-render.md — host-side docker run … render

recipes for rig offsets and export-custom-rig-trajectory.

• references/carline-adaptation.md — augmented target-rig trajectory export,

local render, and optional nurec-fixer Harmonizer handoff.

• references/NRE_RenderClient/README.md — warm-server thin

Python gRPC client.

• references/grpc-api.md — sensorsim gRPC server flags + Python

client cookbook.

• references/nre-image-notes.md — cached-image discovery, 26.04+

vs 26.03 vs pre-26.03 flags.

• references/ngc-and-registry.md — NGC API key resolution.
• references/mp4-encoding.md — host-side ffmpeg recipe.
• references/asset-editing.md — export-external-assets +

edit-assets.json schema.

• references/physical-ai-render.md — recipe for rendering the

HuggingFace NuRec dataset.

• references/example-workflows/ — bash, Hydra, and OSMO

templates.

• references/rig-json/ — bundled rig.json and

augmented_rig.json.

• references/custom-rig-trajectories/ — pre-baked

export-custom-rig-trajectory outputs.

• references/long-running-tasks.md — background-job + 5-minute

status reporting convention.

• references/nurec-skill-catalog.md — routing table for sibling

NuRec-stack skills.

• Public product page: <https://www.nvidia.com/en-us/omniverse/nurec/>;
• HF dataset: <https://huggingface.co/datasets/nvidia/PhysicalAI-Autonomous-Vehicles-NuRec>;
• HF Fixer model: <https://huggingface.co/nvidia/Difix3D>;
• NGC Fixer model card: <https://catalog.ngc.nvidia.com/orgs/nvidia/teams/nre/models/nurec-fixer>;

Prerequisites

Linux x8664 + NVIDIA GPU + Docker 23+ + NVIDIA Container Toolkit 1.13+ + NGCAPIKEY. Full matrix (driver minimums per arch, shm-size, file ownership, GPU-tier guidance) lives in [references/install.md](references/install.md). Always verify via scripts/validatesetup.py before pulling the image.

Limitations

• Linux x86_64 only. aarch64 (e.g. Jetson) is not supported.
• Internal source not redistributable. Use only the public

NGC containers and the public NuRec docs.

• Multi-GPU defaults are conservative. Set

trainer.worldsize / trainer.numnodes explicitly to scale out; SLURM is auto-detected when both are 0. Quality plateaus past ~6 GPUs; per release notes, multi-GPU + dataset.aux_data=false is a known crash combination.

• --config-name paths differ between train and val/export.

Training uses container-bundled recipes; validation and exports re-pass the parsed.yaml written under <output_dir>/<RUN-ID>/config/.

• Validation may prompt for wandb. Choose option 3 to skip

in non-interactive runs, or pass logger=tensorboard / logger=dummy.

• Render gRPC is data-format-pinned. Older releases warn /

reject artifacts that pre-date them; check release notes when mixing client / server versions.

• Asset-Harvester input only. export-external-assets

requires AH outputs; raw .ply files won't carry the per-asset cuboid metadata.

• Difix variants are pluggable. The container ships both the

Cosmos Difix variant (default since 25.09 — difix=cosmosdifix) and the legacy Stable-Diffusion variant (difix=sddifix). The newer Cosmos-Predict-based Fixer variants live in the nurec-fixer skill.

• render ↔ render-grpc overlap. render runs

in-container without a server; render-grpc requires an active serve-grpc. Use render for batch novel-view jobs and render-grpc when you need actor editing, LiDAR rendering, or a long-lived service.

Troubleshooting (top 4)

Full matrix in [references/troubleshooting.md](references/troubleshooting.md).

Teardown

Full inventory, ownership-recovery, and post-teardown verification commands live in [references/teardown.md](references/teardown.md). Headline: stop serve-grpc containers, docker image rm nre/nre-tools, rm -rf ${HOME}/.cache/nre and your <outputdir>/<RUN-ID>/. Do not revoke NGCAPI_KEY unless you suspect it has been leaked.