NCore V4 Data Conversion

Purpose

Convert any sensor recording (cameras, LiDAR, radar, IMU, depth, stereo, COLMAP/SfM, ROS 2 bag) into a valid NVIDIA NCore V4 store so it can be consumed by NuRec / Asset Harvester / ncorevis, or wired into a robotics-to-sim ("r2s") pipeline. Drive the existing in-tree converters (PAI, Waymo, COLMAP/ScanNet++) or author a new converter from ncoretemplate/.

Use this skill when: the user has raw sensor data (any rig) that NuRec or Asset Harvester needs to ingest, or when an existing converter is failing validate.py / producing NuRec data-quality complaints.

Do NOT use this skill when:

• The user is already on V4 and only wants to train or render

(use the nre skill).

• The user wants per-object 3D assets from sparse views (use

asset-harvester).

• The user only needs to browse / pick an existing NVIDIA dataset

(use physical-ai-datasets).

This skill teaches an agent to take any sensor dataset and produce a valid NCore V4 store that NuRec / Asset Harvester / ncore_vis will accept. It covers both driving the existing in-tree converters (PAI, Waymo, COLMAP/ScanNet++) and writing a new one for unsupported formats (PandaSet, NuScenes, KITTI, stereo, mono+depth, mono+lidar, custom robotics rigs).

Table of Contents

1. [When to use which path](#when-to-use-which-path)
2. [Install & references](#install--references)
3. [Mental model — the V4 store](#mental-model--the-v4-store)
4. [Path A — drive an existing in-tree converter](#path-a--drive-an-existing-in-tree-converter)
5. [Path B — author a new converter from the template](#path-b--author-a-new-converter-from-the-template)
6. [V4 conventions you must obey](#v4-conventions-you-must-obey)
7. [Format recipes (AV)](#format-recipes-av)

- [PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)](#pai-nvidia-physical-ai-autonomous-vehicles-huggingface) - [Waymo Open](#waymo-open) - [PandaSet](#pandaset) - [NuScenes](#nuscenes)

1. [Format recipes (non-AV / sensor-only)](#format-recipes-non-av--sensor-only)

- [Mono camera (no depth, no LiDAR) → COLMAP track](#mono-camera-no-depth-no-lidar--colmap-track) - [Stereo cameras](#stereo-cameras) - [Multi-stereo rig (surround stereo)](#multi-stereo-rig-surround-stereo) - [Mono + depth (RGB-D / learned depth)](#mono--depth-rgb-d--learned-depth) - [Mono + LiDAR (handheld / robot)](#mono--lidar-handheld--robot) - [Solid-state / non-repetitive LiDAR (Livox)](#solid-state--non-repetitive-lidar-livox) - [IMU + camera (visual-inertial)](#imu--camera-visual-inertial) - [ROS2 bag (MCAP / SQLite3)](#ros2-bag-mcap--sqlite3) - [Aerial / drone](#aerial--drone)

1. [Robotics pipeline shards (r2s)](#robotics-pipeline-shards-r2s)
2. [Validation & end-to-end NuRec](#validation--end-to-end-nurec)
3. [Common failure modes (and the fix file)](#common-failure-modes-and-the-fix-file)
4. [Additional resources](#additional-resources)

When to use which path

If a candidate path exists in upstream, prefer it. Hand-rolling a Waymo or PAI converter on top of the template is wasted work and almost certainly wrong (rolling-shutter timing, FTheta intrinsics, Waymo camera-frame rotation, etc).

Prerequisites

• Linux host with Python ≥ 3.10 and pip.
• git for the upstream converter sources.
• bazel only if you run the in-tree converters (Path A); pure-Python

in-process writes (ncore.data.v4) need no Bazel.

• Disk: budget tens of GB per converted clip; pre-zarr scratch can be

larger than the final .zarr.itar.

• HuggingFace token (HF_TOKEN) only if pulling a gated PAI clip.

Verifying secrets safely

Always verify prerequisites with the upstream validate.py or by running the converter against a tiny test slice; never write ad-hoc bash that interpolates HF_TOKEN values. The common one-liner

# BAD — leaks the secret to the terminal when the variable is set
echo "HF_TOKEN: ${HF_TOKEN:+yes}${HF_TOKEN:-no}"

prints yes<token-value> whenever HF_TOKEN is set, because ${VAR:-no} only falls back to "no" when the variable is empty. Use a length-only check, which never echoes the value:

# OK — prints "set (N chars)" or "missing", never the value
test -n "$HF_TOKEN" && echo "HF_TOKEN: set (${#HF_TOKEN} chars)" || echo "HF_TOKEN: missing"

Rotate any token you suspect was echoed at <https://huggingface.co/settings/tokens>;.

Install & references

pip install nvidia-ncore        # pure-Python API for in-process writes
git clone --depth 1 https://github.com/NVIDIA/ncore.git   # for upstream converters (Bazel)

• Source + upstream converters: <https://github.com/NVIDIA/ncore>;
• V4 spec / conventions: <https://nvidia.github.io/ncore/data/conventions.html>;
• API reference: <https://nvidia.github.io/ncore/apis/data.v4.html>;
• Conversion guides: <https://nvidia.github.io/ncore/conversions/index.html>;
• Sensor models (camera, LiDAR, windshield): <https://nvidia.github.io/ncore/data/sensor_models.html>;
• The template scaffold (every method documented inline):

[ncoretemplate/impl/dataconverter/exampleconverter.py](ncoretemplate/impl/dataconverter/exampleconverter.py)

• End-to-end NCore → NRE training and rendering: see the sibling

[nre](../nre/SKILL.md) skill (Workflow A). NVIDIA's reference OSMO recipe that wires PAI → NCore → NuRec → USDZ training lives in the upstream NCore repo at <https://github.com/NVIDIA/ncore/tree/main/tools/data_converter/pai>; and the NRE container docs at <https://www.nvidia.com/en-us/omniverse/nurec/>;.

Mental model — the V4 store

Every V4 sequence is one store (itar archive or plain directory) holding component groups. The required components for NuRec are:

Frames of reference (these are non-negotiable — wrong frames = silent NuRec failure):

• Rig: +X forward, +Y left, +Z up. Origin at the middle of the rear axle on nominal ground for AV, or any natural body-fixed point for non-AV. All extrinsics are T_sensor → rig.
• Camera sensor: +X right, +Y down, +Z forward (optical axis). NCore's convention. Waymo (X-fwd) and similar must be rotated before storing extrinsics.
• LiDAR model frame: azimuth 0° = +X, 90° = +Y, +Z up. Independent of the raw sensor's native azimuth — you choose how columnazimuthsrad maps physical columns.
• World: sequence-local. Re-reference all Trigworld to the first ego pose so origin is near the vehicle start (raw UTM/ECEF at 10+ km loses precision in float32). Carry the original first pose into Tworldworld_global (float64) if you need a global anchor.
• Image pixels: u right, v down, origin at the top-left corner of the top-left pixel (so pixel centers are at 0.5, 0.5).
• Units: timestamps in µs everywhere, distances in metres, angles in radians.

The single source of truth is the spec — when in doubt, open it: <https://nvidia.github.io/ncore/data/conventions.html>;.

Sequence-level metadata

Some downstream pipelines need metadata beyond the per-component data — carry it on the sequence's genericmetadata (passed to SequenceComponentGroupsWriter(...)):

• Stereo pairs — required by stereo-depth modules (Foundation Stereo) and

multi-camera training configs to discover left/right pairings:

``json {"stereopairs": [{"left": "camerafrontleft", "right": "camerafront_right"}]} ``

Multiple pairs are allowed for surround-stereo rigs.

• Source tag — distinguishes real from synthetic data. Renderer-output

shards (NuRec sim) set {"source": "simulation", "model_checkpoint": "..."}; real-sensor shards omit the key or set {"source": "real"}. Downstream validators key off this to skip "expected vs measured" checks on sim data.

• Calibration / egomotion provenance — the example template writes

calibrationtype and egomotiontype on the PosesComponent's genericmetadata. Use this to track the upstream tool (e.g. egomotion_type: "cuvslam-stereo" or "kiss-icp" or "vio:orbslam3") so later modules can pick refinement strategies that match the input quality.

Instructions

Pick one of the two paths below.

• Path A if the user's dataset format is already supported in

ncore/tools/data_converter/ (PAI, Waymo, COLMAP/ScanNet++) — bootstrap the upstream repo and drive the existing binary.

• Path B if the format is unsupported (PandaSet, NuScenes, KITTI,

custom rig, robotics bag, …) — copy ncore_template/ next to the dataset and fill in the four hand-written hooks. The V4 conventions in the Mental model section above are mandatory; the recipes further down show typical configurations per rig.

After conversion, always run the Validation & end-to-end NuRec section below before handing the store to NRE.

Path A — drive an existing in-tree converter

The upstream tools/data_converter/<format> modules are Bazel targets. Build once, run per dataset.

Bootstrap

git clone --depth 1 https://github.com/NVIDIA/ncore.git
cd ncore
bazel build //tools/data_converter/pai:convert     # or waymo, or colmap

Each convert binary takes shared base flags (--root-dir, --output-dir, --no-cameras, --camera-id, --no-lidars, --lidar-id, --verbose) followed by a subcommand (pai-v4, pai-stream-v4, waymo-v4, colmap-v4, scannetpp-v4) with format-specific flags.

Standard sub-flags worth knowing

When to script vs run interactively

For a one-off conversion, the bare bazel run form in each format recipe below is enough. For repeatable cluster runs, wrap the same two steps (bazel build then bazel run) inside an OSMO / Slurm / Kubernetes task that clones NCore at a pinned ref, runs the convert step, and chains the result into the [nre](../nre/SKILL.md) training and aux-data containers (see nre's Workflow A). The upstream NVIDIA/ncore repo ships reference converter targets that you can pin by Git commit for reproducibility.

Path B — author a new converter from the template

The scaffold is intentionally minimal but writes every required component type with placeholder data. Treat it as a checklist: every # FILL IN is a correctness gate — none can be skipped.

Scaffold

# Copy the scaffold next to your dataset (run from this skill's folder)
cp -r ncore_template /path/to/ncore-myformat

Then rename the package and class (ExampleConverter → MyFormatConverter) and implement the contract:

Inside convert_sequence the canonical order is Poses → Intrinsics → Masks → Camera → LiDAR → Radar → Cuboids → finalize. This order is not required by the writer but it surfaces calibration / pose / timing bugs before you've spent minutes encoding image bytes.

The skeleton walks each step explicitly and lists every silent-correctness trap inline — read these before filling them in:

• Spinning-LiDAR pitfalls (spinningdirection, non-uniform rowelevations_rad,

columnazimuthsrad ordering, Ouster rowazimuthoffsetsrad): [exampleconverter.py:61-120](ncoretemplate/impl/dataconverter/example_converter.py#L61-L120)

• Pose trajectory density + float64 → float32 + re-referencing rules:

[exampleconverter.py:336-492](ncoretemplate/impl/dataconverter/exampleconverter.py#L336-L492)

• Camera intrinsics for Pinhole / Fisheye / FTheta + shutter type:

[exampleconverter.py:524-578](ncoretemplate/impl/dataconverter/exampleconverter.py#L524-L578)

• Per-ray LiDAR timestamps and the three data shapes (range image / sensor-frame

XYZ / world-frame XYZ requiring decompensation): [exampleconverter.py:746-829](ncoretemplate/impl/dataconverter/exampleconverter.py#L746-L829)

• Cuboid centroid convention (geometric center, not bottom-center):

[exampleconverter.py:876-980](ncoretemplate/impl/dataconverter/exampleconverter.py#L876-L980)

In-process API (no Bazel)

If you already have parsed arrays in Python and don't need a CLI, skip FileBasedDataConverter entirely and call the V4 writer directly:

from ncore.data.v4 import SequenceComponentGroupsWriter, PosesComponent, ...
writer = SequenceComponentGroupsWriter(
    output_dir_path=out / seq_id,
    store_base_name=seq_id,
    sequence_id=seq_id,
    sequence_timestamp_interval_us=interval,
    store_type="itar",
)
poses_writer = writer.register_component_writer(PosesComponent.Writer, ...)
# … write each component …
paths = writer.finalize()

The contract (component order, dtype rules, timestamp constraints) is identical.

V4 conventions you must obey

These are the rules that turn into runtime asserts (or worse: silent NuRec artefacts). Cross-reference the spec before relaxing any of them.

Time

• Microseconds, uint64, everywhere. np.uint64, not np.int64.
• The sequence interval is half-closed [start, stop). Build it with

HalfClosedInterval.fromstartend(start, end_inclusive) — do not pre-add 1 to end.

• Dynamic poses must exactly span the interval: timestamps[0] == start and

timestamps[-1] == stop - 1. Sensors with timestamps slightly outside this range are clamped at write time.

• Per-sensor frame timestamps are [exposurestart, exposureend] (cameras) or

[sweepstart, sweepend] (LiDAR/radar). They must lie within the sequence interval and the end must be unique within that sensor's writer.

• Rolling-shutter cameras: start = trigger + half_shutter,

end = readoutdone - halfshutter. Global shutter: start == end is OK.

• LiDAR frametimestampsus[0] is the sweep start (not midpoint). Per-ray

timestampus must lie in [sweepstart, sweep_end]. Treating the dataset's frame timestamp as the midpoint and subtracting half a sweep duration introduces a ~50 ms shift on a 10 Hz LiDAR and produces motion-comp blur.

Pose graph

• All intermediate pose math is float64. Cast to float32 only as the very

last step before storedynamicpose / storestaticpose. Exception: world → worldglobal stays float64 (NuRec's RigTrajectories.Tworld_base is float64).

• Re-reference dynamic poses to the first ego pose:

posesf64 = inv(posesf64[0]) @ poses_f64. Without this, GPS/UTM/ENU at 10+ km loses sub-cm precision once cast to float32.

• The pose trajectory must be dense (waypoint spacing < LiDAR sweep

duration, e.g. <50 ms for 10 Hz). Combine every available source: per-camera per-frame poses, per-LiDAR per-sweep poses, IMU/GPS/odometry. Concatenate, np.unique by timestamp, sort.

• Tcamerarig and Tlidarrig are float32 (NuRec's transform_poses leaks

input dtype through an internal matmul; float64 here collides with the float32 trajectory and crashes "Get Lidar Point Clouds" with RuntimeError: double != float).

LiDAR

• Direction vectors are unit-norm, in sensor coordinates, at each ray's

measurement time (not the sweep-start time). Three input shapes: 1. Polar range image with beam geometry → derive directly from columnazimuthsrad[col], rowelevationsrad[row], rowazimuthoffsetsrad[row]. Direction: (cos(elev)cos(azi), cos(elev)sin(azi), sin(elev)). 2. Sensor-frame XYZ → direction = xyz / |xyz|. 3. World-frame or ego-compensated XYZ → decompensate with MotionCompensator.motiondecompensate_points, then normalise.

• Spinning LiDAR parameter traps (silent — wrong values pass write but fail

NuRec): - spinningdirection: nearly all automotive spinning LiDARs (Velodyne, Hesai, Ouster, Robosense) are "cw". The template defaults to "ccw" to force the question. Wrong value mirrors Y → Z-flip in NuRec. - rowelevationsrad: non-uniform on every common sensor. Read real per-beam angles from the sensor calibration. Strictly descending (highest beam first); reverse if your source is ascending. NCore asserts np.diff > 0 after projecting through relativeangle. - columnazimuthsrad: must reflect the actual per-column azimuth at frame start (not a synthetic linspace). NCore validates strict ordering via relativeangle(azim[0], azim, spinningdirection). If raw per-column heading is available, use it; otherwise linspace(0, ±2π, N, endpoint=False) is acceptable for "ccw" / "cw" respectively. - rowazimuthoffsetsrad: zero for most sensors; non-zero on Ouster (beamazimuth_angles from the HTTP API). Skipping this on Ouster misaligns LiDAR-to-camera projection.

• Per-ray timestamp_us is required for motion compensation; supply real

per-column firing times (typically column-linear across the sweep).

Cuboids

• BBox3.centroid is the geometric center. Many AV datasets use

bottom-center → add dim_z / 2 to z. Verify empirically: mean(z) - mean(h)/2 near 0 = geometric center; near mean(h)/2 above ground = bottom-center.

• referenceframeid ∈ {"rig", "world", <sensor_id>}.

referenceframetimestampus and timestampus must lie within the sequence interval.

• LabelSource: tag the origin pipeline, not quality. Third-party

dataset labels (Waymo, NuScenes, PandaSet) → EXTERNAL, even if the upstream is human GT. Use GT_ANNOTATION only when this converter's team owns the annotation.

Camera

• Image bytes stored verbatim (JPEG/PNG); NCore does not re-encode.
• Use the model that matches your sensor: OpenCVPinholeCameraModelParameters

(most AV cameras), OpenCVFisheyeCameraModelParameters (Kannala-Brandt fisheye, e.g. ScanNet++ / GoPro), FThetaCameraModelParameters (NVIDIA Hyperion / equidistant-radial). Resolution must match the actual image bytes bit-for-bit.

• Map shutter direction by enum name, not integer cast. The five

ShutterType values are ROLLINGTOPTOBOTTOM=1, ROLLINGLEFTTORIGHT=2, ROLLINGBOTTOMTOTOP=3, ROLLINGRIGHTTOLEFT=4, GLOBAL=5. Build a source-enum-to-name dict.

• Provide ego masks. NuRec's data-quality guide requires binary

ego-vehicle masks per camera. Without them the hood/roof rack leaks into the reconstruction.

NuRec data-quality minimums

Per the NuRec "Ensure Data Quality" guide, hitting these is the difference between a clean reconstruction and visible artefacts:

• Camera extrinsics: < 0.5° rotation, < 2 cm translation (relative to rig).
• Camera intrinsics: < 1 px reprojection error.
• Egomotion: < 0.5° / < 2.5 cm consecutive-frame error; trajectory must cover

every sensor frame's start and end timestamp.

• Cuboids: < 1° / < 5 cm position / < 5 cm dimension; per-track stable

track_ids.

• Original sensor resolution at full FPS, no undistortion / rectification.

Format recipes (AV)

PAI (NVIDIA Physical AI Autonomous Vehicles, HuggingFace)

Built-in. Streaming or local. Uses Hyperion 8 / 8.1 sensor IDs.

# Streaming (no download — recommended for cloud / OSMO)
bazel run //tools/data_converter/pai:convert -- \
    --output-dir <OUT> \
    --camera-id camera_front_wide_120fov \
    --camera-id camera_cross_left_120fov \
    --camera-id camera_cross_right_120fov \
    --camera-id camera_front_tele_30fov \
    pai-stream-v4 \
        --clip-id <CLIP_ID> \
        --hf-token "$HF_TOKEN" \
        --store-type itar \
        --profile separate-sensors \
        --sequence-meta

Notes:

• HF dataset: nvidia/PhysicalAI-Autonomous-Vehicles. License must be accepted

on HF before HF_TOKEN works.

• 7 cameras (FTheta + Fisheye + Pinhole depending on FOV), 1 top LiDAR

(lidartop360fov, spinning, see datasheet for elevations).

• Camera intrinsics include shutterdelayus for per-row rolling-shutter

timestamping, plus optional BivariateWindshieldModelParameters for windshield refraction.

• Output path: <output-dir>/pai<clip-id>/pai<clip-id>.ncore4.zarr.itar.
• Discover clip IDs via the clip_index.parquet blob in the HF dataset

(nvidia/PhysicalAI-Autonomous-Vehicles) — the [physical-ai-datasets](../physical-ai-datasets/SKILL.md) skill has the download recipe and toolkit pointers.

• Subset by time with --seek-sec / --duration-sec; subset sensors with

--no-lidars / --camera-id (repeatable).

• car2sim_6cam sim configs target CARLA sensor names; PAI exports use real

Hyperion IDs — when feeding NRE training, override dataset.cameraids / dataset.lidarids on the NRE Hydra command line (see [nre](../nre/SKILL.md) Workflow A and references/configuration.md).

Waymo Open

Built-in. Reads .tfrecord segment files.

bazel run //tools/data_converter/waymo:convert -- \
    --root-dir <DIR_OF_TFRECORDS> \
    --output-dir <OUT> \
    waymo-v4 \
        --store-type itar \
        --profile separate-sensors \
        --world-global-mode localized   # or 'identity' / 'none'

Notes:

• 5 cameras (camerafront50fov, camerafrontleft_50fov,

camerafrontright50fov, camerasideleft50fov, camerasideright50fov) + 1 top LiDAR (lidartop).

• Camera intrinsics → OpenCVPinholeCameraModelParameters. Waymo's local

camera frame is +X principal axis; the converter rotates to NCore's +Z principal axis. If you re-derive extrinsics manually, apply the same rotation to Tcamerarig.

• Cuboid classes: unknown, vehicle, pedestrian, sign, cyclist. Map → NCore

class_id strings; tag LabelSource.EXTERNAL.

• Multi-return LiDAR (primary + secondary). Stack into the [R, N] distance /

intensity arrays.

PandaSet

No upstream converter — author with Path B. PandaSet (Hesai + Scale-AI-labelled) ships JSON metadata with each sequence.

Sensor inventory:

• 6 cameras (frontcamera, frontleftcamera, frontright_camera,

backcamera, leftcamera, right_camera) — pinhole, global shutter. Intrinsics in meta/intrinsics/<id>.json.

• 2 LiDARs:

- frontlidar: Hesai PandarGT (mechanical, 60° HFOV, 150° VFOV forward-facing). Treat as a partial spinning sensor (columnazimuthsrad spans the 60° wedge). - toplidar (key sensor for AV reconstruction): Hesai Pandar64, spinningdirection="cw", 64 beams with non-uniform elevations (Pandar64 datasheet table — copy the 64 angles, sort descending), rowazimuthoffsetsrad = zeros(64), 1800 columns.

Data layout per sequence (extract):

<seq_id>/
├── meta/
│   ├── intrinsics/<camera_id>.json     # focal, principal point, distortion
│   ├── timestamps.json                 # per-camera-frame µs
│   └── poses.json                      # per-frame ego pose (UTM)
├── camera/<camera_id>/<frame>.jpg
├── camera/<camera_id>/poses.json       # per-frame camera pose in world
├── lidar/<frame>.pkl.gz                # XYZ + intensity + timestamp + ring
└── cuboids/<frame>.pkl.gz              # bottom-center xyz + dim + yaw

Conversion checklist:

• Pose trajectory: union of per-camera per-frame poses (6 × ~80 = ~480

waypoints) and per-LiDAR-frame poses, deduped + sorted, re-referenced to the first ego pose.

• LiDAR points are world-frame XYZ already ego-compensated → run

MotionCompensator.motiondecompensatepoints to recover sensor-frame per-ray XYZ, then normalise to direction. Per-ray µs from the timestamp column.

• Map raw Pandar64 ring → NCore model_element[:, 0] via a

ringid → rowindex permutation that sorts elevations descending (raw ring IDs are firing order, not beam index).

• Cuboids: bottom-center → geometric center (centroidz += dimz/2).

Yaw-only rotation → rot=(0, 0, yaw_rad). LabelSource.EXTERNAL.

• Cameras: global shutter (ShutterType.GLOBAL). frametimestampsus = [t, t].

NuScenes

No upstream converter — author with Path B. NuScenes ships nested JSON tables (sample, sampledata, egopose, calibratedsensor, sensor, sampleannotation, instance).

Sensor inventory:

• 6 cameras (CAMFRONT, CAMFRONTLEFT, CAMFRONTRIGHT, CAMBACK,

CAMBACKLEFT, CAMBACKRIGHT) — pinhole. Rolling shutter; consult the device datasheet for shutter direction (rename to lowercase NCore IDs).

• 1 LiDAR (LIDARTOP): Velodyne HDL-32E, spinningdirection="cw",

32 beams, 1800 columns at 0.2°. Use VLP/HDL-32 datasheet elevations (descending). rowazimuthoffsets_rad = zeros(32).

• 5 radars (RADARFRONT, RADARFRONTLEFT, RADARFRONT_RIGHT,

RADARBACKLEFT, RADARBACKRIGHT). Continental ARS 408. Optional — NuRec ignores radar.

Conversion checklist:

• A NuScenes "scene" → one V4 sequence. Iterate samples in the scene to

enumerate sample_data per sensor.

• Timestamps in sample_data.timestamp are µs already.
• ego_pose is rig→world (translation + quaternion). Build the dense

trajectory from the union of egopose entries across all sampledata (cameras at 12 Hz, LiDAR at 20 Hz, radars at 13 Hz → ~50 ms spacing).

• calibratedsensor.translation + .rotation is Tsensor_rig (named

T_calib in their docs — verify direction by transforming a sensor-frame test point and checking it lands in the expected rig position).

• LiDAR .pcd.bin files are already-ego-compensated sensor-frame XYZ + ring

index + intensity. Per-ray timestamps are not stored — synthesise from columnindex (azimuth bin) and the sweep duration (50 ms at 20 Hz): tray = sweepstart + (azimbin / 1800) * sweep_duration.

• sample_annotation: bottom-of-box origin → add size[2]/2 to z.

rotation quaternion → XYZ-Euler radians. instancetoken → trackid, categoryname → classid. LabelSource.EXTERNAL.

• Ego masks: NuScenes does not ship them. Generate via SAM2 / off-the-shelf

ego segmentation per camera — strongly recommended for NuRec quality.

Format recipes (non-AV / sensor-only)

These do not have a vehicle rig; pick a body-fixed origin and apply the same rig conventions (+X forward, +Y left, +Z up).

Mono camera (no depth, no LiDAR) → COLMAP track

You only have RGB images. Run COLMAP first to produce poses + sparse points, then feed COLMAP into the upstream converter:

colmap automatic_reconstructor \
    --workspace_path <SCENE> --image_path <SCENE>/images \
    --camera_model OPENCV --single_camera 1
bazel run //tools/data_converter/colmap:convert -- \
    --root-dir <SCENE> --output-dir <OUT> \
    colmap-v4 --include-3d-points --start-time-sec 0

Notes:

• COLMAP timestamps are synthetic (1 FPS by default; tune with

--start-time-sec and the FPS embedded in your image filenames).

• The COLMAP camera frame already matches NCore (+Z optical) — no rotation

required.

• SfM points become a PointCloudsComponent named sfm_points. Use this when

no LiDAR is available so NuRec has a sparse geometric prior.

• Ego masks (<imagebasename>mask.png next to images, or --masks-dir).

Stereo cameras

Two synchronised cameras at known baseline. No LiDAR.

• Rig origin: midpoint between the two camera optical centres (or the left

camera — pick one and stay consistent).

• Static extrinsics: Tleftrig, Trightrig from your stereo calibration

(OpenCV stereoCalibrate outputs R, T from right-to-left → invert/compose to get rig-relative).

• Trajectory: stereo-VIO (ORB-SLAM3, OpenVSLAM) or COLMAP run on left

images → propagate to rig with Tleftrig.

• No LiDAR component. Two options:

1. Skip LiDAR entirely (set lidarids = []). NuRec falls back to image-only reconstruction (lower quality, more views needed). 2. Compute disparity per stereo pair → 3-D point cloud per frame in the left-camera frame → write as a PointCloudsComponent per frame (analogous to COLMAP sfmpoints but dense). This gives NuRec a geometric prior without faking a spinning LiDAR.

• Ego masks: render or hand-paint a static mask of the rig body if visible in

the FOV.

Multi-stereo rig (surround stereo)

Multiple synchronised stereo pairs on one rig (e.g. NVIDIA Hyperion 8.1 surround stereo, AV1, custom inspection robots). Each pair feeds Foundation Stereo independently to produce dense depth around the platform.

• Encode each camera as its own CameraSensorComponent with a standard

Tcamerarig extrinsic — do not pre-rectify or fuse a pair into a single virtual sensor. NCore does not have a "stereo pair" component type; pairing is metadata, not structure.

• Declare the pairings on the sequence-level genericmetadata so

downstream tools (Foundation Stereo, NuRec aux-data) can discover them:

``json {"stereopairs": [ {"left": "camerafrontleft", "right": "camerafrontright"}, {"left": "camerarearleft", "right": "camerarearright"}, {"left": "camerasidelefta", "right": "camerasideleft_b"} ]} ``

• Pose trajectory must include every (camera × frame timestamp) sample —

surround stereo at 8 cameras × 100 frames yields 800 trajectory waypoints, more than dense enough for per-ray motion compensation. Do not subsample.

• Calibration: stereo intrinsics + extrinsics from cv2.stereoCalibrate give

you R, T from right-to-left. Compose with your chosen rig origin to get Tleftrig and Trightrig separately; do not store only the baseline.

• Scale is metric by construction (calibrated baseline). Skip scale refinement

(or run it as a sanity check only).

Mono + depth (RGB-D / learned depth)

Single RGB camera + per-frame depth (sensor: RealSense, Kinect, ZED depth, or learned mono-depth like Marigold / DepthAnythingV2 / MoGe-2).

RGB-D specifics:

• RealSense D4xx / L515: active IR stereo (D4xx) or LiDAR-class TOF (L515).

Depth and RGB are co-triggered but not pixel-aligned out of the box — use the rs2_align filter (or pre-aligned topics) before treating depth as RGB-frame metric. Intrinsics: read the colour stream's intrinsics for the RGB component; ignore the depth-stream intrinsics (depth is aligned into the colour frame).

• Microsoft Kinect Azure / Kinect v2: TOF depth with non-trivial

invalidation near object edges. Mask invalidated pixels (depth == 0) before encoding as a point cloud.

• Stereolabs ZED 2 / X: stereo with a built-in disparity engine. Either

store both raw left/right images and treat as a stereo rig (preferred — Foundation Stereo can re-derive depth at higher quality), or store the ZED-native depth as a per-frame PointCloudsComponent.

• Learned mono-depth (Marigold, DepthAnythingV2, MoGe-2) is **scale

ambiguous**. Either anchor with one absolute reference (a known object size, ground-plane height, IMU + visual-inertial scale) before storing, or accept that scale refinement (r2s module 6) will run downstream.

• Rig origin: camera optical centre (or device body if you have a static

IMU offset).

• Trajectory: ARKit/ARCore pose stream, IMU+camera VIO, or depth-aided

RGB-D SLAM (Open3D, Spectacular AI).

• No LiDAR — depth is not a LiDAR. Two valid encodings:

1. Per-frame PointCloudsComponent (preferred for learned/stereo-quality depth, where reliability is uneven). Convert depth + intrinsics → camera-frame XYZ, transform to world via Tcameraworld(t), store as a point cloud per frame. Carry the dense depth into generic_data if downstream consumers want it. 2. Synthetic spinning LiDAR (only if your depth is dense and reliable). Sample a fixed grid of azimuths/elevations, ray-cast against the depth map at each frame, and write a LidarSensorComponent. This is more work and less honest than option 1 — prefer point clouds unless NuRec specifically needs a LiDAR component.

• Camera intrinsics: pinhole or fisheye depending on the lens. Depth-camera

manufacturers ship calibration JSON — copy fx, fy, cx, cy and distortion coefficients verbatim.

• Frame timestamps: depth and RGB are usually co-triggered. Use the RGB

exposure timestamp; depth has no separate component.

Mono + LiDAR (handheld / robot)

Single camera + spinning or solid-state LiDAR, e.g. handheld scanner, ground robot, drone.

• Rig origin: choice driven by mechanical mounting. If the LiDAR is the

reference for ego-motion (LiDAR-inertial SLAM), set rig = LiDAR (so Tlidarrig = I). Otherwise pick the IMU body frame or camera centre.

• Trajectory: LIO-SAM, FAST-LIO, or any LiDAR-inertial pipeline.

Re-reference to the first frame; cast last (float64 → float32).

• Static extrinsics: Tcamerarig from camera-LiDAR calibration

(kalibr, lidar_align). Apply the NCore camera-frame rotation if your calibration target uses a different convention.

• LiDAR:

- Spinning (Velodyne / Ouster / Hesai / Robosense / Livox Mid-360 in repetitive mode): use RowOffsetStructuredSpinningLidarModelParameters with the sensor's real elevations (datasheet) and "cw". For Ouster populate rowazimuthoffsetsrad from beamazimuthangles. - Solid-state non-repetitive (Livox Avia, Mid-40, Mid-70 in non-repetitive mode): the spinning model does not fit. Either replay each scan as a fake "spinning" sweep (stash directions in columnazimuths_rad per-scan — fragile) or, preferred, write the points as a PointCloudsComponent per frame in the sensor frame and skip LidarSensorComponent. NuRec can consume point clouds.

• Cuboids: usually unavailable for non-AV — leave the component empty or

unregistered.

• Ego masks: render the rig body if it intrudes on the FOV (drone arm,

robot chassis); empty {} if not.

Solid-state / non-repetitive LiDAR (Livox)

Livox Avia / Mid-40 / Mid-70 — and Mid-360 in non-repetitive mode — produce a point cloud per scan that does not lay out on a row-major spinning grid. The RowOffsetStructuredSpinningLidarModel does not fit; forcing it (synthetic columns, fake row bins) breaks NCore validation and motion-compensation alignment.

Encoding rules:

• Preferred — write each scan as a per-frame PointCloudsComponent

instance in the sensor frame, with per-point µs timestamps. Skip LidarSensorComponent entirely. NuRec consumes point clouds and r2s depth refinement (module 7) treats them as it would LiDAR sweeps.

• Fallback for repetitive Mid-360 — the Mid-360 in repetitive mode does

produce a structured grid; treat it as a spinning LiDAR with the datasheet's beam table and spinning_direction="cw". This is the only Livox variant the spinning model fits.

Per-point timestamps: Livox custom messages carry offsettime (ns from sweep start). Convert to absolute µs: timestampus = sweepstartus + offsettimens // 1000.

IMU + camera (visual-inertial)

IMU is not stored as its own NCore component — it densifies the pose trajectory (and, optionally, anchors metric scale on monocular setups). Two paths:

1. VIO trajectory (preferred): run a visual-inertial pipeline (cuVSLAM

stereo-inertial, ORB-SLAM3, OpenVSLAM, OpenVINS, Spectacular AI) → IMU-rate (100–200 Hz) Tworldrig poses → store as the dynamic pose. The trajectory is already dense enough that per-ray motion compensation works without further densification.

1. IMU integration only (bootstrap / fallback): pre-integrate IMU

(linear accel + angular vel) over short windows, anchor each window with the next available camera- or LiDAR-rate pose. Useful for filling gaps or extrapolating to per-LiDAR sweep timestamps when the SLAM stack only emits poses at frame rate.

Either way, IMU sample timestamps go into the pose trajectory, not into a separate component. If a downstream consumer wants raw IMU, stash the samples in the sequence-level genericmetadata["imu_samples"] (compact) or in a sidecar file referenced from there.

Calibration: IMU-to-camera and IMU-to-LiDAR extrinsics live in the static pose graph as Timurig = I if you take the IMU body frame as the rig (common for legged robots and drones), with Tcamerarig / Tlidarrig from kalibr / lidaralign outputs. Pick rig = IMU when the IMU is the trajectory reference; otherwise pick the mechanical body frame and store Timu_rig for downstream tools that want IMU-frame data.

ROS2 bag (MCAP / SQLite3)

Most robotics datasets ship as ROS2 bags. The standard path is the rosbags Python library — no rclpy and no native ROS install required:

pip install rosbags pyav   # pyav decodes H.264 video chunks; rosbags handles .mcap and .db3

Convert in two passes: first enumerate sensors and collect calibration / the static TF tree, then stream frames into the V4 writer. Common topic → component mapping:

Notes:

• Rolling-shutter ROS2 cameras: Image.header.stamp is the frame trigger.

Compute [exposurestart, exposureend] from CameraInfo exposure metadata if present, otherwise read the device datasheet. Setting frametimestampsus = [trigger, trigger] is acceptable for well-synchronised global-shutter rigs, wrong for rolling-shutter unless the readout time is small enough to ignore.

• TF tree → rig frame: pick base_link (or whichever frame is configured

as the robot body) as rig. ROS REP 105 (x forward, y left, z up) matches NCore's rig convention exactly — no rotation needed.

• The rosbags API opens both MCAP and SQLite3 with the same code path;

only the file extension differs.

• For multi-bag datasets (one bag per take), one bag = one V4 sequence.
• Keep the bag's recording timestamps (not wall-clock playback) — bag

timestamps are µs-quantised already, perfect for NCore.

Aerial / drone

Drones add three failure modes on top of the standard mono+lidar / stereo recipes:

• Fast rotation: pose trajectory must densify enough to keep angular

drift per waypoint below ~0.5°. At 100 °/s yaw rate, that means waypoints every 5 ms — much tighter than a typical 10 Hz LiDAR. Use IMU samples (200 Hz+) as the trajectory backbone; do not rely on per-LiDAR poses alone.

• Rolling-shutter cameras are common on consumer drones. Set

ShutterType.ROLLINGTOPTOBOTTOM, supply real [exposurestart, exposureend] per frame, and carry the per-row shutter delay into the intrinsics if your model supports it (shutterdelay_us on OpenCVPinholeCameraModelParameters for FTheta-like models).

• Wide altitude / range: a 5+ km loop in raw UTM at the end of the

flight loses the same precision that the AV first-pose re-referencing fixes. For continuous takes under ~2 km, re-reference once to the takeoff pose; for longer flights, segment the bag and emit one V4 sequence per segment.

Rig origin on a drone: IMU body frame is the canonical choice. All extrinsics are then Tsensorimu. Rotor masks are usually unnecessary — rotors blur out of the FOV at flight RPM — but include a static mask of any fixed gimbal arm visible in the FOV.

Robotics pipeline shards (r2s)

The r2s ("robotics-to-sim") pipeline chains NCore through pose, depth, mask, and refinement steps before NuRec. Each step writes its own NCore shard so that any module can be swapped, re-run, or skipped without rebuilding the preceding stages. If you're authoring a converter that feeds r2s — or a downstream module that reads/writes intermediate shards — follow this naming + content contract.

Shard naming and contents

Component encoding until V4 ships first-class depth / mappoints

The r2s spec references two components that are not yet in upstream V4:

• DepthComponent (per-pixel uint16 depth, mm, per camera per keyframe).
• MapPointsComponent (sparse SfM points + per-frame observations).

Until they ship, the safe encodings are:

• Dense depth → per-frame PointCloudsComponent, instance name

depth<cameraid>, points in the camera frame at the keyframe timestamp. Carry the raw uint16 depth array in genericdata["depthmm"] if downstream tools need pixel-aligned access.

• SfM map → a single PointCloudsComponent instance named

sfmpoints (already produced by the upstream COLMAP converter). Per-frame observation tracks, when needed, go in genericmetadata["trackobservations"].

Treat these as forward-compatible encodings: when V4 adds the first-class components, the migration path is mechanical (copy the same arrays into the new writer) and the data shape is already correct.

Validator hooks

Each module's validator runs after its shard is written. The spec assertions that map onto NCore writer-time checks are already enforced (timestamp ordering, valid intrinsics, pose-trajectory completeness, strict-increasing LiDAR azimuths). The remaining checks (depth value range, mask temporal consistency, scale factor bounds, etc.) are the module's responsibility — see the r2s modules.md for the canonical list of per-step validations.

Validation & end-to-end NuRec

After every conversion, before assuming success:

1. ncore_vis — visualise the store. Wrong sensor extrinsics, mirrored

LiDAR (spinning_direction flip), or rotated cameras are obvious here.

1. ncoreprojectpctoimg — projects LiDAR onto camera frames. Crisp

alignment confirms Tlidarrig, Tcamerarig, intrinsics, per-ray timestamps, and pose-trajectory density are all correct simultaneously. Smearing or doubling = motion-comp error (per-ray timestamps wrong, or pose trajectory too sparse).

1. NuRec end-to-end. Hand the converted store to the

[nre](../nre/SKILL.md) sibling skill — Workflow A wires together aux-data generation (via nre-tools), 3DGUT training, USDZ export, and novel-view rendering, and references/configuration.md documents how to override dataset.cameraids / dataset.lidarids so the training Hydra recipe matches your sensor IDs. For repeatable cluster runs, wrap the convert + train + export sequence in an OSMO / Slurm / Kubernetes pipeline that pins both the NVIDIA/ncore and NRE container versions.

NuRec's "Ensure Data Quality" doc (in the NuRec image) also lists checklidarcamerasweepalignment and other validators you can run post-conversion.

Troubleshooting (common failure modes and the fix file)

The canonical fixes live in code (example_converter.py inline comments) and in the spec; most "NuRec produced garbage" complaints reduce to one of:

Limitations

• No first-class IMU component. V4 has no IMU sensor type — IMU

samples must ride in the pose trajectory or in genericmetadata (never as a sensor writer).

• Per-ray LiDAR is the only motion-comp-safe option for spinning

LiDARs. If the dataset only gives frame-level timestamps, expect motion-blur smearing in NuRec; reconstruct per-ray timestamps from azimuth or accept the artefact.

• Sub-cm scene detail at world scale requires float64 pose math

followed by re-referencing (poses = inv(poses[0]) @ poses) before writing as float32. Raw UTM/ECEF casts to float32 will silently lose centimetre-scale geometry.

• Rolling-shutter cameras need per-camera per-frame

[exposurestart, exposureend] and the correct ShutterType by enum name — a single global timestamp will produce ghosting.

• Solid-state non-repetitive LiDAR (Livox-style) must use

PointCloudsComponent, not LidarSensorComponent (the latter assumes a column grid).

• The converter is one-way. There is no v4 → original tool;

always keep the source dataset alongside.

• This skill is convert + validate only. Training, rendering, or

USDZ packaging is the nre skill's job.

Additional resources

• Install: pip install nvidia-ncore
• Source, docs, examples: <https://github.com/NVIDIA/ncore>;
• V4 spec: <https://nvidia.github.io/ncore/data/conventions.html>;
• Sensor models: <https://nvidia.github.io/ncore/data/sensor_models.html>;
• API reference: <https://nvidia.github.io/ncore/apis/data.v4.html>;
• Conversion guides (PAI, Waymo, COLMAP, ScanNet++):

<https://nvidia.github.io/ncore/conversions/index.html>;

• Skeleton converter (FILL IN walkthrough):

[ncoretemplate/impl/dataconverter/exampleconverter.py](ncoretemplate/impl/dataconverter/exampleconverter.py)

• NRE training / rendering / USDZ export from a converted store:

[../nre/SKILL.md](../nre/SKILL.md)

• HuggingFace dataset catalog and download recipes:

[../physical-ai-datasets/SKILL.md](../physical-ai-datasets/SKILL.md)

• Upstream NCore PAI / Waymo / COLMAP converter targets:

<https://github.com/NVIDIA/ncore/tree/main/tools/data_converter>;