Rerun parquet ingestion

ParquetReader is a pure reader: it maps a flat table onto the Rerun model by turning raw columns into grouped, time-indexed chunks of struct and scalar components. Column-name prefixes become entities, grouped columns become a single struct component, designated columns become timelines. The reader does not assemble archetypes anymore — mapping struct fields into typed Rerun components (Transform3D, Scalars, Points3D) is done with lenses on the reader's .stream(). The whole reader job is configuration; fill in the rerun-data-model mapping table first, then express it through stream(). Stream mechanics after .stream() are in rerun-chunk-processing.

The whole table is configuration, not code. If you find yourself building Chunk.from_columns from a parquet, or munging it in pandas first, stop — ParquetReader plus a lens almost certainly expresses it. Anything the reader cannot express (per-row entity routing, derived values, unit conversion) belongs in lenses downstream, not in pre-pandas munging; keep the pipeline columnar.

The API

from rerun.chunk import DeriveLens, IndexColumn
from rerun.experimental import ParquetReader

reader = ParquetReader(table_path)  # a lightweight handle over the file
stream = reader.stream(
    entity_path_prefix="/world",  # prepended to every entity path
    column_grouping="prefix",  # "prefix" | "individual" | "explicit_prefixes"
    delimiter="_",  # split for column_grouping="prefix"
    prefixes=None,  # required for "explicit_prefixes"
    use_structs=True,  # pack grouped columns into one struct component
    static_columns=["robot_type"],  # constant-per-file values, logged static
    index_columns=[IndexColumn.timestamp("timestamp", input_unit="us"), IndexColumn.sequence("frame_index")],
)

The constructor takes only path; all loading options live on stream(), and every one is keyword-only. Each stream() call is independent, so one reader can drive several differently-configured streams over the same file. The config is validated against the file's schema at stream() (missing index/static columns raise ValueError there, not mid-iteration). There is no column_rules kwarg — typed-component assembly moved to lenses (below).

What the reader emits

The reader turns the table into chunks, one chunk per group, then leaves the data as generic struct/scalar components for lenses to map. The naming is the key thing the rest of the pipeline keys off:

• A grouped multi-column prefix X → entity /X, with a single struct

component named data. The struct's fields are the column names with the prefix (and delimiter, for "prefix") stripped. So Aposx, Aquatw under prefix A land as struct data with fields posx, quatw on entity /A.

• A lone column with no group → its own entity named after the column, and

a raw component named after the column — not a data struct. So a speed column becomes entity /speed, component speed.

• A /__properties metadata chunk built from the parquet file's schema

metadata. You typically drop it right after .stream():

``python stream = reader.stream().drop(content="/__properties/**") ``

Column grouping: which columns share an entity

• "prefix" (default): split each column name on delimiter, group by the

first segment. gripperposx, gripperposy → entity /gripper, struct data{posx, posy}.

• "explicit_prefixes": group by the exact strings in prefixes, tried

longest-first; the prefix is stripped from each struct field name (a raw string match, no delimiter — foo + a → field a). Columns matching no prefix become individual groups. Use this when names contain the delimiter ambiguously (observation.state vs observation.images.top: pass the full prefixes).

• "individual": every column is its own chunk/entity with a raw component

named after the column — no struct packing at all, even for columns sharing a prefix. use_structs is ignored here. Rarely the model you want; reach for it only as a debugging baseline.

usestructs=True (default) packs a group's columns into a single Arrow struct component (the data field) for "prefix"/"explicitprefixes"; False emits one component per column (the pre-struct flat layout, what queries see as separate columns).

Timelines: index_columns

Each entry is built with IndexColumn: IndexColumn.timestamp(name, inputunit=...) (since epoch), IndexColumn.duration(name, inputunit=...) (elapsed), or IndexColumn.sequence(name) (ordinal int).

• input_unit describes what the raw integers in the column are ("ns" default,

"us", "ms", "s"); Rerun rescales to ns internally. Sequences take no unit.

If omitted, a synthetic row_index sequence timeline is generated. That is almost never the timeline you want to query or align against; always name the real time columns. Stamp both a timestamp and a sequence timeline when the table has both (multi-rate alignment, see rerun-data-model).

Static columns: static_columns

Listed columns are constant across all rows; they are emitted once as a single static (timeless) chunk, separate from the temporal data. A listed column that actually varies raises an error when the stream runs — that error is a data-quality signal, not a reason to drop the static declaration.

Typed components via lenses

The reader's grouped output is generic struct (data) and scalar data. A DeriveLens reads that struct's fields, packs and casts them into real Rerun components, and writes them to an output entity — this is what the old column_rules API used to do, now done downstream on the stream.

Construct a lens against the reader's struct component ("data" for grouped prefixes, or the column name for a lone/individual column), then add one or more .to_* builder methods. Each builder returns a fresh lens, so they chain.

topackedcomponent packs the named struct fields (in order, at least one required) into the fixed-size list the component expects, and by default auto-casts f64→f32 to match component types. The totranslation, toquaternion, toscale helpers are convenience wrappers over it, so they all auto-cast. torotationaxisangle builds a Struct{axis, angle} and hard-casts axis and angle to f32 internally. to_scalars with a single field emits a plain scalar per row (not a 1-element list); with multiple fields it emits one scalar series per field at the same entity.

Apply lenses with .stream().lenses([lens], content="/A", outputmode="dropunmatched"):

• content is a pre-filter on the source entity path — it scopes which

chunks the lens may touch. Out-of-scope chunks pass through unchanged. Set it to the reader's grouped entity (e.g. "/A").

• output_mode decides the fate of in-scope-but-unmatched chunks:

"dropunmatched" (default, keep only lens output), "forwardunmatched" (output replaces matched, other originals survive), or "forward_all" (output plus all originals).

• The lens's own output_entity= sets the destination entity — independent

of content, which gates the input side.

End-to-end Transform3D example. The reader groups A* columns into a data struct at /A; the lens reads the prefix-stripped field names (posx, quat_w), packs and casts them, and writes a full Transform3D to /pose:

from rerun.chunk import DeriveLens, IndexColumn
from rerun.experimental import ParquetReader

lens = (
    DeriveLens("data", output_entity="/pose")
    .to_translation("pos_x", "pos_y", "pos_z")
    .to_quaternion("quat_x", "quat_y", "quat_z", "quat_w")
)

chunks = (
    ParquetReader(table_path)
    .stream(index_columns=[IndexColumn.sequence("frame_index")])
    .lenses([lens], content="/A", output_mode="drop_unmatched")
    .to_chunks()
)

Chaining several .to* on one lens with a shared outputentity accumulates multiple component columns into the same archetype at that entity — above, both Transform3D:translation and Transform3D:quaternion land on /pose, forming a complete Transform3D. For a generic fixed-size-list component, pass the descriptor to topackedcomponent:

import rerun as rr
from rerun.chunk import DeriveLens
from rerun.experimental import ParquetReader

lens = DeriveLens("data", output_entity="/points").to_packed_component(
    rr.Points3D.descriptor_positions(), "x", "y", "z"
)

Selectors

Lens field paths use Selector, a jq-like grammar over Arrow columns (.field to access a struct field, [] to iterate a list, [N] to index, ? to suppress errors on absent fields, ! to assert non-null, | to pipe, and pack(.x, .y, .z) to zip paths into a fixed-size list). The to* helpers build these selectors for you; reach for tocomponent(component, Selector(".x")) when you need a custom field path. Field paths reference the prefix-stripped struct field names — the lens sees posx, not Apos_x.

Gotchas

1. No indexcolumns → synthetic rowindex timeline only. Queries that

expect a timestamp timeline find nothing.

1. The unit is the raw column's unit, not a desired output unit; a

microsecond column declared "ns" lands 1000x in the past.

1. static_columns raises if a listed column actually varies; that error is a

data-quality signal, not a reason to drop the static declaration. Uniformity is data-dependent, so it is raised lazily when the stream runs; a listed column that doesn't exist raises eagerly at stream().

1. A grouped prefix's struct component is named data — that is the

input_component string a DeriveLens matches against. A lone or "individual" column is instead a raw component named after the column.

1. Selector field paths reference the prefix-stripped struct field names

(posx, not gripperpos_x).

1. Drop the /__properties metadata chunk the reader emits from parquet schema

metadata: .stream().drop(content="/__properties/**").

1. Quaternion column order is x, y, z, w in to_quaternion; check the source's

convention before wiring fields.

1. topackedcomponent (and the transform helpers built on it) auto-casts

f64→f32 to match component types; this is usually what you want for parquet's double columns.

1. Anything the reader cannot express (per-row entity routing, derived values,

unit conversion) belongs in lenses downstream, not in pre-pandas munging; keep the pipeline columnar (rerun-chunk-processing).

References

• Lens builder source with full docstrings: rerun/experimental/_lens.py in

the installed rerun-sdk package (totranslation, toquaternion, toscale, torotationaxisangle, toscalars, topackedcomponent, tocomponent, to_timeline).

• Reader source: rerun/experimental/parquetreader.py, or

python -c "from rerun.experimental import ParquetReader; help(ParquetReader)"

• Canonical worked examples: the integration tests

rerunpy/tests/integration/testparquetreader.py (grouping, index/static columns, and the Transform3D / Points3D / Scalars lens flows) and rerunpy/tests/integration/testlazychunkstream.py (lens application, content/outputmode, selectors).

• rerun-lerobot — LeRobot datasets store episodes as parquet; that skill

covers the built-in importer route vs reading the parquet directly with this reader.

• rerun-data-model (mapping decisions), rerun-chunk-processing (stream

mechanics after .stream())