Image Grounding Pipeline

Turn (image, caption) pairs into per-image grounded annotations: cleaned captions, referring expressions with character spans, and pixel-space bounding boxes for each expression. A single VLM (Gemini or any OpenAI-compatible endpoint) handles both steps.

Purpose

Generate phrase-grounded training data for referring-expression and grounding models. The VLM acts as a "teacher" annotator: Step 0 extracts referring expressions from the caption while looking at the image; Step 1 returns one bbox set per expression for each image.

Pipeline Architecture

Step 0: Expression extraction  → VLM cleans caption, extracts referring expressions + char spans
Step 1: Phrase grounding       → VLM returns pixel bboxes + scores per expression

Steps are individually selectable via workflow.steps. Each step writes a per-sample checkpoint to step<N>*/.ckpt/<sampleid>.json and skips already-processed records on re-run. Set workflow.forcereprocess: true to ignore checkpoints and reprocess from scratch.

Instructions

Initial setup

When a user wants to run this pipeline, walk through these steps:

1. Input JSONL: Ask for the JSONL path. Each line must be one object like {"imagepath": "...", "caption": "..."}. imagepath can be absolute or relative.
2. Image root: If any imagepath values are relative, set data.imageroot to the directory they should resolve from.
3. API access: Ask the user which VLM endpoint they want to use. Present these five options and act on the choice:

1. Gemini — set vlm.backend: "gemini"; require GOOGLEAPIKEY (env var or vlm.gemini.apikey). 2. NIM (e.g. https://inference-api.nvidia.com/v1) — set vlm.backend: "openai"; collect baseurl, modelname, and apikey. 3. TAO inference microservice (self-hosted, OpenAI-compatible). Confirm whether the server is already running: - Running — collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". - Not running — guide the user through the skills/applications/tao-run-inference-service skill, which stands up a local TAO inference microservice with an OpenAI-compatible API. Before promising a specific model, check skills/applications/tao-run-inference-service/references/service.yaml for validnetworkarchconfigbasenames. Once the server is up, collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". 4. vLLM (self-hosted, OpenAI-compatible). Confirm whether the server is already running: - Running — collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". - Not running — follow [references/vllmserver.md](references/vllmserver.md) to install and launch a vLLM server, then collect baseurl, modelname, and (optionally) apikey; set vlm.backend: "openai". 5. Custom (any other OpenAI-compatible endpoint) — set vlm.backend: "openai"; collect baseurl, modelname, and (optionally) api_key.

If the user has no endpoint and does not want to set one up, stop and help resolve API access first.

1. Workflow steps: Choose one of:

- Full pipeline: ["0", "1"] - Expression extraction only: ["0"] - Grounding only: ["1"], which requires existing step-0 output at resultsdir/step0expressionextraction/annotations.jsonl

1. Resume vs fresh run: By default, the workflow reuses checkpoints and skips completed records. To reprocess everything, set imagegrounding.workflow.forcereprocess=true.

Running the pipeline

The pipeline runs inside the TAO Toolkit container via the auto_label CLI:

auto_label generate -e /path/to/spec.yaml \
    results_dir=/results \
    image_grounding.data.input_jsonl=/data/captions.jsonl \
    image_grounding.data.image_root=/data/images \
    image_grounding.vlm.gemini.api_key=$GOOGLE_API_KEY

Generate a default spec: autolabel defaultspecs resultsdir=/results modulename=autolabel, then set autolabeltype: "image_grounding". All fields support Hydra dot-notation overrides on the command line.

See [references/configuration.md](references/configuration.md) for the full YAML structure, all parameters, model/endpoint setup, and error patterns.

Recommended pilot workflow

1. Run on 5-10 images with both steps
2. Inspect step0expressionextraction/annotations.jsonl — are cleanedcaption and expressions[] accurate? Are the right noun phrases captured?
3. Inspect step1grounding/annotations.jsonl — do the bboxes in expressions[].instances[] look right? Are confidence scores reasonable?
4. If quality is insufficient, switch the VLM to a stronger model (e.g. gemini-2.5-pro) or raise mediaresolution/maxoutputtokens, then re-run with forcereprocess=true.
5. Scale to the full dataset once satisfied.

Configuration

Key configuration fields (full reference in [references/configuration.md](references/configuration.md)):

Inputs

A single JSONL file at data.input_jsonl. One JSON object per line:

Outputs

All outputs go to results_dir/:

• step0expressionextraction/annotations.jsonl — per-record output enriched with cleanedcaption and expressions[] (each with text, expressionid, charspan, noun_chunk, empty instances[]).
• step1grounding/annotations.jsonl — same records with expressions[].instances[] filled in (each instance has bbox: [x1,y1,x2,y2] in pixel space, score in [0.0, 1.0], and bbox_id).
• results_dir/annotations.jsonl — copy of the last step's output for convenience.
• step<N>*/.ckpt/<sample_id>.json — per-sample checkpoints used for resume.

Prerequisites

• Container: nvcr.io/nvidia/tao/tao-toolkit:6.26.3-pyt
• API access: At least one VLM endpoint (Gemini API key or OpenAI-compatible endpoint capable of image input)