Visual ChangeNet

Visual ChangeNet is a TAO Toolkit model for visual inspection and defect detection. It supports two tasks:

• Classify — Binary image classification using a siamese-style architecture with a shared backbone (C-RADIO ViT) and a learnable difference module. Compares image pairs to classify defects as PASS/NO_PASS.
• Segment — Pixel-level change segmentation using a ViT-Large NVDINOv2 backbone. Compares before/after image pairs to produce a binary change mask.

The backbone weight (cradiov2vitbasepatch16224) is the nvidia/C-RADIOv2-B model from HuggingFace, distributed as model.safetensors (~393 MB). The TAO 7.0.0-rc container does not auto-fetch from HF URLs — ptmutils.loadpretrainedweights() hands the pretrainedbackbonepath value to torch.load(path) / safetensors.torch.loadfile(path) directly. Passing an https://huggingface.co/... URL or a repo id produces FileNotFoundError and the run fails with Execution status: FAIL within a few seconds. Stage the file locally before launch:

python3 -c "from huggingface_hub import hf_hub_download; import shutil; \
shutil.copy(hf_hub_download('nvidia/C-RADIOv2-B', 'model.safetensors'), '<workspace>/backbone/c_radio_v2_b.safetensors')"

Mount it into the container (-v <workspace>/backbone/cradiov2b.safetensors:/data/pretrainedmodels/C-RADIOv2B.safetensors) and set the spec model.backbone.pretrainedbackbonepath to the container path. HFTOKEN is only needed at staging time, not at training time.

Segment specs use model.backbone.type: vitlargenvdinov2 and the NVDINOv2 checkpoint family. Keep the checkpoint architecture aligned with the backbone type: NVDINOV251816256.ckpt is compatible with the packaged segment templates, but it must not be used with fansmall12p4hybrid. If you switch to a different segment backbone, use a matching checkpoint or leave model.backbone.pretrainedbackbonepath empty for default initialization.

Dataclass Schemas

Generated TAO Core schemas are packaged in schemas/<action>.schema.json, with schemas/manifest.json listing available actions. Each generated schema also emits references/spectemplate<action>.yaml from the schema top-level default field. AutoML enablement is declared at the model layer in references/skillinfo.yaml via automlenabled. Runnable AutoML still requires schemas/train.schema.json and references/spectemplatetrain.yaml to exist and parse. Use the packaged train schema for automldefaultparameters, automldisabledparameters, defaults, min/max bounds, enums, option weights, math conditions, dependencies, and popular parameters. Do not expect ~/tao-core at runtime; maintainers regenerate schemas/templates before packaging the skill bank.

Train Action Policy

This model is AutoML-enabled at the model layer. Before handling any train-stage request, read references/skillinfo.yaml and resolve the run override from either an explicit automlpolicy value or the user's workflow request. Use automlpolicy: on by default and only expose on / off in new launch prompts. Treat phrases like "turn off AutoML", "disable AutoML", "no HPO", or "plain training" as automlpolicy: off for this run only. When automlpolicy: on, automlenabled: true, and both schemas/train.schema.json and references/spectemplatetrain.yaml are packaged, route the train action through tao-skill-bank:tao-run-automl by default with this model's skilldir. Preserve workflow/application overrides for datasets, specs, output directories, GPU/platform settings, parent checkpoints, and automlpolicy. Use direct model training only when automl_policy: off or the packaged train schema/template is missing; in the missing-schema case, report that AutoML is enabled but not runnable for this model until schemas are generated.

Non-train actions declared by this model skill (evaluate, inference, export, quantize, segmentevaluate, and segmentinference) stay in this model skill. Do not present segmentexport or segmentquantize as runnable parent-skill actions until matching entries are packaged in schemas/manifest.json. Prune and retrain are not declared in the current parent references/skillinfo.yaml; do not present them as runnable parent-skill actions unless the metadata is extended with matching action wiring and schemas. The per-run automlpolicy override does not change model metadata.

For TAO Deploy TensorRT actions (gentrtengine, TensorRT evaluate, and TensorRT inference for classify and segment variants), read references/tao-deploy-visual-changenet.md first. Deploy spec templates live in this skill's references/ folder with the spectemplatedeploy*.yaml prefix. Deploy requires an exported ONNX artifact as parentmodel. If no ONNX artifact exists and the main skill does not expose an export action, report deploy as blocked instead of inventing an artifact.

Training Requirements

Visual ChangeNet has two separate task modes with different dataset types and data source structures.

Classify

• Dataset type: visualchangenetclassify
• Formats: default
• Accepted dataset intents: training, evaluation, testing, calibration
• Monitoring metric: val_loss

Per-Action Dataset Requirements (Classify)

The quantize and gentrtengine rows below describe TAO spec data requirements only. They are not parent-skill actions unless the corresponding action is declared in references/skillinfo.yaml or deploy/skillinfo.yaml.

Segment

• Dataset type: visualchangenetsegment
• Formats: default
• Accepted dataset intents: training, calibration
• Monitoring metric: val_loss

Segment uses a paired directory structure (A/, B/, list/, label/) instead of CSV + images. The root_dir spec key points to the top-level directory containing all four subdirectories.

Required files per dataset: A.tar.gz, B.tar.gz, list.tar.gz, label.tar.gz

Per-Action Dataset Requirements (Segment)

The quantize and gentrtengine rows below describe TAO spec data requirements only. They are not parent-skill actions unless the corresponding action is declared in references/skillinfo.yaml or deploy/skillinfo.yaml.

Typical Spec Overrides

Data source overrides are mandatory for every action — the agent MUST construct data source paths from the Per-Action Dataset Requirements table above and include them in spec_overrides.

S3_TRAIN = "s3://bucket/data/train"
S3_EVAL = "s3://bucket/data/eval"

train (classify, mandatory data sources):

{
    "train.num_epochs": 30,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "train.num_gpus": 1,
    "train.use_distributed_sampler": False,
    "train.sync_batchnorm": False,
    "dataset.classify.train_dataset.images_dir": f"{S3_TRAIN}/images.tar.gz",
    "dataset.classify.train_dataset.csv_path": f"{S3_TRAIN}/dataset.csv",
    "dataset.classify.validation_dataset.images_dir": f"{S3_EVAL}/images.tar.gz",
    "dataset.classify.validation_dataset.csv_path": f"{S3_EVAL}/dataset.csv",
}

train (segment, mandatory data sources):

{
    "train.num_epochs": 30,
    "train.checkpoint_interval": 10,
    "train.validation_interval": 10,
    "train.num_gpus": 1,
    "train.use_distributed_sampler": False,
    "train.sync_batchnorm": False,
    "dataset.segment.root_dir": f"{S3_TRAIN}",
}

export (classify):

{
    "export.input_height": 896,
    "export.input_width": 224,
}

export (segment):

{
    "export.input_height": 224,
    "export.input_width": 224,
}

quantize (classify, mandatory data sources):

{
    "dataset.classify.train_dataset.images_dir": f"{S3_TRAIN}/images.tar.gz",
    "dataset.classify.train_dataset.csv_path": f"{S3_TRAIN}/dataset.csv",
    "dataset.classify.validation_dataset.images_dir": f"{S3_EVAL}/images.tar.gz",
    "dataset.classify.validation_dataset.csv_path": f"{S3_EVAL}/dataset.csv",
    "dataset.classify.quant_calibration_dataset.images_dir": f"{S3_TRAIN}/images.tar.gz",
}

evaluate (classify, mandatory data sources):

{
    "dataset.classify.validation_dataset.images_dir": f"{S3_EVAL}/images.tar.gz",
    "dataset.classify.validation_dataset.csv_path": f"{S3_EVAL}/dataset.csv",
    "dataset.classify.test_dataset.images_dir": f"{S3_EVAL}/images.tar.gz",
    "dataset.classify.test_dataset.csv_path": f"{S3_EVAL}/dataset.csv",
}

inference (classify, mandatory data sources):

{
    "dataset.classify.infer_dataset.images_dir": f"{S3_EVAL}/images.tar.gz",
    "dataset.classify.infer_dataset.csv_path": f"{S3_EVAL}/dataset.csv",
}

gentrtengine (classify, mandatory data sources):

{
    "gen_trt_engine.tensorrt.calibration.cal_image_dir": [f"{S3_TRAIN}/images.tar.gz"],
}

quantize (segment, mandatory data sources):

{
    "dataset.segment.root_dir": f"{S3_TRAIN}",
    "dataset.segment.quant_calibration_dataset.images_dir": f"{S3_TRAIN}",
}

evaluate (segment, mandatory data sources):

{
    "dataset.segment.root_dir": f"{S3_TRAIN}",
}

inference (segment, mandatory data sources):

{
    "dataset.segment.root_dir": f"{S3_TRAIN}",
}

gentrtengine (segment, mandatory data sources):

{
    "dataset.segment.root_dir": f"{S3_TRAIN}",
    "gen_trt_engine.tensorrt.calibration.cal_image_dir": [f"{S3_TRAIN}/images.tar.gz"],
}

Optional: running via the TAO SDK

When running without the TAO SDK (local docker), resolve the TAO pyt image from versions.yaml and invoke visualchangenet <train|evaluate|inference|export|quantize> directly. --shm-size=8g is required, the C-RADIO .safetensors must be mounted to /data/pretrainedmodels/C-RADIOv2B.safetensors, and checkpoint/resultsdir can be overridden on the command line. See references/local-docker.md for the full docker run command, mounts, and overrides.

Tasks

Classify (default)

Uses actions: train, evaluate, inference. Defaults template: references/spectemplatetrain.yaml.

Segment

Uses skill action names segmenttrain, segmentevaluate, and segmentinference. When invoking local Docker directly, run TAO CLI subcommands train, evaluate, and inference with task: segment in the spec. The schema-driven action templates are references/spectemplatesegmenttrain.yaml, references/spectemplatesegmentevaluate.yaml, and references/spectemplatesegmentinference.yaml; the compact direct-Docker example template is references/spectemplatesegment.yaml.

Segmentation requires compiling custom CUDA ops (MultiScaleDeformableAttention) on first run, which takes ~5 minutes. The ViT adapter backbone uses these for multi-scale feature extraction.

Dataset structure for segmentation differs from classify — uses paired directories (A/, B/, list/, label/) instead of CSV files. See dataset.segment.root_dir in the defaults.

Data Format

Classify needs a 4-column CSV (inputpath,goldenpath,label,objectname) plus an images directory; segment uses a paired directory structure (A/, B/, list/, label/) under dataset.segment.rootdir instead of CSV. The imageext field (default .jpg) must match the actual file extensions; if images are .png, set dataset.classify.imageext: .png. Multi-lighting input is configured via dataset.classify.inputmap (each lighting name maps to a channel index) with dataset.classify.numinput set to match. See references/data-formats.md for the per-field input tables (classify train/eval/inference, segment), CSV column semantics, lighting/path-concatenation conventions, the segment directory layout, and inputmap/gridmap examples.

Important Parameters

Key knobs include train.validationinterval (default 50, must be ≤ numepochs), train.checkpointinterval (default 200, must be ≤ numepochs), train.numepochs (default 100), model.classify.evalmargin (default 0.3, the precision/recall threshold), model.classify.trainmargineuclid (default 2.0), model.classify.embeddingvectors (default 5), dataset.classify.batchsize (default 16, must be > 1), dataset.classify.fpratiosampling (default 0.25), and train.classify.clsweight (default [1.0, 10.0]). Hardware: minimum 1 GPU with 16GB+ VRAM, recommended 8 GPUs (DDP); do not set gpuspeckey (GPU count is managed internally by TAO), num_nodes (default 1) controls multi-node. See references/tuning-parameters.md for the full per-parameter guidance and hardware detail.

Error Patterns

For checkpoint-not-found, CSV format mismatch, image extension mismatch, OOM, low evaluation accuracy, the contrastive-loss AssertionError, checkpoint load key mismatch at evaluate/inference, non-convergence, segment-only backbone dimension mismatch, the MultiScaleDeformableAttention OSError, the Lightning MisconfigurationException, ModuleNotFoundError: nvidiataopytorch, and epoch defaults, see references/troubleshooting.md for the full symptom-and-fix list.

Spec Param / Parent Model Inference

Model-specific parent-model mappings are declared in references/skillinfo.yaml under specparams, so generated runners and agents resolve checkpoints before createjob() instead of guessing file names. For parentmodel or parentmodelfolder, pass the upstream train/export/AutoML child job id as parentjobid; the SDK lists the parent result folder, filters checkpoint artifacts, and returns the selected model file or folder. See references/parent-model-inference.md for the full per-action spec-field-to-inference-function mapping table.

Deployment

• [tao-deploy-visual-changenet](references/tao-deploy-visual-changenet.md)