NV-Generate-VAE-Finetune

Purpose

• Used for finetuning the NV-Generate-CTMR MAISI VAE/autoencoder from user-supplied CT or MRI NIfTI training volumes.
• Not for clinical interpretation, regulatory use, or approving synthetic data for production training.
• Upstream currently documents VAE training in trainvaetutorial.ipynb and provides configs/helpers, but not a scripts.train_vae CLI. This skill does not execute the notebook; it stages the required config/datalist glue locally and uses upstream helper APIs.
• Manifest I/O: inputs are datalist and databasedir; outputs are autoencodercheckpoint, discriminatorcheckpoint, and result_json.
• The underlying training contract is the upstream config/env JSON (configmaisivaetrain.json + environmentmaisivaetrain.json, as used in trainvaetutorial.ipynb). The wrapper stages those JSON files for you and exposes the most-tuned fields as CLI flags; the sections below document the fields, their defaults, and how to monitor/tune a run.

Instructions

• Read skill_manifest.yaml before changing arguments, side effects, or validation gates.
• Run scripts/runvaefinetune.py from the Medical AI Skills repo root.
• If a host agent exposes runscript, use runscript("scripts/runvaefinetune.py", args=[...]); otherwise run the Bash/Python command below.
• Use --preflight first when checking a new datalist; remove --preflight only when the user explicitly wants to launch GPU finetuning.
• For a staged preflight input bundle directory, use BUNDLE/preflightdatalist.json as the datalist and BUNDLE/preflightdataset as --data-base-dir when those files are present.

Examples

Validate and stage a preflight finetune check from an input bundle (the recommended first step — no GPU, no training). This is the single canonical command; replace INPUTBUNDLE and OUTDIR with your paths:

export NV_GENERATE_ROOT="${NV_GENERATE_ROOT:-$HOME/.cache/nvidia-skills/upstreams/NV-Generate-CTMR-61c4ec7}" && \
python skills/nv-generate-vae-finetune/scripts/run_vae_finetune.py \
  INPUT_BUNDLE/preflight_datalist.json \
  --data-base-dir INPUT_BUNDLE/preflight_dataset \
  --output-dir OUT_DIR \
  --modality mri \
  --preflight

For real GPU finetuning and other variations, see [Usage](#2-usage-one-line-training) below.

Available Scripts

Prerequisites

• An explicit NVGENERATEROOT may point to the caller's local checkout and

must contain configs/configmaisivae_train.json, scripts/transforms.py, and scripts/utils.py. The result records its current commit.

• If NVGENERATEROOT is unset, the wrapper searches .workbench_data/upstreams/NV-Generate-CTMR.
• CUDAVISIBLEDEVICES is optional and can be used to select the GPU for real training.
• Runtime requirements: NVIDIA CUDA GPU for real training, Python packages from the upstream requirements.txt, lpips, and downloaded VAE weights unless using --train-from-scratch.
• Side effects: writes staged configs, checkpoints, TensorBoard logs, and run summaries under the caller-provided --output-dir; may write model caches under the upstream checkout, ~/.cache/huggingface/, and ~/.cache/torch/; may contact https://huggingface.co, https://github.com, and https://download.pytorch.org.
• The datalist is a MONAI-style JSON object with non-empty training[] and validation[] or testing[]. Each entry has an image path relative to --data-base-dir and optional class or modality of ct or mri.

When no local checkout is supplied, create the recommended pinned default checkout once:

if [ -z "${NV_GENERATE_ROOT:-}" ]; then
  export NV_GENERATE_COMMIT=61c4ec709b84cad468852243c48e250bec732074
  export NV_GENERATE_ROOT="$HOME/.cache/nvidia-skills/upstreams/NV-Generate-CTMR-61c4ec7"
  if [ ! -d "$NV_GENERATE_ROOT/.git" ]; then
    git clone https://github.com/NVIDIA-Medtech/NV-Generate-CTMR.git "$NV_GENERATE_ROOT"
    git -C "$NV_GENERATE_ROOT" checkout --detach "$NV_GENERATE_COMMIT"
  fi
fi

1. Config and environment JSON (adapt to your data)

The wrapper copies the upstream VAE config/env JSON from $NVGENERATEROOT/configs, rewrites the fields below, and writes the staged copies under OUT_DIR/workflow/configs/. You normally only set your datalist and data root; the listed CLI flags override individual fields when you need to.

Environment JSON (environmentmaisivae_train.json):

Training fields (configmaisivae_train.json):

--modality (ct or mri, default mri) fills the per-entry class for datalist items missing one. Validation/testing entries are required because the training loop runs a validation pass.

For an end-to-end reference including example data download, see the upstream tutorial trainvaetutorial.ipynb.

2. Usage (one-line training)

Preflight only:

export NV_GENERATE_ROOT="${NV_GENERATE_ROOT:-$HOME/.cache/nvidia-skills/upstreams/NV-Generate-CTMR-61c4ec7}" && \
python skills/nv-generate-vae-finetune/scripts/run_vae_finetune.py \
  PATH_TO_DATALIST.json \
  --data-base-dir PATH_TO_DATA_ROOT \
  --output-dir runs/nv_generate_vae_finetune_preflight \
  --preflight

Preflight bundle input:

export NV_GENERATE_ROOT="${NV_GENERATE_ROOT:-$HOME/.cache/nvidia-skills/upstreams/NV-Generate-CTMR-61c4ec7}" && \
python skills/nv-generate-vae-finetune/scripts/run_vae_finetune.py \
  PATH_TO_INPUT_BUNDLE/preflight_datalist.json \
  --data-base-dir PATH_TO_INPUT_BUNDLE/preflight_dataset \
  --output-dir runs/nv_generate_vae_finetune_preflight \
  --preflight

GPU finetuning:

export NV_GENERATE_ROOT="${NV_GENERATE_ROOT:-$HOME/.cache/nvidia-skills/upstreams/NV-Generate-CTMR-61c4ec7}" && \
python -m pip install -r "$NV_GENERATE_ROOT/requirements.txt" && \
python -m pip install lpips tensorboard && \
python skills/nv-generate-vae-finetune/scripts/run_vae_finetune.py \
  PATH_TO_DATALIST.json \
  --data-base-dir PATH_TO_DATA_ROOT \
  --output-dir runs/nv_generate_vae_finetune \
  --epochs 1 \
  --modality mri \
  --patch-size 64,64,64 \
  --download-model-data

Replace PATHTODATALIST.json and PATHTODATA_ROOT with the user's actual paths. Do not use the fixture datalist for real training; it is a preflight-only placeholder.

3. Monitor training (TensorBoard)

The runner writes TensorBoard scalars (per-iteration and per-epoch reconsloss, klloss, ploss, adversarial/real/fake losses, and a validation scalefactor) under OUT_DIR/artifacts/tfevent/autoencoder. Launch TensorBoard against the output directory:

python -m pip install tensorboard && \
tensorboard --logdir runs/nv_generate_vae_finetune/artifacts/tfevent

The same per-epoch loss history is also captured in OUTDIR/artifacts/workflowsummary.json and echoed in the JSON the wrapper prints to stdout (losshistory, best-checkpoint paths, exitcode, stderr_tail).

4. Hyperparameter tuning and common pitfalls

• Reconstructions blurry — raise --perceptual-weight (default 0.3); try --recon-loss l2 if edges look washed out.
• Posterior collapse / over-regularized latents — --kl-weight is intentionally tiny (1e-7); increasing it too much degrades reconstruction.
• Adversarial training unstable — lower --adv-weight (default 0.1) or --lr; a warmup schedule already ramps the LR over the first 20 epochs.
• Out-of-memory — reduce --patch-size (e.g. 48,48,48) and --val-sliding-window-patch-size, keep --batch-size 1, and lower --cache-rate.
• datalist must include non-empty validation[] or testing[] — the validation loop is mandatory; add validation[] (or testing[]) entries.
• Single-GPU only — the runner asserts exactly one CUDA GPU; set CUDAVISIBLEDEVICES to pick which one.

5. Evaluate the finetuned VAE

Validation reconstruction loss (lowest-valweightedloss epoch) is tracked automatically and the best autoencoder is saved as autoencoderepochN.pt under OUTDIR/artifacts/models. To evaluate downstream:

• Compare validation reconsloss/ploss curves across runs in TensorBoard, and
• Plug the finetuned autoencoder into a diffusion finetune/generation run (e.g. [nv-generate-mr-brain-finetune](../nv-generate-mr-brain-finetune/SKILL.md) via --trained-autoencoder-path) to confirm latents still decode to usable volumes.

This skill gates file accounting and reconstruction bookkeeping only — image quality and downstream utility must be judged by a domain expert.

Limitations

• Requires a current upstream NV-Generate-CTMR checkout with VAE configs and helper APIs. The skill owns the runner glue and does not depend on the notebook.
• Full training can be expensive and is not deterministic across hardware, CUDA, and package versions.
• The wrapper gates file accounting and command provenance, not anatomical realism, reconstruction quality, or downstream model utility.
• Not for clinical deployment, clinical interpretation, autonomous diagnosis, regulatory submission, or production training-data approval.

Troubleshooting