GPU-Only and Quality-First Processing Principles (Notebooks)

Experiment Overview

Scope: Notebooks Only

IMPORTANT: This skill applies to interactive notebook processing only, NOT to SLURM batch processing.

For SLURM processing, see the slurm-concurrent-processing skill which describes how to maximize throughput using both GPU and CPU resources concurrently.

Context

During performance optimization of Notebook 2 (Cycle Processing), a "fast mode" was proposed that would reduce BaSiC iteration parameters to speed up processing. The user explicitly rejected this approach, establishing core principles for KINTSUGI notebook processing.

Scientific imaging requires quality-first processing. Unlike consumer applications where "good enough" may be acceptable, multiplex immunofluorescence analysis depends on accurate quantification. Quality degradation compounds through the pipeline: illumination correction errors affect stitching, which affects deconvolution, which affects segmentation, which affects all downstream analysis.

Core Principles

1. NEVER Sacrifice Quality for Speed

Quality parameters must remain at their scientifically-validated defaults unless the quality impact is negligible (verified, not assumed).

# CORRECT: Quality parameters (do not reduce)
BASIC_IF_DARKFIELD = True
BASIC_MAX_ITERATIONS = 500
BASIC_OPTIMIZATION_TOLERANCE = 1e-6
BASIC_MAX_REWEIGHT_ITERATIONS = 25
BASIC_REWEIGHT_TOLERANCE = 1e-3

2. ALWAYS Use GPU When Available - No CPU Fallback (Notebooks Only)

In notebook/interactive mode: If a GPU is available, it must be used. CPU fallback options should be disabled.

# CORRECT for NOTEBOOKS: GPU enforcement
if not USE_GPU:
    raise RuntimeError(
        "GPU not available but required for processing.\n"
        "Check GPU status with: from kintsugi.gpu import get_gpu_manager; "
        "print(get_gpu_manager().summary())"
    )
use_gpu = True  # Always True - GPU required in notebooks

In SLURM mode: CPU processing is allowed and encouraged to maximize throughput. See slurm-concurrent-processing skill.

3. Remove CPU Options in Notebook Code

In notebook code, don't provide usecpu or usegpu=False options. If the system has a GPU, use it.

# WRONG for notebooks: Providing CPU option
def process(use_gpu=True):  # Allows use_gpu=False
    ...

# CORRECT for notebooks: GPU-only
def process(device_id=None):  # GPU assumed, only device selection
    if device_id is None:
        device_id = GPU_DEVICE_IDS[0]
    ...

Note: SLURM job scripts DO support CPU mode via KINTSUGIDEVICEMODE environment variable - this is intentional for concurrent GPU+CPU processing.

Failed Attempts (Critical)

Acceptable Optimizations

These optimizations improve speed WITHOUT sacrificing quality:

Key Insights

• Quality is non-negotiable - Scientific imaging requires accurate quantification
• Speed comes from better hardware, not shortcuts - Invest in GPUs, not reduced iterations
• Errors compound - A 5% error in illumination correction becomes 10%+ by segmentation
• "Fast mode for testing" is a trap - Test with production parameters or you'll miss production issues
• Notebook GPU enforcement - In notebooks, if no GPU, the user should know immediately
• SLURM is different - Headless batch processing uses concurrent GPU+CPU to maximize throughput (see slurm-concurrent-processing skill)

Implementation Pattern

# GPU enforcement at module level
if not USE_GPU:
    raise RuntimeError("GPU required for KINTSUGI processing")

# Function signatures - no CPU options
def process_zplane(
    ...,
    device_id: int = None,  # Which GPU, not whether to use GPU
):
    """GPU is REQUIRED - no CPU fallback."""
    if device_id is None:
        device_id = GPU_DEVICE_IDS[0]

    # Use validated quality parameters
    corrector = KCorrectGPU(use_gpu=True, device_id=device_id)
    flatfield, darkfield = corrector.fit(
        images,
        max_iterations=500,           # Quality parameter - DO NOT REDUCE
        max_reweight_iterations=25,   # Quality parameter - DO NOT REDUCE
        optimization_tolerance=1e-6,  # Quality parameter - DO NOT REDUCE
    )

References

• KINTSUGI Notebook 2: Cycle Processing
• BaSiC paper: Peng et al., Nature Communications 2017
• Skills Registry: basic-caching-evaluation (another quality-compromising approach that failed)