Incremental rebuilds silently use STALE CUTLASS fused-MHA kernels

The general false-green principles (stale binary, wrong-artifact mtime) are summarised
in the ort-test skill's "False-green taxonomy". This skill is the CUDA/CUTLASS-specific
detail.

The gotcha (verification-integrity bug)

nvcc-generated depfiles do not track the CUTLASS fused-MHA headers under onnxruntime/contribops/cuda/bert/cutlassfmha/ (e.g. kernelforward.h, fmhalaunchtemplate.h). These headers are #included by the fmhasm*.cu translation units, but the build system does not record that dependency.

Consequence: after you edit one of those headers, an incremental build.sh:

• does not recompile fmha_sm*.cu,
• reports [100%] Built target ... and exits 0,
• leaves the recompiled artifacts — the fmha_sm*.cu.o objects and the

libonnxruntimeproviderscuda.so they link into — unchanged (same mtime as the pre-edit build).

(Do not use the gtest test-exe mtime as the stale symptom: in the shared-provider build the exe dlopens the .so and is not relinked, so its mtime stays old even after a correct rebuild — see "How to confirm" below. The reliable diagnostic signal is the fmha_sm*.cu.o / .so mtime.)

So your "successful" rebuild is running the old kernel. Tests that should now pass (or fail) reflect the previous code, not your edit. This silently invalidates any FAIL→PASS / PASS→FAIL verification.

The fix — force recompile the .cu units

Before rebuilding after editing any cutlass_fmha/*.h header:

touch onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/*.cu

Then run the normal build command. This forces the fmha_sm*.cu translation units (and downstream binaries) to recompile against your header change.

How to confirm the rebuild was real (don't trust "[100%] Built")

Confirm that the artifact which actually links the recompiled fmha_sm*.cu.o is newer than your header edit.

⚠️ Do NOT just check the test EXE mtime — it can falsely flag a good build as stale. In the shared-provider build configuration (the default here), the CUDA execution provider is a shared module: the recompiled fmhasm.cu.o link into libonnxruntimeproviderscuda.so, and the onnxruntimeprovider_test executable dlopens that .so — it is not relinked. So after a correct* rebuild the test exe mtime stays old while the .so advances. Checking the exe alone would wrongly conclude the build was stale.

Check the right artifact for your link mode:

• Shared-provider build (default): the .so that links the recompiled .o —

build/<dir>/<cfg>/libonnxruntimeproviderscuda.so

• Statically-linked provider: the test exe itself (onnxruntimeprovidertest)

Safest check — stat both the recompiled object and the .so, and confirm BOTH are newer than the header edit:

stat -c '%y %n' onnxruntime/contrib_ops/cuda/bert/cutlass_fmha/kernel_forward.h
# in your build dir, e.g. build/Debug_quickbuild/Debug/:
stat -c '%y %n' libonnxruntime_providers_cuda.so
# and the actual recompiled object (path varies by build dir):
find . -name 'fmha_sm80.cu.o' -exec stat -c '%y %n' {} +

If the .so (and the fmha_sm*.cu.o) timestamps are older than (or equal to) the header edit, the build was stale — touch the .cu files and rebuild. The most reliable signal of all is behavioral: a test that was failing now passes (a stale binary cannot flip its result).

Related: pick the right test binary

This is the CUDA/CUTLASS instance of false-green mode 1 (zero-match / wrong binary) — see the ort-test skill's "False-green taxonomy" for the general principle. In short: attention/MEA/Flash boundary gtests (e.g. FlashStructuralEmptyRows, AttentionCausalNonPadKVSeqLenMEA) live in onnxruntimeprovidertest, which CI runs; onnxruntimetestall does not contain them and gives a false green. Verify the MEA/Flash boundary fix against onnxruntimeprovidertest.

Related: disk frugality on shared GPU dev boxes

Full ORT CUDA builds are large (test binaries ~1 GB each; a build dir can reach tens of GB). On a shared box, /home filling to 100% makes builds fail in non-obvious places — e.g. git submodule sync reporting No space left on device or a config.lock error, not an obvious "disk full" at the compile step.

Before a big rebuild, check free space and clean only clearly-stale, regenerable build directories (old dated experiment dirs). Never delete another agent's active build dir or anything ambiguous:

df -h /home
du -sh build/* | sort -h