Differential Fuzzer

Always load [Debugging skill for reference](../debugging/)

The differential fuzzer compares Turso results against SQLite for generated SQL statements to find correctness bugs.

Location

testing/differential-oracle/fuzzer/

Running the Fuzzer

Single Run

# Basic run (100 statements, random seed)
cargo run --bin differential_fuzzer

# With specific seed for reproducibility
cargo run --bin differential_fuzzer -- --seed 12345

# More statements with verbose output
cargo run --bin differential_fuzzer -- -n 1000 --verbose

# Keep database files after run (for debugging)
cargo run --bin differential_fuzzer -- --seed 12345 --keep-files

# All options
cargo run --bin differential_fuzzer -- \
  --seed <SEED>           # Deterministic seed
  -n <NUM>                # Number of statements (default: 100)
  -t <NUM>                # Number of tables (default: 2)
  -c <NUM>                # Columns per table (default: 5)
  --verbose               # Print each SQL statement
  --keep-files            # Persist .db files to disk

Continuous Fuzzing (Loop Mode)

# Run forever with random seeds
cargo run --bin differential_fuzzer -- loop

# Run 50 iterations
cargo run --bin differential_fuzzer -- loop 50

Docker Runner (CI/Production)

# Build and run from repo root
docker build -f testing/differential-oracle/fuzzer/docker-runner/Dockerfile -t fuzzer .
docker run -e GITHUB_TOKEN=xxx -e SLACK_WEBHOOK_URL=xxx fuzzer

Environment variables for docker-runner:

• TIMELIMITMINUTES - Total runtime (default: 1440 = 24h)
• PERRUNTIMEOUT_SECONDS - Per-run timeout (default: 1200 = 20min)
• NUM_STATEMENTS - Statements per run (default: 1000)
• LOGTOSTDOUT - Print fuzzer output (default: false)
• GITHUB_TOKEN - For auto-filing issues
• SLACKWEBHOOKURL - For notifications

Output Files

All output goes to simulator-output/ directory:

Reproducing Errors

Always follow these steps

1. Find the seed and profile in the error output:

`` INFO: Starting differentialfuzzer with config: SimConfig { seed: 12345, ..., weightprofile: Writes } ``

1. Re-run with that seed and profile (a seed only replays under the same profile):

``bash cargo run --bin differential_fuzzer -- --seed 12345 --profile writes --verbose --keep-files ``

1. Read the minimized reproduction first. On an oracle failure the fuzzer

writes these files to simulator-output/: - minimized.sql - a shrunken state script plus the shrunken failing statement, produced automatically. Start here. - turso-state.sql / sqlite-state.sql - each engine's full state as a replayable script, when you need more than the minimized version kept. - test.sql - every executed statement (the failing one is marked -- FAILED:). The minimizer falls back to replaying this history when the failure depends on how the state was built, not just its contents. - schema.json - table structure at failure time.

1. Probe the reproduction with differential_probe. It runs a

statement-per-line script on Turso and SQLite side by side, prints both outcomes for every statement, marks divergences, and compares the final table contents. Exit code 1 means something diverged. ``bash cargo run -q -p differential-fuzzer --bin differentialprobe -- \ simulator-output/minimized.sql ` Use it instead of piping SQL into the two shells: the tursodb shell cannot ATTACH ':memory:' AS aux, so fuzzer reproductions with an aux schema only run correctly through the probe. Reading from stdin also works: echo "SELECT ~X'96';" | cargo run -q -p differential-fuzzer --bin differentialprobe`.

1. Bisect by editing the script. Copy minimized.sql, simplify one thing

at a time (replace an expression with a constant, drop a column, drop a state line), and re-run the probe after each edit. The divergence marker tells you immediately whether the edit kept the bug. This loop usually ends at a one-line kernel you can hand to EXPLAIN on both engines.

1. Create a regression test in .sqltest (preferred) or .rs from the

kernel. Always load the [Debugging skill for reference](../debugging/).

Understanding Failures

Oracle Failure Types

1. Row set mismatch - Turso returned different rows than SQLite
2. Turso errored but SQLite succeeded - Turso rejected valid SQL
3. SQLite errored but Turso succeeded - Turso accepted invalid SQL
4. Schema mismatch - Tables/columns differ after DDL

Warning (non-fatal)

• Unordered LIMIT mismatch - LIMIT without ORDER BY may return different valid rows

Key Source Files

Tracing

Set RUST_LOG for more detailed output:

RUST_LOG=debug cargo run --bin differential_fuzzer -- --seed 12345