Benchmark Runner

Standardizes performance comparison methodology: metric selection, test case design, environment capture, result formatting, and tradeoff analysis. Produces reproducible benchmark reports that support informed decisions — not just "A is faster than B" but "A is faster for small inputs while B scales better."

Reference Files

Prerequisites

• Clear candidates to compare (at least 2)
• Access to run or observe the candidates (code, API, or existing results)
• Representative workload definition

Workflow

Phase 1: Define Scope

1. What are the candidates? — Name each candidate precisely, including version.

"Python dict vs Redis" is too vague. "Python 3.12 dict (in-process) vs Redis 7.2 (localhost, TCP)" is testable.

1. What claims need validation? — "A is faster" → faster at what? For what input

size? Under what load? Benchmark design flows from the specific claim.

1. What is the decision context? — Why does this comparison matter? This determines

which metrics are most important.

Phase 2: Select Metrics

Choose metrics that match the decision context:

Select 2-4 metrics. More than 4 makes comparison tables unreadable.

Phase 3: Design Test Cases

Create a matrix of inputs that reveal performance characteristics:

1. Scale variation — Small, medium, large inputs. Performance often changes

non-linearly with scale.

1. Representative data — Use realistic inputs, not synthetic best-case data.
2. Edge cases — Empty input, maximum size, adversarial input.
3. Warmup — Exclude JIT compilation, cache warming, and connection establishment

from measurements. Run N warmup iterations before recording.

Phase 4: Specify Environment

Record everything needed to reproduce the results:

1. Hardware — CPU model, core count, RAM size, GPU model (if applicable)
2. Software — OS version, language runtime version, dependency versions
3. Configuration — Thread count, batch size, connection pool size, cache settings
4. Isolation — What else was running? Background processes affect results.

Phase 5: Structure Results

Produce comparison tables with clear winners per metric, followed by tradeoff analysis.

Output Format

````text

Benchmark: {Descriptive Title}

Date: {YYYY-MM-DD} Hardware: {CPU}, {RAM}, {GPU if applicable} Software: {runtime versions} Configuration: {key settings that affect results}

Candidates

Test Cases

Results

Latency (ms, lower is better)

Memory (MB, lower is better)

Analysis

Overall Winner

{Candidate} wins on {N} of {M} metrics across all test cases.

Tradeoff Summary

• Choose A when: {conditions where A is the better choice}
• Choose B when: {conditions where B is the better choice}

Caveats

• {Limitation of this benchmark}
• {Condition under which results may differ}

Reproduction

# Environment setup
{commands to recreate the environment}

# Run benchmark
{commands to execute the benchmark}
````

Configuring Scope

Calibration Rules

1. Measure, don't guess. Intuition about performance is unreliable. "Obviously

faster" is not a benchmark result.

1. Apples to apples. Candidates must be compared under identical conditions.

Different hardware, configuration, or input data invalidates the comparison.

1. Report variance, not just means. A mean of 50ms with std dev of 100ms is not

the same as a mean of 50ms with std dev of 2ms. Always report spread.

1. Warm up before measuring. First-run performance includes JIT, cache warming,

and connection setup. Exclude warmup iterations from results.

1. Representative inputs only. Benchmarking with synthetic best-case input is

misleading. Use data that resembles production workloads.

1. State the winner per metric, not overall. "A is better" is lazy. "A has lower

latency; B uses less memory" is useful.

Error Handling

When NOT to Benchmark

Push back if:

• The comparison is not performance-related (feature comparison → use a decision matrix or ADR instead)
• The candidates are fundamentally different tools (comparing a database to a message queue)
• The user wants to benchmark trivial operations (comparing two string concatenation methods in Python)
• Results from others already exist and conditions match — link to existing benchmarks instead