A/B Test Design & Analysis

Quick Reference (30 seconds)

A/B Testing Specialist - Design tests that produce statistically valid, actionable results.

Core Principles:

• Hypothesis First: Not just "let's see what happens"
• Single Variable: Test one thing at a time
• Statistical Rigor: Pre-determine sample size, don't peek and stop early
• Business Impact: Measure what matters to the business

Key Deliverables:

• Hypothesis documentation
• Sample size calculation
• Test design specification
• Results analysis

Implementation Guide (5 minutes)

Hypothesis Framework

┌─────────────────────────────────────────────────────────────────┐
│  HYPOTHESIS TEMPLATE                                             │
├─────────────────────────────────────────────────────────────────┤
│  Because [observation/data],                                     │
│  we believe [change]                                             │
│  will cause [expected outcome]                                   │
│  for [audience].                                                 │
│  We'll know this is true when [metrics].                        │
└─────────────────────────────────────────────────────────────────┘

Weak Hypothesis:

"Changing the button color might increase clicks."

Strong Hypothesis:

"Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."

Good Hypotheses Include

Test Types

Sample Size Calculation

Quick Reference Table

Calculation Inputs

Test Duration Formula

Duration = (Sample size × Number of variants) / (Daily traffic × Conversion rate)

Minimum: 1-2 business cycles (usually 1-2 weeks)
Maximum: Avoid running too long (novelty effects, external factors)

Calculator Resources

• Evan Miller: evanmiller.org/ab-testing/sample-size.html
• Optimizely: optimizely.com/sample-size-calculator/

Metrics Selection

Metric Hierarchy

┌─────────────────────────────────────────────────────────────────┐
│  METRIC HIERARCHY                                                │
├─────────────────────────────────────────────────────────────────┤
│  PRIMARY METRIC (1 only)                                        │
│  └── Single metric that matters most                            │
│  └── Directly tied to hypothesis                                │
│  └── What you'll use to call the test                          │
│                                                                  │
│  SECONDARY METRICS (2-4)                                        │
│  └── Support primary interpretation                             │
│  └── Explain why/how change worked                              │
│                                                                  │
│  GUARDRAIL METRICS (1-3)                                        │
│  └── Things that shouldn't get worse                            │
│  └── Revenue, retention, satisfaction                           │
│  └── Stop test if significantly negative                        │
└─────────────────────────────────────────────────────────────────┘

Metric Examples by Test Type

Designing Variants

What to Vary

Variant Documentation Template

Control (A):
- Screenshot: [image]
- Description: Current state

Variant (B):
- Screenshot: [mockup]
- Specific changes: [list changes]
- Hypothesis: Why this will win

Traffic Allocation

Allocation Considerations

• Consistency: Users see same variant on return
• Segment sizes: Ensure segments are large enough
• Time balance: Balanced exposure across times of day/week

Implementation Approaches

Tools Reference

Running the Test

Pre-Launch Checklist

□ Hypothesis documented
□ Primary metric defined
□ Sample size calculated
□ Test duration estimated
□ Variants implemented correctly
□ Tracking verified
□ QA completed on all variants
□ Stakeholders informed

During Test Rules

The Peeking Problem

Looking at results before sample size and stopping when significant leads to:

• False positives
• Inflated effect sizes
• Wrong decisions

Solution: Pre-commit to sample size and stick to it.

Analyzing Results

Result Interpretation Matrix

Analysis Checklist

1. Did you reach sample size?
   → If not, result is preliminary

2. Is it statistically significant?
   → Check confidence intervals, p-value < 0.05

3. Is the effect size meaningful?
   → Compare to MDE, project business impact

4. Are secondary metrics consistent?
   → Do they support primary?

5. Any guardrail concerns?
   → Did anything get worse?

6. Segment differences?
   → Mobile vs desktop, new vs returning

Test Documentation Template

# A/B Test: [Name]

## Test Details
- Test ID: [ID in testing tool]
- Dates: [Start] - [End]
- Owner: [Name]

## Hypothesis
[Full hypothesis using framework]

## Test Design
- Type: A/B / A/B/n / MVT
- Duration: X weeks
- Sample size: X per variant
- Traffic allocation: 50/50

## Variants
[Control and variant descriptions with screenshots]

## Metrics
- Primary: [metric and definition]
- Secondary: [list]
- Guardrails: [list]

## Results
- Sample size: [achieved vs. target]
- Primary metric: [control] vs. [variant] ([% change], [confidence])
- Secondary metrics: [summary]
- Segment insights: [notable differences]

## Decision
Winner / Loser / Inconclusive

## Action
[What we're doing next]

## Learnings
[What we learned, what to test next]

Common Mistakes

Works Well With

Skills:

• jikime-marketing-page-cro - Generate test ideas from CRO analysis
• jikime-marketing-copywriting - Create variant copy
• jikime-marketing-analytics - Set up test measurement
• jikime-marketing-psychology - Apply psychological hypotheses

Version: 1.0.0 Last Updated: 2026-01-25 Attribution: Enhanced from marketingskills by Corey Haines (MIT License)