When the user wants to plan, design, or implement an A/B test or experiment. Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," or "how long should I run this test." Use this whenever someone is comparing two approaches and wants to measure which performs better. For tracking implementation, see analytics-tr…
When the user wants to plan, design, or implement an A/B test or experiment.
Also use when the user mentions "A/B test," "split test," "experiment," "test this change," "variant copy," "multivariate test," "hypothesis," "should I test this," "which version is better," "test two versions," "statistical significance," or "how long should I run this test." Use this whenever someone is comparing two approaches and wants to measure which performs better.
For tracking implementation, see analytics-tracking.
For page-level conversion optimization, see page-cro.
Stronger alternatives
This repository is archived — consider an actively maintained alternative.
Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.
Claude CodeNot declared
CursorNot declared
CodexNot declared
GitHub CopilotNot declared
WindsurfNot declared
Gemini CLINot declared
ClineNot declared
OpenCodeNot declared
Repository health
Stars1.1K
LicenseLICENSE
Default branchmain
Open issues0
Status
Archived
Package contents
Files included with this skill beyond the listing page.
skill mdSKILL.md7,404 B
docsSUMMARY.md581 B
History
First seen on skills.sh
First recorded snapshot · 1 installs
SKILL.md
A/B Test Setup
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Initial Assessment
Check for product marketing context first: If .agents/product-marketing-context.md exists (or .agents/product-marketing-context.md in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
Test Context - What are you trying to improve? What change are you considering?
Current State - Baseline conversion rate? Current traffic volume?
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Example
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
For detailed sample size tables and duration calculations: See [references/sample-size-guide.md](references/sample-size-guide.md)
Metrics Selection
Primary Metric
Single metric that matters most
Directly tied to hypothesis
What you'll use to call the test
Secondary Metrics
Support primary metric interpretation
Explain why/how the change worked
Guardrail Metrics
Things that shouldn't get worse
Stop test if significantly negative
Example: Pricing Page Test
Primary: Plan selection rate
Secondary: Time on page, plan distribution
Guardrail: Support tickets, refund rate
Designing Variants
What to Vary
Category
Examples
Headlines/Copy
Message angle, value prop, specificity, tone
Visual Design
Layout, color, images, hierarchy
CTA
Button copy, size, placement, number
Content
Information included, order, amount, social proof
Best Practices
Single, meaningful change
Bold enough to make a difference
True to the hypothesis
Traffic Allocation
Approach
Split
When to Use
Standard
50/50
Default for A/B
Conservative
90/10, 80/20
Limit risk of bad variant
Ramping
Start small, increase
Technical risk mitigation
Considerations:
Consistency: Users see same variant on return
Balanced exposure across time of day/week
Implementation
Client-Side
JavaScript modifies page after load
Quick to implement, can cause flicker
Tools: PostHog, Optimizely, VWO
Server-Side
Variant determined before render
No flicker, requires dev work
Tools: PostHog, LaunchDarkly, Split
Running the Test
Pre-Launch Checklist
Hypothesis documented
Primary metric defined
Sample size calculated
Variants implemented correctly
Tracking verified
QA completed on all variants
During the Test
DO:
Monitor for technical issues
Check segment quality
Document external factors
Avoid:
Peek at results and stop early
Make changes to variants
Add traffic from new sources
The Peeking Problem
Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.
