SkillsAdd

mohitagw15856/pm-claude-skills

ab-test-readout

Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call.

First seen Jun 21, 2026

Installation

$ npx skills add mohitagw15856/pm-claude-skills --skill ab-test-readout
View on skills.sh ↗ GitHub repository ↗

Summary

  • Analyse a finished A/B test and write the readout — the result, whether it's statistically and practically significant, what it means, and the ship/no-ship call.
  • Use when asked to analyse experiment results, write an A/B test readout, interpret test data, or decide whether to ship a variant.
  • Produces a clear verdict with the lift and confidence, segment cuts, the risks (peeking, novelty, sample), and a recommendation.
  • Distinct from planning a test — this reads results.

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

agent-browser Shared topics
vercel-labs/agent-browser

Browser automation CLI for AI agents. Use when the user needs to interact with websites, includ…

810.4K installs
azure-aigateway Shared topics
microsoft/azure-skills

Configure Azure API Management as an AI Gateway for AI models, MCP tools, and agents. WHEN: sem…

566.3K installs
azure-compute Shared topics
microsoft/azure-skills

Azure VM/VMSS router. WHEN: create / provision / deploy / spin-up VM, recommend VM size, compar…

510K installs
supabase-postgres-best-practices Shared topics
supabase/agent-skills

Postgres best practices maintained by Supabase, for Postgres running anywhere. Load this skill …

391.6K installs
prisma-cli Shared topics
prisma/skills

Prisma ORM CLI commands reference covering init, generate, migrate, db, dev, complete, studio, …

267K installs
supabase Shared topics
supabase/agent-skills

Use when doing ANY task involving Supabase. Triggers: Supabase products (Database, Auth, Edge F…

263K installs

Also in this package

Other skills from mohitagw15856/pm-claude-skills · top by installs.

npx skills add mohitagw15856/pm-claude-skills
teaching-lesson-plan 150 team-offsite-planner 98 sop-writer 72 patient-communication 67 user-research-synthesis 66 financial-due-diligence 65 competitive-analysis 65 prd-template 64 meeting-notes 62 project-status-report 62

Browse all from mohitagw15856/pm-claude-skills

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Not declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Repository health

Stars 1.3K
License LICENSE
Default branch main
Open issues 7
Status Active

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 3,027 B
  • docs SUMMARY.md 500 B

History

  1. First seen on skills.sh
  2. First recorded snapshot · 14 installs

SKILL.md

A/B Test Readout Skill

The hard part of an experiment is the readout: not "B won" but "is this real, is it big enough to matter, and should we ship?" This skill turns results into an honest decision — and flags the ways A/B results lie.

Working from a brief

Given results (even partial), write the full readout anyway. If significance isn't provided, reason about it from the numbers and flag what's needed to confirm. Mark assumed figures. Never declare a winner without addressing significance and sample.

Required Inputs

Ask for (if not already provided):

  • The hypothesis and the primary metric
  • Results — control vs variant: conversions/rate, sample size per arm, duration
  • Guardrail metrics (revenue, retention, latency, complaints) that mustn't regress
  • Pre-registered decision rule (what would count as a win) if one exists

Output Format

1. Verdict (one line)

Ship / Don't ship / Inconclusive — keep running — with the headline number.

2. The result

Metric Control Variant Relative lift Significant?
Primary p / CI
Guardrail(s)

State statistical significance (p-value / confidence interval) and practical significance (is the lift big enough to matter given the cost?).

3. Did it really win?

Address the ways A/B tests mislead:

  • Sample / power — was the test adequately powered, or under-sampled?
  • Peeking — was the call made early, inflating false positives?
  • Novelty / primacy — could the effect fade?
  • Segments — does the win hold across key segments, or is it driven by one?

4. Segment cuts

Where the effect is strong vs flat vs negative (new vs returning, platform, geography).

5. Recommendation & next step

Ship / iterate / re-run, plus what to monitor post-launch or what the follow-up test should isolate.

Quality Checks

  • Distinguishes statistical from practical significance
  • Checks guardrail metrics, not just the primary
  • Flags peeking, power, novelty, and segment-driven wins
  • Recommendation follows from the evidence, with a monitoring/next-test step
  • Doesn't declare a winner on an underpowered or peeked result

Anti-Patterns

  • "B won by 8%!" with no significance or sample size
  • Calling a result early (peeking) and shipping
  • Ignoring a guardrail regression because the primary went up
  • A statistically significant but practically meaningless lift treated as a win