smithery/rileyhilliard

fixing-flaky-tests

Diagnose and fix tests that pass in isolation but fail when run concurrently. Covers shared state isolation, resource conflicts, and timing-based flakiness.

Installation

$ npx skills add smithery/rileyhilliard --skill fixing-flaky-tests

Similar popular skills

Related neighbors and high-traction skills in the same topics — useful to compare before installing.

Also in this package

Other skills from smithery/rileyhilliard.

npx skills add smithery/rileyhilliard

Browse all from smithery/rileyhilliard

More details

Agent compatibility

Declared targets from SKILL.md / docs. Unmarked agents are not listed — the skill may still install via the CLI.

Claude Code Declared
Cursor Not declared
Codex Not declared
GitHub Copilot Not declared
Windsurf Not declared
Gemini CLI Not declared
Cline Not declared
OpenCode Not declared

Skill metadata

Parsed from SKILL.md frontmatter.

Declared agents claude-code

Package contents

Files included with this skill beyond the listing page.

  • skill md SKILL.md 4,295 B
  • docs SUMMARY.md 211 B

History

  1. First recorded snapshot · 0 installs

SKILL.md

If the current repo has its own rules/skills covering this topic (check .claude/rules/ and repo CLAUDE.md), those take precedence — apply this skill only where they're silent.

Fixing Flaky Tests

Target symptom: Tests pass when run alone, fail when run with other tests.

Diagnose first

Test passes alone, fails with others?
    │
    ├─ Same error every time → Shared state
    │   └─ Database, globals, files, singletons
    │
    ├─ Random/timing failures → Race condition
    │   └─ See async waiting patterns in `writing-tests` skill
    │
    └─ Resource errors (port, file lock) → Resource conflict
        └─ Need unique resources per test/worker

Quick diagnosis:

  1. Run failing test 10x alone - does it always pass?
  2. Run failing test 10x with the suite - same error or different?
  3. Check error message - mentions port/file/connection?

Shared state (deterministic failures)

Tests pollute state that other tests depend on. Fix by isolating state per test.

State Type Isolation Pattern
Database Transaction rollback, savepoints, worker-specific DBs
Global variables Reset in beforeEach/afterEach
Singletons Provide fresh instance per test
Module state jest.resetModules() or equivalent
Files Unique paths per test, temp directories
Environment vars Save/restore in setup/teardown

Database isolation (most common):

# Python: Savepoint rollback - each test gets rolled back
@pytest.fixture
async def db_session(db_engine):
    async with db_engine.connect() as conn:
        await conn.begin()
        await conn.begin_nested()  # Savepoint
        # ... yield session ...
        await conn.rollback()  # All changes vanish
// Jest: Reset mocks between tests
beforeEach(() => {
  jest.clearAllMocks()
  jest.resetModules()  // Clear module cache before test
})

afterEach(() => {
  jest.restoreAllMocks()  // Restore spied functions
})

See language-specific references for complete patterns.

Race conditions (random failures)

Tests don't wait for async operations to complete.

See the writing-tests skill for async waiting patterns:

  • Framework-specific waiting (Testing Library findBy, Playwright auto-wait)
  • Custom polling helpers
  • When arbitrary timeouts are acceptable

Quick summary: Wait for conditions, not time:

// Bad
await sleep(500)

// Good
await waitFor(() => expect(result).toBe('done'))

Resource conflicts (port/file errors)

Multiple tests or workers compete for same resource.

Worker-specific resources:

# Python pytest-xdist: unique DB per worker
@pytest.fixture(scope="session")
def database_url(worker_id):
    if worker_id == "master":
        return "postgresql://localhost/test"
    return f"postgresql://localhost/test_{worker_id}"
// Jest/Node: dynamic port allocation
const server = app.listen(0)  // OS assigns available port
const port = server.address().port

File conflicts:

import tempfile

@pytest.fixture
def temp_dir():
    with tempfile.TemporaryDirectory() as d:
        yield d

Language-specific isolation patterns

Stack Reference
Python (pytest, SQLAlchemy) [references/python.md](references/python.md)
Jest / Testing Library [references/jest.md](references/jest.md)
Playwright E2E [references/playwright.md](references/playwright.md)
Async waiting patterns (TypeScript) [writing-tests waiting-typescript](../writing-tests/references/waiting-typescript.md)
Async waiting patterns (Python) [writing-tests waiting-python](../writing-tests/references/waiting-python.md)

Verification

After fixing, verify the fix worked:

# Run the specific test many times
pytest tests/test_flaky.py -x --count=20

# Run with parallelism
pytest -n auto

# Jest equivalent
jest --runInBand  # First verify serial works
jest              # Then verify parallel works