mem0-test-integration

Verifies what /mem0-integrate produced. Runs in the same workspace, on the same feature branch. Loose coupling — fast, catches compile and runtime bugs, does not catch logical errors.

Canonical sources (use these, not ambient knowledge)

All static checks and smoke-test shapes validate against these URLs. WebFetch each before running step 3.

• Scope-tagged docs index: https://docs.mem0.ai/llms.txt
• OpenAPI (Platform REST): https://docs.mem0.ai/openapi.json
• Published SDK skill (canonical call patterns): https://raw.githubusercontent.com/mem0ai/mem0/main/skills/mem0/SKILL.md
• Vercel AI SDK skill (if the target repo uses @ai-sdk/*): https://raw.githubusercontent.com/mem0ai/mem0/main/skills/mem0-vercel-ai-sdk/SKILL.md
• SDK source (cross-check version against frontmatter mem0testedversions):

- Repo root: https://github.com/mem0ai/mem0 - Python: https://github.com/mem0ai/mem0/tree/main/mem0 - TypeScript: https://github.com/mem0ai/mem0/tree/main/mem0-ts

Read the Delegated skill: field in .mem0-integration/plan.md — if it names a skill URL, fetch that skill and use its example blocks as the reference for both static checks (step 3) and the smoke test (step 5).

Non-invasiveness contract

Every check in this skill assumes the integration is additive and feature-flagged (see /mem0-integrate "Integration principles"). Specifically:

• product.json must contain a feature_flag field.
• Steps 4–6 run in two passes:

- Pass A — flag unset. All pre-existing tests must pass, smoke/E2E skip. The repo must behave like main. Any failure here is a hard fail — do not let the self-heal loop attempt a patch. - Pass B — flag set. New tests must pass, smoke and E2E run.

• If Pass A fails, the scorecard marks non_invasive: false and sets

overall: fail with a distinct reason code the integrator's heal loop refuses to touch.

Preconditions

Refuse to start unless ALL of the following are true:

• .mem0-integration/ directory exists in the repo root.
• .mem0-integration/product.json, goal.md, and plan.md are readable

and internally consistent (JSON parses, docs non-empty).

• Current branch name begins with mem0-integrate/ (set by the companion

skill). Prevents accidental runs on unrelated branches.

• Working tree is clean. The skill never modifies source files; any dirty

state means the integration is mid-edit and not ready to verify.

• The same API key the integration used is available in the environment

(MEM0APIKEY for Platform, OPENAIAPIKEY for OSS — read which from product.json). Interactive mode asks if missing; CI mode exits 2.

Exit with a written rationale on any precondition failure. Never attempt to "fix up" state.

Pipeline

1. Read the contract

Load:

• product.json → which language, which product (Platform vs OSS), which

mem0 version, writesite, readsite.

• plan.md → the mechanical contract (write pattern, read pattern,

preserved behavior).

• goal.md → the intent (displayed in the scorecard only; not tested).

2. Install dependencies

Route by language from product.json:

If install fails → exit code 2 with stderr tail. Never move to testing if dependencies don't resolve.

3. Static sanity checks (fast, local, no API calls)

• Import check: does the write-site file import the expected Mem0

surface? Authoritative list comes from ## Identify the User's Setup in https://docs.mem0.ai/llms.txt: - Platform Python → from mem0 import MemoryClient - Platform TS → import MemoryClient from "mem0ai" - OSS Python → from mem0 import Memory - OSS TS → import { Memory } from "mem0ai/oss"

If plan.md names a delegated skill (e.g., Vercel AI), use that skill's import signature instead of the list above. Mismatch → fail with line number.

• Version check: installed mem0ai version falls in the range from

this skill's mem0testedversions. Out of range → warn but continue.

• Type check (TS tracks only): run tsc --noEmit or tsup --dts.

Non-zero → fail.

• Lint (if the repo has a linter configured): run the repo's own

lint command. Lint failures from this skill's changes → fail; pre-existing lint failures → surface as a warning.

• Eager-init check: grep the writesite and readsite files (paths

from product.json) for MemoryClient( or Memory( at module scope — i.e., not inside a function, method, or class body. MemoryClient() validates the API key in init (network call) and OSS Memory() can eagerly initialize embedding/LLM providers — module-level instantiation hits the wire on import and breaks Pass A's test collection whenever the key is unset. Hit → fail with file:line and the lazy-init guidance from /mem0-integrate step 8 constraint #7.

4. Run the repo's native test suite (two passes)

Pass A — featureflag unset. Run the entire pre-existing suite (excluding the new testmem0* files). Must be 100% green. Any failure here marks noninvasive: false in the scorecard and is a hard fail — the integrator's self-heal loop refuses to touch it.

Pass B — feature_flag set (value from product.json). Run the full suite including the new tests. All must pass.

Isolate integration-introduced failures using git diff main..HEAD --name-only. A test file that exists on main and fails only under the integration branch (flag set or unset) counts against the scorecard regardless of pass. A test file that already failed on main is surfaced as preexistingunrelated and does not count — but is still reported so the user can clean it up.

Capture output to .mem0-integration/test-stdout-flag-off.log and .mem0-integration/test-stdout-flag-on.log. Scorecard reports pass/fail per pass.

5. Smoke test (real API call, shortest round-trip)

Scripted end-to-end flow tailored to product.json. The call shapes below are the minimal ones; if plan.md names a delegated skill, use that skill's minimal example verbatim instead — it is the canonical shape for the detected stack.

Platform (Python):

from mem0 import MemoryClient c = MemoryClient() # uses MEM0APIKEY uid = f"mem0-test-integration-{os.urandom(4).hex()}" c.add([{"role": "user", "content": "I prefer aisle seats"}], userid=uid) hits = c.search("seat preference", userid=uid) assert any("aisle" in h.get("memory", "") for h in hits), hits c.deleteall(userid=uid) # clean up

Platform (TS): same shape with MemoryClient from "mem0ai".

OSS (Python / TS): uses Memory() / new Memory() with default config (OpenAI LLM via OPENAIAPIKEY, local Qdrant). If the repo ships a docker-compose.yml with a Qdrant service, the skill starts it first and tears it down after. If no backing store is reachable → fail with a clear message naming the fix.

The smoke test always uses a disposable random user_id prefixed with mem0-test-integration- so a failed cleanup doesn't pollute the user's real data. A background tidy step deletes any prefix-matching entries older than 24 hours on the next run.

Capture output to .mem0-integration/smoke-stdout.log.

6. E2E integration test (run the app, exercise the flow)

Unit tests + smoke prove the SDK works in isolation. This step is the real signal: does memory actually appear in the app's user-visible output when the integration runs end-to-end?

Requires plan.md to contain an E2E recipe: section (authored by /mem0-integrate step 5). If absent → status skipped (not fail), note in scorecard that the repo has no runnable entry point.

Recipe fields the skill reads:

• start — shell command to launch the app using $PORT for any network

port. Run in background with stdout/stderr teed to .mem0-integration/e2e-app.log.

• ready_probe — how to detect readiness. url=... status=... polls an

HTTP endpoint; log="..." waits for a substring in e2e-app.log; sleep=N waits N seconds (last resort). 60-second hard timeout.

• compose_services — optional. If set, bring them up via

docker compose up -d <services> before start, tear them down with docker compose down at the end.

• write_call — triggers the Mem0 write path exactly once. Output is

captured and surfaced on failure. 60-second hard timeout.

• writeasyncwaitms — pause after writecall to let async memory

flushes land. Default 0.

• read_call — triggers the Mem0 read path. Typically a fresh session

or new request that should surface the stored memory.

• read_assert — substring, regex=..., or jsonpath=<expr>=<value>

that must appear in read_call's stdout. This is the E2E pass gate.

Execution order:

1. Allocate an ephemeral TCP port; export as PORT.
2. Set MEM0USERID to a disposable mem0-test-integration-<rand> value

and export it, so the app can use the same scoping the smoke test does if the recipe wants cleanup.

1. Bring up compose_services if named.
2. Run start in the background.
3. Poll ready_probe until success or 60s timeout. Timeout → fail.
4. Run write_call. Non-zero exit → fail (but continue to cleanup).
5. Sleep writeasyncwait_ms.
6. Run read_call.
7. Evaluate readassert against readcall's stdout. Miss → fail.
8. Cleanup (always, even on failure): SIGTERM the app, SIGKILL after

5s, docker compose down if services were started, delete_all memories matching mem0-test-integration-* on Platform scenarios.

On any failure, the scorecard includes:

• Last 40 lines of e2e-app.log
• Full write_call output
• Full read_call output
• The expected vs actual for read_assert

7. Scorecard

Write .mem0-integration/scorecard.md and .mem0-integration/scorecard.json:

{ "timestamp": "2026-04-20T14:03:11Z", "branch": "mem0-integrate/remember-user-preferences", "product": "platform", "language": "python", "mem0version": "2.0.0", "noninvasive": true, "featureflag": "MEM0ENABLED", "results": { "install": {"status": "pass", "durationms": 12043}, "staticchecks":{"status": "pass", "durationms": 812}, "unittestsflagoff": {"status": "pass", "durationms": 3920, "count": 47, "reason": "all pre-existing tests green with flag unset"}, "unittestsflagon": {"status": "pass", "durationms": 4321, "count": 49}, "smoketest": {"status": "pass", "durationms": 2890, "memoryid": "mem..."}, "e2etest": {"status": "pass", "durationms": 14200, "readyprobems": 3100, "writeexit": 0, "readassertmatched": true} }, "friction": { "dependencyinstallretries": 0, "preexistingtest_failures": 0, "warnings": ["mem0ai 2.0.0 pinned; consider 2.0.1 for fix X"] }, "overall": "pass" }

The markdown version is human-readable and includes:

• Goal doc + plan doc reprinted at top (so reviewers don't have to hunt).
• Each check with pass/fail + log excerpt.
• Friction summary.
• Verbatim warnings from mem0 SDK (if any — e.g., deprecated field usage).
• Explicit "NOT checked" section listing what loose coupling misses:

"Whether the stored data is what the user wants stored. Whether search runs at the right moment. Whether user_id matches the actual session scope. Human review required."

8. Report + exit

• Print the scorecard path + overall pass/fail to stdout.
• Do not commit the scorecard files. They live in .mem0-integration/,

which is gitignored. The user can inspect and optionally pin.

• On fail: print the first failing step's log tail (last 40 lines) and

stop. Do not attempt to fix anything.

Artifacts (all under .mem0-integration/)

Modes

Invocation

/mem0-test-integration # interactive, all steps /mem0-test-integration --ci # non-interactive /mem0-test-integration --skip-smoke # no API calls, no E2E /mem0-test-integration --skip-e2e # unit + smoke only (faster CI) /mem0-test-integration --only-smoke # just smoke /mem0-test-integration --only-e2e # just E2E (assumes deps installed)

Composition: --skip- can stack (--skip-smoke --skip-e2e = static + unit only, zero API cost). --only- is mutually exclusive with all other flags.

Exit codes

Explicitly out of scope

• Modifying source files. The skill is read-only against the repo.

If verification exposes a bug, re-run /mem0-integrate on the same goal + plan; do not hand-patch.

• Fixing broken tests. Failing unit tests are a signal that the

integration is wrong, not that the tests are wrong. The skill does not "try a different test."

• Deep logical correctness. The E2E step proves "something the user

said earlier comes back later," which is a useful but shallow signal. It does NOT prove the integration picks the right facts to store, scopes user_id correctly across real users, or handles conflict resolution well. That's human review territory.

• Self-healing. This skill never modifies source files. The paired

/mem0-integrate skill in its default --heal mode consumes the scorecard produced here and drives its own remediation loop. Exit code 7 (non-invasiveness violation) is the explicit signal the heal loop must stop and surface to the user.

• Cross-branch comparisons. No main baseline diffing. The

scorecard reflects this branch only.

• Running against production data. Every smoke test uses a disposable

random user_id and cleans up after. Never touches any other user's data.