Deployment Procedures

Invoke with /tzurot-deployment for Railway operations.

Auto-Deploy: branch-to-environment

Both Railway environments auto-deploy from their respective branches. No manual deploy step is required for either environment.

For prod, the trigger is the release-PR merge from develop into main (procedure in tzurot-git-workflow skill). Schema changes do NOT auto-apply — see the migration step below.

What a dev deploy proves (and does NOT prove)

Dev has NO organic traffic. Its only user is the project owner, and only during explicit testing sessions. A change that has "been on dev for a day" has, in the common case, executed exactly zero times beyond service boot.

Never cite "soaked in dev" as release-safety evidence. For this project, prod is the soak environment — the honest release-safety basis is: per-PR CI (all tiers) + adversarial review, the holistic release review, blast-radius analysis of the runtime-unverified paths, and logging good enough to diagnose the first real occurrence post-hoc. When a change's failure path genuinely needs pre-prod runtime verification, that means an explicit dev testing session with the user driving Discord (see /tzurot-testing), not passive deploy time.

Deployment Procedure

1. Merge PR (triggers auto-deploy)

For feature PRs to develop:

gh pr merge <PR-number> --rebase --delete-branch

For release PRs to main: see tzurot-git-workflow skill (no --delete-branch).

2. Run Prisma migration if the PR includes one

Migrations are NOT auto-applied, and the timing differs by environment because every service auto-deploys in parallel:

# Dev (develop pushes): apply promptly after the push — the brief window is low-stakes
pnpm ops db:migrate --env dev

# Prod (release): migrate BEFORE merging the release PR, so auto-deploy lands into a
# ready schema. This is part of the release flow (tzurot-git-workflow skill, step 4):
pnpm ops release:premigrate

Additive migrations are safe to premigrate; destructive ones (drop/rename/tighten) need a maintenance window — release:premigrate refuses them without --allow-destructive.

Destructive-release sequence (maintenance mode quiesces traffic so the old still-live code never reads the changed schema):

pnpm ops maintenance on --env prod      # bot replies 🔧, gateway 503s (health stays green), BullMQ drains
pnpm ops maintenance status --env prod  # REQUIRED: confirm ON before migrating (see below)
pnpm ops release:premigrate --allow-destructive
gh pr merge <release-PR> --rebase       # Railway auto-deploys into the ready schema
# verify /health + bot boot, then:
pnpm ops maintenance off --env prod     # traffic resumes within the 5s flag-cache window

Verify status shows ON before premigrating — do not proceed if Redis is down. It reports the flag state plus queue depth. MaintenanceFlag fails OPEN on Redis errors, which is right for normal operation (a Redis blip must not 503 the whole API) but inverts here: if Redis is unreachable during the window, traffic is NOT quiesced precisely when quiescing is what protects the still-live old code from the changed schema. maintenance on returning success is not proof the gate is active — read the status.

The flag lives in Redis (not an env var — env changes trigger Railway redeploys, and Postgres is the thing in flux). Note: the maintenance CHECK must already be deployed in prod for the gate to work — it protects releases after the one that ships it. See .claude/rules/03-database.md § Deployment.

3. Monitor deployment

railway status --json
railway logs --service api-gateway -n 100

4. Verify health

curl https://api-gateway-<your-deployment>.railway.app/health

Rollback Procedure

git revert HEAD
git push origin develop
# Railway auto-deploys the revert

Log Analysis

Incident digs: reach for pnpm ops logs correlation flags first. They encode the reliable pull-and-grep-locally pattern (the server-side --filter DSL routinely misses hyphenated UUIDs) and sweep all three app services in one command:

# Cross-service trace of one request (5000-line window per service, local match)
pnpm ops logs --env prod --request-id <uuid>

# BullMQ job trace, floored to the last 2 hours (local pino-time filter)
pnpm ops logs --env prod --job-id <id> --since 2h

# Both flags AND together; --since accepts ISO-8601 or 45m/6h/2d
pnpm ops logs --env prod --request-id <uuid> --job-id <id> --since 6h

Correlation mode reads the CURRENT deployment; for older windows use the deployment-ID lookup below.

Raw CLI equivalents (single service, manual grep):

# Tail specific service (CURRENT deployment only)
railway logs --service bot-client -n 50

# Trace request across services
railway logs | grep "requestId:abc123"

⚠️ NEVER grep prod logs by level. Railway renders every pino line as [INFO] regardless of its actual level, and "level":50-style greps return false-clean while real logger.error failures exist (this wrongly read a broken deploy as "clean" once). Grep by message text or err= content instead:

# ✅ Find errors by content, not level
railway logs --service api-gateway | grep -iE 'failed|error:|err='

# ❌ Both of these give false-clean on Railway
# railway logs | grep "ERROR"
# railway logs | grep '"level":"error"'

Pulling logs from PAST deployments

railway logs defaults to the most recent successful deployment. Logs from before a deploy event aren't visible by default — but Railway DOES retain them, and the CLI supports an explicit deployment-ID lookup.

When you need to investigate an event that happened before the most recent deploy:

# 1. List recent deployments to find the relevant ID + timestamp
railway deployment list --service ai-worker --environment production --limit 10
# Output: <deployment-id> | SUCCESS|REMOVED | <local-timestamp>

# 2. Pull logs from a specific deployment by ID
railway logs <DEPLOYMENT_ID> --service ai-worker --environment production --lines 5000 --filter "<search>"

The --filter flag uses Railway's query DSL (@level:error, quoted phrases like "vision AND 404", etc.). For multi-token literal substring searches, single tokens often work better than quoted phrases — Railway's filter syntax doesn't always behave like grep.

When this matters: investigations into bugs that surfaced just before a release deploy, or cross-deployment timeline traces.

"Did X ever log?" sweeps back to the feature's ship date — not the current deployment. The current deployment only answers "is X logging NOW"; a negative from it says nothing about whether X ever fired (a ~30-minute-old deployment once backed a "no cascade logs" claim that a 5-deployment, ~12.5k-line sweep then overturned; a later 15-deployment sweep retroactively closed several smoke items). Enumerate deployments back to when the log line shipped and sweep each by ID before stating a negative.

Empty log results = debug the query, not the retention

Self-serve first: before asking the owner for runtime facts, exhaust what's already retrievable — Railway CLI (incl. ended deploys), dev probe logs, and any /inspect output the owner already posted (its timestamps anchor log queries). Routing retrievable facts through the owner is the recurring anti-pattern ("you should have all the info you need already").

Railway retains logs; an empty or suspiciously short result almost always means the query is wrong, not that the logs "rolled off" or "aged out" (this is a named anti-pattern in .claude/rules/00-critical.md § "Don't Present Speculation as Fact"). The real culprits, in order of how often they bite:

• --lines has a cap (~5000). Values like 6000/8000 fail with Error in limit - Invalid input and return zero rows — which looks identical to "no matching logs." Stay at --lines 5000 or below; to reach further back, narrow by deployment ID (above) rather than raising --lines.
• Filter by the field the log actually carries. A vision/image error logs attachmentId and url, not the top-level requestId — so --filter "<requestId>" silently misses it and you see only the job-start/complete lines. When the error isn't where you expect, it's logged under a different field: pull a window and grep locally instead of trusting one --filter.
• The --filter DSL is finicky. Hyphenated tokens (ref1-image), quoted phrases ("failed after"), and special characters often match nothing. Prefer a single bare token, or pull --lines 5000 into a file and grep -iE locally — local grep is predictable; the DSL is not.
• Ended/removed deploys are still queryable by deployment ID (above). A REMOVED status does not mean the logs are gone.

(Every one of these has produced a false "logs rolled off" conclusion at least once; --lines 5000 + local grep resolved each immediately.)

Environment Variables

# Preview changes (ALWAYS dry-run first)
pnpm ops deploy:setup-vars --env dev --dry-run

# Apply to dev
pnpm ops deploy:setup-vars --env dev

# List all for a service
railway variables --service api-gateway --json

# Set single variable
railway variables --set "KEY=value" --service ai-worker --environment development

# DELETE - the CLI cannot; use pnpm ops deploy:var-delete (needs TZUROT_RAILWAY_API_TOKEN), dashboard as fallback

Database Migration Procedure

⚠️ Prisma uses LOCAL migrations folder! Checkout the branch matching deployed code first.

# 1. Checkout correct branch
git checkout main  # For production

# 2. Check status
pnpm ops db:status --env prod

# 3. Apply migrations
pnpm ops db:migrate --env prod --force

Running Scripts Against Railway

# Generic pattern
pnpm ops run --env dev <command>

# One-off script
pnpm ops run --env dev tsx scripts/src/db/backfill.ts

# Prisma Studio
pnpm ops run --env dev npx prisma studio

Service Restart

railway redeploy has no --environment flag — it acts on whatever environment the CLI is currently LINKED to, which is not necessarily the one you were just working in. A blind redeploy --service <name> --yes has come within one keystroke of bouncing PROD because the link happened to be production. Always check the link, switch explicitly, then restore it:

# Note: 'railway restart' doesn't exist, use redeploy
railway status                          # shows the currently LINKED environment — note it
railway environment <target>            # switch explicitly — never assume the link
railway redeploy --service bot-client --yes
railway environment <the one status showed>   # restore the link you found, not a guess

Same hazard class as any env-implicit CLI command against shared infra: the command looks identical whether it is about to touch dev or prod.

Cost Impact in Infra Recommendations

Any recommendation that changes hosting posture — keep-warm toggles, serverless on/off, replica counts, timeout bumps that hold instances alive — MUST state its hosting-cost impact alongside the technical rationale. A keep-warm toggle was once recommended purely on latency grounds and overruled on cost ("will cost me a lot more money on hosting"); prefer the cost-neutral alternative (e.g., a timeout bump) unless the user opts into spend.

Troubleshooting Checklist

References

• Railway CLI: docs/reference/RAILWAYCLIREFERENCE.md
• Railway Operations: docs/reference/deployment/RAILWAY_OPERATIONS.md