Rust Systems & Services

Covers modern application-layer Rust (edition 2024): CLIs, web services, libraries. Not no_std/embedded.

Tooling

• Pin rust-toolchain.toml per repo so every contributor and CI uses the same compiler.
• cargo update -p <crate> for single-package upgrades. cargo update rewrites everything — avoid in PR diffs.
• Cargo.lock goes in version control for binaries and libraries (modern guidance; reproducibility wins).
• cargo install <crate> from a registry or git source no-ops silently when the installed version matches — it prints "package is already installed" and keeps the old binary; pass --force in install scripts. cargo install --path . always rebuilds and replaces regardless of --force. Either way, certify the installed artifact (which <bin> + version/behavior probe), not target/release/<bin> — the two can diverge when a stale env override points tests at the wrong one.

Workspaces

Multi-crate projects use a workspace with layered crates. Dependencies point inward only.

Cargo.toml                  # [workspace] members + [workspace.dependencies]
crates/
  protocol/    # Shared types, no deps on other workspace crates
  storage/     # Persistence, depends on protocol
  service/    # Business logic, depends on protocol + storage
  cli/        # Binary, depends on everything

• Centralize versions in [workspace.dependencies], reference as foo = { workspace = true } in members.
• Keep the leaf-most crate (protocol / types) dependency-free so every other crate can depend on it without cycles.
• Feature flags belong on the crate that introduces the dependency, not re-exported through the workspace root.
• Library crates expose one stable facade: a thin lib.rs with a //! purpose doc and pub use re-exports — one import path per concept, internals free to reorganize without breaking callers.
• pub alone does not prove an item is externally reachable. Reachability runs through the re-export graph: a pub item inside a private module that is never re-exported is free to change, while the same item surfaced through a pub use at the crate root is not — even though its containing module stays private. (A pub(crate) item cannot be re-exported outside the crate: pub use on one is E0364, while pub(crate) use compiles.) Trace the facade before calling a reorganization internal. On a library crate with a published baseline, cargo semver-checks settles it mechanically.
• Defining a macro_rules! or proc macro, or handling paths, process output, or on-disk state? Load [macros-and-os-boundaries.md](./references/macros-and-os-boundaries.md) — $crate resolution, single-interpolation of $x:expr, $t:tt precedence, item-name collisions across invocations, syn::Error over panic, non-UTF-8 Path/OsStr, and write-then-rename. These type-check cleanly and fail on a caller's machine.
• Document public items at the point of exposure. /// on every public item (purpose, params, return, plus # Examples / # Errors / # Panics / # Safety where they apply); //! for modules and crates. Doc examples compile and run under cargo test --doc, so they are regression tests, not decoration. Enforce with #![deny(missing_docs)] on library crates; see [rustdoc.md](./references/rustdoc.md).
• Feature gates must error, never silently degrade. If runtime config requests a capability the binary wasn't compiled with (e.g. device = "gpu" on a non-CUDA build), fail at startup — silent fallback diverges from operator config unnoticed.
• Centralize lints at the workspace root with [workspace.lints.*] — every member crate inherits the same ruleset, no per-crate #![deny(...)] drift:

``toml [workspace.lints.clippy] all = { level = "warn", priority = -1 } pedantic = { level = "warn", priority = -1 } ``

Each member crate opts in with [lints] workspace = true.

Build Profiles

When tuning Cargo build profiles (release LTO, release-dbg symbols, release-min for distributable binaries) or adding dev-machine speedups (mold linker, target-cpu=native, share-generics), load [build-profiles.md](./references/build-profiles.md).

Error Handling

Split by crate role:

• Libraries / lower crates: define typed errors with thiserror. Consumers can pattern-match.
• Binaries / top-level crates: use anyhow::Result with .context("what was being attempted"). Human-readable error chains.
• Never return Box<dyn Error> from library APIs — it erases variant information.
• Use ? liberally. Never .unwrap() or .expect() outside tests and main. An expect("...") is acceptable only when the invariant is provably upheld and the message explains why.
• Convert at boundaries: #[from] on thiserror variants for auto-conversion; .map_err(MyError::from) when explicit.
• bail!("...") / ensure!(cond, "...") in application code for early exits.
• Prefer Result<T, E> over panics for any recoverable error. Panics are for programmer bugs (broken invariants), not runtime failures.
• #[mustuse] on fallible APIs: annotate functions returning Result or newtype-wrapped results that callers frequently ignore. Catches let = validate(x); at compile time instead of shipping a silently-dropped error.
• Make illegal call-sequences unrepresentable — the type-state pattern: encode a mandatory call order as distinct types (Client<Uninitialized> → Client<Connected>) so an out-of-order call fails to compile instead of erroring at runtime.
• fs::readtostring(p).unwrapordefault() to mean "an absent file is an empty config" swallows every read error, not just NotFound. A file that exists but cannot be read — permission denied, invalid UTF-8, transient I/O — collapses to empty, and the next step writes a fresh file over the comments and unrelated entries the read never surfaced. Match the kind: Err(e) if e.kind() == ErrorKind::NotFound => Ok(default), everything else propagates with context. Test it by writing invalid UTF-8 bytes to the path and asserting the operation returns Err and leaves the bytes untouched.

Ownership Discipline

• Take &str over &String, &[T] over &Vec<T> in function signatures — accepts more call sites for free.
• Return owned (String, Vec<T>) from constructors and public APIs. Borrow in hot paths where lifetimes are obvious.
• Reach for Arc<T> only when sharing across threads. Single-threaded sharing uses Rc<T> or references.
• Cow<'_, str> when a function sometimes allocates and sometimes borrows (e.g. normalization).
• Rely on lifetime elision. More than one signature needing an explicit 'a is a signal the type should own its data — convert the borrow to owned before adding lifetimes.
• Reducing hot-path allocations (SmallVec, ArrayVec, string interning, Bytes, vectored writes): profile first, then load [performance.md](./references/performance.md).
• str::lines() splits on \n only. A line-oriented scanner ported from a language with universal newlines (Python, Ruby) silently merges a bare-\r file into one line — in a redaction or filtering tool that is a security divergence, not a formatting one: the whole body rides through on whatever classification the merged first line matched. Write the splitter explicitly over CR, LF and CRLF, and emit the original bytes for every line the rules did not change rather than re-encoding a decoded copy — a round-trip through lossy decoding transcodes lines the tool was supposed to pass through untouched.
• The regex crate has no look-around. If a rule is defined by a lookbehind or lookahead, reach for fancy-regex rather than hand-rolling boundary checks, which drift from the reference on the one input nobody tried. Three neighbours that type-check and still diverge: regex::bytes still applies Unicode \b (a byte-oriented token scan wants (?-u:\b), or cafémb-x1z passes a boundary check ASCII \b would have failed); str::tolowercase() is not case folding (ß maps to ss only under casefold, so a hash key derived from case-folded text differs between implementations — use caseless); and char::iswhitespace() excludes U+001C–U+001F, which Python's \s and str.strip() include.

Async with Tokio

• Default runtime: #[tokio::main] with features = ["full"] for apps; features = ["rt", "macros", "sync"] for libraries that need to stay slim.
• tokio::spawn for independent tasks. JoinSet for a dynamic group awaited together with cancellation.
• tokio::select! for racing futures (timeouts, cancellation, first-wins).
• Never block the runtime: tokio::task::spawn_blocking for sync CPU work or blocking I/O libs.
• tokio::sync::Mutex only when the guard must be held across .await. Otherwise std::sync::Mutex is faster.
• tokio::sync::RwLock when reads dominate writes (config snapshots, route tables, hot caches). Many readers proceed in parallel; Mutex serializes them. For snapshot-swap semantics (rarely-updated config), arc-swap::ArcSwap is faster still — no lock on the read path.
• Cancellation: CancellationToken (from tokio-util) propagates shutdown. Long-running tasks must check it.
• Backpressure via bounded mpsc channels — unbounded channels hide memory growth until OOM.
• Semaphore for hard concurrency limits on spawn paths that don't fit a channel model (e.g. "at most 50 concurrent outbound HTTP calls"). let _permit = sem.acquire().await?; inside the task; dropping the permit releases the slot. Pair with Arc<Semaphore> shared across spawners.
• Don't mix async runtimes. Pick tokio and stick with it; async-std and smol don't interop cleanly.
• A manually-constructed Runtime's Drop joins already-running spawnblocking tasks. A daemon whose shutdown must not wait on wedged blocking work (long inference, stuck I/O) has to finish its cleanup and std::process::exit(0) rather than let the runtime drop, or use shutdowntimeout. JoinSet::abort_all does not help — abort takes effect at an await point, and a blocking closure that has already started has none.
• A panic inside a spawned per-request task is worse than an error. Without a catchunwind the panic unwinds that one task: the connection survives, no response is ever sent for that request id, and the caller waits until its own timeout. So every reachable unwrap/expect/slice index in a handler — a DB row with an unexpected enum string, a model output of unexpected shape, an index derived from untrusted input — is a client hang rather than a crash anyone would notice. Running the handler body under spawnblocking gives the boundary for free: a panic arrives as a JoinError you convert into an error response, and the same call offloads the blocking work.

CLI Tools (clap)

• Use the derive API: #[derive(Parser)] + #[derive(Subcommand)]. Less boilerplate, types drive the help text.
• One enum Commands variant per subcommand; flatten shared flags into a #[command(flatten)] struct CommonArgs.
• --json flag on query commands for agent/pipe consumption. Emit via serdejson::tostring(&value)?.
• Exit codes: 0 success, 1 for errors main returned, 2 for argparse (clap handles this), reserve 3+ for domain meanings documented in --help.
• Provide --version automatically via #[command(version)].

See [cli-tools.md](./references/cli-tools.md) for config layering, logging setup, progress reporting, and shell completions.

HTTP Services (axum)

• Framework default: axum (tokio-native, tower middleware, extractor-based handlers). Pick actix-web only if an existing codebase uses it.
• Handlers return Result<impl IntoResponse, AppError>. Implement IntoResponse for AppError to centralize error → status mapping.
• Validate input at the boundary: axum::extract::Json<T> where T: Deserialize + Validate (use validator crate). Internal services trust input was validated.
• Share state via State<Arc<AppState>> — not globals, not lazy_static.
• Middleware via tower::ServiceBuilder: tracing → timeout → auth → CORS → handler. Order matters.
• Resilience layers (outbound clients, shared services): combine LoadShed + ConcurrencyLimit for backpressure, not unbounded queueing; full tower stack in [production-resilience.md](./references/production-resilience.md).

See [axum-service.md](./references/axum-service.md) for project layout, extractors, error types, graceful shutdown, and OpenAPI generation.

Concurrency

rayon and tokio coexist — use tokio::task::spawnblocking to call a rayon pool from async code. Never call .blockon() from inside a tokio task; it deadlocks the runtime.

Testing

• Built-in #[test]. Prefer cargo nextest run --workspace over cargo test — it runs tests in parallel processes with proper isolation.
• Unit tests live in mod tests { ... } at the bottom of the file (access to private items).
• Integration tests in tests/ directory. One file per public surface area.
• #[tokio::test] for async tests. Add flavor = "multi_thread" when the code under test spawns tasks.
• rstest for parametrized tests and fixtures. proptest / quickcheck for property-based tests on pure logic.
• insta for snapshot testing CLI output, serialization, large structs. Review diffs with cargo insta review.
• assert_cmd + predicates for CLI integration tests (invokes the binary, asserts on stdout/stderr/exit code).
• Assert on error variants with matches!: assert!(matches!(result.unwraperr(), MyError::Validation())) — no match arms to update when unrelated variants are added.
• Coverage: cargo llvm-cov --workspace --html. Target 70%+ on application code, higher on library crates.
• Fuzzing for parsers: cargo fuzz + libfuzzer-sys on any code parsing untrusted input; nightly runs surface panics and UB unit tests miss.
• Never mutate process-global state in a test. set_var("TMPDIR", …) in one test makes every concurrent tempfile::tempdir() create its scratch dir inside that test's TempDir — recursively deleted when it drops. The smoking gun is nested temp paths (/tmp/.tmpXXXX/.tmpYYYY/…) and ENOENT on files a victim just created, with the failing test rotating between runs. A mutex around the env-mutating tests does not fix it: the victims never take the mutex. Refactor the function under test into a thin env-reading wrapper over an env-free core that takes the values as parameters, and test the core.
• A concurrent Command::spawn briefly extends the lifetime of every open file descriptor. fork duplicates the whole parent fd table and only exec's CLOEXEC closes the copies, so in that window a sibling test holds the caller's lock fd or its just-written script's write fd. Two symptoms, one cause: an flock that outlives its guard's drop (a test asserting "drop released it, re-acquire succeeds immediately" fails roughly 1 run in 12 next to spawn-heavy tests, 0 in N alone — fix with a bounded poll-acquire to a deadline, not a one-shot assert) and ErrorKind::ExecutableFileBusy on exec'ing a file just chmod +x'd (fix with a bounded retry around the spawn). The already-held assertion needs no change in either case.

For generic test discipline (anti-patterns, mock rules, rationalization resistance), see the ia-writing-tests skill.

Unsafe Discipline

• Default: no unsafe. If clippy flags it, don't #[allow] it — refactor. The #[expect] escape hatch below does not apply here; unsafe findings get fixed, not annotated.
• Every unsafe block gets a // SAFETY: comment above it explaining why each invariant holds. No comment = reviewer rejects.
• Keep unsafe blocks minimal — wrap in a safe abstraction at module boundary, mark the module pub(crate).
• Use miri (cargo +nightly miri test) on any crate containing unsafe or raw pointer arithmetic — catches UB that optimizers mask.
• Prefer bytemuck, zerocopy, bytes over hand-rolled transmutes for zero-copy patterns.
• Env-var writes are unsafe in edition 2024. Write them only in main, before the runtime starts or any thread spawns. Concurrent getenv is UB; OnceLock does not make it safe. Watch for lazy LDLIBRARYPATH-style writes on first use — hoist them to startup.

Production Resilience

When productionizing a service (config validation, /health + /ready endpoints, graceful shutdown, retries/timeouts/jitter, deny-by-default fallback when the call is the security decision, connection pools, diagnostic secret redaction), load [production-resilience.md](./references/production-resilience.md).

Observability

For logging (tracing + tracing-subscriber with init recipe), #[instrument] spans, correlation IDs, metrics, and distributed tracing patterns, load [observability.md](./references/observability.md). Never use println! or log:: in new code.

CI

General CI design lives with the ia-infrastructure-engineer agent. For Rust-specific callouts (rustsec/audit-check, cargo-llvm-cov, Swatinem/rust-cache, taiki-e/install-action, matrix coverage guidance, doc-test step), load [ci-pipeline.md](./references/ci-pipeline.md).

Discipline

• Simplicity first — every change as simple as possible, impact minimal code.
• Only touch what's necessary — avoid unrelated changes in a PR.
• No #[allow(clippy::...)] as a shortcut — fix the underlying issue. When a suppression is genuinely warranted, write #[expect(clippy::lint_name, reason = "...")] instead: expect warns once the lint stops firing, so a suppression that has outlived its cause reports itself, where allow rots silently forever. (expect needs Rust 1.81+; edition 2024 clears that floor.)
• Before adding a trait or generic, verify it's used in 3+ places. Otherwise a concrete type is clearer.
• bool::thensome(x) takes x by value — the argument is computed before the bool is consulted, so a guard written as a condition plus a fixed-width slice panics on exactly the inputs the condition was checking for: (b.len() >= 19 && b[4] == b'-').thensome(&v[..19]) panics on any shorter value, exiting 101 inside the one function written to report the case as undetermined. Use then(|| …), which is lazy. Clippy does not flag the difference. Grep then_some( for an argument that indexes, slices, unwraps, or allocates. Related: a fixed-width slice is not a parse — &v[..19] also panics mid-character on non-ASCII, and comparing two such prefixes lexicographically drops the timezone offset, so 01:00+02:00 sorts after 00:00Z while being an hour earlier. Parse and normalize, or reject.

Verify

• cargo fmt --all -- --check passes with zero diffs
• cargo clippy --workspace --all-targets --all-features -- -D warnings passes
• cargo nextest run --workspace (or cargo test --workspace) passes with zero failures
• cargo deny check passes (licenses, advisories, duplicates) for any crate going to production
• No new unsafe without // SAFETY: comment