Routing Architecture Guide

Overview

The router (src/core/router/) selects among deployments — concrete provider+model pairs registered via Router::adddeployment / Router::setmodellist — using one of 7 strategies. There is no per-strategy router struct, no Router trait, and no createrouter factory: strategy dispatch is a match on the RoutingStrategy enum calling free functions in src/core/router/strategy_impl.rs.

Key Design Principles

• Snapshot isolation: deployments, the model index, and aliases live in an immutable

RoutingSnapshot, published through ArcSwap (src/core/router/unified.rs:61,308). Readers load one generation lock-free; writers clone-modify-store under a parking_lot::Mutex that only serializes writers (unified.rs:312,379-398).

• Atomic deployment state: per-deployment runtime state (DeploymentState) is plain

atomics with Relaxed ordering (deployment.rs:210-252); RoundRobin uses a DashMap<String, AtomicUsize> of per-model counters (unified.rs:321).

• Health-aware: candidates are filtered by cooldown, health status, parallel limits,

and RPM/TPM limits before a strategy picks among them.

• Fallback chains: the built-in execution path tries the model-keyed General

fallback list after retries are exhausted. Typed context-window, content-policy, and rate-limit lists require explicit caller lookup/wiring today.

Selection Flow

Entry point Router::selectdeploymentlease (selection.rs:95) delegates to selectdeploymentmatching (selection.rs:226). The real flow:

1. Load the current snapshot; resolve model aliases via resolvemodelname

(max MAXALIASHOPS = 16 hops, unified.rs:26).

1. Look up the resolved model in snapshot.model_index to get candidate DeploymentIds.
2. One filter pass builds Vec<RoutingContext> (strategy_impl.rs:17), skipping

deployments that are in cooldown (isincooldown), unhealthy (ishealthy), at their maxparallelrequests limit, or at their rpmlimit / tpmlimit (selection.rs:287-344). Each context copies weight, priority, activerequests, tpmcurrent/tpmlimit, rpmcurrent/rpmlimit, avglatencyus.

1. Router::selectfromrouting_contexts dispatches on the configured strategy

(selection.rs:195-224):

match strategy {
    RoutingStrategy::SimpleShuffle => strategy_impl::weighted_random_from_context(routing_contexts),
    RoutingStrategy::LeastBusy     => strategy_impl::least_busy_from_context(routing_contexts),
    RoutingStrategy::UsageBased    => strategy_impl::lowest_usage_from_context(routing_contexts),
    RoutingStrategy::LatencyBased  => strategy_impl::lowest_latency_from_context(routing_contexts),
    RoutingStrategy::PriorityBased => strategy_impl::lowest_priority_from_context(routing_contexts),
    RoutingStrategy::RateLimitAware => strategy_impl::rate_limit_aware_from_context(routing_contexts),
    RoutingStrategy::RoundRobin    => strategy_impl::round_robin_from_context(
        model_name, routing_contexts, round_robin_counters),
}

1. The winner is reserved by tryreservedeployment — an atomic increment of

activerequests (CAS loop when maxparallel_requests is set). If another caller wins the last slot, that candidate is removed from the contexts and selection retries (selection.rs:378-415).

1. Returns a DeploymentLease; dropping it decrements active_requests (RAII release,

selection.rs:64-70). The deprecated ID-returning select_deployment still exists but converts the lease to an ID without release-on-drop.

Routing Strategies

Enum RoutingStrategy (src/core/router/config.rs:22-39) serializes as snakecase (simpleshuffle, roundrobin, leastbusy, latencybased, prioritybased, usagebased, ratelimitaware); PriorityBased also accepts the serde alias "costbased".

1. SimpleShuffle (runtime default)

Weighted random selection: draws a point in 0..totalweight and walks candidates until the cumulative weight covers it; uniform random when total weight is 0 (weightedrandomfromcontext, strategy_impl.rs:56-85). Weights come from DeploymentConfig.weight (default 1).

Use when: general traffic where deployments have different capacity.

2. RoundRobin (gateway YAML default)

Per-model counter in roundrobincounters: DashMap<String, AtomicUsize> cycles through candidate order (roundrobinfromcontext, strategyimpl.rs:247-274). Note the defaults diverge: GatewayRouterConfig defaults to round_robin (src/config/models/router.rs:48-50) while runtime RouterConfig defaults to SimpleShuffle.

Use when: predictable distribution needed, debugging provider issues.

3. LeastBusy

Single pass for the fewest activerequests; ties are broken randomly with reservoir sampling so equal-load deployments share traffic (leastbusyfromcontext, strategy_impl.rs:88-116).

Use when: high concurrency, need to prevent deployment overload.

4. LatencyBased

Lowest avglatencyus wins. Deployments reporting 0 latency (no success yet) inherit the average of non-zero latencies in the pool, so new deployments neither always win nor starve (lowestlatencyfromcontext, strategyimpl.rs:145-182). Latency is recorded per request via recordsuccess into DeploymentState.avglatency_us.

Use when: response time is critical, deployments have varying latencies.

5. PriorityBased

Lowest priority value wins (lower = higher priority; u32, default 0). This is tier ordering, not cost — despite the legacy "costbased" serde alias (lowestpriorityfromcontext, strategy_impl.rs:185-203).

Use when: primary/backup tiering, e.g. production vs backup deployments (see gateway.yaml.example provider priority).

6. UsageBased

Lowest TPM usage percentage wins: (tpmcurrent * 100) / tpmlimit; deployments with no limit count as 0% usage (lowestusagefromcontext, strategyimpl.rs:119-142).

Use when: spreading load relative to token budgets, avoiding TPM exhaustion. Requires tpm to be configured on deployments to be meaningful.

7. RateLimitAware

Picks the deployment furthest from its rate limits: score is the minimum of remaining TPM fraction and remaining RPM fraction; unlimited axes score 1.0 (ratelimitawarefromcontext, strategy_impl.rs:206-241).

Use when: high request volume against deployments with strict TPM/RPM limits.

References

• [reference/health-and-fallbacks.md](reference/health-and-fallbacks.md) — health states, cooldown/circuit-breaker mechanics, probe tasks, and the FallbackConfig execution flow
• [reference/router-configuration.md](reference/router-configuration.md) — real router: YAML keys, gateway-to-runtime config mapping, and per-provider routing knobs
• [reference/performance-and-practices.md](reference/performance-and-practices.md) — verified complexity table and routing best practices