Agentic Routing for the Enterprise
Why most routing layers fail at scale — and the three failure modes we encountered in our first 12 deployments.
A routing layer that sends everything to GPT-4o works fine in a demo. It falls apart at 10,000 requests per hour with a cost ceiling and a 99.9% uptime SLA. We know because we watched it happen.
This is the reference architecture we've used across 12 enterprise deployments. It's not a product pitch. Every component has survived real traffic.
The three things that break first
Latency spikes — one of your upstream model providers degrades for 20 minutes. Latency doubles. Without observability, you don't know why.
Retry storms — a rate limit triggers a naive retry. Within seconds, you've multiplied the problem by your retry count and burned your cost budget for the hour.
Hallucination cascades — a slow model returns a partial response. A downstream tool call gets corrupted input. The synthesis layer treats it as fact.
These aren't edge cases. In our first three deployments, at least one of these happened within the first two weeks.
The routing layer
The router sits between the user request and the model fleet. Before anything else, it classifies the request by complexity. Not by rules — by a lightweight model that runs in 8–40ms and outputs an intent class.
Intent classes map to models. Simple extractions go to DeepSeek V3. Classification goes to GPT-4o Mini. Complex reasoning goes to GPT-4o. The savings on routine tasks: 10–40× versus routing everything to GPT-4o.
This is the part most teams skip. They hard-code rules for what goes where, and the rules work for the 60% of requests that look like what they tested. They fail silently for the other 40%.
Fallback chains
A fallback chain is a prioritized list of models with health thresholds. The router monitors error rate and p95 latency per model over a 5-minute sliding window. If either crosses the threshold, the circuit opens and traffic routes to the next model preemptively.
The user never hits a degraded endpoint. This is the distinction that matters: pre-emptive re-route, not retry logic. Retry logic reacts to errors that have already happened. Circuit breaking prevents them.
Workload identity
Every request carries a SPIFFE SVID — an X.509 certificate that encodes the calling service's cryptographic identity. The inference server validates it before processing. Downstream services validate it before accepting the response.
This sounds like security theater until you need it. Then it matters for every request: audit logging, downstream access control, upstream mTLS.
What the observability looks like
Every routing decision emits a span: intent class, selected model, fallback state, latency, error rate. These flow to ClickHouse via OTLP.
The latency budget we target per task type: 200ms for classification, 2000ms for inference, 500ms for tool calls, 200ms for synthesis. Total: under 5 seconds p95 for complex multi-step tasks.
Three things we got wrong
Rules-based intent classification worked for 60% of requests. It failed silently for the other 40%. We switched to a model-based classifier. It catches the edge cases that rules miss.
We didn't monitor fallback hit rate. A fallback hit rate above 2% turned out to be an early signal that a primary model was degrading — hours before it caused user-visible errors.
We treated the routing layer as infrastructure that couldn't fail. If it goes down, nothing routes anywhere. The fix: stateless design backed by Redis with three replicas. The router fails independently of the model fleet.