Three-Stage Architecture for Model Routing and Full-Stack Classification Above Claude Code

01No commercial router is ideal for coding-specific routing.

RouteLLM (LMSYS) provides the strongest open-source primitives — its matrix-factorisation router achieves 95% of GPT-4 performance using only 26% strong-model calls on MT-Bench, and Anthropic's preference-data preprocessing trick generalises to other model pairs without retraining. Martian and NotDiamond are commercial black boxes optimised for general queries; OpenRouter is a marketplace, not a learned router. Coding tasks have very different difficulty signals than general LLM-arena prompts, so any commercial router must be benchmarked on Optivia's own logs before being trusted.

02The MVP gateway is LiteLLM, not Portkey or OpenRouter.

LiteLLM is open-source (MIT/Apache 2.0), self-hostable, sustains approximately 1,000 RPS, adds only 10–20ms latency, has a unified OpenAI-compatible interface for 100+ providers, and crucially supports custom routing logic written in Python. OpenRouter charges a 5.5% fee with no infrastructure control, and Portkey's per-log pricing scales linearly with volume. Portkey is reasonable as a secondary observability layer; LiteLLM is the right primary substrate.

03Claude Code can be intercepted cleanly via ANTHROPIC_BASE_URL.

Multiple proven patterns (agentgateway, seifghazi/claude-code-proxy, lrgs/claude-code-proxy, fuergaosi233/claude-code-proxy) demonstrate that pointing Claude Code at a localhost proxy lets you see every prompt, tool call, and file read in real time, rewrite the system prompt, route subagents to different models, and inject Optivia's classification and synthesis pipeline upstream of Anthropic's API — without forking Claude Code itself. Anthropic's Agent SDK exposes the same harness programmatically with hooks, settingSources, and subagent control.

04Coding-task difficulty has well-validated signal sources.

SWE-bench Verified's annotator-time tiers (under 15 minutes, 15 minutes to 1 hour, 1 to 4 hours, and over 4 hours) and Multi-SWE-bench's Easy/Medium/Hard categorisation map directly to Optivia's complexity score. The empirical signal: lines changed scales 11× from Easy to Hard; files modified scales 2×. These map cleanly to features Optivia can extract at prompt time.

05ModernBERT, not DistilBERT, is the right Stage 2 classifier.

ModernBERT has an 8K context window (versus BERT's 512), is trained on 2T tokens including code, runs 2–4× faster than older BERT, and outperforms the original BERT by approximately 3% on classification while training 3× faster. This makes it the clear choice for the coding-task classifier that must handle long prompts with attached file context.

06Clarifying questions should use Expected Information Gain, not zero-shot prompting.

The BED-LLM and Active Task Disambiguation papers prove that maximising expected information gain over a sampled candidate-question set substantially outperforms naive "ask the LLM what to clarify." For Stage 1, a structured-output LLM call is acceptable. Stage 2 explicitly scores candidates by EIG over the model's posterior — EIG scoring alone (without learned question generation) delivers most of the quality gain.

07The economics of routing are dominated by Anthropic prompt caching, not raw model selection.

Anthropic prompt caching gives 90% read-cost reduction at the cost of a 25% write premium, breaking even after two cache hits. For Optivia, the master-prompt prefix should be cache-controlled, making the marginal cost of an extra Haiku or Sonnet call collapse substantially. The router design must account for this — naive cost models that ignore caching will misroute.

08The Sonnet–Opus gap on coding has nearly closed in the current generation.

As of early 2026, Sonnet 4.6 scores 79.6% and Opus 4.6 scores 80.8% on SWE-bench Verified — a 1.2-point quality gap for approximately 1.7× cost ($3/$15 vs $5/$25 per million tokens). Haiku 4.5 sits at 73.3% at $1/$5. Optivia's router therefore has only three rational tiers: Haiku for trivial, Sonnet for default, Opus for genuine architectural reasoning. The Sonnet/Opus threshold is the single most valuable routing prediction.

09Langfuse is the right Stage 1 observability platform.

Open-source, MIT-licensed, OpenTelemetry-native, span-level tracing for multi-step LangGraph workflows, free up to 50K events per month on cloud, and cleanly self-hostable on Postgres. Helicone's proxy approach is great for cost dashboards but sees only HTTP traffic (no span-level visibility into LangGraph nodes). LangSmith is too LangChain-locked for a router product.

10Stage 2 routing is a learning-to-rank problem with three heads, not one.

The literature (RouteLLM, FrugalGPT, Hybrid LLM, LLMRec, EmbedLLM, Router-R1, RouterArena) converges on this pattern: (a) predict each candidate model's win probability for the prompt, (b) predict expected output length, (c) compute expected utility under a cost constraint, and (d) pick the argmax. Optivia should ship three small heads on top of a shared encoder, not a single end-to-end "magic" router.

11The MVP stack must be Python.

The entire AI tooling ecosystem — LangGraph, LiteLLM, instructor, RouteLLM, DSPy, ModernBERT fine-tuning, Aurelio Semantic Router, HuggingFace, Langfuse SDK — is Python-first. FastAPI, Pydantic, and asyncio form the established production pattern. Use Next.js for the workflow visualisation UI (React Flow), but the backend is unambiguously Python.

12The Stage 1 → Stage 2 migration succeeds or fails on the Stage 1 schema.

Every commercial router-replacement project that has been published (Hybrid LLM, RouteLLM, Martian's RouterBench) was data-bottlenecked, not algorithm-bottlenecked. Optivia's MVP must over-instrument: every prompt logged with full classification vector, every clarification interaction logged, every master prompt logged with diff versus user prompt, every routing decision logged with alternatives considered, every Claude Code outcome logged with token counts, exit signals, file diffs, error patterns, and user feedback.

ModernBERT Task Classifier

ModernBERT-base (149M parameters) fine-tuned on the 8-class taxonomy with a regression head for the 5 scoring dimensions and 3 length-estimation outputs (multi-task learning). Loss: cross-entropy for classification, MSE for scoring, ranking loss for SWE-bench-aligned tier classification. Training recipe: 10,000 trace-derived examples + 5,000 LLM-judge-augmented examples + 2,000 hand-labelled gold examples. HuggingFace Trainer, mixed-precision, on a single A100 for approximately 30 minutes. This replaces the Stage 1 instructor-on-Haiku classifier, dropping classification cost from approximately $0.001/call to under $0.0001/call and latency from approximately 500ms to under 100ms.

Matrix-Factorisation Router

A three-head model on a shared encoder, following the consensus from the literature (RouteLLM, Hybrid LLM, EmbedLLM, LLMRec). Head 1 (win-rate prediction): for each candidate model m, predict P(model m succeeds | prompt encoding x) using a bilinear scoring function. Head 2 (output-length prediction): a small regressor predicting est_tokens_output; critical for cost-aware routing because output tokens are 5× input cost across all Anthropic models. Head 3 (fallback-needed prediction): a binary classifier predicting whether this prompt would require a retry/fallback on the cheapest viable model. Decision rule: utility(m) = P_success(x,m) − λ_cost · E[cost(x,m)], where λ_cost is per-tenant configurable, enabling cost-mode/quality-mode toggles backed by a single model.

DSPy EIG Clarifying Question Generator

Stage 2 implements the BED-LLM/BALAR/Active-Task-Disambiguation pattern explicitly. Step 1: generate k=8 plausible task interpretations from a solution-sampler. Step 2: generate m=5 clarifying questions from a fine-tuned question-generator. Step 3: score each question by EIG — for each (question, candidate-answer) pair, compute the posterior over interpretations; EIG is the expected reduction in posterior entropy, approximated via Monte Carlo. Step 4: select the question that maximises EIG, or the top two if EIG values are close. Training uses Stage 1's logged (prompt, asked-questions, answers, outcome) tuples, weighted by realised EIG.

Embedding-Based Retrieval

A new node retrieve_similar_traces is added between deep_classify_and_score and synthesize_master_prompt. It does pgvector kNN over raw_prompt_emb with filters on workspace ID, task type, and outcome.user_accepted = true, pulling the top 3 historically successful master prompts as exemplars for the synthesiser. This is RAG over Optivia's own history; exemplars from the same workspace/repository are far more useful than generic best-practices. For embeddings, voyage-code-3 is the recommended choice.

Shadow Deployment and Promotion

The Stage 2 evaluation methodology follows RouterBench/RouterArena. Hold out 20% of the trace store as a frozen evaluation set, stratified by task type and time period. For each router, compute the deferral curve: fraction of calls routed to the strong model on the x-axis, aggregate success rate on the y-axis. Promotion criterion: OptivaRouter must dominate or match the best wrapped router on the Pareto frontier across three or more cost-level slices, with statistical significance, on both the eval set and two weeks of shadow data. Shadow percentages start at 1% and ramp: 1% → 5% → 25% → 50% → 100%.

Caching Strategy

Three caching layers work together. Anthropic prompt caching: the master-prompt system preamble (approximately 1,500 tokens of coding-agent best practices, taxonomy descriptions, and output-format instructions) is marked cache_control="ephemeral" on every call. With a 5-minute TTL extended to 1 hour with regular hits, this reduces input cost by approximately 85% for the bulk of the master prompt. Optivia-managed semantic cache: Redis-backed, keyed by embedding of (raw_prompt + workspace_signature). Cosine similarity ≥ 0.97 combined with the same task type and workspace constitutes a cache hit — returns the cached master prompt, plan, and routing decision, but not the cached final output (user-acceptance of a previous outcome is not transitive). Plan cache: for repeat operations (e.g. "add a test for X" where X is parameterised), cache the workflow plan template separately from the prompt.

Cost Optimisation

Tiered model selection per subagent: within a single Claude Code session, subagents (file readers, lint runners, simple edit applicators) can be routed to Haiku via the proxy's agent-routing configuration. A long Opus session that delegates 80% of token consumption to Haiku-tier subagents can be 5× cheaper than naive Opus-everything. Output-token budget per tier: Haiku jobs are capped at 4K output tokens, Sonnet at 16K, and Opus at 32K. Capping prevents runaway costs from a misrouted prompt. Batch API usage: for nightly retraining and offline evaluation, use Anthropic's Batch API at 50% off.

Online Learning Loop

The router and classifier are retrained weekly on the previous week's traces. The pipeline runs as Modal scheduled functions: snapshot Postgres to S3-Parquet on Friday night; build training splits on Saturday morning; train on a single A100 for approximately 4 hours; validate on held-out traces; if F1 ≥ 0.85 and cost/quality metrics pass, promote the new checkpoint by updating the Postgres models table and reloading Modal endpoints. Rollback is a SQL update, not a code deploy.

Full Claude Code Integration

By Stage 3, Optivia ships all four Claude Code integration surfaces: (1) CLI wrapper — sets ANTHROPIC_BASE_URL and launches claude with the master prompt prepopulated; highest control, sees everything. (2) MCP server — users add it via claude mcp add optivia and call /mcp__optivia__plan from inside Claude Code; lower friction, does not intercept downstream traffic. (3) Skill — users run npx optivia install and Claude Code automatically invokes Optivia when it detects a vague prompt; most natural UX. (4) Hooks integration — Optivia ships a PostToolUse hook that scores Claude Code's actions and a PreToolUse hook that can block destructive operations not in the approved plan.

Reliability Patterns

Circuit breakers per model tier: trip at P95 > 500ms or error rate > 5%; auto-reset after 30 seconds of healthy traffic. Exponential backoff with jitter for all LLM calls. Fallback chain: primary model → secondary tier → HeuristicRouter (the always-available baseline). Health checks with synthetic probes every 60 seconds on all tiers. Langfuse wired at three levels: trace level (every Optivia request is one trace), span level (every LangGraph node is a span), and score level (cost_usd, wall_time_seconds, outcome_success, user_feedback, retry_count, plan_edited_by_user).