SCX Router:
Choosing the Right Model Mid-Conversation

An interactive guide to streaming zero-shot model selection, decoder-KV caching, and policy-aware routing.

Authors

  • Ihor Stepanov1, 
  • Aleksandr Smechov2, 
  • Mykhailo Shtopko1, 
  • Dmytro Vodianytskyi1, 
  • Oleksandr Lukashov1

Affiliations

  1. Knowledgator
  2. SCX.ai Holdings Limited

Published

Sep. 02, 2026

PDF

The model choice is part of the request

An LLM request does not arrive with a model already attached. A user may need code execution, a long context window, low latency, a specific region, or a careful reasoning pass. The available endpoints differ on every one of those axes.

SCX Router treats model choice as a streaming classification problem. Given the request and a roster of natural-language candidate labels, it scores each candidate without generating an answer. A separate application policy then removes ineligible endpoints and chooses among the survivors.

The separation that matters

The classifier predicts signals. The deployment policy enforces context, tools, safety, privacy, sovereignty, cost, and cache rules. A high score cannot override a hard constraint.

One request, several decisions
zero-shot / streaming

Route the turn.
Keep the context.

A decoder-KV classifier scores candidate labels without generating a route token. Policy chooses only from eligible endpoints.

01 / turnnew request
→
02 / cachedialogue KV
→
03 / routepolicy winner
SCX ROUTER / 0.6Blabels stay transient
SCX Router turns a dialogue turn into model scores, task attributes, and a policy decision. The label text can change with the candidate roster.

The paper introduces a roughly 0.6B-parameter GLiClass checkpoint built on a Qwen3 decoder, together with a real-world task ontology and a cache-aware routing design. This article makes those moving parts visible without pretending that a toy demo is the released checkpoint.

Scores are signals, not destinations

The router receives a request, conversation history, and a set of candidate labels such as “fast coding model” or “long-context reasoning model.” For each label mm, it returns a logit and a probability-like suitability score:

Q(m∣xt,h<t)=σ(sm(xt,h<t;Mt)).Q(m\mid x_t,h_{<t})=\sigma\left(s_m(x_t,h_{<t};\mathcal{M}_t)\right).

This is multi-label classification. Several candidates may be suitable, so the scores are not forced to sum to one. The same label-conditioned interface can also predict task type, difficulty, reasoning mode, expected output length, or a custom taxonomy.

Route a request by evidence
Choose a task and toggle hard constraints. The bars are an illustrative classifier output, not a call to the released checkpoint. Policy filtering happens after semantic scoring.

A new endpoint name can be scored because labels are inputs rather than fixed output neurons. That is useful, but it is not automatic calibration. Semantic compatibility does not replace outcome data for measuring whether a new endpoint actually performs well.

Why multi-label instead of one softmax?

A roster can contain multiple models that are all good enough. Independent scores preserve that possibility and let a policy apply a threshold, a shortlist, or a downstream cost and latency objective. Single-label auxiliary tasks, such as task type, can still use a softmax and an argmax.

Keep the dialogue, discard the labels

A stateless encoder rereads the full conversation whenever a new turn arrives. SCX Router instead uses a causal decoder with a persistent text-only key–value cache.

On a new turn, only the newly appended request tokens extend the persistent cache. Candidate labels are then evaluated as a transient suffix. Their keys and values are discarded after scoring, so changing the roster does not contaminate the next conversation turn.

(Htx,Kt,Vt)=Dθ(Δct;Kt−1,Vt−1),Htℓ=Dθ(q(L);Kt,Vt).(H_t^x,K_t,V_t)=D_\theta(\Delta c_t;K_{t-1},V_{t-1}), \qquad H_t^\ell=D_\theta(q(\mathcal{L});K_t,V_t).
Persistent context, transient candidates
Add dialogue turns and change the candidate roster. The cached path reuses earlier context states; label tokens are scored and then released.

The cache saves recomputation, not every routing cost. The transient labels still attend to retained history, and a large roster still has to be scored. The router’s cache is also separate from the selected generation model’s prompt cache. Switching endpoints may require replaying history downstream.

Streaming is a contract

Cache update and classification trigger are separate. A turn can extend the persistent context without producing a route until a delimiter, token count, or other trigger fires.

Give routing a vocabulary for real work

A label like “hard task” is too vague to route a production request. SCX Router organizes intent as a hierarchy:

family→task type→routable subtype.\text{family}\rightarrow\text{task type}\rightarrow\text{routable subtype}.

The paper defines 23 families, 115 task types, and 345 subtypes. An orthogonal domain axis adds 30 reusable contexts. Cross-cutting dimensions describe reasoning level, risk, freshness, interaction mode, modality, output mode, route kind, and instruction complexity.

Explore the routing ontology
Select a family, then inspect task types and subtypes. Domains stay orthogonal so “summarize” can be reused across a clinical handoff, a code review, or a policy memo.

This separation prevents a combinatorial label list. “Summarize a clinical handoff” and “summarize a deployment incident” share an operation, while their domain and risk signals remain available to the policy.

Why boundaries matter

The paper calls out confusable categories such as analysis, evaluation, critique, verification, and problem solving. Explicit definitions make the ontology more useful for supervision and make failure analysis more honest.

The last step is a policy decision

Model suitability alone is not a route. First construct the eligible set Et\mathcal{E}_t by enforcing context length, tools, modalities, safety, privacy, residency, and capacity constraints.

Only then compare utility. The paper’s illustrative selector combines normalized performance and cost:

Ut(m)=αUperf(m)+(1−α)Ucost(m).U_t(m)=\alpha U_{\mathrm{perf}}(m)+(1-\alpha)U_{\mathrm{cost}}(m).

For a cache-warm conversation, switching models can also replay history at the new endpoint’s input price. The next-request cost depends on cached tokens, new tokens, expected output length, and whether the current model’s cache remains reusable.

Move the policy along the quality–cost frontier
Adjust the performance weight and eligibility constraints. The selected endpoint changes because policy preferences and hard filters change, even when the classifier scores stay fixed.

This is why a router should expose interpretable signals instead of burying product policy inside model weights. Prices, latency, endpoint versions, and sovereignty rules can change without retraining the semantic classifier.

Never trade away eligibility

Cost savings or a high predicted score cannot compensate for a model that lacks the required tool, context, region, privacy, or safety capability.

What the paper actually shows

The released checkpoint reports macro F1 of 0.759 for eight-candidate model suitability, 0.837 for 28-way task type, 0.789 for five-level difficulty, 0.897 for reasoning mode, and 0.788 for expected output length. These are classification metrics, not a single end-to-end quality score.

On the paper’s selected 1,000-task subset, top-1 routing scores 0.707 compared with 0.696 for the strongest fixed endpoint. The result is mixed across datasets: routing helps where candidate outcomes disagree and can lose where a fixed model is already strong.

Read the result by benchmark, not by slogan
The selected-subset values from the paper show both gains and losses. A router is useful when task semantics predict meaningful model specialization.
Missing is not failure

The expanded eleven-endpoint collection has unequal coverage. An unobserved task–model outcome is missing data, not a negative label. Comparisons need a paired intersection or an explicit missing-outcome model.

The paper distinguishes released paths, implemented components, and proposed compositions. Direct endpoint routing is released and evaluated. Hard attribute-profile routing is implemented without an end-to-end result. Posterior, hybrid, adaptive, and planner–worker variants remain proposals or future evaluation targets.

The routing loop

SCX Router makes four boundaries explicit:

  1. Classify the request against dynamic natural-language labels.
  2. Reuse the dialogue cache while evaluating transient candidates.
  3. Describe the task with an ontology that separates intent from domain and policy dimensions.
  4. Decide with hard eligibility filters and a cost-aware utility.

The central engineering lesson is modest and useful: model selection is not a single argmax. It is a stateful decision made from semantic evidence, empirical outcomes, operational constraints, and cache state.

The paper’s results support targeted gains over fixed baselines, not a universal claim that routing always wins. The next credible step is paired, versioned evaluation that measures quality, cost, latency, calibration, drift, and regret under the same candidate roster.

Read the source

Ihor Stepanov, Aleksandr Smechov, Mykhailo Shtopko, Dmytro Vodianytskyi, and Oleksandr Lukashov. “SCX Router: Streaming Zero-Shot Model Selection with a Decoder-KV Classifier and a Real-World Task Ontology.” arXiv:2609.02292, 2026. Paper · Checkpoint · Code