How Cloud Score routes a request

The fixed sixteen-stage pipeline every gateway request runs, what each stage decides, the three outcomes, how failover works, and which admin page configures which stage.

On this page

Prompt Optimization: optimize, review, then run

Prompt Optimization has one ON/OFF control, defaults ON, and accepts any prompt length. In Playground, Optimize prompt prepares a complete optimized prompt before answer routing, at the level you pick — Standard, Balanced or Thorough. Live counts use a real backend tokenizer. Review the original, optimized version and diff, then choose Run original or Run optimized. Our checks cannot guarantee that meaning is preserved, which is why you review the result. Editing the request invalidates its comparison. Workspace defaults are under Optimization › Settings.

Playground explicitly requests a bounded optimization and shows its recorded cost. A rejected, unavailable or unhelpful result falls back to your original without hiding its expense. Requests sent ON without a prepared proposal are optimized by structural rules only. Comparison details show the outcome, the rules applied and the measured work time. Infrastructure still consumes resources; unallocated compute keeps fully loaded savings unknown. Historical optimization expenses remain recorded and are not reset to zero.

The execution receipt distinguishes the selected prompt from the prompt actually sent for the answer, including no provider send when a cache supplies it. Optimization can have a recorded cost even when the original is selected or no answer is generated. Unknown infrastructure cost keeps fully loaded net unknown. Fewer tokens do not establish a latency or quality improvement. Analytics, ROI and Reports reuse Prompt Optimization evidence with production/demo filters and daily, weekly and monthly views.

See Playground for comparison and recovery, and Analytics for measurement coverage. Historical compression receipts do not establish choices or costs that were never recorded.

What it is

Every request your application sends to Cloud Score, whether to the native endpoint POST /v1/requests or the OpenAI-compatible POST /v1/chat/completions, passes through one fixed sequence of sixteen stages. Each stage makes one decision, appends one event to the request's decision trace, and hands the request to the next stage. Every path ends at Record.

Guard inscan the prompt

Prompt Optimizationapply the reviewed choice

Analyzecount tokens

Classifytask type

Matchcapable models

Estimateprice each one

Policyrules and budget

Optimizewrite the plan

Resolvecache lookup

Dedupcoalesce twins

Selectscore and pick

output_budgetno board label

Attempt / Executecall the provider

Guard outscan the answer

Validatenon-empty, JSON

Recordpersist and end

The rail uses the station names the Route board shows on Playground (live) and in the Observability Request drawer (recorded). One stage, output_budget, has no station label on the Route board; the board shows the raw stage name for any stage it has no label for. The Analyze station renders with an empty summary line.

Errors travel with the request.

A stage that refuses or fails does not throw; it writes an error type into the request and the graph routes straight to Record. On the native endpoint a blocked or failed request still answers HTTP 200 with the status inside the envelope by default; the OpenAI-compatible endpoint answers 403 for a block.

The sixteen stages

One row per stage, in the order the graph runs them.
Stage	What it decides	Where it can go next
`Guard in` guard_input	Scans every text part of the prompt against the API key's security profile (six categories, each set to off, log, redact, or block); a redaction rewrites the prompt before anything downstream sees it.	To `Record` on a block; otherwise to `Prompt Optimization`. With no profile bound the station reads `off`.
`Prompt Optimization` prepare_prompt	Verifies and applies the reviewed original or optimized proposal before routing. Playground's earlier Optimize prompt action prepares a complete hosted rewrite; this execution stage does not change the reviewed candidate again. ON requests without a prepared proposal use the deterministic engine; OFF preserves the original.	To `Record` when proposal verification fails; otherwise to `Analyze`.
`Analyze` analyze	Counts the input tokens, sets the expected output length (the request's max_tokens, or 500), and notes whether the prompt has images or asks for JSON. No model call.	To `Classify`, always.
`Classify` classify	Decides the task type, a complexity from 1 to 5, the capabilities the request needs, and where that came from: a stored earlier result (cache), one call to the classifier model (llm), or built-in keyword rules (heuristic).	To `Match`, always. Routing never blocks on the classifier; any failure falls back to the rules.
`Match` match	Builds the candidate set: every enabled model that has the required capabilities and a context window large enough for the estimated input plus expected output.	To `Record` as no_candidates when nothing matches; otherwise to `Estimate`.
`Estimate` estimate	Prices each candidate from its input and output price per million tokens and the token estimates. The station shows the cheapest figure.	To `Policy`.
`Policy` policy	Runs every enabled policy in priority order, then the budget gate. A policy block or an exhausted budget ends the request before any provider is called. Cost conditions are judged against the cheapest candidate.	To `Record` on a block; otherwise to `Optimize`.
`Optimize` optimize	Writes the request's optimization plan: each mechanism's mode (off, observe, or auto), the reference model savings are measured against, and the learned routing adjustments. Never fails the request.	To `Resolve`, always.
`Resolve` resolve	Looks the request up in the response cache when a policy allows it: first by exact fingerprint, then by meaning when the policy's cache mode is semantic. A hit is served at cost $0. Every miss records why.	To `Guard out` on a hit; otherwise to `Dedup`.
`Dedup` dedup	Coalesces identical requests in flight on the same API key: the first is the leader, a follower waits and is served the leader's answer. Waits only in auto; observe (the default) records what would have been coalesced.	To `Guard out` when a follower is served; otherwise to `Select`.
`Select` select	Filters the candidates (failed models, provider allow and deny lists, a pinned model, a cost cap), resolves the scoring weights, scores each survivor on cost, latency, and quality, applies learned-signal penalties and the affinity bonus, and picks the winner plus ranked alternates.	To `Record` as no_candidates when the filters leave nothing; otherwise to `output_budget`.
`output_budget`	Predicts how long the answer will be from this workspace's history for the task type and, in auto, caps max_tokens accordingly. observe (the default) records the cap it would have set.	To `Execute`, always.
`Attempt` / `Execute` execute	Calls the selected model with the provider's stored credentials, retries transient errors on the same model, and accumulates tokens and cost across every billed call. Each provider call is one Attempt station; Execute is the settled result.	To `Guard out` on success; back to `Select` on a hard provider error with attempts left (failover); to `Record` when attempts are exhausted; back to `output_budget` once if an applied cap truncated the answer.
`Guard out` guard_output	Scans the answer, or the cached text on a hit, under the serving key's profile. A block withholds the answer; a redact replaces it with one redacted chunk. This is the one point that releases buffered output.	To `Record` on a block or when a cache or dedup hit was served; otherwise to `Validate`.
`Validate` validate	Checks the answer is non-empty and, when the request set a response_format, that it is valid JSON (or matches the schema). Allows exactly one re-prompt.	Back to `Execute` when invalid and the re-prompt is still available; otherwise to `Record`.
`Record` record	Persists the request row: status, classification, decision, the full trace, usage, cost, latency, cache and dedup verdicts, and the optimization plan. Writes the response-cache entry for an eligible completed miss.	The end. Every request reaches this stage.

Resolve runs after Policy, so a request that a policy or budget blocks can never be answered from the cache. A served cache hit still passes Guard outunder the serving key's profile; a hit that Guard out blocks claims no savings.

Three outcomes

The Record station, the Playground status badge, and the Gateway status column in Observability all show one of three words.

Status	When	Error types that map to it
`completed`	A model answered and passed Guard out and validation, or a cache or dedup hit was served at cost $0.	None.
`blocked`	Refused before spending (a policy block, an exhausted budget, a Guard in block), or an answer withheld by Guard out.	blocked, security_blocked
`failed`	Cloud Score tried and could not deliver.	no_candidates, provider_error, validation_failed, deadline_exceeded

The error message travels in the envelope and is shown on the Playground error line and in Observability. For security_blocked the Playground shows a guardrail verdict chip, such as Blocked before sending: PCI, instead of the error line.
Error type	Message you see	Produced by
no_candidates	`no enabled model matches the required capabilities and context window`	`Match`
no_candidates	`no candidate models remain after constraints, policies, and failover exclusions`	`Select`
blocked	`blocked by policy '<name>'` unless the policy sets its own message	`Policy`
blocked	`budget exhausted: <scope> <scope_value> at <utilization_pct>% (model_spend)` from the default budget gate	`Policy`
security_blocked	The profile's blocked reason, else `blocked by AI Security`	`Guard in` or `Guard out`
provider_error	`<model_key>: <scrubbed provider error>`, at most 300 characters with secrets removed	`Execute`
validation_failed	The validator's reason, for example `empty response`, with the first 500 characters of the answer	`Validate`
deadline_exceeded	`<model_key>: request deadline exhausted before another attempt could start`	`Execute`

Retries and failover

Two independent budgets bound every request: 3 billed provider calls (attempts), and 6 setup failures that never reach a provider and bill nothing. Either one exhausted ends the request as provider_error.

Situation	What Cloud Score does	Counts as an attempt
Transient error on the selected model (rate limit, provider unavailable, timeout)	Retries the same model up to 2 times, after 0.5 s then 1.0 s, or after the provider's retry-after hint capped at 30 s.	No. Retries stay inside the current attempt.
Hard error on the selected model (authentication, invalid request, content filtered)	No retry. The model is excluded and the graph returns to `Select`, which re-scores the remaining candidates. Scores are relative to who is left, so the normalization is recomputed.	Yes. The next model's call is a new attempt.
Answer fails validation	One re-prompt to the same model, then `Record` as validation_failed.	Yes.
An applied output cap truncated the answer	One rollback through `output_budget`, then `Execute` runs again.	Yes.
Fewer than 5 s left on the request deadline	No further billed call starts. The request ends deadline_exceeded and deliberately does not fail over.	Not started.
A pinned model (model_override, or a concrete model name on the OpenAI-compatible endpoint)	The filter chain leaves one candidate, so a hard error has nothing to fail over to. The request fails honestly.	Yes, for the one call made.

What a failure does to the next request.

Every counted failure is written to a cooldown table. Three failures within 120 s open a cooldown of 30 s that doubles with each further failure, capped at 600 s; a rate limit with a retry-after hint opens one immediately. Open cooldowns reach the next request's Select as exclusions through Routing adjustments, and act only when that mechanism is in auto (its default is observe). An exclusion is never allowed to empty the candidate set.

Streaming and a failed attempt.

On the native endpoint with streaming on (Stream the answer in the Playground), text streams live, so chunks already shown from a failed attempt are retracted. The Playground shows Attempt failed. Rerouting… or Output failed validation. Retrying… above the answer until the next attempt streams. The OpenAI-compatible endpoint buffers until Guard out and Validate settle. A security profile that enforces output (redact or block on any output category) also switches live streaming off for that key.

In the product

You watch the pipeline on Playground (the Route board fills in as the request runs) and re-read it in Observability (the Request drawer shows the recorded board). The Optimization decision card, on Playground and in Observability, shows the plan Optimize wrote and the candidates Select scored; the full request page adds the Savings receipt.

Which admin page changes which stage's decision.
Stage	Where you meet it	What the page controls
`Guard in`, `Guard out`	`Gateway` › `Guardrails` (/governance/guardrails); `Admin` › `API keys` (/admin/keys)	Security profiles (categories and actions, locale, Confidential dictionary, fail mode, Scan Preview); which profile each key binds.
`Analyze`	Nowhere.	Nothing is configurable.
`Classify`	`Admin` › `Organization` (/admin/organization); `Providers`	The classifier provider, model, and timeout; the connected OpenAI provider the default classifier resolves to.
`Match`	`Models`; `Providers`	The enabled switch, capabilities, and context window per model; which providers are connected.
`Estimate`	`Models`	Input and output price per million tokens from the synced catalog.
`Policy`	`Policies`; `Admin` › `API keys` (/admin/keys); `Budgets`	IF conditions and THEN actions (block, prefer a tier, allow or deny providers, force a model, cap cost, cache settings); a key's monthly budget; scoped budgets with hard actions.
`Optimize`	`Optimization settings` › `Settings`	The mode per mechanism (off, observe, auto), the confidence bar, the reference model, and each mechanism's knobs.
`Resolve`	`Policies`; `Optimization settings` › `Settings` (`Exact response cache`, `Semantic response cache`, and the `Response cache` section)	Turning caching on per scope (cache mode, TTL, scope, similarity bar); the workspace-level kill switch and confidence gate.
`Dedup`	`Optimization settings` › `Settings` (`In-flight dedup`)	Mode, maximum wait, leader TTL.
`Prompt Optimization`	`Playground` › `Prompt Optimization`, or the request Prompt Optimization ON/OFF control	Enabled by default with a workspace setting and request override. Any prompt length is eligible. Explicit Playground preparation requests a Balanced rewrite and requires review and selection; requests without a proposal use deterministic rules without rewrite inference. OFF preserves the original.
`Select`	`Admin` › `Organization`; `Policies`; `Optimization settings` › `Settings` (`Routing adjustments`); `Optimization settings` › `Routing signals`; `Admin` › `Developer`	The default scoring weights; policy weights and filters; learned-signal penalties; the strategy your client code sends.
`output_budget`	`Optimization settings` › `Settings` (`Output budget`)	Mode, floor tokens, headroom, minimum samples, model-cap enforcement, rollbacks.
`Attempt` / `Execute`	`Providers`; `Models`	Provider credentials (stored encrypted) and health; the model's max output tokens, which the hard clamp uses.
`Validate`	Nowhere.	Nothing in the admin app; the caller's response_format drives it.
`Record`	`Observability`; `Usage & cost`; `Optimizations`; `Overview`	Read-only views of the recorded row: the trace, decision card and savings receipt; spend and attribution; savings by mechanism; headline tiles.

To read why a request picked its model:

Click Observability in the sidebar, then click the request's row.
The Request drawer opens with the recorded Route board.
Read the stations left to right.
Classify shows the task type and its source, Match how many models qualified, Estimate the cheapest price, Policy how many policies ran or blocked, Resolve the cache verdict, Select the winner.
Expand the Select detail.
It lists the weights used, the strategy if your client sent one, the reasons, and each candidate with its estimated cost and the difference against the winner.
Check the filtered counts.
The five gates are excluded_failover, allowed_provider, denied_provider, model_override, and cost_cap; the count next to each says which gate removed a model.

Defaults that decide whether a mechanism acts.

Exact response cache and Semantic response cache default to auto. In-flight dedup, Output budget, and Routing adjustments default to observe: their stations and plan rows record what would have happened, and nothing changes for the caller. Prompt Optimization defaults ON through its separate workspace setting, supports a request override, and does not require a compression policy. Context caching is designed but not live.

Example

A provider outage during a request (failover)

What you do: Send a prompt from Playground with Routing on auto and Stream the answer on, as in Your first request. The provider behind the winning model answers 503 to every call.
What Cloud Score does: Select picks the winner. Execute calls it, retries after 0.5 s, retries after 1.0 s, then excludes the model and returns to Select, which re-scores the survivors. The new winner answers. The request ends completedwith attempts 2, and the failing provider's cooldown counter moves up.
What you see: In Details › Routing, an Attempt station drops below the rail on a red connector with a summary ending in failed, the amber line Attempt failed. Rerouting… shows above the answer, then a second Select station appears followed by Attempt, Execute, Guard out, Validate, and Record reading completed. Attempts in Details › Usage reads 2.

Selectfirst winner

Attempt503, three times

Selectre-scored survivors

Attemptsecond model

Executesettled cost

Recordcompleted

An answer that is not the JSON you asked for (validation re-prompt)

What you do: Your application sends a request with a response_format of type json_schema and a schema.
What Cloud Score does: Analyze notes that JSON was requested, so Match keeps only json_mode models. The first answer wraps the JSON in prose, so Validate fails it and sends one re-prompt to the same model (attempt 2). The second answer parses and matches the schema.
What you see: Validate reads retry, then a second Attempt and Execute, then Validate reads ok and Record reads completed. Had the second answer failed too, the status would be failed with the error type validation_failed and the first 500 characters of the answer.