Cut Cost Without Losing Quality

Price the cost of quality on one real prompt, write the two policies that pay for themselves, promote an optimization mechanism only when the observe counts justify it, then read the ledgers that prove it worked.

On this page

Prompt Optimization: optimize, review, then run

Prompt Optimization has one ON/OFF control, defaults ON, and accepts any prompt length. In Playground, Optimize prompt prepares a complete optimized prompt before answer routing, at the level you pick — Standard, Balanced or Thorough. Live counts use a real backend tokenizer. Review the original, optimized version and diff, then choose Run original or Run optimized. Our checks cannot guarantee that meaning is preserved, which is why you review the result. Editing the request invalidates its comparison. Workspace defaults are under Optimization › Settings.

Playground explicitly requests a bounded optimization and shows its recorded cost. A rejected, unavailable or unhelpful result falls back to your original without hiding its expense. Requests sent ON without a prepared proposal are optimized by structural rules only. Comparison details show the outcome, the rules applied and the measured work time. Infrastructure still consumes resources; unallocated compute keeps fully loaded savings unknown. Historical optimization expenses remain recorded and are not reset to zero.

The execution receipt distinguishes the selected prompt from the prompt actually sent for the answer, including no provider send when a cache supplies it. Optimization can have a recorded cost even when the original is selected or no answer is generated. Unknown infrastructure cost keeps fully loaded net unknown. Fewer tokens do not establish a latency or quality improvement. Analytics, ROI and Reports reuse Prompt Optimization evidence with production/demo filters and daily, weekly and monthly views.

See Playground for comparison and recovery, and Analytics for measurement coverage. Historical compression receipts do not establish choices or costs that were never recorded.

The path

Playgroundprice the trade-off

Policiescheap tasks, cached repeats

Optimization settingsobserve, then promote

Optimizationsthe savings

By mechanismcache savings

Cost falls in three places and they stack: routing a task to a cheaper model that can still do it, serving repeats from the response cache instead of a provider, and letting the optimization engine apply a mechanism once you have watched it in observe mode. Each stop below does one of those on real traffic; the last stop is where the effect becomes visible.

Measure

one prompt, two strategies

Bias

a prefer_tier cost policy

Reuse

a cache_mode exact policy

Promote

observe counts first

Verify

three ledgers

Every savings figure here is an estimate.

Cloud Score marks them (est.) wherever they appear. Only cost is measured: provider-reported tokens at the price card of the model that ran. A figure with no underlying data renders — with a cause, never a zero. See Costs, savings, and estimates.

Price the trade-off in the Playground

Before changing anything, find out what quality actually costs on one prompt you care about. The Cost and Intelligence chips are the two ends of the scoring range, and the decision card prices every candidate that lost.

Open Playground and paste a key into Platform API key.
Switch on Save for session to keep it for this tab.
Type a representative prompt into Message.
Under Advanced options, leave Quality on Auto and Max cost (USD) and Required capabilities empty. Quality maps to a weight preset of its own, and the other two drop candidates before scoring, so any of them would change the comparison you are making.
Under Routing click cost, then Send.
The caption reads cheapest within 8% of the best; the button reads Running… while it runs. This is a real request: routed, billed, recorded.
Open Details › Routing and read the decision summary under the stations.
It names the selected model, a badge score N.NNNN, the strategy, a band N% badge, the execution profile when one was chosen, and the selection reasons.
Click intelligence (caption the best model, cheapest among ties), then Send again.
Same prompt, same candidates, different weights.
Open Details › Decision and read the Candidates table of the Optimization decision card.
The winner carries a selected badge. Every other row shows what it would have cost instead, green when cheaper and amber when pricier.

Column	What it tells you
`Δ vs selected`	The signed money difference against the winner, `±$0.00` when there is none. That is the price of the quality you asked for. Older rows replace the whole table with `Candidates not recorded for this request — trace predates the candidates array.`
`Tiers`	Rendered `Q3 · L4` under the header title `quality / latency tiers (priors, 1–5)`. Priors are what Cloud Score assumes, not what it measured. See Models, tiers, and scoring.

A caller strategy outranks your policy.

Weights resolve in one order: the request strategy, then Quality, then a latency ceiling, then a policy Prefer tier, then the workspace scoring weights on Admin › Organization, then the default 0.4 / 0.2 / 0.4. Only auto lets the policy you are about to write apply. See Routing strategies.

Write the two policies

Two rules do most of the work: one biases a cheap task type toward price, the other turns the response cache on for traffic that repeats. The page subtitle states the ordering: Evaluated in priority order; all matching policies apply, later ones win on conflict.

Bias one cheap task type toward price.

Click New policy and type a Name, for example "Summaries go cheap".
The draft opens with Priority at 100. A name is the only field the browser validates; Save policy stays disabled without one.
Under Conditions (all must match), set Task type to summarization.
Leave the rest blank. A blank field sends no constraint, so the rule matches on task type alone.
Under Actions, set Prefer tier to cost.
That one word maps to the weight vector 0.7 cost / 0.1 latency / 0.2 quality for matching requests.
Click Save policy, then preview it in Test against a sample.
Paste a prompt into Sample input, leave Estimated cost (USD) at 0.05, click Run test. The result shows the classified task type and complexity and a green Allowed pill; the line weights c/l/q 0.7/0.1/0.2 appears only when the sample classified as summarization and the rule fired.

Serve repeat traffic from the response cache.

Click New policy, name it, and set the condition that matches the repeating traffic.
A task type is the usual choice. You can scope it to one caller instead with API key ID (UUID), which the backend requires to parse as a UUID.
Set Response caching to exact — serve identical repeats.
Two fields appear. Cache TTL (seconds) has the placeholder 3600, which is also the default when blank; a value of 0 means never write, not write and expire immediately.
Choose a Cache scope.
per API key keeps entries private to the key that produced them. shared across your keys lets every key be served the same stored answer, which raises the hit rate and widens the reach of a stale answer.
Click Save policy, then send the same prompt twice from the Playground.
The second run shows a Resolve station reading hit · <model> · saved $<usd>, a from cache · saved N tok · $X chip on the response, and a ledger that collapses to Served · saved (est.).

Cache setting	What it does	Watch out for
`exact — serve identical repeats`	Serves a byte-identical repeat from the stored answer, with no provider call.	The cheapest and safest mode. Start here.
`semantic — also serve close matches`	A superset: exact lookup first, then the nearest embedded entry at or above `Similarity bar (0–1)` (placeholder `0.90`).	Needs a provider that can embed. The effective bar is the higher of the policy bar and the workspace floor, so a policy can raise it but never lower it. Semantic lookup is skipped when the request carries a response format, a model override, required capabilities, or a forced model.
`off — force-disable`	Force-disables caching for a scope a broader policy turned on.	Use it to carve one key back out, at a higher priority number than the rule that turned it on.
`inherit (off unless a broader policy enables it)`	Defers to the `Response cache` workspace default on the Optimization settings page.	A request with no cache policy on a workspace whose default is off records `default_off`.

When caching cannot happen at all

No policy overrides these. The workspace cache mode is off, the key has body storage off, AI Security redacted the input, the request carries tools, temperature is non-zero, or the caller sent constraints.cache = "no-store". Each case records its own verdict in the request's Cache section in Observability. These are not misses: nothing could have served them from cache.

What the sample test cannot tell you

It classifies with the heuristic rules, not the classifier a real request may use, so task type and complexity can differ from production. It evaluates enabled policies only, always at text modality with no capabilities and no key, so conditions on API key ID (UUID), Modality or Required capability never match there, and cache actions are not reported in the result at all. Deleting a policy is a hard delete with no confirmation. See Policies.

Watch in observe, then promote

The engine has a mode ladder per mechanism. off runs nothing. observe runs the full decision and records what it would have done without changing a single answer. auto applies it when the confidence clears the floor. Leave everything in observe until the counts justify moving.

Open Optimization settings under Gateway, and on the Settings tab set Reference model.
This is what every request is compared against for savings. The first option is auto; the rest are enabled models rendered as gpt-4o — openai. Without a declared reference the savings picture stays empty.
Read the mechanism table without touching it.
Columns are Mechanism, Mode, Min confidence and Last 7 days. Each Last 7 days cell reads N applied · M would with the hover title saved $… · would save $…. That would-count is the evidence you are waiting for.
Wait until one mechanism earns it, then set its Mode to auto.
A row sitting at 0 applied · 96 would in Last 7 days has measured 96 opportunities and taken none. Only Semantic response cache, Output budget and Routing adjustments take a Min confidence; the rest show —. It accepts 0.5 to 1 in steps of 0.05, and 0.5 is a hard floor.
Click Save optimization settings and read the dialog before confirming.
A promotion opens Start applying these mechanisms automatically?, which names the mechanisms and warns that clients may receive cached or coalesced answers, a smaller output budget, or a different model. Turning the workspace cache default on instead opens Turn the response cache on for this workspace?. Confirm with Save and apply, or step back with Cancel.

Where	Column	What it means
`By mechanism`	`Applied` / `Saved`	Measured counts and money per mechanism on Optimizations, over its range and filters. A figure no mechanism records reads `—` with its cause, never a zero.
`By mechanism`	`Would apply` / `Would save`	What a mechanism in observe would have done, where it records it: potential, never added to savings. The `Settings` tab's `Last 7 days` cells count it per mechanism.

Two mechanisms change what your callers receive.

Setting In-flight dedup or Output budget to auto raises the banner Auto dedup and output budgets change what clients receive. Coalesced followers stream nothing until the leader settles, and a capped call that truncates bills one rollback attempt. The output budget claims no dollars at all: it reports caps, a safety rate and rollbacks.

Read the badge before you trust the switch.

A control badged Observe only stays editable but nothing acts on it yet; a control badged Not wired is disabled and keeps its value without ever being read. Hover either badge for the backend line that proves it. See Optimization settings.

Read the savings

Optimizations puts the savings, the cost they came from and what the engine recommends side by side. Pick 30 days, 90 days, 180 days or 1 year, then narrow with the filter bar if you are asking about one team, key or model.

Read Saved (net) and Requests optimized.
Saved (net) is gross savings minus Cloud Score's known overhead, an estimate against the reference model. Requests optimized, under Optimization engine, counts the requests where at least one optimization applied, out of every request in the range.
Follow the Savings waterfall.
Routing, Compression and Cache + dedup step the reference cost down to Cost. Provider cache (est.) then steps it down to the Estimated bill; it is never Cloud Score savings.
Read Recommendations.
Each carries a confidence and, where it can, an estimated monthly effect, never added to savings. Top-tier model carrying simple tasks sends you back to a Prefer tier rule; Exact repeats are being re-executed and Cache-eligible traffic rarely hits send you back to a Response caching rule.

Check the cache savings

Caching only pays when it hits. On Optimizations, the Exact response cache and Semantic response cache rows of By mechanism count the hits and what they saved.

On 30 days, read the response cache rows of By mechanism.
Applied counts the hits and Saved what they saved; the Cache-served requests tile counts every request answered from the cache or by dedup. Never add the provider cache discount into the saving: the waterfall keeps it as its own step on purpose.
Check Recommendations for Cache-eligible traffic rarely hits.
The engine raises it when eligible traffic rarely hits. Requests that could not use the cache at all are not misses: read their reason in each request's Cache section in Observability.
Watch Savings breakdown over the following days.
Its Cache + dedup segment per UTC day shows whether the change keeps paying.

Reason	What to change
`Caching not enabled by any matching policy`	Shown per request, not a miss: no policy set a cache mode for that traffic. Widen the condition on your `Response caching` rule.
`Caching off by workspace default (no policy set a cache mode)`	Shown per request, not a miss: no policy matched and the workspace default is off. Add a policy, or set the `Response cache` workspace default on the Optimization settings page.
`Closest match below similarity bar`	Semantic matching ran and found nothing close enough. Lower `Similarity bar (0–1)` on the policy, remembering the workspace floor still applies.
`No connected provider can embed for semantic matching`	Semantic caching cannot run at all. Connect an embedding-capable provider, or fall back to exact.

What to check afterwards

Three places prove the change landed, from one request up to the whole workspace.

Where	What to open	What tells you it worked
Observability	Click a row for the `Request` drawer and read the `Optimization decision` card; the full request page adds the `Savings receipt`.	The ledger reads `Reference cost (est.)`, `− Routing (est.)`, `− Compression (est.)`, `= Cost` on a routed request, or collapses to `Served · saved (est.)` on a cache hit. A mechanism row reading `—` is unrecorded, not skipped, and a pre-engine row reads `No plan recorded for this request (pre-engine row).`
Optimizations	`By mechanism`, plus `Saved (net)` and the `Savings waterfall`, over the range you measured before.	The mechanism you promoted reads `auto`; one that claims dollars also shows a rising `Applied` count and money under `Saved`, and its step in the waterfall grows. With no reference-priced traffic the waterfall reads `No requests with a reference model in this range.`
Usage & cost	The `Breakdown by` table on `Model`, and `Recommendations` on Optimizations.	`Success` holds while `$/completed request` falls, and the recommendations you acted on drop out.

Give it a full period, and compare like for like.

Counts accumulate, so a change made this morning barely moves a 30d range. Use the same range and the same filters before and after.

If the numbers did not move

Check in this order. Callers may be sending an explicit strategy, which outranks your Prefer tier. Another policy may be setting off — force-disable at a higher priority number, or blocking the request before the cache is ever reached. The promotion may still be at observe because you clicked Cancelin the confirm dialog. Or no request in range is reference-priced, in which case Optimizations' Savings breakdown reads No requests with a reference model in this range and you need a Reference model.