If you want to reduce LLM API costs, start with measurement, not model switching. Log cost per feature, request type, input tokens, output tokens, retries, and failure rate. Then cut in this order: remove unnecessary calls, shorten context, cap outputs, cache deterministic work, route simple tasks to cheaper models, batch background jobs, and add quality evals before every cost-saving change. The goal is lower cost per successful task, not just lower spend.
The cost equation to track
For each AI feature, track four numbers:
- Calls per user action: how many model calls one workflow triggers.
- Input tokens: instructions, user input, retrieved context, examples, hidden templates.
- Output tokens: generated answer, JSON, explanations, retries.
- Failure cost: retries, fallbacks, human review, support tickets, rework.
A cheap call that fails often can cost more than an expensive call that succeeds once.
Use this internal metric:
Cost per successful task = total LLM cost for workflow / number of accepted completed tasks
Accepted means the output passed your validation, eval, user approval, or downstream system check.
If you already run production evals, connect cost to the same test set. If not, build a small one first. For a practical setup, see LLM Evals: How to Set Them Up for Production AI.
Decision table: which lever to try first
| Symptom | Likely waste | First fix | Trade-off |
|---|---|---|---|
| Same user action calls the model 3-6 times | Workflow design | Merge steps or remove non-essential calls | Less modular debugging |
| Long policy, contract, or knowledge text in every prompt | Context bloat | Retrieve only relevant sections | Needs retrieval quality checks |
| Outputs are verbose when you need fields | Output bloat | Use strict schemas and max token caps | May truncate weak prompts |
| Many repeated summaries or classifications | Repeated work | Cache by normalized input and prompt version | Cache invalidation required |
| Simple tasks use the strongest model | Over-routing | Route by difficulty | Needs confidence thresholds |
| Failed JSON causes retries | Format failure | Validate, repair once, then fail safely | More engineering work |
| Background tasks run one-by-one | Inefficient execution | Batch where latency is not critical | Slower individual completion |
Step 1: Put every AI feature on a token budget
Create a budget before optimizing. It should be visible to product and engineering.
Example budget fields:
- Feature name
- User action
- Maximum calls per action
- Maximum input tokens
- Maximum output tokens
- Expected success rate
- Maximum cost per successful task
- Accepted fallback path
A good budget forces product decisions. For example, a contract upload feature may justify deeper review. A button that rewrites a short email may not.
Do not average everything across the product. Costly workflows hide inside averages.
Step 2: Remove calls before changing models
The fastest saving is often deleting a model call.
Look for:
- Separate calls for classification, extraction, summary, and formatting when one structured call can do the job.
- Calls made before you know whether the user needs the result.
- Calls repeated when the same document or prompt is reopened.
- Calls triggered during autosave, preview, typing, or background refresh.
- Calls made for records that later fail permission, plan, or validation checks.
Rule of thumb: validate cheap things before expensive things. Check file type, size, permissions, account status, and required fields before calling a model.
Step 3: Shrink input context carefully
Long prompts are usually a mix of useful context and accidental baggage.
Cut in this order:
- Remove repeated instructions.
- Replace long examples with one compact example.
- Move static rules into a short numbered policy.
- Retrieve only the relevant document sections.
- Summarize long conversation history into task-specific state.
- Drop metadata that the model does not need.
For document-heavy products, retrieval quality matters more than prompt cleverness. Bad retrieval can reduce cost and accuracy at the same time. If you are deciding how much knowledge to retrieve versus train into a model, read How to Build a Production RAG System in 2026.
Step 4: Cap output tokens and demand compact formats
Output tokens can become a silent cost leak.
Bad instruction:
Explain your reasoning and provide a detailed answer.
Better instruction:
Return only valid JSON matching this schema. Keep each explanation under 25 words. If unknown, return null.
For structured features, ask for only the fields you use. If the UI shows three bullets, do not ask for a page of analysis and then trim it later.
Also set hard output limits. A max token cap is not a substitute for a good prompt, but it prevents runaway responses.
Step 5: Cache only when the answer is safe to reuse
Caching can cut cost, but unsafe caching can leak stale or incorrect answers.
Good cache candidates:
- Document text extraction results.
- Stable document summaries.
- Clause classification for the same document version.
- Help-center answers for public content.
- Embeddings for unchanged chunks.
Poor cache candidates:
- Personalized legal, finance, or HR recommendations.
- Permission-sensitive answers.
- Outputs that depend on current account settings.
- Time-sensitive facts.
- Anything affected by policy changes.
Cache key should include:
- Normalized input hash
- Prompt version
- Model or route version
- Retrieval corpus version
- User or tenant boundary where relevant
- Expiry time
Never cache across customers unless the content is public and non-sensitive.
Step 6: Route by task difficulty
Not every task needs the same model.
A simple routing setup can use three lanes:
| Lane | Use for | Example | Guardrail |
|---|---|---|---|
| Low-cost lane | Simple classification, labels, formatting | Tag an inbound support message | Schema validation |
| Standard lane | Extraction, summaries, grounded answers | Extract payment terms from a contract | Confidence checks and citations to source text |
| High-accuracy lane | Ambiguous, high-risk, or high-value tasks | Review indemnity or termination risk | Human review or stricter eval threshold |
Routing can reduce cost, but it adds complexity. You need monitoring for false confidence. If the router sends hard tasks to the cheap lane, quality drops quietly.
Start with simple rules before building complex routing:
- Short text plus low-risk task: low-cost lane.
- Long document plus business decision: standard lane.
- Legal, finance, security, or compliance impact: high-accuracy lane or human review.
Step 7: Use retrieval instead of stuffing documents into prompts
If users upload contracts, policies, invoices, or support histories, do not send the whole file every time.
Use this pattern:
- Split the document into meaningful chunks.
- Store chunk text with page, section, and document version.
- Retrieve only chunks relevant to the question.
- Ask the model to answer using only retrieved chunks.
- Return source references for review.
This lowers token usage and improves traceability. The limitation is that retrieval can miss relevant sections. Add tests for known questions where the right answer appears in different parts of the document.
For contract extraction workflows, Contract Metadata Extraction Methods for Business Teams gives a related business-side view.
Step 8: Batch background work
Batching helps when users do not need instant output.
Good batch jobs:
- Nightly document tagging.
- Bulk metadata extraction.
- Backfilling summaries.
- Re-indexing documents.
- Periodic quality checks.
Avoid batching for:
- Interactive chat.
- Signing flows.
- Approval blockers.
- Security-sensitive checks that must complete immediately.
Batching trades latency for cost and operational simplicity. Make the delay visible in the product so users do not keep retrying.
Worked example: reducing cost per document review
Hypothetical example. Assume an AI document review feature processes 10,000 documents per month.
Before optimization:
| Item | Assumption |
|---|---|
| Calls per document | 4 |
| Average input tokens per call | 12,000 |
| Average output tokens per call | 1,500 |
| Successful completion rate | 85% |
| Monthly model bill | $2,400 |
| Cost per successful document | $2,400 / 8,500 = $0.282 |
Changes:
- Merge classification and extraction into one structured call.
- Retrieve only relevant clauses instead of sending the full document each time.
- Cap output to required JSON fields.
- Cache results by document version.
- Send only high-risk documents to the expensive review lane.
After optimization:
| Item | Assumption |
|---|---|
| Calls per document | 2 |
| Average input tokens per call | 5,000 |
| Average output tokens per call | 700 |
| Successful completion rate | 88% |
| Monthly model bill | $950 |
| Cost per successful document | $950 / 8,800 = $0.108 |
The useful number is not the 60% lower bill. It is the cost per accepted review dropping from $0.282 to $0.108 while success rate improves. Your numbers will differ by vendor, model, prompt, traffic, and task difficulty.
Cost-control checklist for small teams
Before launch:
- [ ] Define cost per successful task for each AI feature.
- [ ] Log input tokens, output tokens, model route, retries, and failures.
- [ ] Add max token limits for every call.
- [ ] Validate permissions and inputs before model calls.
- [ ] Keep prompts versioned.
- [ ] Build a small eval set with realistic success and failure cases.
After launch:
- [ ] Review top 10 most expensive workflows weekly.
- [ ] Track cost by tenant, plan, and feature.
- [ ] Alert on token spikes and retry loops.
- [ ] Cache stable outputs with prompt and document version keys.
- [ ] Route simple tasks to cheaper lanes only after eval approval.
- [ ] Re-run evals before changing prompts, models, retrieval, or schemas.
Common mistakes to avoid
Switching models before measuring workflows
This may reduce unit price but leave call volume, retries, and context bloat untouched.
Optimizing for tokens while hurting acceptance rate
If users regenerate outputs more often, your bill can rise even after each call gets cheaper.
Caching without version control
A cache that ignores prompt version, document version, or policy version can return stale answers.
Sending full documents for every question
This is expensive and harder to audit. Retrieve the relevant sections instead.
Treating cost controls as only an engineering problem
Product choices drive cost. A feature that generates five alternatives, explains each one, and stores all drafts costs more than a feature that returns one accepted output.
Where Zettaura fits
Zettaura builds AI employees for business workflows such as documents, contracts, events, work assistance, and professional presence. In our own product work, cost control has to be designed into the workflow: what the AI employee is allowed to do, when it should ask for review, and when a cheaper deterministic step is enough.
FAQ
Should I fine-tune to reduce API costs?
Sometimes, but not first. Fine-tuning can reduce prompt length or improve consistency for narrow tasks, but it adds training, evaluation, versioning, and maintenance work. First remove waste, reduce context, cache stable outputs, and route by task difficulty.
Is a smaller model always cheaper?
No. If it fails more often, needs longer prompts, or causes more retries, the total cost per successful task may be higher.
How often should I review LLM costs?
For an early product, review weekly. Once usage stabilizes, keep alerts for spikes and review major workflows before pricing, plan, or feature changes.
What is the safest first cost-saving change?
Add logging and output token caps. These usually improve visibility without changing product behavior much.
Closing note
Pick one expensive workflow and calculate its cost per successful task this week. Then apply one low-risk change: remove an unnecessary call, cap output length, or cache a stable result.
From the Zettaura team: we build AI-native products for businesses and founders, connected by one passwordless Zettaura ID. You can see the product suite at zettaura.com/products.
