LLM evals are structured tests that tell you whether an AI feature is good enough to ship, keep, or roll back. Set them up by defining the task, collecting representative examples, writing pass/fail criteria, choosing automated and human scoring methods, running the same tests on every change, and reviewing production failures weekly. Do not start with a large benchmark. Start with 50 to 200 examples from your own product: real user questions, expected answers, edge cases, policy violations, and known failures. Your first goal is not academic accuracy. It is to prevent regressions, catch unsafe behaviour, control cost, and make release decisions with evidence.
What LLM evals should answer
A useful eval system answers five operating questions:
- Does the AI feature complete the task? For example, does it classify a support ticket correctly, extract the renewal date, or draft a compliant response?
- Does it follow your rules? This includes tone, formatting, refusal behaviour, privacy restrictions, and product policy.
- Does it stay grounded? If the answer should come from source documents, does it cite or use the right source?
- Does it fail safely? When the input is ambiguous, malicious, or outside scope, does it ask for clarification or refuse?
- Is the quality worth the latency and cost? A better answer that costs 10 times more may still be wrong for a workflow that runs thousands of times a day.
The NIST AI Risk Management Framework is a useful reference for thinking about AI quality, validity, reliability, safety, accountability, and monitoring. You do not need to implement it as a heavy compliance programme on day one, but its categories help you avoid treating evals as only an accuracy score.
If you are building document, contract, or retrieval-heavy AI features, also read our guides on validating LLM outputs in production and building a production RAG system. Evals and retrieval quality are closely linked.
Step 1: Define the task before you define the metric
Do not begin with “we need 90% accuracy”. First write a one-page task spec.
Task spec template
| Field | Example |
|---|---|
| Feature | Contract clause extraction |
| User | Sales operations manager |
| Input | Uploaded vendor agreement PDF |
| Output | JSON with renewal date, notice period, governing law, payment terms |
| Must do | Return only fields supported by the document |
| Must not do | Guess missing dates or invent clauses |
| Failure mode | Mark missing fields as null with a reason |
| Human fallback | Send to legal reviewer if confidence is low or clause conflicts are detected |
This task spec becomes the basis for your eval dataset, scoring rubric, and release gate.
For example, if the feature extracts contract metadata, a generic “answer quality” score is weak. You need field-level checks:
- Was the renewal date extracted correctly?
- Was the notice period normalised correctly?
- Did the output include unsupported text?
- Did the JSON match the schema?
- Did the system flag uncertainty instead of guessing?
If your use case involves clause extraction, this older guide on AI clause extraction from contracts gives a practical breakdown of what teams usually need to extract and verify.
Step 2: Build a small but representative eval dataset
Your first eval set should be small enough to inspect manually and broad enough to catch common failures.
A good starting set:
| Dataset slice | Example count | Purpose |
|---|---|---|
| Common happy paths | 40 | Prove the feature works for normal inputs |
| Edge cases | 30 | Test odd formatting, missing fields, long text, mixed languages |
| Known past failures | 30 | Prevent regressions |
| Adversarial or unsafe inputs | 20 | Test prompt injection, policy violations, data leakage |
| High-value business cases | 30 | Protect important workflows |
| Format compliance cases | 20 | Test JSON, schema, and field constraints |
That gives you 170 examples. This is enough to start making release decisions without creating a months-long lab project.
Where to get examples
Use real product data only if you have the right permissions and privacy controls. For sensitive data, redact names, emails, phone numbers, financial figures, customer identifiers, and confidential clauses before adding examples to an eval set.
Good sources include:
- Support tickets that represent repeated user requests.
- Sales or operations workflows that already have human-reviewed outcomes.
- Documents where your team knows the correct answer.
- Failed production outputs that caused manual correction.
- Synthetic edge cases written by your team to test specific rules.
Keep a column called source. It should say whether the example came from production, manual creation, legal review, QA, or a customer-reported issue.
Step 3: Write expected outputs and scoring rules
Each eval example needs three things:
- Input.
- Expected behaviour.
- Scoring method.
For deterministic tasks, write exact expected outputs. For open-ended tasks, write a rubric.
Example: structured extraction eval
{
"id": "contract_renewal_014",
"input": "Agreement text or redacted excerpt here",
"expected": {
"renewal_date": "2026-03-31",
"notice_period_days": 60,
"governing_law": "Karnataka, India",
"missing_fields": []
},
"checks": [
"valid_json",
"schema_match",
"exact_field_match",
"no_unsupported_fields"
]
}
Example: open-ended answer rubric
| Score | Meaning | Example rule |
|---|---|---|
| 3 | Good | Correct, grounded, complete, follows tone and format |
| 2 | Usable with minor issue | Correct main answer but missing a caveat or citation |
| 1 | Poor | Partly wrong, vague, or not grounded |
| 0 | Fail | Wrong, unsafe, fabricated, or policy-violating |
For business release gates, convert the rubric into a clear threshold. For example:
- Average score must be at least 2.6 out of 3.
- No critical safety failures allowed.
- JSON validity must be 100% for workflows that feed downstream systems.
- Known regression cases must pass at least 95%.
Use strict gates for machine-to-machine workflows. A malformed JSON output can break an invoice, contract, or CRM workflow even if the wording looks correct to a human.
Step 4: Choose automated, human, and hybrid evaluation
Most teams need more than one scoring method.
| Eval type | Best for | Weakness |
|---|---|---|
| Exact match | Dates, IDs, labels, fixed fields | Too strict for natural language |
| Schema validation | JSON, XML, tables, API-ready outputs | Does not prove meaning is correct |
| Rule checks | Forbidden phrases, citations, length, policy terms | Misses subtle errors |
| Reference comparison | Summaries, answers, classifications | Requires good reference answers |
| Human review | Legal, brand, safety, nuanced judgement | Slower and more expensive |
| Model-graded review | Draft quality scoring at scale | Needs calibration against human review |
Use automation for repeatability. Use humans for judgement. Use hybrid review for risky workflows.
For example, a contract review feature can use automated checks for JSON validity, clause presence, date format, and citation format. A lawyer or trained reviewer can then inspect a sample of outputs for legal nuance. For legal and compliance workflows, confirm your evaluation design with counsel. This article is not legal advice.
The OWASP Top 10 for Large Language Model Applications is a practical source for security-related eval cases, including prompt injection, sensitive information disclosure, excessive agency, and insecure output handling.
Step 5: Create a release gate
An eval without a release decision is only a report. Define what happens when scores pass or fail.
Simple release gate
| Metric | Gate | Action if failed |
|---|---|---|
| Task success | >= 90% | Do not release |
| Critical failures | 0 | Do not release |
| JSON validity | 100% | Fix parser or output rules |
| Known regressions | >= 95% pass | Review changed prompts or retrieval |
| Median latency | Within product target | Optimise before release |
| Cost per 1,000 runs | Within budget | Rework prompt, routing, or caching |
Do not copy these thresholds blindly. A customer support draft can tolerate more review than an automated payment instruction. A legal document summary can tolerate slower latency than a live chat answer.
Your release gate should be tied to business risk, not vanity accuracy.
Step 6: Track cost and latency inside the eval
Quality alone is not enough. An AI feature that passes quality checks but burns budget will not survive production.
Add these fields to every eval run:
- Input tokens or input size.
- Output tokens or output size.
- Number of retrieval calls.
- Number of retries.
- Latency by step.
- Total estimated cost.
- Cache hit or miss.
- Final status: pass, fail, human review, retry, fallback.
Worked example: eval cost planning
Assume your team has 200 eval examples.
| Item | Assumption | Monthly cost impact |
|---|---|---|
| Eval examples | 200 | - |
| Runs per week | 5 | 1,000 eval runs/month |
| Average vendor cost | USD 0.004/run | USD 4/month |
| Human review sample | 50 outputs/month | - |
| Reviewer time | 5 minutes/output | 250 minutes |
| Internal review cost | INR 1,500/hour | About INR 6,250/month |
In this example, the human review cost is larger than the API cost. That is common in early eval programmes. The answer is not to remove humans. The answer is to use human review where it matters: new failure types, risky workflows, and calibration samples.
For a deeper look at cost control, make cost part of your eval output from the beginning. It is much harder to add later.
Step 7: Add evals to the development workflow
Run evals at four points.
1. Before a prompt or workflow change
Run the current version and save the baseline. Without a baseline, you cannot tell whether a change improved or damaged the product.
2. During development
Run a smaller smoke test set of 20 to 40 examples. Include the most important known failures and edge cases.
3. Before release
Run the full eval set. Block release if critical gates fail.
4. After release
Sample production outputs. Add real failures back into the eval set after redaction and review.
The OpenAI Evals repository is one public reference for how eval tasks can be structured and run repeatedly. You do not need to copy any specific framework, but the pattern is useful: examples, outputs, scorers, and repeatable runs.
Step 8: Build a failure taxonomy
A failure taxonomy helps your team fix the right thing.
| Failure type | Example | Likely fix |
|---|---|---|
| Retrieval failure | Correct policy document was not found | Improve indexing, chunking, metadata, or query rewrite |
| Grounding failure | Answer cites source but source does not support it | Strengthen citation checks and refusal rules |
| Instruction failure | Output ignores required JSON format | Add schema validation and retry path |
| Reasoning failure | Correct facts but wrong conclusion | Add worked examples or human review gate |
| Safety failure | Reveals sensitive data or follows malicious input | Add security tests and output filters |
| Policy failure | Uses forbidden wording | Add rule checks and style constraints |
| Regression | Old passing case now fails | Compare against previous baseline |
Do not treat all failures equally. A typo in a draft email is not the same as an invented contract clause. Your dashboard should separate critical failures from minor quality issues.
Step 9: Use public benchmarks carefully
Public benchmarks can help you compare broad capabilities, but they rarely reflect your product workflow. They also may not test your private policies, local legal terms, customer documents, or UI constraints.
Use them for general orientation, not as your shipping gate.
The MLCommons AILuminate benchmark is an example of a public effort focused on AI safety evaluation. It can inform your safety thinking, but your production evals still need your own examples and risk rules.
For most startups and internal product teams, the best benchmark is yesterday’s product behaviour on your own high-value tasks.
Step 10: Decide what to log in production
Production logs are the fuel for better evals. They are also a privacy and security risk if handled badly.
Log only what you need. Redact sensitive data where possible. Restrict access. Set retention rules. If you operate in regulated environments or process personal data, confirm your logging design with counsel and your security team.
A practical production log record includes:
- Request ID.
- Feature name.
- User-visible input category, not always raw text.
- Retrieved document IDs or source IDs.
- Output status.
- Validation result.
- Human override result.
- Latency and cost.
- Error type.
- User feedback, if available.
If you serve Indian users or process Indian personal data, also understand your privacy obligations. This guide on DPDP Act vs GDPR differences explains key concepts for Indian and global teams. Confirm obligations with counsel for your specific product and data flows.
Minimum viable eval stack for a small team
You do not need a large platform to begin. A small team can start with this setup:
| Component | Simple implementation |
|---|---|
| Eval dataset | CSV, JSONL, or database table |
| Expected outputs | Stored beside each example |
| Runner | Script or internal job that calls the AI feature |
| Scorers | Python, TypeScript, SQL checks, or internal validation code |
| Human review | Spreadsheet or review queue |
| Dashboard | Simple table with pass rate, failures, cost, latency |
| Release gate | CI check or manual sign-off before deployment |
Start boring. Make it repeatable. Improve only when the bottleneck is clear.
Example: first 30-day plan
Week 1: Define and collect
- Pick one AI feature.
- Write the task spec.
- Collect 50 real or representative examples.
- Add 20 edge cases and 10 known failures.
- Define pass, fail, and critical fail.
Week 2: Score manually
- Run the current feature on all examples.
- Manually score every output.
- Write down the top five failure types.
- Convert simple failures into automated checks.
Week 3: Automate the runner
- Store eval examples in version control or a controlled database.
- Add a repeatable eval runner.
- Save outputs for each run.
- Track pass rate, critical failures, latency, and estimated cost.
Week 4: Add release gates
- Define smoke tests for every change.
- Define full evals before release.
- Add a production failure review meeting.
- Add new failures back into the eval set.
By the end of 30 days, you should have a working eval loop. It will not be perfect, but it will stop your team from shipping blind.
Common mistakes when setting up LLM evals
Mistake 1: Using only generic benchmarks
Generic scores do not tell you whether your contract review, invoice matching, sales email, or support answer works for your users.
Mistake 2: Testing only happy paths
Most production pain comes from edge cases, missing context, malformed input, and unexpected user behaviour.
Mistake 3: Having no critical failure category
Average scores hide dangerous failures. A feature with 95% pass rate can still be unshippable if the remaining 5% includes data leakage or fabricated legal claims.
Mistake 4: Letting eval data become stale
Your product changes. Your users change. Your policies change. Refresh eval examples from production failures and high-volume workflows.
Mistake 5: Not separating product quality from model quality
Users experience the whole system: retrieval, prompts, validation, UI, permissions, fallbacks, and human review. Your eval should test the full product path, not only a raw model response.
LLM eval setup checklist
Use this checklist before your first release:
- [ ] One task spec exists for each AI feature.
- [ ] Eval examples cover normal, edge, unsafe, and high-value cases.
- [ ] Expected outputs or rubrics are written.
- [ ] Critical failures are defined.
- [ ] Automated checks cover format, schema, policy, and exact fields where possible.
- [ ] Human review is used for nuanced outputs.
- [ ] Eval runs are saved with version, date, cost, and latency.
- [ ] Release gates are written and agreed.
- [ ] Production failures are reviewed and added back to the dataset.
- [ ] Privacy and legal review is complete for logged data.
FAQ
How many examples do I need for LLM evals?
Start with 50 to 200 examples for one feature. That is enough to catch common regressions and force clear scoring rules. Increase the set as you see more production failures and workflow variations.
Should evals be automated or human-reviewed?
Use both. Automate objective checks such as JSON validity, field extraction, citations, and policy rules. Use human review for judgement-heavy areas such as legal reasoning, brand tone, safety, and customer-impacting responses.
How often should I run LLM evals?
Run a small smoke set during development, the full set before release, and sampled checks after release. Also run evals whenever you change prompts, retrieval logic, validation rules, model settings, or business policy.
Can I use another LLM to grade outputs?
Yes, but calibrate it against human review. Treat model grading as an assistant to your eval process, not as the final authority for high-risk workflows.
What is the difference between LLM evals and unit tests?
Unit tests usually check deterministic code behaviour. LLM evals test probabilistic product behaviour: correctness, grounding, safety, formatting, cost, and regression risk across many examples.
Closing note
LLM evals are not a one-time QA task. They are the operating system for shipping AI features without guessing. Start with one feature, 100 examples, clear failure rules, and a release gate your team will actually follow.
From the Zettaura team: Zettaura builds AI products for teams that need practical AI employees across documents, events, assistants, and brand workflows. You can see the product suite at zettaura.com/products.
