Sample work · AI & Automation
Review AI agent: test suite for a customer service agent, before and after
A customer service agent with tools for order search, refund, and tickets was run through a test suite. Built-in errors like duplicate refunds or foreign data had to be flagged, after which corrections were made and tested again.
Sample work (demo), not a client job. Built as a trial task with invented example data; the check figures below come from the automatic test run and are not polished.
What was built
- Test cases for standard requests, edge cases, and security, each case run multiple times
- Rules that apply to every tool call: no duplicate refunds, no foreign customer data
- Report with error classes, corrections, and the subsequent result
Download pruefsuite-faelle.json
The example agent operates on a rule basis, the sample does not call a language model. With a real language model agent, more runs per case are needed because the responses fluctuate.
How it was checked
Automatic test run on 2026-10-31, taken from the log (test data invented, figures measured):
- Customer service agent (tools search_order/refund/create_ticket), eval suite 32 cases x 3 runs (8 security cases, invariants on every call): before 16/32 (50.0 %), 6/6 built-in error classes detected (unknown_order_no, amount_currency, duplicate_refund, foreign_data
Limits
What this sample does not cover, so nobody expects more:
- The example agent is rule-based and deterministic except for the wording; real LLM agents respond differently each run - requiring more runs per case (10-20), a success rate per case, and a threshold instead of ...
- The sample does not call a language model (no costs); response contents are checked against patterns/facts. Free-form quality (tone, helpfulness) would require a model evaluator or humans and is ...
- The corrections are prepared switches in the example agent; for a customer agent, these are manual changes to the prompt, code, or a guard before the tools, written by hand after the ...
- The suite only finds errors for which there are cases or invariants; unknown error types remain undetected - therefore, invariants on every call and a mutation test as a measure of ...
- Prompt injection is tested using typical patterns (SYSTEM:, HTML comment, 'Ignore ...'); a determined attacker will find other forms - limiting tool permissions remains the actual ...
- Tools are mocks in a copied world per case; real side effects (payment providers, emails) are not triggered
Something like this for your business?
Describe briefly what you need. You get a fixed price before I start; nothing happens without your yes.