Agent & automation build
that leaves the decisions with your people.
Building means plain automations for the steps that follow a rule and agents for the steps that need judgement, connected to the tools you already use. It matters when the map shows where the hours go and you want them back without handing over the decisions.
The real question
Which decisions the system may take on its own, and which it must bring to a person.
Step 02 of 07 · AI Agent Quality & Evaluation
The path
From process map
to release decision.
Seven steps, each feeding the next: the first two automate the work, the other five verify it. This page is one of them.
- Process mapping
- Agent & automation build (this page)
- Agent evaluations
- Workflow and handoff testing
- Deterministic checks
- Evaluation harnesses
- Release criteria
Running exampleAn illustration, not a case study.
A support agent is asked to refund a duplicated charge. The agent reads the ticket and the payment record, drafts the refund and the reply, and waits for a person to approve; a plain automation then issues the refund and logs it.
01Questions
The questions
it answers.
Which steps need an agent, and which a plain automation?
A step with a fixed rule gets a fixed automation, which is cheaper to run and easier to test; an agent is kept for steps that read free text or need judgement.
Where does a person approve?
Every step that moves money, changes a customer record or makes a commitment waits for an approval, and the approval shows exactly what will be done.
Which systems may it read, and which may it change?
Each agent and automation gets the narrowest access its step needs, so a mistake cannot reach further than the step.
What happens when a step fails halfway?
Each step has a fallback: retry, undo what was done, or hand the case to a person with the work done so far.
02Pitfalls
Where it
usually breaks.
An agent where a rule would do
A model decides something a fixed rule could have decided. It is slower, costs more on every run and sometimes gives a different answer to the same case.
Approvals that do not show what they approve
The person is asked to approve a refund without seeing the amount or the account. Approving becomes a habit, and the control exists only on paper.
Broad access granted to save time
The agent can write to every system because narrowing its access was fiddly. One wrong step changes records the process never needed to touch.
Going live without the tests that come next
The automation is tried on the cases it was built from. Nobody checks the cases where it should stop, so the first unusual request is handled live.
03Approach
How I work
on it.
Automate the fixed steps first
I build the steps that follow a rule as plain automations in the tools you already use, so the simplest time savings come early and are easy to check.
Add agents where judgement is needed
For steps that read messages or documents I build agents with narrow access and an approval before anything consequential, and log every action with the input behind it.
Hand over a system your team can run
I document what runs, what to watch and how to change it, and agree with you how the first period of real use is watched while the evaluations catch what the build missed.
What you leave with
- Automations for the steps that follow a rule, built in the tools you already use.
- Agents for the steps that need judgement, each with narrow access and an approval before any consequential action.
- A log of every action an agent takes and the input it acted on, so a person can see what ran and why.
- A fallback for each step: retry, undo, or hand the case to a person with the work done so far.
- A handover: what runs, what to watch, where a person approves and how to change it.
04Measures
What I
would measure.
Signals to read, not targets to hit: what counts as good enough is your team’s decision.
Approvals changed or refused
A rise means the agent proposes the wrong action more often; almost none can also mean approvals are being clicked through unread.
Cases handed back to a person
Shows how much of the work the automation really covers, and which kinds of case it still cannot handle.
Time from a case arriving to its resolution
Against the baseline from the map, it shows whether the hours came back or only moved to someone else’s queue.
A process to automate, or
an agent to check?
Describe the process or the workflow, what it should achieve and where it goes wrong today.