How to evaluate AI agent economics: cost per case and operator effort
A proposed evaluation worksheet for comparing accepted outcomes, operator hours, and process costs before and after introducing an AI agent.
Write the decision before building the dashboard
Should an AI agent handle a larger share of a workflow, stay within a limited pilot, or leave that workflow? We propose answering with a decision worksheet covering accepted outcomes, process costs, and human effort. The method below is our editorial recommendation. Its definitions and formulas are proposed accounting conventions for an operational evaluation, rather than evidence of a particular deployment's performance.
Start with a narrowly described task, such as preparing a draft knowledge-base entry from approved reference material. Name the person who will decide whether to expand the pilot. Ask that person to record the acceptable cost, required review, and permitted operator involvement before the evaluation begins.
Use the worksheet to distinguish observations from assumptions. Label a measured expense as measured, an estimated allocation as estimated, and missing information as missing. We recommend leaving an unresolved field visible instead of filling it with an optimistic forecast. Treat every instruction that follows as part of this proposed workflow, to be adapted and agreed by your team.
Define the outcome you will count
For this worksheet, we propose defining a case as one requested outcome with a persistent identifier and a documented acceptance decision. In the knowledge-base example, one case might cover a single entry with specified sections and approved supporting material. Do not use the number of generated drafts as the denominator unless draft production itself is the agreed outcome.
Write an acceptance checklist before counting cases. For the example workflow, your proposed requirements might include coverage of requested questions, traceability to the supplied material, and approval by an assigned reviewer. Specify who can accept a case and how disagreements will be recorded.
Choose an observation window for corrections. Within that window, we recommend attaching revisions for the original request to its existing case identifier. Treat a separate request as a new case only under a rule agreed before the pilot. Select a window appropriate to your own work; this article does not prescribe a universal duration.
Use three final statuses in the proposed ledger: accepted, still requiring work, and withdrawn from scope. Record why each withdrawal occurred. Add a separate completion-route field: agent alone, agent with review, or operator takeover. Under our convention, an accepted case completed after takeover remains part of the agent-enabled process, while its route remains explicitly recorded.
Our proposed counting rules
Specify an accepted outcome
- 1
Identify
Give one requested outcome a persistent case identifier.
- 2
Specify
Write the acceptance checklist and name the reviewer.
- 3
Observe
Choose how long corrections remain attached to the case.
- 4
Record
Enter the final status and level of operator involvement.
Reconstruct the process before automation
Prepare a baseline worksheet describing the previous workflow. List its tasks, acceptance rules, participating roles, and included expenses. Apply the same case definition to the baseline and the pilot. Ask a named owner to approve any difference in scope before interpreting the comparison.
If historical records lack operator time, mark that field as unavailable or as an estimate with an explanation. Our recommendation is to keep estimated baseline figures separate from directly observed figures. Alternatively, arrange a defined observation period for the existing process before starting the pilot, subject to your team's operational constraints.
We propose grouping cases using characteristics selected in advance: task type, completeness of input, and a team-defined difficulty category. Compare matching groups and show their case counts. Where a group has no baseline counterpart, label its result as an unmatched pilot observation.
Also record the process version used during each measurement period. If you change instructions, staffing rules, or acceptance requirements, annotate the change and decide whether to start a separate comparison. Avoid quietly combining versions in the proposed report. Give the decision owner an explicit choice about which comparisons remain suitable for the decision.
Build an explicit cost ledger
Our proposed ledger has five categories: operator labor, metered services and tools, recurring process maintenance, outcome review, and setup work. Assign each expense to one category, with a source record and an owner. Where an activity could fit more than one category, choose one location and document the choice.
Define labor cost in this worksheet as recorded time multiplied by the agreed rate for each role. State what each rate includes and apply the same convention to both workflows. Record rates separately from time, so the reviewer can examine the assumptions without changing the activity log.
We propose the following unit measure: cost per accepted case = included process cost for the evaluation scope / accepted cases within that same scope. Keep the included spending on unfinished and unaccepted work in the numerator. Count only accepted cases in the denominator. If there are no accepted cases, report the total spending and mark the unit measure as unavailable.
Prepare two cost views. The operating view contains the recurring expenses included by your agreed rules. The full pilot view also includes setup and preparation. If you allocate setup spending across a chosen period, show that period beside the result and retain a view of the original setup amount.
For sensitivity review, we recommend recalculating the worksheet under explicitly chosen assumptions about case volume, operator time, and setup allocation. Label each version as a scenario, not a forecast. Keep the observed result separate from these scenarios and name the assumption responsible for each difference.
Proposed reporting views
Separate operations from pilot setup
- Recurring expenses
- Include agreed operating categories
- Setup work
- Display separately
- Case count
- Accepted cases in the evaluation scope
- Assumptions
- State rates and included activities
- Recurring expenses
- Include the same operating categories
- Setup work
- Include under the stated allocation rule
- Case count
- Accepted cases in the same scope
- Assumptions
- Also state the setup allocation period
Account for operator effort separately
In our proposed activity log, separate input preparation, monitoring, review, correction, takeover, and instruction maintenance. Give operators a short description of each category and a place to record work that does not fit. Agree how to handle overlapping activities before collecting totals.
Record active operator time separately from elapsed case time. Establish a written rule for periods when a person must remain available but is not actively working on a case. Apply that rule consistently to the baseline and pilot, with any exceptions documented.
Report total operator hours and operator minutes per accepted case. Break out the totals by role and by case group. Alongside the average, we recommend showing the underlying case count and the distribution of recorded effort using intervals selected by your team. Leave low-volume groups identifiable rather than presenting their results as settled expectations.
Keep three distinct fields in the decision worksheet: measured change in effort, documented change in spending, and the team's proposed use of available capacity. Do not automatically label a difference in recorded hours as money saved. Require the owner to write a concrete staffing or workload decision before entering an expected financial consequence.
Set a quality gate and respect the evidence boundary
Google's guidance on generative AI content says generated content may contain inaccuracies and calls for manual fact-checking and review before publication. Its scope is content published on websites. source-2
Google's separate helpful-content guidance includes self-assessment questions about original analysis, completeness, and clear sourcing. source-3
For the knowledge-base workflow, our proposed checklist therefore includes an explicit review task and a recorded acceptance decision. Record that review time in the ledger under the convention already agreed. For a different workflow, write acceptance criteria for that task rather than treating web-publishing guidance as a universal evaluation method. Neither cited document is offered here as evidence of agent cost savings.
The Max AI registration page describes the product as an operating system for AI agents. source-1 We make no product-specific cost or workload claim in this evaluation. Our proposed next step is to prepare your own case definitions and measurement worksheet before assessing a deployment.
Finish with a decision and a next observation
Use a one-page decision record containing the baseline comparison, cost per accepted case, operator effort, acceptance results, and unresolved assumptions. Include the original decision conditions beside the findings. Have the owner sign the interpretation, including any qualification caused by missing records or unmatched case groups.
We recommend three possible decisions: expand within a named scope, revise and repeat the pilot, or end the pilot. For expansion, specify the next case group and its measurement owner. For a repeat, name the change being tested and the next review date. For closure, retain the worksheet and the written reason.
If essential baseline or operator records are missing, our recommendation is to commission a defined measurement task before making an expansion claim. End the evaluation with an accountable next action, rather than an unsupported return-on-investment statement.
Suggested evaluation stages
Give each stage an owner
Agree
Approve scope, definitions, and decision conditions.
Measure
Collect expense records and operator activity logs.
Close
Complete the observation window and record acceptance.
Decide
Assign expansion, a revised pilot, or closure to a named owner.
Sources
- Max AI - Operating system for AI agents — maxaiagent.app; accessed November 13, 2026
- Google Search's Guidance on Generative AI Content on Your Website | Google Search Central | Documentation | Google for Developers — developers.google.com; published October 1, 2026
- Creating Helpful, Reliable, People-First Content | Google Search Central | Documentation | Google for Developers — developers.google.com; published October 5, 2026
Start working with your own agent
The agent knows this article context and still follows the assigned department checklist.
Related articles
Designing AI agent intake: clarify the request and carry the case forward
A proposed design workshop for turning an ambiguous request into an agreed deliverable, with explicit questions, case notes, corrections, and restart rules.
AI Agent Permissions: A Proposed Workflow for Data, Tools, and Action Approval
Define what an agent may read, prepare, change, and publish. Our proposed workflow includes a permission brief, a review packet, and a staged publishing example.
Monitoring AI Agents After Launch: A Practical Review and Response Workflow
Build an explicit review process for service quality, investigate failed conversations, and define the evidence required to change or restore an agent’s scope.
Designing an AI-to-human handoff: triggers, context and ownership
An editorial playbook for defining escalation triggers, preparing a handoff record and assigning responsibility through acceptance and follow-up.