Testing AI Agents Before Release: Conversations, Evidence, and Acceptance

Before releasing a documentation assistant, we recommend preparing a review packet that another team member can inspect: the conversation, the permitted documents, the expected behavior, and the actual answer. Make the release decision against that packet. The workflow below is our editorial proposal, rather than a published testing standard or a guarantee of performance.

Google's guidance for AI-generated website content says outputs can contain inaccuracies and calls for manual fact-checking before publication. Google's generative AI guidance. We use that concern as a starting point for the proposed evidence review here. Google's page is about website content; the agent acceptance procedure in this article is our own recommendation.

Write the acceptance agreement first

For our proposed workflow, begin with a narrow assignment: answering questions about an internal equipment-booking guide. Write down what the assistant should explain, what it should ask, and which actions belong outside this test. Give the assignment an owner who can settle questions about the expected answer.

Prepare a case sheet before running the conversation. Include a case ID, user objective, document snapshot, message sequence, required answer elements, prohibited additions, and review notes. Specify whether the test covers an explanation, a request for clarification, or an action. If you choose to test an action, define the expected observable result separately from the assistant's description of it.

As a fictional fixture, suppose a guide permits equipment collection on Tuesday and Thursday but says nothing about delivery. Our proposed acceptance agreement would require the assistant to explain collection arrangements and leave delivery availability unresolved. Do not quietly add delivery rules to the expected answer after reading the output. Revise the fixture explicitly if the intended assignment changes.

Design a conversation matrix around uncertainty

Our proposed matrix uses four dimensions: evidence availability, question clarity, conversation history, and requested output. Choose cases deliberately across those dimensions. Treat this as a planning convention for your team, not an exhaustive taxonomy of agent behavior.

For evidence availability, prepare one case with a direct answer, one with only part of the answer, and one with no answer in the supplied guide. For question clarity, contrast a named piece of equipment with an ambiguous request such as “Can I collect it then?”. For conversation history, include a correction to the requested day. For output, ask separately for a short reply and a structured checklist.

Write the expected behavior for each chosen combination. In our fictional collection example, a follow-up saying “Actually, I meant Wednesday” should trigger the behavior specified for Wednesday in your fixture. If the guide does not establish an exception, our proposed rule is to avoid inventing one. Keep that rule visible on the case sheet so reviewers assess the same requirement.

Separate task completion from evidence support

We propose two independent review tracks. The task track asks whether the answer addresses the user's request and includes the required elements. The evidence track asks whether each factual statement is supported by the permitted documents. Add format compliance as a separate field when a case requests a particular structure.

Use explicit acceptance statements. For example: “Names both collection days listed in the fixture”; “Does not assert that delivery is available”; and “Asks which item the user means when the reference is unresolved”. Mark each statement as passed, failed, or not applicable, and attach a brief explanation. These labels are our suggested scoring convention.

Google's helpful-content self-assessment includes questions about clear sourcing, readily verifiable factual errors, and whether readers have learned enough to achieve their goal. Google's helpful-content questions. Our recommendation is to borrow those questions for a separate editorial review, while retaining your own case-specific acceptance requirements. Do not turn a favorable overall impression into an automatic pass on every field.

Audit the answer one assertion at a time

For this workflow, define an unsupported assertion as a factual statement for which the reviewer cannot identify sufficient evidence in the permitted document snapshot. This is our operational definition for the exercise. It does not declare the statement false everywhere outside that snapshot.

Our proposed review process is to split the response into individual assertions, record an exact supporting passage for each, and classify the relationship. Use three labels: supported, contradicted, and not established. Apply the labels to the complete assertion, including its quantities, conditions, and scope. If a sentence contains two assertions, create two entries.

Consider the fictional answer “Collection is available on Tuesday and Thursday, and delivery is free.” Under our fixture, the collection statement passes the evidence check. The delivery statement receives “not established”. A link to the collection guide would not meet our proposed acceptance rule for the delivery assertion. Ask the reviewer to identify what supports the delivery terms specifically, rather than accepting a generally related document.

For a correction, require either removal of the unsupported detail or an explicit statement that the guide does not establish it. Record the revised wording alongside the original answer.

Test clarification and partial answers deliberately

In our proposed rubric, a well-scoped limitation can satisfy the case. Define exactly what that means before execution. An acceptable answer to a delivery question might state that the guide describes collection and does not specify delivery arrangements. If you want a follow-up question, write that requirement into the case instead of rewarding any question the assistant happens to ask.

Also include a case where the assistant has enough information to answer. In that case, our proposed criterion would reject an unnecessary blanket refusal. Score the answer against the actual evidence condition, rather than treating caution itself as the objective.

For longer conversations, write a turn-by-turn expectation ledger. Record which user corrections should carry forward, which references remain unresolved, and when the assistant should answer directly. If you run the same conversation several times, retain each complete result. In our proposed report, show the individual outcomes alongside any aggregate score and identify the number of executions included.

Give each language its own review cases

For a multilingual release, our recommendation is to author a separate set of conversations for each language. Choose locally natural requests and write acceptance statements in that language. Review terminology, clarification wording, and the language of the final response as explicit editorial criteria. These are proposed testing choices, not claims about measured performance differences.

Keep website publication checks separate. Google recommends different URLs for different language versions and links that let users choose another language version. Google's multilingual-site guidance. Those recommendations concern website organization; they do not supply an agent conversation score.

In our proposed language review, give the reviewer the localized source material as well as the localized conversation. Ask them to mark unsupported additions using the same evidence labels, while judging natural wording against a language-specific brief. If a source is intentionally available in only one language, declare that condition on the case sheet and specify the expected response language before running the test.

Assemble a release record with named decisions

Our proposed release record has five parts: the scope agreement, the document snapshots, the executed conversations, the scored case sheets, and the unresolved issues. For each issue, name an owner and the evidence required to close it. Establish your blocking conditions before the review begins; for example, your team might decide that invented collection exceptions block acceptance of that task.

After an edit, rerun the cases your team identifies as affected and preserve the earlier results. Record whether the change satisfied the original requirement. If you change the requirement itself, document that as a separate decision. We recommend keeping the release decision distinct from the numerical summary and attaching a named reviewer to it.

Max AI describes itself on its registration page as an operating system for AI agents. Max AI registration. The review packet described here is our proposed working method, rather than a statement that the product provides particular testing features.

To adopt the method, start with one task and a small, explicitly named case set. Write the expected behavior, execute the conversations, inspect every factual assertion, and document the team's acceptance decision.