How to evaluate a document Q&A assistant

Give a reviewer one familiar procedure and its current source file. Ask them to write down what a correct answer must contain before they open the assistant. That gives the pilot a standard independent of how convincing the answer sounds.

Download the document Q&A pilot scorecard (CSV). It opens in a spreadsheet, with sample questions, expected evidence, and space to record the actual result and person responsible for a fix.

Run three different tests on one guide

Start with a known answer. In the sample employee handbook, an employee requests time off in the HR portal and submits dates for manager approval. Ask “How do I request time off?” A passing answer identifies where to go, what to submit, and who approves it. Open the citation and check all three details on the page.

Change a detail. Ask “I selected my dates. Is my time off already approved?” A useful answer distinguishes submitting a request from getting approval. A reply that treats submission as approval changes what the employee might do next, even if it links to the right handbook.

Ask for something the guide never states. Try “What is our interplanetary travel reimbursement rate?” Record whether the assistant identifies the missing guidance. A plausible dollar amount or an unrelated citation is a reason to revise the assistant before giving it to the team.

Score the answer and the source separately

Record the answer the reviewer saw, the page the link opened, and the sentence that supports the next action. Use Pass when the instruction and citation both hold up. Use Revise when the action is right but the evidence is incomplete or points to a different page. Use Block for a fabricated policy, an approval the source never grants, or a link the intended reader should not be able to open.

Suppose an assistant correctly names the HR portal but cites the receiving SOP. The employee may still submit the request, but the reviewer cannot verify the instruction from that citation. Mark the source check as needing revision. This distinction tells the maintainer whether to examine retrieval or the wording of the answer.

Decide what goes into the pilot

Run the same questions after a source or model change. Keep the source edition, test date, observed reply, and decision in separate columns. Give each unresolved item an owner. If the source itself is silent, that owner can add approved guidance or decide that a person should handle the question. Use the completed sheet to agree which questions the first team can rely on.