The setting
Compare two dated GDS accounts of GOV.UK Chat. They are first-party reports, not an independent audit or a statement of present service performance.
The decision context
A retrospective evidence review, not a decision on the current GOV.UK service. Use the reports to decide what they can justify and what would require another evaluation.
The evidence to examine
Reported evidence
18 January 2024
Nearly 70% of 157 survey respondents found answers useful. Expert assessment nevertheless found answers fell short of the accuracy demanded for GOV.UK. Some users placed undue confidence in the government brand.
Source: Government Digital Service, The findings of our first generative AI experiment: GOV.UK Chat (2024) · Our early findings; survey n=157 and separate expert assessment ↓Reported evidence
16 March 2026
GDS reported accuracy scores increasing from an early 76% benchmark to 90%, assessed through subject experts and automated evaluation. It noted that an inaccurate answer might address only part of a question.
Source: Government Digital Service, 5 things we learned testing GOV.UK Chat (2026) · Headline finding 2: answer accuracy and assessment rules ↓AIPEX analysis
AIPEX reading question
Usefulness, completeness and accuracy are distinct. Before interpreting change over time, check evaluation questions, assessment rules and system versions.
What this does not establish
Survey respondents are a subset of users, not the full invited population. The 2026 post does not provide the evaluation-set denominator behind its headline accuracy in the text. Neither report supplies a confusion matrix: do not put these percentages into the binary calculator. Evaluation-set or scoring changes could affect comparisons over time; causal service benefits and rare serious failures need separate evidence.
Try the decision first
Your task.
What would you request before using these findings to justify a different public-service chatbot?
- Make a claim ledger: reported result, population/denominator, assessment method and the claim it can support. Do not merge survey usefulness with answer accuracy.
- Specify a same-question comparison with the existing information route: successful task completion, serious wrong advice and total user/staff time. Identify exclusions and unanswered questions.
- Write a bounded next step. Name the evidence needed before expanding to a different service, and who would have to approve it.
Read one defensible approach
AIPEX analysis for discussion. Other conclusions can be defensible if their evidence, assumptions and trade-offs are explicit.
- Keep a separate row for each dated claim. A usefulness survey does not validate accuracy. Ask for the evaluated question set, scoring rubric, assessor agreement, exclusions and severity breakdown; the published percentages alone do not supply these.
- Request a comparison against the existing route on the same tasks, including incorrect actions and unanswered questions. Do not manufacture a trend by subtracting percentages from different measures or evaluation sets.
- Use the accounts as evidence for questions to test, rather than approval for another chatbot. Require a locally relevant evaluation and a recorded decision on acceptable consequences, support and fallback before expanding scope.
Read the evidence / Reviewed 18 October 2026
Sources & scope.
AIPEX’s prompts and worked approaches are our own synthesis. Use the linked material to examine the method and its limits.
- Government Digital Service
The findings of our first generative AI experiment: GOV.UK Chat (2024) ↗
Our early findings; expert evaluation and user surveys
The delivery team’s account of an early experiment, not an independent evaluation or a description of the current service.
- Government Digital Service
5 things we learned testing GOV.UK Chat (2026) ↗
User testing, answer evaluation and the next phase
A later team account. Read alongside the 2024 report to avoid treating early limitations as an unchanging property of the service.
- UK Government
Guidance on the impact evaluation of AI interventions ↗
Evaluating AI interventions in their operational setting
Helps frame evaluation of the intervention and its workflow, alongside technical model testing.