In plain words
A government service assistant needs to answer questions about procedures without making up fees or handling services outside its authority. This paper presents a system that runs on state infrastructure, looks up official procedure records, and can answer, ask the citizen to clarify, or abstain.
On a 200-query test, the full system more often put the right procedure first, or abstained when a request was outside the state catalog, than a simpler cutoff system. The comparison changes several parts of the system at once, so it does not show which part made the difference.
The research
A citizen-service assistant must give useful answers without inventing fees or requirements, and it must recognize requests beyond the state government’s authority. Can a system do this while keeping queries on local infrastructure and giving citizens a way to clarify uncertain requests?
The paper evaluates a local government-service assistant that retrieves official procedures, routes uncertain requests, and checks generated numbers against records.
The paper presents a local system that retrieves official procedures and decides whether to answer, ask for clarification, or abstain. The question is whether it can avoid false rejections while handling out-of-jurisdiction requests and limiting invented fees.
On 200 historical queries, the full system scored 86.50% strict Top-1 accuracy, against 76.50% for a binary-cutoff baseline without clarification. The measure requires the right procedure first for catalog queries or immediate abstention for out-of-catalog ones.
Separately, auditors found no numeric inventions in 40 production answers that used generation, supporting a bounded account of performance in the tested settings.
These results cover one state portal and its tested configurations. The production audit found no numeric inventions in 40 generated answers, but its confidence bound allows a population rate up to 8.81%; users also often did not reply to clarification menus.
What the evidence shows
- The service question
The paper presents a governed, sovereign retrieval-augmented generation (RAG) system for citizen services on the official portal of Tlaxcala, Mexico. It retrieves official procedure records and routes requests to an answer, a clarification request, or an abstention.
The design addresses the tension between answering useful questions, avoiding invented fees or requirements, keeping queries on local infrastructure, and handling requests outside state jurisdiction.
- How it handles uncertainty
The system checks whether retrieved procedures provide enough coverage and context before answering. If it proceeds toward a specific answer, a second stage compares candidate procedures and may offer a menu when alternatives are close, allowing the citizen to clarify rather than forcing a choice.
A post-generation check compares generated fees and validity figures with the retrieved record and falls back to an official template when a figure is absent.
- The benchmark evidence
On a stratified test of 200 historical queries, the full C7 system scored 86.50% strict Top-1 accuracy, compared with 76.50% for B2, a binary-cutoff baseline without a clarification route. This measure counts a query as correct when the true procedure ranks first for an in-catalog request or the system immediately abstains for an out-of-catalog request. B2 falsely rejected 38 of 130 valid in-catalog queries, illustrating the baseline’s tradeoff.
- What the evidence supports
The comparison is between whole configurations, not a test of one component in isolation: C7 includes tripartite routing and portal-specific rules, while B2 uses a binary cutoff. Separately, two analysts audited 40 production answers that used generation and found no numeric inventions relative to retrieved records.
The two-sided 95% upper bound was 8.81%, so that sample does not rule out a nonzero population rate.
- Where the answer stops
Clarification also has a practical limit in the observed cases: 16 of 30 citizens offered a menu did not reply at turn two. And the numeric guardrail checks whether figures occur in the retrieved record; it cannot ensure that the right procedure was retrieved, that concepts within a record were matched correctly, or that the record reflects later statutory changes. The portal-specific rules also need validation before transfer to other administrations.
Deep
- Original paper authors
- Andres Cruz Degante
- Submitted to arXiv
- License
- CC BY 4.0