Project 01 — Production
Horong · KEIAdminSuperv
On-prem RAG for internal regulations — every answer cites its source, or refuses.
In administration, finance and audit, a wrong answer is an incident. So the goal was never "sounds plausible" — it was not being wrong. 599 documents are chunked at the article level, embedded with KURE-v1, retrieved dense top-20, reranked on-prem to top-5, then passed through a trust gate before generation. If the evidence isn't there, it refuses.
FastAPI · Ollama (Qwen3.5-9B GGUF) · Chroma · KURE-v1 · bge-reranker-v2-m3 · Next.js 14 · MLflow · Cloudflare Zero Trust
100%
of answers carry the closing disclaimer. The model dropped it 19% of the time; code guarantees it now. Up from 80.6%.
82.9%
strict Hit@1 after adding the reranker, from 60.0%.
6,044
chunks across 599 documents, plus 4,116 statute chunks kept physically separate.
[regulation, article] source, with evidence cards you can open in place
Serving under load — measured, then explained.
A concurrency sweep against the dev instance, not a guess. Throughput saturates at two concurrent requests; everything above that becomes queueing. Generation speed stays flat at 14–15 tok/s regardless of load — which is what isolates the bottleneck to non-batched, serial generation rather than raw GPU capacity.
| Concurrency | p50 | p95 | Throughput | Generation |
|---|---|---|---|---|
| 1 | ~15 s | 18.0 s | 3.9 req/min | 13.9 tok/s |
| 2 | 28.3 s | 30.3 s | 4.3 req/min | 14.4 tok/s |
| 4 | 50.0 s | 55.2 s | 4.3 req/min | 15.0 tok/s |
| 8 | 74.9 s | 111.2 s | 3.8 req/min | 15.0 tok/s |
Enough for the few dozen internal users it serves. If it needed to scale, the order is clear: move to a server with continuous batching, then resolve the driver constraint pinning it to Vulkan, then add replicas. Token figures are character-based approximations — the endpoint doesn't return usage.
Determinism over goodwill.
Amounts, percentages and deadlines missing from the evidence get a warning attached by code, not at the model's discretion. Tables broken by HWP conversion are barred from being quoted numerically. Repealed articles are demoted as evidence.
It grades itself, daily.
200 questions on weekdays, 500 × 3 rounds at weekends — generated, answered, scored and published automatically, with results tracked in MLflow.
Rejection is a result too.
Hybrid BM25 retrieval measured no gain, so it ships off by default. A model swap A/B on identical evidence — 57.3% vs 40.8% — decided which model runs.
War story — the metric was the bug. Daily accuracy fell 90 → 68 overnight. The service hadn't regressed: 85% of the question sample rotates each day, so the aggregate could never prove improvement in the first place. Splitting retake and new-question cohorts showed errors on repeated questions going 7 → 1, and identified the drop as a sampling artefact within 30 minutes. Two separate measurement bugs surfaced the same way — a refusal regex counting correct answers as refusals, and a grader marking exact answers "partial" for three days.
