RAG Autocomplete System for Accessibility Audits

Every accessibility issue an auditor logs needs a structured "where it happened" field — a precise description of the page, device, and element affected, written in a consistent format the whole team can rely on. Multiplied across a busy audit cycle, this is a real drag on throughput: the same kinds of statements get typed out fresh every time, even when a near-identical issue has already been documented dozens of times before.

The underlying data already existed — years of accumulated audit history sitting in YouTrack. It just wasn't being used to help write the next issue faster.

The system embeds a new issue's context and retrieves the most similar past incidents from a real vector index, then drafts a completion grounded in them. The page/device clause is inserted deterministically by code — never written by the model — with an explicit instruction against inventing detail the auditor never gave. The auditor reviews and commits rather than writing from scratch.

Raw multi-year audit history is ingested from YouTrack, then split at the incident level — each issue's "where it happened" field parsed into individual location statements, with train/val/test splits partitioned by parent issue so one issue's incidents never leak across splits.

Embeddings are computed incrementally: content-hashing means only new or changed records are ever re-sent to the embedding API, keeping the index current without re-processing the whole corpus on every run.

Retrieval runs behind one shared interface with a switchable backend — NumPy, Qdrant, or Chroma — so the choice of retrieval strategy is a decision made on evidence, not assumption. The three backends were benchmarked head-to-head on real queries to pick the right one for production.

A separate evaluation framework runs structured comparisons between the RAG approach and an earlier in-context-learning (ICL) method, producing reports that directly inform prompt and retrieval design decisions.

The service runs as a FastAPI app, integrated directly into the audit team's YouTrack workflow — an auditor triggers the workflow on a real issue, and the drafted completion is written straight back to the "where it happened" field. It's been tested through repeated live runs against real audit data, currently in beta ahead of full rollout to the wider auditing team.

Observability is built in throughout — live request tracing plus batch pipeline-step logging — so accuracy, latency, and cost are all visible as the system moves from prototype to full production use.

• Full RAG pipeline built end-to-end — ingestion, incremental embedding, retrieval, and generation — grounded in a real, growing incident corpus
• Three retrieval backends (NumPy, Qdrant, Chroma) implemented behind one interface and benchmarked head-to-head on real queries
• Structured RAG-vs-ICL evaluation framework in place, producing evidence rather than guesswork for prompt and retrieval decisions
• Tested through repeated live runs integrated with the real YouTrack workflow, now in beta ahead of full rollout
• Full observability — request-level tracing and batch pipeline logging — built in from the start