NLP Analytics Pipeline for Unstructured Text

Any large collection of free-text records — reported by many people, in natural language, with varying quality and consistency — is hard to make sense of at scale. What themes actually exist in the data? How do they cluster? Where are the gaps? How does retrieval even behave on this particular corpus? Without a dedicated analytical layer, none of these questions have real answers — only assumptions.

Rather than building a one-off analysis for a single dataset, this is a general-purpose NLP analytics pipeline — designed to make any large unstructured text corpus legible, not tied to any one client's data or domain. The pipeline runs a fixed sequence: ingest, prepare, embed, reduce, topic model, and thematically allocate — feeding three analytical surfaces (exploratory data analysis, a topic map, and a RAG explorer) from the same pre-computed outputs.

Records are cleaned and structured, embedded, then passed through UMAP for dimensionality reduction before topic modelling (Turftopic, with HDBSCAN clustering) surfaces the themes actually present in the data. An agentic LLM-based allocation step then builds a theme hierarchy from the raw topic clusters — automating what would otherwise be manual qualitative coding.

Over 22,000 records are indexed into a Qdrant vector store, powering a KNN retrieval explorer that surfaces the nearest neighbours to any query. An integrated RAG generation layer sits on top of this, producing thematic synthesis grounded directly in the corpus rather than a generic summary.

A multi-page Plotly Dash dashboard — exploratory data analysis, a topic map, and a RAG explorer — makes the data directly accessible to stakeholders, so they can explore and understand it themselves rather than waiting on a manual analysis every time a question comes up. Observability is built in throughout, tracking cost and performance across every pipeline run.

• Full pipeline built and operational — ingestion through thematic allocation, all running against a real, growing corpus
• 22,000+ records indexed in Qdrant, powering live KNN retrieval and RAG-based thematic synthesis
• Multi-page Dash dashboard shipped, giving stakeholders direct, self-service access to themes and data quality
• Agentic thematic allocation automating what was previously manual qualitative coding
• Project reporting and stakeholder summaries generated directly from the pipeline's outputs