RAG & Data Pipeline

Vector DB: Qdrant

Qdrant is the vector store:

• Ops simplicity: single binary/container, no external dependencies (Milvus needs etcd + MinIO + Pulsar — a distributed system a 2-person team should not carry).
• Payload filtering is first-class and fast — our per-user ACL and department/sensitivity filters run inside the vector search, which is the security model (security-and-operations.md).
• Native snapshots slot into the backup plan; built-in hybrid (dense + sparse) matches BGE-M3’s dual output.

When pgvector would suffice: if we only ever ran one collection of <1–2M chunks with simple filters, pgvector inside our existing PostgreSQL would be one less service — genuinely tempting under the boring-tech principle. We expect multiple collections, heavy filtered search, and sparse+dense hybrid, where Qdrant clearly earns its keep. Decision stands; pgvector remains the documented fallback if Qdrant ever feels like overkill in practice.

PostgreSQL: the relational backbone

One PostgreSQL 16 instance (with WAL archiving) hosts, in separate databases: Open WebUI state, n8n, Langfuse, the ingestion catalog (every document: source, hash, chunk map, sensitivity, owner, last-synced), the SoftOne semantic layer (integrations-m365-softone.md), and the audit log. One engine to tune, back up, and secure.

Ingestion pipeline

flowchart LR
    SRC[Connectors<br/>git · Graph API · SoftOne · e-shop · scans] --> EXT[Extraction / OCR<br/>docling · Tesseract]
    EXT --> CH[Chunking<br/>per doc type]
    CH --> EM[Embedding<br/>BGE-M3 dense+sparse]
    EM --> META[Metadata enrichment<br/>dept · sensitivity · source · date · lang]
    META --> UP[Upsert<br/>Qdrant + ingestion catalog in PostgreSQL]

Orchestrated by n8n (schedules, webhooks, retries, failure alerts to Teams); the extract/chunk/embed stages are LlamaIndex worker code. Idempotent by content hash — re-syncs only touch changed documents; deletes propagate (source gone ⇒ chunks tombstoned within one sync cycle).

Chunking strategy per document type:

Doc type	Strategy	Size / overlap
Vault markdown (SOPs, KB)	Structure-aware: split on headings, keep heading breadcrumb in chunk	~400–800 tok, 10% overlap
Tenders / contracts	Section-aware (numbered clauses); never split a clause; parent-section retrieval	~600–1,000 tok, clause-aligned
Product sheets / PPE specs	One chunk per product/spec block + structured fields to metadata	~200–500 tok
Emails (selective)	One chunk per message, thread ID in metadata	≤800 tok
Meeting transcripts	Topic-segmented (from summarizer), speaker-tagged	~500–800 tok
Scanned archive	OCR → same as source type; low-confidence OCR flagged for human review	—

Sources & sync cadence

Source	Method	Cadence	Notes
Obsidian vault (this repo)	git pull → structure-aware ingest	On push (webhook) + nightly	The curated RAG core — highest-trust collection; see ../06-knowledge-base/obsidian-vault-blueprint.md
SharePoint / Teams files	Graph API delta queries	Hourly delta, nightly full reconcile	Scoped to approved sites/libraries only
Mailboxes (selective)	Graph API, named shared mailboxes (e.g. sales@, tenders@)	15-min delta	Never all mail; per-mailbox owner sign-off
SoftOne ERP	Views/API → PostgreSQL semantic layer; text fields (product names/descriptions) also embedded	Nightly + on-demand	Read-only; integrations-m365-softone.md
PPE e-shop catalog	Platform API/feed export	Nightly	Product Q&A + IMPA matching support
Scanned archive (certs, customs docs)	Watched folder → OCR pipeline	On drop	Backfill as a bounded project, not a default

Retrieval design

• Hybrid: BGE-M3 dense + sparse in one query (Qdrant hybrid), fused (RRF), then bge-reranker-v2-m3 over top ~25 → final top 5–8 to the LLM with source citations mandatory.
• ACL filtering at query time: every chunk carries department, sensitivity, source_system, allowed_groups. The retrieval service resolves the user’s Entra ID groups and injects a filter into the Qdrant query — the model never sees chunks the user couldn’t open in the source system. Sensitivity tiers: public-internal / department / restricted (tenders, HR, finance — explicit group lists).
• Cross-lingual (Greek query → English doc and vice versa): BGE-M3 handles cross-lingual retrieval natively; the answering model cites the source language and answers in the user’s language. Eval set (M4) includes cross-lingual pairs explicitly.
• Parent-document retrieval for clause-aligned types (retrieve chunk, feed parent section) — matters for tenders.

Data governance

• Source of truth rules: the vault beats SharePoint beats email; ERP data is only ever read from the semantic layer, never from stale exports. Conflicting duplicates: highest-trust source wins, others get deduped by content hash + a superseded_by pointer in the catalog.
• Who can ingest what: adding a source (site, mailbox, folder) requires data-owner sign-off recorded in the ingestion catalog; the knowledge manager approves vault-collection changes; anything restricted needs the department head + security officer.
• PII / GDPR: ingestion catalog doubles as the processing inventory feed (what personal data, from where, why). Retention: emails 24 months in the index, transcripts 12 months, then automatic tombstone; right-to-erasure = delete in source → propagates via reconcile, plus manual purge runbook. DPIA covers the pipeline (security-and-operations.md).
• No customer PII in dev/staging — staging syncs use masked or truncated copies.

Quality feedback loop

• Thumbs-down triage: Open WebUI feedback lands in PostgreSQL; weekly n8n digest to the knowledge manager, tagged by collection.
• Retrieval-miss log: answers with low retrieval scores or “no source found” are logged as candidate KB gaps.
• Monthly KB gap report to the knowledge manager: top unanswered/poorly-answered topics → new vault pages or ingestion-scope changes. This loop is a program KPI (../02-program/kpi-framework.md).