LLM Serving Stack

Serving: vLLM (prod) + Ollama (dev)

vLLM is the serving engine for everything on the GPU server:

• Continuous batching + PagedAttention — the only way 10–15 concurrent users share a 70B model responsively; naive serving (one request at a time) would collapse at 3 users.
• OpenAI-compatible API — every downstream tool (Open WebUI, n8n, LiteLLM, LlamaIndex) speaks it natively; zero adapter code.
• First-class AWQ/GPTQ/FP8 support, tensor parallelism across the 4 GPUs, de-facto industry standard ⇒ hiring/support story.

Two vLLM instances: one for the primary 70B (TP=2–4), one for Krikri-8B. Embeddings/reranker run under a lightweight server (vLLM embedding mode or Text-Embeddings-Inference); Whisper large-v3 via faster-whisper as a small service.

Ollama on developer laptops and the dev VM only — instant model pulls, runs on consumer hardware, same OpenAI-ish API. Never in prod (no continuous batching at our concurrency).

Rejected: TGI (weaker quant/model coverage momentum), NVIDIA Triton/TensorRT-LLM (best raw perf, but engine-build complexity a 2-person team should not own; revisit if latency ever becomes the bottleneck), llama.cpp server (great for dev, wrong tool at 15 concurrent).

Gateway: LiteLLM proxy

One LiteLLM proxy in front of all model endpoints:

• Single virtual API: apps call prime-large, prime-greek, prime-fast, prime-embed — models swap behind the alias without touching apps.
• Per-team/per-app API keys, budgets, and rate limits; central request/audit logging shipped to Langfuse (security-and-operations.md).
• If a cloud model is ever approved for a non-sensitive edge case, it’s a gateway config entry with its own key and audit trail — not a new integration.

Open WebUI’s built-in multi-model routing was considered and rejected as the system gateway: it only governs traffic through the UI, while n8n and future agents also need governed access. One chokepoint, one audit log.

UI: Open WebUI

Open WebUI over building custom: OIDC SSO (Entra ID), per-group RBAC and model visibility, built-in RAG hooks (we point them at our Qdrant pipeline rather than its internal store for prod collections), prompt/“workspace” sharing, admin usage panel, active project. Building a custom UI would burn the entire AI-engineering budget on table stakes. Rejected: LibreChat (comparable, weaker RAG/admin fit), AnythingLLM (desktop-oriented), custom (no).

Model matrix (Greek + English)

Model names current as of mid-2026 — re-verify exact HF repo IDs and license terms at deployment; newer checkpoints (e.g. Qwen3.x, Krikri updates) may supersede via the quarterly review.

Role	Model	Quant	Ctx (deployed)	Why
Primary generalist (prime-large)	Llama-3.3-70B-Instruct	AWQ 4-bit, FP8 KV	32k	Strongest open generalist at this size; solid English, workable Greek; huge ecosystem. Evaluate head-to-head vs Qwen3-32B on the Greek eval set in M4 — if 32B matches on our tasks, we free half the VRAM.
Greek specialist (prime-greek)	ILSP Llama-Krikri-8B-Instruct (Athena RC)	FP8/AWQ	32k	Continual-pretrained on Greek; best-in-class Greek fluency/tone at its size — Greek drafting, translation polish, customer-facing Greek text.
Fast/triage (prime-fast)	Qwen3-4B/8B-Instruct class	AWQ/FP8	16k	Email triage, classification, routing, title generation — pennies of VRAM, milliseconds of latency.
Embeddings (prime-embed)	BGE-M3	FP16	8k	Multilingual incl. Greek, dense+sparse in one model (feeds hybrid retrieval), strong cross-lingual Greek↔English. Chosen over multilingual-e5-large: comparable quality, but M3’s built-in sparse output simplifies hybrid search.
Reranker	bge-reranker-v2-m3	FP16	—	Same family as embedder, multilingual, cheap accuracy boost on top-k.
Speech-to-text	Whisper large-v3 (faster-whisper)	INT8/FP16	—	Best open Greek transcription — meeting scribe pipeline (integrations-m365-softone.md).
Document OCR/layout	Docling + Tesseract/PaddleOCR (grc/ell); optional Qwen-VL-class vision model later	—	—	Classic OCR + layout parsing is deterministic and cheap for scanned tenders/certificates; add a vision LLM only when eval shows classic OCR failing (stamps, handwriting).

Quantization strategy: AWQ 4-bit for the 70B (quality loss negligible vs the VRAM it buys), FP8 for ≤8B models and KV cache. GPTQ only as fallback where an AWQ quant doesn’t exist. No 3-bit/2-bit heroics — quality cliffs.

Context lengths: deploy 32k on large/Greek, 16k on fast — not the models’ 128k maximums. KV cache is the scarce resource (hardware-and-sizing.md); long-document work goes through RAG/chunking, not brute-force context. Raise per-model limits only with a measured use case.

Greek/English considerations

• Tokenizer tax: Greek costs ~2–3× tokens per word on Llama/Qwen tokenizers. Consequences: KV/context sizing assumes Greek-heavy traffic; latency budgets per Greek page are ~2–3× English; prompt templates for Greek tasks stay terse. Krikri’s Greek-extended tokenizer is materially more efficient — another reason it owns Greek drafting.
• Routing policy: default prime-large for reasoning/RAG (retrieval handles cross-lingual), prime-greek for Greek-native drafting and tone-sensitive customer text. n8n flows set the model explicitly; the UI exposes both with plain-language names.
• Evaluation plan (M4): build a ~100–150 item Greek/English eval set from real company documents — tender excerpts, requisition lists, customer emails, SOPs — covering RAG-grounded Q&A (with gold citations), Greek drafting quality (rubric-scored), extraction accuracy (IMPA codes, compliance clauses), and translation fidelity. This set is the promotion gate for every model change and lives in the vault under access control.

Model lifecycle

• Registry: a versioned manifest in Git (model ID, source, revision hash, quant, license, eval scores, deployed alias) — no weight blobs in Git. Weights cached on the hot NVMe tier, N−1 kept for rollback.
• Promotion gate: candidate model → staging vLLM → full eval-set run → score delta review → sign-off → alias flip in LiteLLM. Alias flip makes rollback a one-line config revert.
• Approval: platform engineer + TPL AI engineer propose; program steering approves anything user-visible, per ../02-program/governance-model.md.
• Quarterly review: re-scan the open-model landscape, rerun evals on incumbent vs top challenger, decide keep/upgrade. Time-boxed to 2 days.

Orchestration & automation layer

n8n is the workflow backbone — chosen over Dify/Flowise because for an M365 company the hard part is connectors, schedules, retries, and approvals, not chat-flow drawing: n8n has mature Microsoft (Graph, Teams, Outlook, SharePoint), HTTP/SQL, and webhook nodes; it’s self-hosted, license-friendly at our scale, and low-code enough for IT to own flows without a dev team. It handles: scheduled ingestion, email triage, Teams notifications, ERP extract jobs, and human-approval steps for agents.

LlamaIndex (Python services called by n8n or run as ingestion workers) owns everything retrieval-shaped: parsing, chunking, embedding, Qdrant upserts, hybrid query + rerank (rag-and-data-pipeline.md). Low-code tools do RAG badly at the margins (chunking per doc type, ACL filters); code does it well and testably.

LangChain/LangGraph only where a true multi-step agent needs stateful tool loops (e.g. Tender Extraction Agent) — not as a general framework. Division of labor: n8n = when and with-what-approvals; LlamaIndex = retrieval; LangGraph = the rare agent brain.

Deliberately NOT adopted at start

Not adopted	Why	Revisit when
Full-fat Kubernetes / K8s operators	Ops tax with zero single-node benefit	Second server / HA (then k3s)
Dify / Flowise	Overlaps n8n+Open WebUI; adds a third UI paradigm to a tool-zoo we said no to	If business users must self-build chat apps and n8n proves too technical
Fine-tuning (incl. LoRA)	Prompting + RAG + Krikri covers Greek; tuning needs eval discipline we’re only building in M4	A high-value task where the eval set proves prompting has plateaued
Multi-agent frameworks (CrewAI etc.)	Every wave-1 use case is single-agent with human approval	An approved use case genuinely requires agent-to-agent handoff
Cloud LLM APIs	Confidentiality posture (tenders/defense)	Non-sensitive, explicitly classified workloads only, via LiteLLM with steering sign-off