Architecture Overview

Goals & constraints

Goals

• One on-prem AI platform serving ~50 daily users (peak ~10–15 concurrent) with Greek + English LLM chat, RAG over company knowledge, and workflow automation.
• Integrate the systems people already live in: Microsoft 365/Teams and SoftOne ERP. Do not create a parallel universe.
• Support the AI portfolio (../08-ai-portfolio/_index.md) from simple assistants to approval-gated agents.
• Operable by a 2–4 person IT team with part-time TPL support — minimal ops burden is a first-class requirement, not a nice-to-have.

Constraints

• On-prem mandatory. Defense/public-sector tender business and customer commercial data make cloud LLM APIs a non-starter for core workloads. Model inference never leaves the building.
• No existing AI hardware, no ML team. Everything below must be buy-and-run, not research.
• Greek language quality is a hard requirement (see llm-serving-stack.md).
• Server room in Piraeus HQ with realistic power/cooling limits — validate before ordering (hardware-and-sizing.md).
• Budget discipline: single-site, single-cluster; HA is a growth option (Option C), not day one.

Guiding principles

1. On-prem first. Sensitive inference and data stay on our hardware. Cloud only for non-sensitive tooling (Entra ID is already there) and explicitly approved edge cases.
2. Open-source first. vLLM, Qdrant, PostgreSQL, n8n, Open WebUI, Prometheus. No per-seat AI licensing; exit costs stay near zero.
3. Boring-tech bias. PostgreSQL over exotic stores, Docker Compose over Kubernetes, one Linux distro (Ubuntu LTS), one database engine. Every extra technology is a permanent tax on a small IT team.
4. One platform, not a tool zoo. One chat UI, one gateway, one vector DB, one workflow engine. New tools enter only by replacing something or by decision record.
5. Everything as code. VM definitions, Compose files, n8n flows (exported), model configs, dashboards — all in Git. The platform must be rebuildable from the repo + backups.

High-level architecture

flowchart TB
    subgraph users["Users (~50, Greek/English)"]
        U1[Office staff]
        U2[Sales / tenders]
        U3[Warehouse / ops]
    end

    SSO[Entra ID SSO<br/>OIDC + MFA]

    subgraph access["Access layer"]
        OWUI[Open WebUI<br/>chat + RAG UI]
        TEAMS[Teams bot / link-out]
        N8N[n8n<br/>workflows & agents]
    end

    GW[LiteLLM gateway<br/>routing · quotas · audit]

    subgraph serving["Model serving (GPU server)"]
        VLLM[vLLM<br/>primary 70B + Krikri-8B]
        EMB[Embedding + reranker server<br/>BGE-M3 · bge-reranker · Whisper]
    end

    subgraph data["Data layer"]
        QD[(Qdrant<br/>vectors + ACL payloads)]
        PG[(PostgreSQL<br/>app DBs · semantic layer · audit)]
    end

    ING[Ingestion pipeline<br/>n8n + LlamaIndex]

    subgraph sources["Source systems"]
        S1[SoftOne ERP]
        S2[M365: SharePoint/Teams/Mail]
        S3[Obsidian vault - git]
        S4[PPE e-shop]
    end

    subgraph mon["Monitoring"]
        PR[Prometheus + Grafana<br/>DCGM GPU metrics]
        LK[Loki logs]
        LF[Langfuse LLM traces]
    end

    users --> SSO --> access
    OWUI --> GW
    TEAMS --> GW
    N8N --> GW
    GW --> VLLM
    GW --> EMB
    OWUI --> QD
    N8N --> PG
    ING --> EMB
    ING --> QD
    ING --> PG
    S1 --> ING
    S2 --> ING
    S3 --> ING
    S4 --> ING
    serving -.metrics.-> mon
    access -.logs/traces.-> mon
    GW -.traces.-> LF

Environment separation: dev / staging / prod

For a company of this size, “three environments” must not mean triple hardware. It means isolation by VM/namespace on the same cluster, plus laptops:

Environment	Where	What it is	GPU access
dev	Engineer laptops + one small dev VM	Ollama with small models (Krikri-8B, Qwen3 small), local Compose stacks, n8n dev instance	None / laptop GPU
staging	Dedicated VMs on the prod Proxmox host, separate VLAN-tagged network + separate PostgreSQL/Qdrant instances	Same Compose files as prod, pointed at copies of data; where model upgrades, prompt changes, and new n8n flows are validated against the eval set	Time-shared slice of the prod GPUs (one GPU reservable off-hours, or MIG-style partition if hardware supports)
prod	Main VMs on the GPU server	The real thing	Dedicated

Rules: prod config changes go through Git; staging must run the exact artifact (image tags, model hashes) being promoted; the eval gate (llm-serving-stack.md § model lifecycle) runs in staging.

Deployment architecture: Docker Compose on Proxmox — not Kubernetes

Recommendation: Proxmox VE on the GPU server(s); Ubuntu LTS VMs with GPU passthrough; each service group as a Docker Compose stack; Compose files + .env templates in Git.

Why not k3s (yet):

• A 2–4 person IT team with no Kubernetes experience will spend its scarce hours on cluster plumbing (ingress, CSI, cert rotation, upgrades) instead of AI use cases. Compose failure modes are ones they can already debug.
• We run a single node. Kubernetes’ core value — scheduling across many nodes — is absent; its complexity is not.
• Every component we run ships first-class Compose deployment docs (Open WebUI, n8n, Qdrant, LiteLLM, Langfuse).

Revisit trigger: second GPU server for HA (Option C in hardware-and-sizing.md), or >3 teams shipping their own services. Then adopt k3s, not full-fat K8s, and migrate stack by stack. The everything-as-code discipline makes this migration mechanical.

VM layout (prod): vm-serving (GPU passthrough: vLLM, embedding server), vm-apps (Open WebUI, LiteLLM, n8n), vm-data (PostgreSQL, Qdrant), vm-mon (Prometheus, Grafana, Loki, Langfuse). Staging mirrors vm-apps/vm-data as smaller VMs.

Technical roles needed

Roles, not names. See ../04-departments/it.md for the department plan.

Role	Who provides	FTE estimate	Responsibilities
Platform engineer	Prime Products IT (upskilled sysadmin)	0.5 → 1.0 by M6	Proxmox, VMs, Compose stacks, backups, patching, monitoring response
AI engineer	TPL (fractional)	0.4 in build phase, 0.2 steady state	Serving config, RAG pipeline code, model evals, agent builds
Knowledge manager	Prime Products (business side, part-time)	0.3	KB curation, ingestion sign-off, gap reports (rag-and-data-pipeline.md)
MLOps / model lifecycle	TPL (fractional)	0.1 steady state	Quarterly model reviews, eval set maintenance, promotion gate
Integration owner (SoftOne/M365)	Prime Products IT + SoftOne partner	0.2 + external partner SLA	Semantic layer extracts, Graph API app registrations
Security officer (part-time)	Prime Products IT lead + TPL review	0.1	Access reviews, DPIA upkeep, incident response lead

Total steady-state internal load: roughly 1.5 FTE across the existing IT team plus business-side knowledge manager — consistent with the minimal-ops requirement. Risks of under-resourcing tracked in ../02-program/risk-register.md.

Maintenance calendar

Cadence	Tasks
Daily (automated, ~10 min human)	Backup job success check, Grafana overview glance, disk/GPU alerts triage, failed n8n executions review
Weekly (~2 h)	OS security patches on non-serving VMs, Compose image updates in staging, restore-log review, LLM usage/cost report to program lead
Monthly (~1 day)	Promote validated image updates to prod, access review (joiners/leavers), Qdrant snapshot verification, KB gap report, KPI extract for ../02-program/kpi-framework.md
Quarterly (~2–3 days)	Restore drill (full stack from backup to staging), model review + eval rerun (promotion/rollback decisions), firmware/driver/CUDA updates in a maintenance window, DPIA and RBAC matrix review, capacity review against growth path

Runbooks for each land in ../../07-Runbooks/ as the platform is built — see security-and-operations.md.