Docs / 08-transformation/05-technical/security-and-operations
Security & Operations
Security model, monitoring/logging stack, backup & DR with RTO/RPO targets, runbook pointers, and patching cadence for the AI platform.
Security & Operations
Security model
Network zoning
- Dedicated AI VLAN for all platform VMs, firewalled from the office LAN; only the reverse proxy (Caddy/Traefik, TLS-terminating) is reachable from the user VLAN. Management (Proxmox, IPMI) on a separate admin VLAN, VPN/bastion-only.
- Egress default-deny. The serving and data VMs have no internet access except during scheduled update windows (package mirrors, model downloads through a proxy allowlist). Models run effectively air-gapped — this is both a security control and the confidentiality story for tender/defense customers.
- Internal TLS everywhere via a private CA (step-ca or smallstep), certs automated.
Identity & access
- Entra ID OIDC + MFA on every human-facing service (integrations-m365-softone.md); Entra groups drive everything below.
| Role (Entra group) | Open WebUI | Restricted RAG collections | n8n | Grafana/Langfuse | Proxmox/SSH |
|---|---|---|---|---|---|
| All staff | chat, default collections | — | — | — | — |
| Department user (per dept) | + dept collection | own dept only | — | — | — |
| Tenders/finance/HR (restricted) | + restricted collection | own restricted set | — | — | — |
| Power user / flow builder | + workspaces | per grant | edit own flows | view | — |
| Platform admin | admin | all (audited) | admin | admin | yes (MFA, logged) |
| TPL engineer | scoped admin | per engagement scope | admin | admin | bastion, time-boxed |
- Per-document ACLs in RAG: enforced as Qdrant payload filters at query time from the user’s live group memberships — see rag-and-data-pipeline.md. Access reviews monthly (joiners/leavers) and quarterly (full matrix).
AI-specific
- Prompt injection: any agent that touches external content (inbound email, supplier docs, web) treats that content as untrusted data: it is delimited and never granted tool authority; tool-using agents run with least-privilege credentials; irreversible actions always sit behind human approval (../08-ai-portfolio/ai-agent-registry.md). Injection test cases go into the M4 eval set.
- System prompts and tool schemas are code-reviewed in Git like any other artifact.
GDPR
- DPIA recommended before go-live (large-scale processing of employee + customer data with novel tech) — TPL drafts, Prime Products DPO/counsel signs.
- Processing-register entries for: RAG index, meeting transcription, email triage, ERP semantic layer. Retention and erasure mechanics per rag-and-data-pipeline.md. Chat logs retained 12 months (troubleshooting + audit), then purged; users are told this in the UI.
Monitoring & logging
| Layer | Tooling | What we watch |
|---|---|---|
| Infra | Prometheus + node_exporter, DCGM exporter (GPU util, VRAM, temp, ECC), Proxmox metrics | GPU saturation/thermals, disk fill, backup job status |
| Services | Grafana dashboards per stack; blackbox probes on user-facing endpoints | Availability, latency P50/P95 |
| Logs | Loki + promtail from all containers | Errors, auth failures, ingestion failures |
| LLM | LiteLLM request logs → Langfuse (user, model, latency, tokens in/out, cost-equivalent, trace) + Open WebUI analytics | Usage by team/use case (feeds ../02-program/kpi-framework.md), slow prompts, error rates, quota burn |
| Alerting | Alertmanager → Teams channel (IT-ops) | Paging tiers: P1 platform down, P2 degraded, P3 job failed |
Baseline alerts: GPU > 90 °C or ECC errors, VRAM sustained > 95%, request P95 > 30 s, backup missed, ingestion failed twice, disk > 80%, cert expiry < 14 d, auth-failure spikes.
Backup & DR
What is backed up (3-2-1 mechanics in hardware-and-sizing.md): PostgreSQL (nightly dump + WAL archiving), Qdrant snapshots (nightly), Open WebUI + n8n + LiteLLM + Grafana configs (in Git + nightly VM backup), model-registry manifest (Git), full VM images via Proxmox Backup Server (nightly, immutable offsite copy). The Obsidian vault’s DR is its git remotes.
| Asset | RPO | RTO | Restore path |
|---|---|---|---|
| PostgreSQL | ≤ 1 h | ≤ 4 h | WAL point-in-time restore |
| Qdrant | ≤ 24 h (re-derivable from sources) | ≤ 8 h | Snapshot restore or full re-ingest |
| Chat/RAG core (user-visible service) | ≤ 24 h | ≤ 4 h | VM restore + alias flip |
| Full platform | ≤ 24 h | ≤ 1 business day | PBS restore, runbook-driven |
Restore drills quarterly: restore the full stack from backup into staging, run the smoke suite + 10 eval questions, log the drill in the vault. An untested backup is a rumor.
Runbooks (to be created)
As the platform lands, each procedure gets a runbook in ../../07-Runbooks/ (TPL runbook format): daily checks; model update & rollback; user onboarding/offboarding; ingestion-source addition; incident response (P1 comms template incl. who tells management); restore drill; update-window procedure. Runbook creation is part of each build milestone’s definition of done — tracked in ../07-mission-control/mission-control-blueprint.md.
Update & patching cadence
| What | Cadence | Path |
|---|---|---|
| OS security patches | Weekly (unattended-upgrades on non-serving VMs; serving VM in monthly window) | dev → staging → prod |
| Container images (apps) | Monthly, pinned tags bumped via Git PR | staging soak ≥ 1 week |
| vLLM / CUDA / NVIDIA driver | Quarterly maintenance window; driver+CUDA+vLLM as one tested bundle | staging eval-set run first |
| Models | Via lifecycle gate (llm-serving-stack.md) | staging eval → alias flip |
| Firmware (BIOS/BMC/GPU) | Quarterly window, vendor-advised only | with restore point |
No unpinned latest tags anywhere in prod. Every change is a Git commit; every window ends with the smoke suite green.