One mini-PC in a bedroom in Mexico City, run as a small production
environment: metrics, logs, alerting, encrypted secrets, least-privilege containers, and
an AI agent that reads the telemetry and answers questions about the system.
liveAOOSTAR mini-PC · Ryzen 9 8945HS · ProxmoxDocker Composebehind Cloudflare Tunnel — no open ports
19
containers across two Compose stacks (12 app · 7 observability)
0ports
exposed to the internet — all ingress via Cloudflare Tunnel
0priv
privileged application containers (down from 2)
15s
metric scrape interval · 15-day retention
7d
container-log retention in Loki
4
alert rules → routed to Discord
Architecture
How a request enters, and how the system watches itself.
Two subsystems. On top, the request path: nothing is port-forwarded — a
Cloudflare Tunnel is the only door in, and internal tools sit behind Zero-Trust Access.
Below, the observability plane the box uses to monitor and explain its own behaviour.
The teal, dashed edges are the AIOps loop: the same agent that serves the public
chat reads live metrics and logs to answer questions like “why did CPU spike?” or
“show errors in the last hour.”*cAdvisor registers only the
root cgroup on this host — see the decisions below.
Engineering decisions
The trade-offs, stated plainly.
A homelab is easy to make impressive and hard to make honest. These are the
calls that shaped it — each one is a choice with a cost, not a feature checkbox.
01 · Least privilege over convenience
Was
The public security service ran privileged: true with host networking — full kernel access, ironically the least-safe container.
Now
cap_drop: ALL plus a single capability (DAC_READ_SEARCH) to read one log file; bridged networking; no-new-privileges.
Trade-off
Lost host network/port visibility. Moved that to node-exporter, which is the correct source anyway.
02 · Secrets encrypted at rest
Was
~40 secrets in a plaintext .env file.
Now
SOPS + age: an encrypted, version-controllable file; the private key can live off the box; deploy decrypts on the fly.
Trade-off
On a single host, key and ciphertext coexist, so the runtime gain is modest. The real wins are no plaintext at rest and clean disaster recovery — and I say so rather than overselling it.
03 · Rejected a tempting shortcut
Option
Mount the host /proc into the app to keep host metrics after dropping host networking.
Chose
Not to. A read-only /proc mount exposes /proc/PID/environ of other processes — a path to other containers’ secrets.
Result
CPU/RAM/disk still read correctly from shared procfs; host network stats come from node-exporter. No new attack surface.
04 · An honest limit, not a forced fix
Problem
cAdvisor won’t register per-container metrics on this host.
Root cause
Docker’s newer overlayfs (containerd-snapshotter) driver; cAdvisor can’t map layers to names. Tested privileged — it doesn’t help, because it isn’t a permissions problem.
Call
Documented it and kept the container unprivileged rather than adding privilege for no gain. Host metrics and logs cover the real questions.
05 · AIOps that is safe to expose
Design
The agent’s tools are strictly read-only (PromQL, LogQL, alert state) with per-day token and USD budgets.
Why
It’s reachable on the public internet. Read-only by construction means a prompt can’t mutate infrastructure; budgets cap runaway cost.
Detail
The model's tool-calling broke on long tool descriptions — fixed by trimming schemas and moving the PromQL cheatsheet into the system prompt.
Metrics, logs, and alerting wired end-to-end, with rules that route to a human channel.
Security by design
Least-privilege containers, encrypted secrets, Zero-Trust edge, and shortcuts refused on purpose.
AI engineering
A tool-using agent grounded in real telemetry, read-only and budget-capped for safe public exposure.
Operational judgment
Failure modes named, trade-offs stated, and one limitation left honest instead of papered over.
On scope: this is a personal homelab — a single mini-PC, not a datacenter.
That’s the point. The constraint is what forces real engineering decisions, and the goal is to
operate it the way a production system is operated, at a scale I fully control.