uclab-dev-prod — The Current Cluster Setup

8 min read

The lab has moved on. The old posts built things up on a Raspberry Pi 5 k3s cluster — GitOps for the blog, hardening it, signing images, Postgres with CloudNativePG. This post is a snapshot of where the new cluster, uclab-dev-prod, actually stands today.

It’s the same ideas, rebuilt properly: Talos Linux on vSphere, provisioned by a self-hosted Sidero Omni, networked with Cilium, and managed end-to-end by Flux. Below is the whole thing, layer by layer, plus the handful of things that bit me during the move.

The shape of it

Sidero Omni (omni.uclab.dev)
   │  clones VMs from a vSphere content-library template
   ▼
Talos v1.13.8  — 3 control-plane + 3 workers, Kubernetes v1.36.3
   │  CNI: none, kube-proxy: disabled
   ▼
Cilium  — CNI + kube-proxy replacement + Gateway API + Hubble
   ▼
Flux  — reconciles infra → apps → monitoring from Git

Everything below the Omni template is declarative and lives in one Git repo. If the cluster burns down, I re-provision from Omni, install Cilium, bootstrap Flux, and it rebuilds itself.

Provisioning: Omni + Talos + vSphere

I stopped hand-rolling nodes. Omni owns the machine lifecycle now — the cluster is defined by a template that pins Talos v1.13.8, Kubernetes v1.36.3, and machine classes for 3 control-plane + 3 worker VMs. The vSphere infra provider clones them from a content-library image.

The one patch worth calling out disables the built-in networking so Cilium can own it:

cluster:
  proxy:
    disabled: true          # Cilium does kube-proxy replacement
  network:
    cni:
      name: none            # no built-in CNI
  extraManifests:
    - https://.../kubelet-serving-cert-approver/.../standalone-install.yaml
    - https://.../metrics-server/.../components.yaml

Talos being an immutable, API-driven OS means there’s no SSH, no config drift, no “what did I change on node 2 last year”. It’s a nice base to build GitOps on top of.

Networking: Cilium with cni: none

Because the CNI is none, there’s a small chicken-and-egg problem: Flux can’t run without pod networking, but I want Cilium managed by Flux. The answer is a two-phase install:

  1. Bootstrap — helm install cilium once, so nodes go Ready.
  2. Adoption — Flux carries a HelmRelease with the same release name (cilium), so it adopts and reconciles the existing install instead of fighting it.

Keep the release name identical and it just works. Cilium then gives me kube-proxy replacement, the Gateway API (all my ingress), and Hubble for flow-level network observability — which feeds straight into Prometheus later.

GitOps: how the repo is laid out

Flux is the spine. The repo splits into three concerns, each a Flux Kustomization, wired with dependsOn:

flux-system
  └─ infra-controllers   (cilium, cert-manager, eso, cloudnativepg,
       │                   longhorn, gateway, cloudflare, policy-controller)
       └─ infra-configs   (ClusterSecretStore, ClusterIssuer, ClusterImagePolicy)
            ├─ apps                    (forgejo, n8n, linkding, hugoblog, runner)
            └─ monitoring-controllers  (kube-prometheus-stack, loki, alloy)
                 └─ monitoring-configs (dashboards, ServiceMonitors, alerts)

The important rule: anything that installs a CRD (CloudNativePG, the Prometheus Operator, the sigstore policy-controller) goes in *-controllers; anything that is an instance of one of those CRDs goes in the dependent *-configs. Otherwise a cold bootstrap fails with “no matches for kind…”. Everywhere follows a base/ + per-cluster overlay pattern.

Secrets: External Secrets + Vault

No secrets in Git. A ClusterSecretStore (vault-backend-global) points ESO at my Vault, and each app declares an ExternalSecret that materializes a native Kubernetes Secret. The only thing I create by hand is a single vault-token — the root of trust. Once that’s seeded, everything else (DB creds, registry pull secrets, the Cosign key, Grafana creds, the Cloudflare tunnel token) flows in on its own.

That includes image-pull secrets, templated from discrete Vault fields into a dockerconfigjson:

target:
  template:
    type: kubernetes.io/dockerconfigjson
    data:
      .dockerconfigjson: |
        {"auths":{"{{ .server }}":{"username":"{{ .username }}",
        "password":"{{ .password }}",
        "auth":"{{ printf "%s:%s" .username .password | b64enc }}"}}}

Ingress: Cloudflare Tunnel → Cilium Gateway

Nothing is exposed directly. A single Cloudflare Tunnel forwards hostnames to the Cilium Gateway, and cert-manager issues a wildcard cert via DNS-01. grafana.uclab.dev, forgejo.uclab.dev, n8n.uclab.dev — all the same path.

The blog lives on the apex uclab.dev, and that one taught me a lesson: a wildcard doesn’t match the bare domain. *.uclab.dev covers blog.uclab.dev but not uclab.dev, and that bites in three independent places:

  • the Gateway listener (NoMatchingListenerHostname),
  • the TLS cert (the wildcard SAN doesn’t cover the apex — add uclab.dev as a second SAN),
  • the Cloudflare tunnel ingress (the *.uclab.dev rule falls through to the 404 catch-all).

Each needed its own explicit apex entry. Obvious in hindsight; annoying to chase across three layers.

Storage: Longhorn

Longhorn provides RWO and RWX volumes. The gotcha I learned the hard way: RWX is NFS-backed (via Longhorn’s share-manager), and some workloads hate that. My Forgejo runner’s Docker-in-Docker daemon kept dying at container-create with:

fstype: overlay ... invalid argument

OverlayFS can’t be mounted on top of NFS. The runner is a single replica, so it never needed RWX in the first place — switching /var/lib/docker to a plain ReadWriteOnce volume (real ext4 block device) fixed it instantly. Rule of thumb now: single-writer workloads get RWO; only reach for RWX when you genuinely need concurrent multi-node writes.

Databases: CloudNativePG, versioned and self-recovering

Every app that needs Postgres gets a CloudNativePG cluster, with WAL + base backups streamed to MinIO through the Barman Cloud plugin. The pattern I settled on — borrowed from how I set up linkding — is worth writing down:

  • The cluster is versioned (forgejo-db-v1) and bootstraps by recovery from the object store rather than initdb.
  • It exposes a stable, un-versioned service (forgejo-db) via managed.services, so the app config never has to change when I bump to -v2.
spec:
  bootstrap:
    recovery:
      source: source
      database: forgejodb
      owner: forgejo
  externalClusters:
    - name: source
      plugin:
        name: barman-cloud.cloudnative-pg.io
        parameters:
          barmanObjectName: minio-objectstore
          serverName: forgejo-db      # recover FROM the old server path
  managed:
    services:
      disabledDefaultServices: [ro, r]
      additional:
        - selectorType: rw
          serviceTemplate:
            metadata: { name: forgejo-db }   # stable app-facing service

The sharp edge here: a Backup object saying completed does not mean you can recover. You need a real base backup and working continuous WAL archiving. I had a cluster whose archiving had been silently broken (it referenced an object store that was never created), so firstRecoverabilityPoint was empty — a rename-and-recover would have restored nothing. Now I always verify the base backup actually exists in the bucket before trusting it:

mc find minio/backups/<server>/ --name data.tar   # base backup present?

Observability: land on the answer, not a menu

The monitoring stack is kube-prometheus-stack (Prometheus, Alertmanager, Grafana) plus Loki for logs. Loki only stores logs, so I run Grafana Alloy as a DaemonSet to actually ship them, relabelled to OTEL conventions (k8s_namespace_name, service_name, …) so the dashboards’ LogQL works.

The part I’m happiest with: Grafana opens straight onto a curated “START HERE — Cluster Performance” dashboard instead of the stock home page. Red cards first — what’s broken, is the cluster full, which workload is eating CPU/RAM — and every workload links into a “Why Is This App Slow?” investigator. It turns “something’s slow” from a dashboard scavenger hunt into two clicks. Dashboards are provisioned as grafana_dashboard ConfigMaps picked up by the sidecar, and the home page is set with:

grafana.ini:
  dashboards:
    default_home_dashboard_path: /tmp/dashboards/performance-cockpit.json

CloudNativePG, Cilium, Hubble, and etcd all have dashboards too, and CNPG ships default alert rules straight into Alertmanager.

One small Grafana trap: its PVC is ReadWriteOnce, and a rolling update tries to attach the volume to the new pod before releasing the old one → a cross-node Multi-Attach deadlock. The fix is strategy: Recreate — correct for any single-replica, RWO-backed Deployment.

The apps, and how the blog ships itself

Running on top: Forgejo (git forge), n8n (self-hosted automation), linkding (bookmarks), a Forgejo Actions runner, and this blog.

The blog’s delivery is the neat bit. It’s a full self-hosted pipeline — the successor to the GitHub-based one I hardened earlier:

push to Forgejo ─► Forgejo Actions runner
   │  docker build ─► push forgejo.uclab.dev/antonis/uclab:<sha>
   │  Trivy scan ─► Cosign sign
   ▼
clone the GitOps repo, bump the image digest, commit, push
   ▼
Flux reconciles ─► rollout on the signed digest

A couple of migration scars here too: Cosign signing failed with invalid pem block until I base64-encoded the key in the CI secret (multi-line PEMs get mangled otherwise), and the cluster needed a pull secret for the Forgejo registry, not the old ghcr.io one.

Supply chain: only signed images run

The last piece extends the image-signing work from the Pi cluster to the new one. The sigstore policy-controller runs as an admission webhook, and a ClusterImagePolicy requires every forgejo.uclab.dev/antonis/uclab image to be Cosign-signed with my key. It’s opt-in per namespace via a label, and I rolled it out safely — mode: warn first, confirmed the signed digest verifies (a matching tag gets rejected “must be an image digest”, the signed digest passes), then flipped to enforce. Because CI signs before it deploys, an unsigned image can never reach the cluster in the first place.

Where it’s at

Six Talos nodes, one Git repo, and a blog that builds, signs, and deploys itself onto a cluster that refuses to run anything unsigned. The whole thing is reproducible from Omni → Cilium → Flux → one Vault token.

The moves that paid off: making Talos + Omni own the machines, pushing every secret through Vault, versioned self-recovering databases, and a Grafana that answers “is anything on fire?” the moment I open it. Next on the list is extending signature enforcement beyond the blog namespace and getting the alerting routes wired to something I’ll actually notice.