Hugo on K3s

i decided to make a website. a static one. this one. with Hugo.
the main reason i have for needing a website is as a learning project, so i have some stuff to host in a Kubernetes cluster i’m running. the k3s cluster is also a learning project.
k3s running in Raspberry Pi 5 control plane and two Intel NUC8i7 worker nodes.
Cilium as cni. Flux as continuous delivery.
gitlab runs as selfhosted in a separate vm.
my repo structure according to fluxcd best practices.

Parts involved in this solution:

  • nginx(running from a stock image);
  • git and bash (running from a stock image)
  • gitlab running as self-hosted locally in a virtual machine.
  • Kubernetes:
    • flux, especially kustomize-controller and helm controller;
    • cloudflared tunnel to handle external trafffic;
    • the bitnami/nginx Helm chart;

Getting Started

i built my site by following the straight-forward Getting Started guide in the Hugo documentation.

i did hugo new site uclab and then cd uclab; git init. and then i picked a theme “inspired by terminal ricing aesthetics”, installing it like git submodule add https://github.com/joeroe/risotto.git themes/risotto; echo “theme = ‘risotto’” » hugo.toml.

at this point, my website is basically finished (i also changed the title in hugo.toml). i probably won’t be putting anything on it, so there’s no point fiddling with other details.

Getting Flux’d

to move my web stack into flux, i create a HelmRepository resource for the bitnami Helm charts:

# bitnami-helm.yaml
apiVersion: source.toolkit.fluxcd.io/v1beta2
kind: HelmRepository
metadata:
  name: bitnami
  namespace: uclab
spec:
  type: oci
  url: oci://registry-1.docker.io/bitnamicharts
  interval: 24h

and add a HelmRelease pointing to the repository/chart.

nginx-uclab.yaml
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: uclab
  namespace: uclab
spec:
  interval: 5m
  chart:
    spec:
      chart: nginx
      version: "22.2.1"
      sourceRef:
        kind: HelmRepository
        name: bitnami
        namespace: uclab
      interval: 1m
  values:
    cloneStaticSiteFromGit:
      enabled: true
      repository: "https://gitlab.uclab8.net/affragak/uclab.git"
      branch: main
      gitClone:
        command:
          - /bin/bash
          - -ec
          - |
            [[ -f "/opt/bitnami/scripts/git/entrypoint.sh" ]] && source "/opt/bitnami/scripts/git/entrypoint.sh"
            git clone {{ .Values.cloneStaticSiteFromGit.repository }} --branch {{ .Values.cloneStaticSiteFromGit.branch }} /tmp/app
            [[ "$?" -eq 0 ]] && shopt -s dotglob && rm -rf /app/* && mv /tmp/app/* /app/

    serverBlock: |-
      server {
        listen 8080;
        root /app/public;
        index index.html;
      }
    service:
      type: ClusterIP

when i push these to my flux [source repository], the Helm release rolls out.

Alt text

Traffic inbound to the cluster is handled by Cloudflare tunnel and direct it to the service.

cloudflare.yaml
---
apiVersion: traefik.io/v1alpha1
apiVersion: apps/v1
kind: Deployment
metadata:
  name: cloudflared
spec:
  selector:
    matchLabels:
      app: cloudflared
  replicas: 2 # You could also consider elastic scaling for this deployment
  template:
    metadata:
      labels:
        app: cloudflared
    spec:
      containers:
        - name: cloudflared
          image: cloudflare/cloudflared:latest
          args:
            - tunnel

            # Points cloudflared to the config file, which configures what
            # cloudflared will actually do. This file is created by a ConfigMap
            # below.
            - --config
            - /etc/cloudflared/config/config.yaml
            - run
          livenessProbe:
            httpGet:
              # Cloudflared has a /ready endpoint which returns 200 if and only if
              # it has an active connection to the edge.
              path: /ready
              port: 2000
            failureThreshold: 1
            initialDelaySeconds: 10
            periodSeconds: 10
          ports:
            - containerPort: 2000
              name: http-metrics
          volumeMounts:
            - name: config
              mountPath: /etc/cloudflared/config
              readOnly: true
            # Each tunnel has an associated "credentials file" which authorizes machines
            # to run the tunnel. cloudflared will read this file from its local filesystem,
            # and it'll be stored in a k8s secret.
            - name: creds
              mountPath: /etc/cloudflared/creds
              readOnly: true
      volumes:
        - name: creds
          secret:
            secretName: tunnel-credentials

        # Create a config.yaml file from the ConfigMap below.
        - name: config
          configMap:
            name: cloudflared
            items:
              - key: config.yaml
                path: config.yaml
---
# This ConfigMap is just a way to define the cloudflared config.yaml file in k8s.
# It's useful to define it in k8s, rather than as a stand-alone .yaml file, because
# this lets you use various k8s templating solutions (e.g. Helm charts) to
# parameterize your config, instead of just using string literals.
apiVersion: v1
kind: ConfigMap
metadata:
  name: cloudflared
data:
  config.yaml: |
    # Name of the tunnel you want to run

    tunnel: uclab

    credentials-file: /etc/cloudflared/creds/credentials.json

    # Serves the metrics server under /metrics and the readiness server under /ready
    metrics: 0.0.0.0:2000
    no-autoupdate: true

    ingress:
    - hostname: uclab.dev
      service: http://uclab-nginx:80

    # This rule sends traffic to the built-in hello-world HTTP server. This can help debug connectivity
    # issues. If hello.example.com resolves and tunnel.example.com does not, then the problem is
    # in the connection from cloudflared to your local service, not from the internet to cloudflared.
    - hostname: hello.example.com
      service: hello_world
    # This rule matches any traffic which didn't match a previous rule, and responds with HTTP 404.
    - service: http_status:404

Cloudflare tunnel secret stored in Vault.

cloudflare-secret.yaml
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: tunnel-credentials
  namespace: uclab
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend-global
    kind: ClusterSecretStore
  target:
    name: tunnel-credentials
    creationPolicy: Owner
  data:
    - secretKey: credentials.json
      remoteRef:
        key: cloudflare-tunnel-uclab
        property: file

And lastly the flux controller kustomization configuration.

# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: uclab
resources:
  - bitnami-helm.yaml
  - nginx-uclab.yaml
  - cloudflare.yaml
  - cloudflare-secret.yaml