Securing the CI/CD Chain
In the previous post we built a full GitOps pipeline: push to Forgejo, build a Hugo image, ship it to k3s via Flux. It worked. But it was naive in a few ways — running nginx as root, no image scanning, no signing, and deploying by mutable tag.
This post covers the fixes. Three independent improvements, each worth doing on its own:
- Unprivileged nginx — drop root from the container entirely
- Trivy — scan the image for HIGH/CRITICAL CVEs before it ever touches the registry
- Cosign — sign the image so the cluster can verify it came from CI
The full updated files are at the bottom. Here is the reasoning behind each change.
1 — Unprivileged nginx
The original Dockerfile used nginx:1.27-alpine. That image runs the master process as root (it needs to bind port 80). The worker processes drop to nginx, but root is still involved at startup.
The fix is nginxinc/nginx-unprivileged:1.27-alpine. This is the official upstream image maintained by the nginx team. It:
- Listens on 8080 instead of 80 (no privileged port, no root needed)
- Runs everything as the
nginxuser from the start - Ships with a conf that already expects unprivileged operation
The only catch: you still want to run apk upgrade to patch OS-level packages. Since the unprivileged image starts as nginx, you need to briefly switch to root to run the upgrade, then drop back:
# --- Builder Stage ---
FROM hugomods/hugo:0.157.0 AS builder
WORKDIR /site
COPY . .
RUN hugo --minify
# --- Final Stage ---
FROM nginxinc/nginx-unprivileged:1.27-alpine
USER root
RUN apk update && apk upgrade --no-cache
USER nginx
COPY --from=builder /site/public /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 8080
CMD ["nginx", "-g", "daemon off;"]
Nothing else in the k8s manifests needs to change — containerPort: 8080 was already there.
2 — Security Headers and nginx Hardening
While touching the nginx config, a few headers worth adding:
server_tokens off; # don't leak nginx version in error pages / headers
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
The always flag ensures headers are sent even on error responses, not just 200s. HSTS only makes sense because Cloudflare terminates TLS in front — the header tells browsers to enforce HTTPS for a year.
3 — Trivy: Scan Before Push
The workflow used to build, push, done. The new order is: build, scan, push only if clean.
Trivy runs against the locally-built image (via digest) before it ever reaches the registry. If it finds any unfixed HIGH or CRITICAL CVEs, the pipeline exits 1 and the image is never pushed.
- name: Build and push image
run: |
docker build -t ${{ env.IMAGE }}:${{ env.TAG }} .
docker push ${{ env.IMAGE }}:${{ env.TAG }}
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${{ env.IMAGE }}:${{ env.TAG }})
echo "IMAGE_WITH_DIGEST=${DIGEST}" >> $GITHUB_ENV
- name: Cache Trivy DB
uses: actions/cache@v3
with:
path: ~/.cache/trivy
key: trivy-db-${{ runner.os }}
- name: Scan image with Trivy
run: |
if ! command -v trivy &> /dev/null; then
apt-get update && apt-get install -y wget apt-transport-https gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor > /usr/share/keyrings/trivy.gpg
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" > /etc/apt/sources.list.d/trivy.list
apt-get update && apt-get install -y trivy
fi
trivy image \
--exit-code 1 \
--severity HIGH,CRITICAL \
--format table \
--ignore-unfixed \
${{ env.IMAGE_WITH_DIGEST }}
A few notes:
--ignore-unfixedskips CVEs where no fix exists yet — these are noise you can’t act on.--exit-code 1is what actually gates the pipeline. Without it, Trivy is just reporting.- The Trivy DB cache step matters on self-hosted runners where the DB has to be downloaded every run. On Forgejo runners it cuts scan time significantly.
- Trivy is installed via the official apt repo rather than
curl | bash, which is the right call in a CI environment.
4 — Cosign: Sign the Image
After scanning, the image gets signed with Cosign using a key stored in Forgejo secrets. This gives you a verifiable chain: this exact digest was produced by this CI run and nobody tampered with it between push and deploy.
- name: Install Cosign
uses: sigstore/[email protected]
- name: Sign image with Cosign
env:
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
run: |
echo "${{ secrets.COSIGN_PRIVATE_KEY }}" > /tmp/cosign.key
cosign sign --yes --key /tmp/cosign.key \
--registry-referrers-mode=legacy \
--registry-username ${{ secrets.REGISTRY_USER }} \
--registry-password ${{ secrets.REGISTRY_TOKEN }} \
${{ env.IMAGE_WITH_DIGEST }}
rm /tmp/cosign.key
--registry-referrers-mode=legacy is needed for Forgejo’s container registry, which doesn’t yet support the OCI 1.1 referrers API. Without this flag, Cosign tries to use the newer spec and fails.
The signature is stored as a separate tag in the same registry, referenced by the image digest.
Generating the key pair
If you haven’t done this yet:
cosign generate-key-pair
# outputs cosign.key (private) and cosign.pub (public)
Add to Forgejo secrets:
| Secret | Value |
|---|---|
COSIGN_PRIVATE_KEY |
contents of cosign.key |
COSIGN_PASSWORD |
the passphrase you set |
Keep cosign.pub in your repo or somewhere accessible — you’ll need it to verify.
Verifying a signature
cosign verify \
--key cosign.pub \
forgejo.uclab.dev/affragak/uclab@sha256:<digest>
5 — Deploy by Digest, Not Tag
The original sed in the workflow replaced the tag portion of the image ref:
# old
sed -i "s|image: ${{ env.IMAGE }}:.*|image: ${{ env.IMAGE }}:${{ env.TAG }}|" $DEPLOY_FILE
The new version replaces the entire image: line with the full digest ref:
# new
sed -i "s|image: .*|image: ${{ env.IMAGE_WITH_DIGEST }}|" $DEPLOY_FILE
A digest ref (image: registry/repo@sha256:abc123...) is immutable. A tag ref is not — someone could push a different image to the same tag. Deploying by digest means Kubernetes will always pull exactly the image that was scanned and signed, nothing else.
This also pairs naturally with Cosign verification: the digest in the manifest is the same one the signature is attached to.
Updated Files
Dockerfile
# --- Builder Stage ---
FROM hugomods/hugo:0.157.0 AS builder
WORKDIR /site
COPY . .
RUN hugo --minify
# --- Final Stage ---
FROM nginxinc/nginx-unprivileged:1.27-alpine
USER root
RUN apk update && apk upgrade --no-cache
USER nginx
COPY --from=builder /site/public /usr/share/nginx/html
COPY nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 8080
CMD ["nginx", "-g", "daemon off;"]
nginx.conf
server {
listen 8080 default_server;
server_name _;
root /usr/share/nginx/html;
index index.html;
server_tokens off;
access_log /dev/stdout;
error_log /dev/stderr warn;
gzip_static on;
add_header X-Content-Type-Options "nosniff" always;
add_header X-Frame-Options "DENY" always;
add_header Referrer-Policy "strict-origin-when-cross-origin" always;
add_header X-XSS-Protection "1; mode=block" always;
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;
location / {
try_files $uri $uri/ $uri.html =404;
}
location ~* \.(css|js|png|jpg|jpeg|gif|ico|svg|woff|woff2)$ {
expires 1y;
add_header Cache-Control "public, immutable";
access_log off;
}
location ~* \.html$ {
add_header Cache-Control "no-cache";
}
location ~ /\.(?!well-known).* {
deny all;
}
}
.forgejo/workflows/build.yml
name: Build and Deploy
on:
push:
branches: [main]
env:
REGISTRY: forgejo.uclab.dev
IMAGE: forgejo.uclab.dev/affragak/uclab
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Checkout Hugo repo (with submodules)
uses: actions/checkout@v4
with:
submodules: recursive
fetch-depth: 0
- name: Set image tag
run: |
SHORT_SHA=$(git rev-parse --short HEAD)
echo "TAG=${SHORT_SHA}" >> $GITHUB_ENV
- name: Login to Forgejo registry
run: echo "${{ secrets.REGISTRY_TOKEN }}" | docker login ${{ env.REGISTRY }} -u ${{ secrets.REGISTRY_USER }} --password-stdin
- name: Build and push image
run: |
docker build -t ${{ env.IMAGE }}:${{ env.TAG }} .
docker push ${{ env.IMAGE }}:${{ env.TAG }}
DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' ${{ env.IMAGE }}:${{ env.TAG }})
echo "IMAGE_WITH_DIGEST=${DIGEST}" >> $GITHUB_ENV
- name: Cache Trivy DB
uses: actions/cache@v3
with:
path: ~/.cache/trivy
key: trivy-db-${{ runner.os }}
- name: Scan image with Trivy
run: |
if ! command -v trivy &> /dev/null; then
apt-get update && apt-get install -y wget apt-transport-https gnupg
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor > /usr/share/keyrings/trivy.gpg
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb generic main" > /etc/apt/sources.list.d/trivy.list
apt-get update && apt-get install -y trivy
fi
trivy image \
--exit-code 1 \
--severity HIGH,CRITICAL \
--format table \
--ignore-unfixed \
${{ env.IMAGE_WITH_DIGEST }}
- name: Install Cosign
uses: sigstore/[email protected]
- name: Sign image with Cosign
env:
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
run: |
echo "${{ secrets.COSIGN_PRIVATE_KEY }}" > /tmp/cosign.key
cosign sign --yes --key /tmp/cosign.key \
--registry-referrers-mode=legacy \
--registry-username ${{ secrets.REGISTRY_USER }} \
--registry-password ${{ secrets.REGISTRY_TOKEN }} \
${{ env.IMAGE_WITH_DIGEST }}
rm /tmp/cosign.key
- name: Setup SSH for GitHub
run: |
mkdir -p ~/.ssh
echo "${{ secrets.GH_DEPLOY_KEY }}" > ~/.ssh/id_ed25519
chmod 600 ~/.ssh/id_ed25519
ssh-keyscan github.com >> ~/.ssh/known_hosts
- name: Clone k3s repo and update image tag
run: |
git clone [email protected]:affragak/pi5cluster.git /tmp/pi5cluster
cd /tmp/pi5cluster
git config user.name "forgejo-ci"
git config user.email "[email protected]"
DEPLOY_FILE="blog/base/uclab/deployment.yaml"
sed -i "s|image: .*|image: ${{ env.IMAGE_WITH_DIGEST }}|" $DEPLOY_FILE
git add $DEPLOY_FILE
git diff --staged --quiet && echo "No changes to commit" && exit 0
git commit -m "chore: deploy hugo ${{ env.TAG }}"
git push
Summary of Changes
| What | Before | After |
|---|---|---|
| Base image | nginx:1.27-alpine (runs as root) |
nginxinc/nginx-unprivileged:1.27-alpine |
| CVE scanning | None | Trivy, gates on HIGH/CRITICAL unfixed |
| Image signing | None | Cosign with key stored in Forgejo secrets |
| Deployment ref | Mutable tag (image:sha) |
Immutable digest (image@sha256:...) |
| nginx hardening | Minimal | server_tokens off + security headers |
None of these are dramatic changes. Together they close the obvious gaps in the original setup: a root-running container, no verification that the image is clean, no way to prove the image wasn’t tampered with after it left CI, and a mutable tag that Kubernetes could silently re-pull to a different image.