Botkube

Botkube Installation Guide

Overview

This guide covers the installation of Botkube on a K3s Raspberry Pi 5 cluster with Slack integration, using HashiCorp Vault for secret management and FluxCD for GitOps deployment.

Architecture

  • Cluster: K3s on Raspberry Pi 5
  • GitOps: FluxCD
  • Secret Management: HashiCorp Vault + External Secrets Operator
  • Communication: Slack (Socket Mode)
  • Storage: Persistent volume for plugin caching

Prerequisites

  • K3s cluster running
  • FluxCD installed and configured
  • HashiCorp Vault deployed with KV secrets engine
  • External Secrets Operator installed with ClusterSecretStore configured
  • Slack workspace with admin access

Directory Structure

infrastructure/
├── controllers/base/botkube/
│   ├── kustomization.yaml
│   ├── kustomizeconfig.yaml
│   ├── namespace.yaml
│   ├── pvc.yaml
│   ├── release.yaml
│   ├── repository.yaml
│   └── values.yaml
└── configs/base/botkube/
    ├── botkube-slack-secrets.yaml
    └── kustomization.yaml

Step 1: Create Slack App

1.1 Create the App

  1. Go to https://api.slack.com/apps
  2. Click “Create New App” → “From scratch”
  3. Name it “Botkube” (or your preferred name)
  4. Select your workspace

1.2 Configure OAuth Scopes

Navigate to OAuth & Permissions and add these Bot Token Scopes:

  • app_mentions:read - View messages that directly mention the bot
  • channels:history - View messages in public channels
  • channels:read - View basic channel information
  • chat:write - Send messages
  • commands - Add shortcuts and/or slash commands
  • files:write - Upload files
  • users:read - View people in the workspace
  • users:read.email - View email addresses (optional)

1.3 Enable Socket Mode

  1. Go to Socket Mode in the sidebar
  2. Enable Socket Mode
  3. Generate an App-Level Token with connections:write scope
  4. Save the token (starts with xapp-)

1.4 Get Bot Token

  1. Go to OAuth & Permissions
  2. Click “Install to Workspace”
  3. Copy the Bot User OAuth Token (starts with xoxb-)

1.5 Configure App Display

  1. Go to Basic Information
  2. Under Display Information:

1.6 Create Slack Channel

  1. Create a channel in Slack (e.g., k3s-pi5cluster)
    • Note: Channel name must be lowercase, no special characters
  2. Invite your Botkube app to the channel: /invite @Botkube

Step 2: Store Secrets in Vault

Store your Slack tokens in Vault:

# Store both tokens in Vault
vault kv put secret/botkube-slack-secrets \
  botToken="xoxb-your-bot-token-here" \
  appToken="xapp-your-app-token-here"

# Verify the secrets
vault kv get secret/botkube-slack-secrets

Note: Adjust the path (secret/botkube-slack-secrets) based on your Vault mount point.

Step 3: Deploy Botkube with FluxCD

3.1 Create Namespace

File: infrastructure/controllers/base/botkube/namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: botkube

3.2 Add Helm Repository

File: infrastructure/controllers/base/botkube/repository.yaml

apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata:
  name: botkube
  namespace: botkube
spec:
  interval: 1h
  url: https://charts.botkube.io/

3.3 Create PersistentVolumeClaim for Plugin Cache

File: infrastructure/controllers/base/botkube/pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: botkube-plugins-cache
  namespace: botkube
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: synology-iscsi-storage  # Adjust to your storage class

Why PVC? Plugin downloads on Raspberry Pi can take 2-3 minutes. Caching plugins in a PVC reduces restart time from minutes to seconds.

3.4 Configure Helm Values

File: infrastructure/controllers/base/botkube/values.yaml

## Format: communications.{alias}
communications:
  "default-group":
    socketSlack:
      enabled: true
      channels:
        "default":
          name: "k3s-pi5cluster"  # Your Slack channel name (lowercase)
          bindings:
            executors:
              - k8s-default-tools
            sources:
              - k8s-err-events
              - k8s-recommendation-events
      botToken: ""  # Will be overridden by existingCommunicationsSecretName
      appToken: ""

## Global Botkube configuration
settings:
  clusterName: k3s-pi5cluster  # Customize your cluster name

# Use existing secret for communications config
existingCommunicationsSecretName: "botkube-slack-secrets"

# Mount PVC for plugin caching
extraVolumes:
  - name: plugins-cache
    persistentVolumeClaim:
      claimName: botkube-plugins-cache

extraVolumeMounts:
  - name: plugins-cache
    mountPath: /tmp

# Increase health check timeouts for slower Pi hardware
deployment:
  livenessProbe:
    initialDelaySeconds: 180  # 3 minutes
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 12
  readinessProbe:
    initialDelaySeconds: 180
    periodSeconds: 10
    timeoutSeconds: 5
    failureThreshold: 12

3.5 Create HelmRelease

File: infrastructure/controllers/base/botkube/release.yaml

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
  name: botkube
  namespace: botkube
spec:
  releaseName: botkube
  interval: 15m
  chart:
    spec:
      chart: botkube
      version: 1.*
      sourceRef:
        kind: HelmRepository
        name: botkube
        namespace: botkube
  valuesFrom:
    - kind: ConfigMap
      name: botkube-values

3.6 Create Kustomize Configuration

File: infrastructure/controllers/base/botkube/kustomizeconfig.yaml

nameReference:
  - kind: ConfigMap
    version: v1
    fieldSpecs:
      - path: spec/valuesFrom/name
        kind: HelmRelease

File: infrastructure/controllers/base/botkube/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: botkube
resources:
  - namespace.yaml
  - release.yaml
  - repository.yaml
  - pvc.yaml
configMapGenerator:
  - name: botkube-values
    files:
      - values.yaml=values.yaml
configurations:
  - kustomizeconfig.yaml

Step 4: Configure External Secrets

4.1 Create ExternalSecret

File: infrastructure/configs/base/botkube/botkube-slack-secrets.yaml

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: botkube-slack-secrets
  namespace: botkube
spec:
  refreshInterval: "15s"
  secretStoreRef:
    name: vault-backend-global  # Your ClusterSecretStore name
    kind: ClusterSecretStore
  target:
    name: botkube-slack-secrets
    creationPolicy: Owner
    template:
      engineVersion: v2
      data:
        comm_config.yaml: |
          communications:
            'default-group':
              socketSlack:
                enabled: true
                channels:
                  'default':
                    name: 'k3s-pi5cluster'
                    bindings:
                      executors:
                        - k8s-default-tools
                      sources:
                        - k8s-err-events
                        - k8s-recommendation-events
                botToken: "{{ .botToken }}"
                appToken: "{{ .appToken }}"
  data:
    - secretKey: botToken
      remoteRef:
        key: botkube-slack-secrets  # Vault path
        property: botToken
    - secretKey: appToken
      remoteRef:
        key: botkube-slack-secrets  # Vault path
        property: appToken

Key Points:

  • The ExternalSecret creates a Kubernetes Secret with a comm_config.yaml file
  • Botkube automatically loads this config via existingCommunicationsSecretName
  • Tokens are templated from Vault into the YAML config
  • Secret refreshes every 15 seconds

4.2 Create Kustomization

File: infrastructure/configs/base/botkube/kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: botkube
resources:
  - botkube-slack-secrets.yaml

Step 5: Deploy

5.1 Commit and Push

# Add all files to git
git add infrastructure/controllers/base/botkube/
git add infrastructure/configs/base/botkube/

# Commit
git commit -m "Add Botkube with Vault secrets integration"

# Push
git push

5.2 Trigger FluxCD Reconciliation

# Reconcile the configuration
flux reconcile kustomization infra-configs

# Reconcile the controllers
flux reconcile kustomization infra-controllers

# Watch the deployment
kubectl get pods -n botkube -w

5.3 Monitor Startup

# Check pod status
kubectl get pods -n botkube

# View logs
kubectl logs -f -n botkube -l app=botkube

# Check if secrets are synced
kubectl get externalsecret -n botkube
kubectl get secret botkube-slack-secrets -n botkube

Expected startup time: 3-5 minutes on first deployment (downloading plugins), then 10-30 seconds on subsequent restarts (using cached plugins).

Step 6: Verify Installation

6.1 Check Logs

Look for these successful initialization messages:

{"component":"Plugin Manager","msg":"Executor plugin registered successfully","plugin":"botkube/kubectl"}
{"component":"Plugin Manager","msg":"Executor plugin registered successfully","plugin":"botkubeExtra/helm"}
{"component":"Plugin Manager","msg":"Source plugin registered successfully","plugin":"botkube/kubernetes"}
{"bot":"SocketSlack","msg":"Botkube connected to Slack!"}

6.2 Test in Slack

Go to your Slack channel and try these commands:

@Botkube ping
@Botkube kubectl get nodes
@Botkube kubectl get pods -n botkube
@Botkube helm list -A

6.3 Check Notifications

Deploy a test pod to trigger Kubernetes event notifications:

kubectl run test-nginx --image=nginx -n default
kubectl delete pod test-nginx -n default

You should see notifications in your Slack channel.

Troubleshooting

Pod Stuck in CrashLoopBackOff

Symptoms: Pod restarts repeatedly, never becomes Ready.

Common causes:

  1. Health checks too aggressive for slow plugin downloads
  2. Invalid Slack tokens
  3. Secret not mounted correctly

Solutions:

# Check logs for errors
kubectl logs -n botkube -l app=botkube --tail=100

# Verify secret exists and has correct data
kubectl get secret botkube-slack-secrets -n botkube -o yaml

# Check if comm_config.yaml has actual tokens (not templates)
kubectl get secret botkube-slack-secrets -n botkube -o jsonpath='{.data.comm_config\.yaml}' | base64 -d

# Verify ExternalSecret is syncing
kubectl describe externalsecret botkube-slack-secrets -n botkube

Slack Connection Issues

Symptoms: “missing_scope” or connection errors in logs.

Solution: Verify all required OAuth scopes are added (see Step 1.2), then reinstall the app to your workspace and update the Bot Token in Vault.

Slow Startup Times

Symptoms: Pod takes 3+ minutes to become Ready.

Solution: This is normal on first start. The PVC caches plugins for faster subsequent starts. To verify caching is working:

# Check PVC is bound
kubectl get pvc -n botkube

# Verify plugins are cached
kubectl exec -n botkube -l app=botkube -- ls -lh /tmp/botkube/ /tmp/botkubeExtra/

ExternalSecret Not Syncing

Symptoms: ExternalSecret shows “SecretSyncedError” status.

Solutions:

# Check ExternalSecret status
kubectl describe externalsecret botkube-slack-secrets -n botkube

# Verify ClusterSecretStore is configured
kubectl get clustersecretstore vault-backend-global -o yaml

# Check Vault path and permissions
vault kv get secret/botkube-slack-secrets

# Verify the Vault path in ExternalSecret matches your Vault mount point

Channel Name Issues

Symptoms: Warning about invalid channel name.

Solution: Slack channel names must be lowercase with no special characters. Update both:

  1. The actual Slack channel name
  2. The channel name in botkube-slack-secrets.yaml

Configuration Options

Customize Notifications

Edit the values.yaml to change which events trigger notifications:

sources:
  "k8s-err-events":
    botkube/kubernetes:
      config:
        event:
          types:
            - error
            - warning  # Add this for warnings

Add Multiple Channels

In botkube-slack-secrets.yaml, add more channels:

channels:
  'default':
    name: 'k3s-pi5cluster'
    bindings:
      sources:
        - k8s-err-events
  'alerts':
    name: 'k3s-alerts'
    bindings:
      sources:
        - k8s-err-events
        - k8s-recommendation-events

Change Cluster Name

Update in values.yaml:

settings:
  clusterName: my-prod-cluster

Maintenance

Update Botkube

FluxCD automatically updates Botkube when new versions matching 1.* are released. To force an update:

flux reconcile helmrelease botkube -n botkube

Rotate Slack Tokens

  1. Generate new tokens in Slack App settings
  2. Update Vault: 
    vault kv patch secret/botkube-slack-secrets \
  botToken="xoxb-new-token" \
  appToken="xapp-new-token"
  3. Wait 15 seconds for ExternalSecret to sync
  4. Botkube automatically reloads (Config Watcher detects changes)

View Cached Plugins

kubectl exec -n botkube -l app=botkube -- ls -lh /tmp/

Clear Plugin Cache

# Delete PVC (will be recreated)
kubectl delete pvc botkube-plugins-cache -n botkube

# Restart pod to download fresh plugins
kubectl rollout restart deployment botkube -n botkube

Security Considerations

  1. Vault Access: Ensure only the External Secrets Operator service account can read the Botkube secrets path
  2. Slack Tokens: Never commit tokens to Git - always use Vault
  3. RBAC: Review Botkube’s RBAC permissions in the Helm chart
  4. Network Policies: Consider adding NetworkPolicy to restrict Botkube’s network access

Resources

Summary

You now have:

  • ✅ Botkube deployed via GitOps
  • ✅ Secrets managed securely in Vault
  • ✅ Automatic secret synchronization
  • ✅ Plugin caching for fast restarts
  • ✅ Slack integration with custom icon
  • ✅ Kubernetes event notifications

Enjoy your automated Kubernetes monitoring! 🚀