You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Danny Bessems bc92ca19b9 Update '' 4 weeks ago
ingress/Traefik2.x OpenCart deploy through plain lighttpd;Add Traefik certificate 1 month ago
services TfState-- 1 month ago
storage Store sealed secrets in version control;Add missing manifest 3 months ago
system Store sealed secrets in version control;Add missing manifest 3 months ago
.gitignore Define PVR apikey rules in separate file; add file to .gitignore 2 years ago Update '' 4 weeks ago

TODO: Files with sensitive data; migrate to SealedSecret

# line ??: services/Mastodon/deploy-Mastodon.yml


3 VM's provisioned with Ubuntu Server 18.04

additional lvm configuration
pvcreate /dev/sdb
vgcreate longhorn-vg /dev/sdb
lvcreate -l 100%FREE -n longhorn-lv longhorn-vg
ls /dev/mapper
mkfs.ext4 /dev/mapper/longhorn--vg-longhorn--lv
#! add "UUID=<uuid> /mnt/blockstorage ext4 defaults 0 0" to /etc/fstab
mkdir /mnt/blockstorage
mount -a

K3s cluster

On first node (replace <floating ip> with the correct value):

curl -sfL | sh -s - server --cluster-init --disable local-storage,traefik --tls-san <floating ip>
cat /var/lib/rancher/k3s/server/token
kubectl config view --raw

Install kube-vip (replace <interface name> and <floating ip> with the correct values):

ctr image pull
cat << EOF > /var/lib/rancher/k3s/server/manifests/kube-vip.yml
$(ctr run --rm --net-host vip /kube-vip manifest daemonset --interface <interface name> --address <floating ip> --inCluster --taint --controlplane --services --arp --leaderElection)

On subsequent nodes (replace <floating ip> and <value from master> with the correct values):

curl -sfL | K3S_URL=https://<floating ip>:6443 K3S_TOKEN=<value from master> sh -s - server --disable local-storage,traefik

0) Configure automatic updates

Install Rancher's System Upgrade Controller:

kubectl apply -f

Apply a server (master node) and agent (worker node) plan:

kubectl apply -f system/UpgradeController/plan-Server.yml # -f system/UpgradeController/plan-Agent.yml

1) Secret management

Prereq: latest kubeseal release

1.1) Install Helm Chart

See Bitnami Sealed Secrets:

helm repo add sealed-secrets
helm repo update
helm install sealed-secrets-controller -n kube-system sealed-secrets/sealed-secrets

Retrieve public/private keys (store these on a secure location!):

kubectl get secret -n kube-system -l -o yaml > BitnamiSealedSecrets.masterkey.yml

2) Persistent storage

2.1) storageClass for SMB (CIFS):


curl -skSL | bash -s master --

Store credentials in secret:

kubectl apply -f storage/csi-driver-smb/sealedSecret-CSIdriverSMB.yml

2.2) flexVolume for SMB (CIFS):

curl -Ls -o storage/flexVolSMB/daemonSet-flexVolSMB.yml

Override drivername to something more sensible (see storage/flexVolSMB/daemonSet-flexVolSMB.yml)

        - image: juliohm/kubernetes-cifs-volumedriver-installer:2.0
            - name: VENDOR
              value: mount
            - name: DRIVER
              value: smb

Perform installation:

kubectl apply -f storage/flexVolSMB/daemonSet-flexVolSMB.yml

Wait for installation to complete (check logs of all installer-pods), then delete daemonSet:

kubectl delete -f storage/flexVolSMB/daemonSet-flexVolSMB.yml

Store credentials in secret:

kubectl apply -f storage/flexVolSMB/sealedSecret-flexVolSMB.yml

2.3) storageClass for distributed block storage:

See Longhorn Helm Chart:

kubectl create namespace longhorn-system
helm repo add longhorn
helm install longhorn longhorn/longhorn --namespace longhorn-system --values=storage/Longhorn/chart-values.yml

Expose Longhorn's dashboard through IngressRoute:

kubectl apply -f storage/Longhorn/ingressRoute-Longhorn.yml

Log on to the web interface and delete the default disks on each node (mounted at /var/lib/longhorn) and replace them with new disks mounted at /mnt/blockstorage.

Add additional storageClass with backup schedule: After specifying a NFS backup target (syntax: nfs://servername:/path/to/share) through Longhorn's dashboard

kind: StorageClass
  name: longhorn-dailybackup
allowVolumeExpansion: true
  numberOfReplicas: "3"
  staleReplicaTimeout: "2880"
  fromBackup: ""
  recurringJobs: '[{"name":"backup", "task":"backup", "cron":"0 0 * * *", "retain":14}]'

Then make this the new default storageClass:

kubectl patch storageclass longhorn-dailybackup -p '{"metadata": {"annotations":{"":"true"}}}'
#kubectl delete storageclass longhorn

3) Ingress Controller

3.1) Create configMap, secret and persistentVolumeClaim

The configMap contains Traefik's static and dynamic config:

kubectl apply -f ingress/Traefik2.x/configMap-Traefik.yml

The secret contains credentials for Cloudflare's API:

kubectl apply -f ingress/Traefik2.x/sealedSecret-Traefik-Cloudflare.yml

The persistentVolumeClaim will contain /data/acme.json (referenced as existingClaim):

kubectl apply -f ingress/Traefik2.x/persistentVolumeClaim-Traefik.yml
3.2) Install Helm Chart

See Traefik 2.x Helm Chart:

helm repo add traefik
helm repo update
helm install traefik traefik/traefik --namespace kube-system --values=ingress/Traefik2.x/chart-values.yml
3.3) Replace IngressRoute for Traefik's dashboard:
kubectl apply -f ingress/Traefik2.x/ingressRoute-Traefik.yaml
kubectl delete ingressroute traefik-dashboard --namespace kube-system

4) GitOps

See ArgoCD:

kubectl create namespace argocd
kubectl apply -n argocd -f

Expose endpoints (see ArgoCD Ingress Configuration):

kubectl patch deployment -n argocd argocd-server --type='json' -p='[{"op": "add", "path": "/spec/template/spec/containers/0/command/-", "value": "--insecure"}]'
kubectl apply -f system/ArgoCD/ingressRoute-ArgoCD.yml

Retrieve initial password:

kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo

Login with username admin and the initial password, browse to User Info and Update Password.

5) Services

5.1) Adminer (SQL management)
kubectl apply -f services/Adminer/configMap-Adminer.yml
kubectl apply -f services/Adminer/deploy-Adminer.yml
kubectl apply -f services/Adminer/sealedSecret-Adminer.yml
5.2) Vaultwarden (password manager)

Requires mount.cifs' option nobrl

kubectl apply -f services/Bitwarden/deploy-Bitwarden.yml
kubectl apply -f services/Bitwarden/sealedSecret-Bitwarden.yml
5.3) DDclient (dynamic dns)
kubectl apply -f services/DDclient/deploy-DDclient.yml
kubectl apply -f services/DDclient/sealedSecret-DDclient.yml
5.4) DroneCI (contineous delivery)
kubectl apply -f services/DroneCI/deploy-DroneCI.yml
kubectl apply -f services/DroneCI/sealedSecret-DroneCI.yml
5.5) Gitea (git repository)
kubectl apply -f services/Gitea/deploy-Gitea.yml
5.6) Gotify (notifications)
kubectl apply -f services/Gotify/deploy-Gotify.yml
5.7) Guacamole (remote desktop gateway)

Requires specifying a uid & gid in both the securityContext of the MySQL container and the persistentVolume

kubectl apply -f services/Guacamole/deploy-Guacamole.yml
kubectl apply -f services/Guacamole/sealedSecret-Guacamole.yml

Wait for the included containers to start, then perform the following commands to initialize the database:

kubectl exec -i guacamole-<pod-id> --container guacamole -- /opt/guacamole/bin/ --mysql > initdb.sql
kubectl exec -i guacamole-<pod-id> --container mysql -- mysql -uguacamole -pguacamole guacamole < initdb.sql
kubectl rollout restart deployment guacamole
5.8) Lighttpd (webserver)

Serves various semi-containerized websites; respective webcontent is stored on fileshare

kubectl apply -f services/Lighttpd/configMap-Lighttpd.yml
kubectl apply -f services/Lighttpd/deploy-Lighttpd.yml
5.9) PVR namespace (automated media management)

Containers use shared resources to be able to interact with downloaded files

kubectl create secret generic --type=mount/smb smb-secret --from-literal=username=<<omitted>> --from-literal=password=<<omitted>> -n pvr
kubectl apply -f services/PVR/persistentVolumeClaim-PVR.yml
kubectl apply -f services/PVR/storageClass-PVR.yml
5.9.1) Overseerr (request management)
kubectl apply -f services/PVR/deploy-Overseerr.yml
5.9.2) Plex (media library)

Due to usage of symlinks, partially incompatible with SMB-share-backed storage

kubectl apply -f services/PVR/deploy-Plex.yml

After deploying, Plex server needs to be claimed (=assigned to Plex-account):

kubectl get endpoints Plex -n PVR

Browse to the respective IP address (http://:32440/web) and follow instructions.

5.9.3) Prowlarr (indexer management)
kubectl apply -f services/PVR/deploy-Prowlarr.yml
5.9.4) Radarr (movie management)
kubectl apply -f services/PVR/deploy-Radarr.yml
5.9.5) Readarr (book management)
kubectl apply -f services/PVR/deploy-Readarr.yml
5.9.6) SABnzbd (download client)
kubectl apply -f services/PVR/deploy-SABnzbd.yml
5.9.7) Sonarr (tv management)
kubectl apply -f services/PVR/deploy-Sonarr.yml
5.10) Shaarli (bookmarks/notes)
kubectl apply -f services/Shaarli/deploy-Shaarli.yml
5.11) Terraform backend (supporting database)
kubectl apply -f services/TfState/deploy-TfState.yml
kubectl apply -f services/TfState/sealedSecret-TfState.yml
5.12) Traefik-Certs-Dumper (certificate tooling)
kubectl apply -f services/TraefikCertsDumper/deploy-TraefikCertsDumper.yml
5.13) Unifi-Controller (network infrastructure management)
kubectl apply -f services/Unifi/deploy-Unifi.yml

Change STUN port to non-default:

kubectl exec --namespace unifi -it unifi-<uuid> -- /bin/bash
sed -e 's/# unifi.stun.port=3478/unifi.stun.port=3479/' -i /data/
kubectl rollout restart deployment --namespace unifi unifi

Update STUN url on devices: doesn't seem to work

ssh <username>@<ipaddress>
sed -e 's|stun://<ipaddress>|stun://<ipaddress>:3479|' -i /etc/persistent/cfg/mgmt

6) Miscellaneous

Various notes/useful links

  • Replacement for not-yet-deprecated kubectl get all -A:

    kubectl get $(kubectl api-resources --verbs=list -o name | paste -sd, -) --ignore-not-found --all-namespaces
  • DaemonSet to configure nodes' sysctl fs.inotify.max-user-watches:

    kubectl apply -f system/InotifyMaxWatchers/daemonSet-InotifyMaxWatchers.yml
  • Debug DNS lookups within the cluster:

    kubectl run -it --rm dnsutils --restart=Never -- nslookup [-debug] [fqdn]


    kubectl run -it --rm busybox --restart=Never --image=busybox:1.28 -- nslookup [-debug] [fqdn]
  • Delete namespaces stuck in Terminating state:
    First check whether there are any resources still present; preventing the namespace from being deleted:

    kubectl api-resources --verbs=list --namespaced -o name \
      | xargs -n 1 kubectl get --show-kind --ignore-not-found -n <namespace>

    Any resources returned should be deleted first (worth mentioning: if you get an error error: unable to retrieve the complete list of server APIs, you should check kubectl get apiservice for any apiservice with a status of False)
    If there are no resources left in the namespace, and it is still stuck terminating, the following commands remove the blocking finalizer (this is a last resort, you are bypassing protections put in place to prevent zombie processes):

    kubectl get namespace <namespace> -o json | jq -j '.spec.finalizers=null' > tmp.json
    kubectl replace --raw "/api/v1/namespaces/<namespace>/finalize" -f ./tmp.json
    rm ./tmp.json