GitOps repository

Hypervisor

1) Harvester Hyperconverged Infrastructure

[...]

Configure Harvester HCI nodes through cloud-init (requires node reboot):

kubectl apply -f system/Harvester/cloudinit-disable-nic-offloading.yaml

Downstream cluster(s)

Cluster configuration for RKE2 cluster using kube-vip with Traefik ingress controller:

apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
  [...]
  rkeConfig:
    chartValues:
      harvester-cloud-provider:
        [...]
        kube-vip:
          env:
            svc_election: 'true'
      [...]
      rke2-traefik:
        service:
          [...]
          spec:
            externalTrafficPolicy: Local

Traefik advanced configuration:

additionalArguments:
  - --providers.file.directory=/etc/traefik/dynamic
  - --providers.file.watch=true
  - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s
certificatesResolvers:
  default:
    acme:
      dnsChallenge:
        propagation:
          delayBeforeChecks: 5m0s
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53
      email: <omitted>
      storage: /data/acme.json
deployment:
  initContainers:
    - command:
        - sh
        - -c
        - touch /data/acme.json; chown 65532 /data/acme.json; chmod -v 600 /data/acme.json
      image: busybox:latest
      name: volume-permissions
      securityContext:
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      volumeMounts:
        - mountPath: /data
          name: traefik-data
  kind: Deployment
env:
  - name: CF_API_EMAIL
    valueFrom:
      secretKeyRef:
        key: CF_API_EMAIL
        name: traefik-cloudflare
  - name: CF_API_KEY
    valueFrom:
      secretKeyRef:
        key: CF_API_KEY
        name: traefik-cloudflare
extraObjects:
  - apiVersion: v1
    data:
      config.yml: |
        http:
          middlewares:
            2fa-authentication:
              forwardAuth:
                address: "https://auth.spamasaurus.com/api/verify?rd=https://auth.spamasaurus.com/"
                trustForwardHeader: true
            security-headers:
              headers:
                forceSTSHeader: true
                stsSeconds: 315360000
                stsIncludeSubdomains: true
                stsPreload: true
        tls:
          options:
            defaults:
              minVersion: VersionTLS12
              sniStrict: false
              curvePreferences:
                - secp521r1
                - secp384r1
                - secp256r1
              cipherSuites:
                - TLS_AES_128_GCM_SHA256
                - TLS_AES_256_GCM_SHA384
                - TLS_CHACHA20_POLY1305_SHA256
                - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
                - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
                - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
                - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
                - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
                - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
                - TLS_FALLBACK_SCSV
    kind: ConfigMap
    metadata:
      name: traefik-file-provider
      namespace: kube-system
ingressRoute:
  dashboard:
    enabled: true
    entryPoints:
      - websecure
    matchRule: Host(`ingress.lab.spamasaurus.com`)
    middlewares:
      - name: 2fa-authentication@file
      - name: security-headers@file
logs:
  general:
    level: INFO
persistence:
  enabled: true
  name: traefik-data
ports:
  web:
    http:
      redirections:
        entryPoint:
          permanent: true
          scheme: https
          to: websecure
  websecure:
    forwardedHeaders:
      insecure: true
    http:
      tls:
        certResolver: default
        domains:
          - main: '*.pvr.spamasaurus.com'
          - main: '*.lab.spamasaurus.com'
          - main: '*.spamasaurus.com'
            sans:
              - spamasaurus.com
          - main: '*.bessems.com'
            sans:
              - bessems.com
          - main: '*.bessems.eu'
            sans:
              - bessems.eu
          - main: '*.gabaldon.eu'
            sans:
              - gabaldon.eu
          - main: '*.gabaldon.nl'
            sans:
              - gabaldon.nl
          - main: '*.itch.fyi'
            sans:
              - itch.fyi
        options: defaults@file
providers:
  kubernetesCRD:
    ingressClass: ""
service:
  annotations:
    cloudprovider.harvesterhci.io/ip-pool: <ippoolname>
    cloudprovider.harvesterhci.io/ipam: pool
  spec:
    externalTrafficPolicy: Local
    type: LoadBalancer
volumes:
  - mountPath: /etc/traefik/dynamic
    name: traefik-file-provider
    type: configMap

2) Persistent storage

2.1) CSI plugin for SMB (CIFS):

kubectl apply -f storage/csi-driver-smb/application-csi-driver-smb.yaml

2.2) Harvester CSI plugin

See Harvester CSI Driver

3) GitOps

3.1) Install Helm Chart

See ArgoCD:

helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argo-cd -n argo-cd --create-namespace argo/argo-cd --values system/ArgoCD/chart-values.yml

Retrieve initial password:

kubectl get secret -n argocd argocd-initial-admin-secret -oyaml | yq e '.data.password | @base64d'

Login with username admin and the initial password, browse to User Info and Update Password.

3.1) Adopt through GitOps

kubectl apply -f system/ArgoCD/application-argo-cd.yaml

4) Secret management

Prereq: latest kubeseal release

kubectl apply -f system/SealedSecrets/application-sealed-secrets-controller.yaml

Retrieve public/private keys (store these on a secure location!):

kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > BitnamiSealedSecrets.masterkey.yml

Restoring public/private keys:

kubectl apply -f BitnamiSealedSecrets.masterkey.yml
kubectl rollout restart deployment -n kube-system sealed-secrets-controller

5) Services

5.1) Gitea (git repository)

Required for all other workloads

kubectl apply -f services/Gitea/application-gitea.yaml

5.2) Argus (release management)

kubectl apply -f services/Argus/application-argus.yaml

5.3) Authelia (single sign-on))

kubectl apply -f services/Authelia/application-authelia.yaml

5.4) Vaultwarden (password manager)

kubectl apply -f services/Vaultwarden/application-vaultwarden.yaml

5.5) DDclient (dynamic dns)

kubectl apply -f services/DDclient/application-ddclient.yaml

5.6) Gotify (notifications)

kubectl apply -f services/Gotify/application-gotify.yaml

5.7) Webtop (remote desktop)

kubectl apply -f services/Webtop/application-webtop.yaml

5.8) Lighttpd (webserver)

kubectl apply -f services/Lighttpd/application-lighttpd.yaml

5.9) PVR toolsuite (automated media management)

API-keys whitelisted in ingressroutes:

spec:
  routes:
  - match: Host(`<fqdn>`) && (Headers(`X-Api-Key`, `<secret>`) || Query(`apikey`, `<secret>`))
    [...]

5.9.1) Jellyfin (media library)

kubectl apply -f services/PVR/Jellyfin/application-jellyfin.yaml

5.9.2) Jellyseerr (media requests management)

kubectl apply -f services/PVR/Jellyseerr/application-jellyseerr.yaml

5.9.3) Prowlarr (indexer management)

kubectl apply -f services/PVR/Prowlarr/application-prowlarr.yaml

5.9.4) Radarr (movie management)

kubectl apply -f services/PVR/Radarr/application-radarr.yaml

5.9.5) SABnzbd (download client)

kubectl apply -f services/PVR/SABnzbd/application-sabnzbd.yaml

5.9.6) Sonarr (tv management)

kubectl apply -f services/PVR/Sonarr/application-sonarr.yaml

6) Miscellaneous

Various notes/useful links

• Replacement for not-yet-deprecated kubectl get all -A:

  kubectl get $(kubectl api-resources --verbs=list -o name | paste -sd, -) --ignore-not-found --all-namespaces
  

• DaemonSet to configure nodes' sysctl fs.inotify.max-user-watches:

  kubectl apply -f system/InotifyMaxWatchers/daemonSet-InotifyMaxWatchers.yml
  

• Debug DNS lookups within the cluster:

  kubectl run -it --rm dnsutils --restart=Never --image=gcr.io/kubernetes-e2e-test-images/dnsutils -- nslookup [-debug] [fqdn]
  

  or

  kubectl run -it --rm busybox --restart=Never --image=busybox:1.28 -- nslookup api.github.com [-debug] [fqdn]
  

• Memory-leak liveness probe:

  livenessProbe:
    exec:
      command:
        - sh
        - -c
        - test $(cat /proc/1/smaps | grep -i pss |  awk '{Total+=$2} END {print int(Total/1024)}') -le <limit>