diff --git a/README.md b/README.md index 2dc4b81..e6f998a 100644 --- a/README.md +++ b/README.md @@ -1,63 +1,46 @@ -# Kubernetes.K3s.installLog -*3 VM's provisioned with Ubuntu Server 22.04* -
additional lvm configuration +# GitOps repository +### 1) Harvester Hyperconverged Infrastructure +[...] + +Configure Harvester HCI nodes through cloud-init (requires node reboot): ```shell -pvdisplay -pvcreate /dev/sdb -vgdisplay -vgcreate longhorn-vg /dev/sdb -lvdisplay -lvcreate -l 100%FREE -n longhorn-lv longhorn-vg -ls /dev/mapper -mkfs.ext4 /dev/mapper/longhorn--vg-longhorn--lv -#! add "UUID= /mnt/blockstorage ext4 defaults 0 0" to /etc/fstab -mkdir /mnt/blockstorage -mount -a +kubectl apply -f system/Harvester/cloudinit-disable-nic-offloading.yaml ``` -
+### 2) Persistent storage -## K3s cluster -On first node (replace `` with the correct value): +#### 2.1) CSI plugin for SMB (CIFS): ```shell -curl -sfL https://get.k3s.io | sh -s - server --cluster-init --disable local-storage,servicelb --tls-san -cat /var/lib/rancher/k3s/server/token -kubectl config view --raw -``` -Install kube-vip (replace `` and `` with the correct values): -```shell -ctr image pull ghcr.io/kube-vip/kube-vip:latest -cat << EOF > /var/lib/rancher/k3s/server/manifests/kube-vip.yml -$(curl https://kube-vip.io/manifests/rbac.yaml) ---- -$(ctr run --rm --net-host ghcr.io/kube-vip/kube-vip:latest vip /kube-vip manifest daemonset --interface --address --inCluster --taint --controlplane --services --arp --leaderElection) -EOF -``` -On subsequent nodes (replace `` and `` with the correct values): -```shell -curl -sfL https://get.k3s.io | K3S_URL=https://:6443 K3S_TOKEN= sh -s - server --disable local-storage,servicelb +kubectl apply -f storage/csi-driver-smb/application-csi-driver-smb.yaml ``` -### 0) Configure automatic updates -Install Rancher's [System Upgrade Controller](https://rancher.com/docs/k3s/latest/en/upgrades/automated/): -```shell -kubectl apply -f https://github.com/rancher/system-upgrade-controller/releases/latest/download/system-upgrade-controller.yaml -``` -Apply a [server (master node)](https://code.spamasaurus.com/djpbessems/Kubernetes.K3s.installLog/src/branch/master/system/UpgradeController/plan-Server.yml) ~~and [agent (worker node)](https://code.spamasaurus.com/djpbessems/Kubernetes.K3s.installLog/src/branch/master/system/UpgradeController/plan-Agent.yml)~~ plan: -```shell -kubectl apply -f system/UpgradeController/plan-Server.yml # -f system/UpgradeController/plan-Agent.yml -``` +#### 2.2) Harvester CSI plugin +See [Harvester CSI Driver](https://docs.harvesterhci.io/v1.5/rancher/csi-driver) -### 1) Secret management -*Prereq*: latest `kubeseal` [release](https://github.com/bitnami-labs/sealed-secrets/releases) - -##### 1.1) Install Helm Chart -See [Bitnami Sealed Secrets](https://github.com/bitnami-labs/sealed-secrets#helm-chart): +### 3) GitOps +##### 3.1) Install Helm Chart +See [ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/#getting-started): ```shell -helm repo add sealed-secrets https://bitnami-labs.github.io/sealed-secrets +helm repo add argo https://argoproj.github.io/argo-helm helm repo update -helm install sealed-secrets-controller -n kube-system sealed-secrets/sealed-secrets +helm install argo-cd -n argo-cd --create-namespace argo/argo-cd --values system/ArgoCD/chart-values.yml +``` +Retrieve initial password: +```shell +kubectl get secret -n argocd argocd-initial-admin-secret -oyaml | yq e '.data.password | @base64d' +``` +Login with username `admin` and the initial password, browse to `User Info` and `Update Password`. + +##### 3.1) Adopt through GitOps +```shell +kubectl apply -f system/ArgoCD/application-argo-cd.yaml +``` + +### 4) Secret management +*Prereq*: latest `kubeseal` [release](https://github.com/bitnami-labs/sealed-secrets/releases) +```shell +kubectl apply -f system/SealedSecrets/application-sealed-secrets-controller.yaml ``` Retrieve public/private keys (*store these on a **secure** location!*): @@ -65,158 +48,72 @@ Retrieve public/private keys (*store these on a **secure** location!*): kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > BitnamiSealedSecrets.masterkey.yml ``` -### 2) Persistent storage - -#### 2.1) `storageClass` for SMB (CIFS): -See https://github.com/kubernetes-csi/csi-driver-smb: -```shell -curl -skSL https://raw.githubusercontent.com/kubernetes-csi/csi-driver-smb/master/deploy/install-driver.sh | bash -s master -- -``` -Store credentials in `secret`: -```shell -kubectl apply -f storage/csi-driver-smb/sealedSecret-CSIdriverSMB.yml -``` - -#### 2.2) `flexVolume` for SMB (CIFS): -```shell -curl -Ls https://github.com/juliohm1978/kubernetes-cifs-volumedriver/blob/master/install.yaml -o storage/flexVolSMB/daemonSet-flexVolSMB.yml -``` -Override drivername to something more sensible (see [storage/flexVolSMB/daemonSet-flexVolSMB.yml](https://code.spamasaurus.com/djpbessems/Kubernetes.K3s.installLog/src/branch/master/storage/flexVolSMB/daemonSet-flexVolSMB.yml)) -```yaml -spec: - template: - spec: - containers: - - image: juliohm/kubernetes-cifs-volumedriver-installer:2.0 - ... - env: - - name: VENDOR - value: mount - - name: DRIVER - value: smb - ... -``` -Perform installation: -```shell -kubectl apply -f storage/flexVolSMB/daemonSet-flexVolSMB.yml -``` -Wait for installation to complete (check logs of all installer-pods), then pause `daemonSet`: -```shell -kubectl patch daemonset juliohm-cifs-volumedriver-installer -p '{"spec": {"template": {"spec": {"nodeSelector": {"intentionally-paused": ""}}}}}' -``` -Store credentials in `secret`: -```shell -kubectl apply -f storage/flexVolSMB/sealedSecret-flexVolSMB.yml -``` - -#### 2.3) `storageClass` for distributed block storage: -See [Longhorn Helm Chart](https://longhorn.io/): -```shell -helm repo add longhorn https://charts.longhorn.io && helm repo update -helm install longhorn longhorn/longhorn --namespace longhorn-system --create-namespace --values=storage/Longhorn/chart-values.yml -``` - -Log on to the web interface and delete the default disks on each node (mounted at `/var/lib/longhorn`) and replace them with new disks mounted at `/mnt/blockstorage`. - - -### 3) Ingress Controller -Reconfigure default Traefik configuration: -See [Traefik 2.x Helm Chart](https://github.com/traefik/traefik-helm-chart) and [HelmChartConfig](https://docs.k3s.io/helm) -```shell -kubectl apply -f ingress/Traefik2.x/helmchartconfig-traefik.yaml -``` - -### 4) GitOps -##### 4.1) Install Helm Chart -See [ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/#getting-started): -```shell -helm repo add argo https://argoproj.github.io/argo-helm -helm repo update -helm install argo-cd -n argo-cd --create-namespace argo/argo-cd --values system/ArgoCD/chart-values.yml -``` - -Retrieve initial password: -```shell -kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d; echo -``` -Login with username `admin` and the initial password, browse to `User Info` and `Update Password`. - ### 5) Services -##### 5.1) [Argus]() (release management) +##### 5.1) [Gitea](https://gitea.io/) (git repository) +*Required for all other workloads* ```shell -kubectl apply -f services/Argus +kubectl apply -f services/Gitea/application-gitea.yaml ``` -##### 5.2) [Authelia]() (single sign-on)) + +##### 5.2) [Argus]() (release management) ```shell -kubectl apply -f services/Authelia +kubectl apply -f services/Argus/application-argus.yaml ``` -##### 5.3) [Vaultwarden](https://github.com/dani-garcia/vaultwarden) (password manager) -*Requires [mount.cifs](https://linux.die.net/man/8/mount.cifs)' option `nobrl`* +##### 5.3) [Authelia]() (single sign-on)) ```shell -kubectl apply -f services/Vaultwarden +kubectl apply -f services/Authelia/application-authelia.yaml ``` -##### 5.4) [DDclient](https://github.com/linuxserver/docker-ddclient) (dynamic dns) +##### 5.4) [Vaultwarden](https://github.com/dani-garcia/vaultwarden) (password manager) ```shell -kubectl apply -f services/DDclient +kubectl apply -f services/Vaultwarden/application-vaultwarden.yaml ``` -##### 5.5) [Gitea](https://gitea.io/) (git repository) +##### 5.5) [DDclient](https://github.com/linuxserver/docker-ddclient) (dynamic dns) ```shell -kubectl apply -f services/Gitea +kubectl apply -f services/DDclient/application-ddclient.yaml ``` ##### 5.6) [Gotify](https://gotify.net/) (notifications) ```shell -kubectl apply -f services/Gotify +kubectl apply -f services/Gotify/application-gotify.yaml ``` -##### 5.7) [Guacamole](https://guacamole.apache.org/doc/gug/guacamole-docker.html) (remote desktop gateway) -*Requires specifying a `uid` & `gid` in both the `securityContext` of the db container and the `persistentVolume`* +##### 5.7) [Webtop](#) (remote desktop) ```shell -kubectl apply -f services/Guacamole +kubectl apply -f services/Webtop/application-webtop.yaml ``` -Wait for the included containers to start, then perform the following commands to initialize the database: -```shell -kubectl exec -n guacamole -i guacamole- --container guacamole -- /opt/guacamole/bin/initdb.sh --postgresql > initdb.sql -kubectl exec -n guacamole -i guacamole- --container db -- psql -Uguacamole -f - < initdb.sql -kubectl rollout restart deployment -n guacamole guacamole -``` - ##### 5.8) [Lighttpd](https://www.lighttpd.net/) (webserver) -*Serves various semi-containerized websites; respective webcontent is stored on fileshare* ```shell -kubectl apply -f services/Lighttpd/configMap-Lighttpd.yml -kubectl apply -f services/Lighttpd/deploy-Lighttpd.yml +kubectl apply -f services/Lighttpd/application-lighttpd.yaml ``` -##### 5.9) PVR `namespace` (automated media management) -*Containers use shared resources to be able to interact with downloaded files* -```shell -kubectl create secret generic --type=mount/smb smb-secret --from-literal=username=<> --from-literal=password=<> -n pvr -kubectl apply -f services/PVR/persistentVolumeClaim-PVR.yml -kubectl apply -f services/PVR/storageClass-PVR.yml +##### 5.9) PVR toolsuite (automated media management) +*API-keys whitelisted in ingressroutes*: +```yaml +spec: + routes: + - match: Host(``) && (Headers(`X-Api-Key`, ``) || Query(`apikey`, ``)) + [...] ``` -###### 5.9.1) [Plex](https://www.plex.tv/) (media library) -*Due to usage of symlinks, partially incompatible with SMB-share-backed storage* +###### 5.9.1) [Jellyfin](#) (media library) ```shell -kubectl apply -f services/PVR/deploy-Plex.yml +kubectl apply -f services/PVR/Jellyfin/application-jellyfin.yaml ``` -After deploying, Plex server needs to be *claimed* (=assigned to Plex-account): +###### 5.9.2) [Jellyseerr](https://sonarr.tv/) (media requests management) ```shell -kubectl get endpoints Plex -n PVR +kubectl apply -f services/PVR/Jellyseerr/application-jellyseerr.yaml ``` -Browse to the respective IP address (http://:32440/web) and follow instructions. -###### 5.9.2) [Prowlarr](https://github.com/Prowlarr/Prowlarr) (indexer management) +###### 5.9.3) [Prowlarr](https://github.com/Prowlarr/Prowlarr) (indexer management) ```shell -kubectl apply -f services/PVR/deploy-Prowlarr.yml +kubectl apply -f services/PVR/Prowlarr/application-prowlarr.yaml ``` -###### 5.9.3) [Radarr](https://radarr.video/) (movie management) +###### 5.9.4) [Radarr](https://radarr.video/) (movie management) ```shell -kubectl apply -f services/PVR/deploy-Radarr.yml +kubectl apply -f services/PVR/Radarr/application-radarr.yaml ``` -###### 5.9.4) [SABnzbd](https://sabnzbd.org/) (download client) +###### 5.9.5) [SABnzbd](https://sabnzbd.org/) (download client) ```shell -kubectl apply -f services/PVR/deploy-SABnzbd.yml +kubectl apply -f services/PVR/SABnzbd/application-sabnzbd.yaml ``` -###### 5.9.5) [Sonarr](https://sonarr.tv/) (tv management) +###### 5.9.6) [Sonarr](https://sonarr.tv/) (tv management) ```shell -kubectl apply -f services/PVR/deploy-Sonarr.yml +kubectl apply -f services/PVR/Sonarr/application-sonarr.yaml ``` ### 6) Miscellaneous @@ -237,4 +134,12 @@ kubectl apply -f services/PVR/deploy-Sonarr.yml or kubectl run -it --rm busybox --restart=Never --image=busybox:1.28 -- nslookup api.github.com [-debug] [fqdn] +* Memory-leak liveness probe: + + livenessProbe: + exec: + command: + - sh + - -c + - test $(cat /proc/1/smaps | grep -i pss | awk '{Total+=$2} END {print int(Total/1024)}') -le diff --git a/services/Gitea/supportingfiles/configmap-gitea-actions-act-runner-config.yaml b/services/Gitea/supportingfiles/configmap-gitea-actions-act-runner-config.yaml index 3a1c628..9b2cff1 100644 --- a/services/Gitea/supportingfiles/configmap-gitea-actions-act-runner-config.yaml +++ b/services/Gitea/supportingfiles/configmap-gitea-actions-act-runner-config.yaml @@ -20,7 +20,10 @@ data: enabled: true container: options: > + --add-host=docker:host-gateway -v /dev/kvm:/dev/kvm privileged: true valid_volumes: - /dev/kvm + runner: + capacity: 2 diff --git a/storage/Longhorn/application-longhorn.yaml b/storage/Longhorn/application-longhorn.yaml deleted file mode 100644 index 38bd5d6..0000000 --- a/storage/Longhorn/application-longhorn.yaml +++ /dev/null @@ -1,23 +0,0 @@ -apiVersion: argoproj.io/v1alpha1 -kind: Application -metadata: - name: longhorn - namespace: argo-cd -spec: - destination: - server: https://kubernetes.default.svc - namespace: longhorn-system - project: default - sources: - - repoURL: https://charts.longhorn.io - chart: longhorn - targetRevision: 1.8.0 - helm: - valueFiles: - - $values/storage/Longhorn/values.yaml - - repoURL: https://code.spamasaurus.com/djpbessems/Kubernetes.K3s.installLog - targetRevision: master - ref: values -# - repoURL: https://github.com/djpbessems/Kubernetes.K3s.installLog -# targetRevision: master -# ref: values diff --git a/storage/Longhorn/values.yaml b/storage/Longhorn/values.yaml deleted file mode 100644 index aeffce5..0000000 --- a/storage/Longhorn/values.yaml +++ /dev/null @@ -1,8 +0,0 @@ -defaultSettings: - defaultDataPath: /mnt/blockstorage/ -ingress: - enabled: true - host: storage.spamasaurus.com - annotations: -# traefik.ingress.kubernetes.io/router.middlewares: 2fa-authentication@file,security-headers@file,compression@file - traefik.ingress.kubernetes.io/router.middlewares: 2fa-authentication@file,security-headers@file diff --git a/system/Harvester/cloudinit-disable-nic-offloading.yaml b/system/Harvester/cloudinit-disable-nic-offloading.yaml new file mode 100644 index 0000000..813652d --- /dev/null +++ b/system/Harvester/cloudinit-disable-nic-offloading.yaml @@ -0,0 +1,15 @@ +apiVersion: node.harvesterhci.io/v1beta1 +kind: CloudInit +metadata: + name: disable-nic-offloading +spec: + matchSelector: + harvesterhci.io/managed: "true" + filename: 91-disable-nic-offloading + contents: | + stages: + network: + - name: Disable all NIC offloading + commands: + - ethtool -K eno1 gso off gro off tso off tx off rx off rxvlan off txvlan off sg off + paused: false