Kubernetes.K3s.installLog/README.md

# GitOps repository

## Hypervisor

### 1) Harvester Hyperconverged Infrastructure
[...]  

Configure Harvester HCI nodes through cloud-init (requires node reboot):
```shell
kubectl apply -f system/Harvester/cloudinit-disable-nic-offloading.yaml
```
## Downstream cluster(s)

Cluster configuration for RKE2 cluster using kube-vip with Traefik ingress controller:
```yaml
apiVersion: provisioning.cattle.io/v1
kind: Cluster
spec:
  [...]
  rkeConfig:
    chartValues:
      harvester-cloud-provider:
        [...]
        kube-vip:
          env:
            svc_election: 'true'
      [...]
      rke2-traefik:
        service:
          [...]
          spec:
            externalTrafficPolicy: Local
```

Traefik advanced configuration:
```yaml
additionalArguments:
  - --providers.file.directory=/etc/traefik/dynamic
  - --providers.file.watch=true
  - --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s
certificatesResolvers:
  default:
    acme:
      dnsChallenge:
        propagation:
          delayBeforeChecks: 5m0s
        provider: cloudflare
        resolvers:
          - 1.1.1.1:53
          - 1.0.0.1:53
      email: <omitted>
      storage: /data/acme.json
deployment:
  initContainers:
    - command:
        - sh
        - -c
        - touch /data/acme.json; chown 65532 /data/acme.json; chmod -v 600 /data/acme.json
      image: busybox:latest
      name: volume-permissions
      securityContext:
        runAsGroup: 0
        runAsNonRoot: false
        runAsUser: 0
      volumeMounts:
        - mountPath: /data
          name: traefik-data
  kind: Deployment
env:
  - name: CF_API_EMAIL
    valueFrom:
      secretKeyRef:
        key: CF_API_EMAIL
        name: traefik-cloudflare
  - name: CF_API_KEY
    valueFrom:
      secretKeyRef:
        key: CF_API_KEY
        name: traefik-cloudflare
extraObjects:
  - apiVersion: v1
    data:
      config.yml: |
        http:
          middlewares:
            2fa-authentication:
              forwardAuth:
                address: "https://auth.spamasaurus.com/api/verify?rd=https://auth.spamasaurus.com/"
                trustForwardHeader: true
            security-headers:
              headers:
                forceSTSHeader: true
                stsSeconds: 315360000
                stsIncludeSubdomains: true
                stsPreload: true
        tls:
          options:
            defaults:
              minVersion: VersionTLS12
              sniStrict: false
              curvePreferences:
                - secp521r1
                - secp384r1
                - secp256r1
              cipherSuites:
                - TLS_AES_128_GCM_SHA256
                - TLS_AES_256_GCM_SHA384
                - TLS_CHACHA20_POLY1305_SHA256
                - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
                - TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
                - TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
                - TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
                - TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
                - TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
                - TLS_FALLBACK_SCSV
    kind: ConfigMap
    metadata:
      name: traefik-file-provider
      namespace: kube-system
ingressRoute:
  dashboard:
    enabled: true
    entryPoints:
      - websecure
    matchRule: Host(`ingress.lab.spamasaurus.com`)
    middlewares:
      - name: 2fa-authentication@file
      - name: security-headers@file
logs:
  general:
    level: INFO
persistence:
  enabled: true
  name: traefik-data
ports:
  web:
    http:
      redirections:
        entryPoint:
          permanent: true
          scheme: https
          to: websecure
  websecure:
    forwardedHeaders:
      insecure: true
    http:
      tls:
        certResolver: default
        domains:
          - main: '*.pvr.spamasaurus.com'
          - main: '*.lab.spamasaurus.com'
          - main: '*.spamasaurus.com'
            sans:
              - spamasaurus.com
          - main: '*.bessems.com'
            sans:
              - bessems.com
          - main: '*.bessems.eu'
            sans:
              - bessems.eu
          - main: '*.gabaldon.eu'
            sans:
              - gabaldon.eu
          - main: '*.gabaldon.nl'
            sans:
              - gabaldon.nl
          - main: '*.itch.fyi'
            sans:
              - itch.fyi
        options: defaults@file
providers:
  kubernetesCRD:
    ingressClass: ""
service:
  annotations:
    cloudprovider.harvesterhci.io/ip-pool: <ippoolname>
    cloudprovider.harvesterhci.io/ipam: pool
  spec:
    externalTrafficPolicy: Local
    type: LoadBalancer
volumes:
  - mountPath: /etc/traefik/dynamic
    name: traefik-file-provider
    type: configMap
```

### 2) Persistent storage

#### 2.1) CSI plugin for SMB (CIFS):
```shell
kubectl apply -f storage/csi-driver-smb/application-csi-driver-smb.yaml
```

#### 2.2) Harvester CSI plugin
See [Harvester CSI Driver](https://docs.harvesterhci.io/v1.5/rancher/csi-driver)

### 3) GitOps
##### 3.1) Install Helm Chart
See [ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/#getting-started):
```shell
helm repo add argo https://argoproj.github.io/argo-helm
helm repo update
helm install argo-cd -n argo-cd --create-namespace argo/argo-cd --values system/ArgoCD/chart-values.yml
```
Retrieve initial password:
```shell
kubectl get secret -n argocd argocd-initial-admin-secret -oyaml | yq e '.data.password | @base64d'
```
Login with username `admin` and the initial password, browse to `User Info` and `Update Password`.

##### 3.1) Adopt through GitOps
```shell
kubectl apply -f system/ArgoCD/application-argo-cd.yaml
```

### 4) Secret management
*Prereq*: latest `kubeseal` [release](https://github.com/bitnami-labs/sealed-secrets/releases)
```shell
kubectl apply -f system/SealedSecrets/application-sealed-secrets-controller.yaml
```

Retrieve public/private keys (*store these on a **secure** location!*):
```shell
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > BitnamiSealedSecrets.masterkey.yml
```

Restoring public/private keys:
```shell
kubectl apply -f BitnamiSealedSecrets.masterkey.yml
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
```

### 5) Services
##### 5.1) [Gitea](https://gitea.io/)    <small>(git repository)</small>
*Required for all other workloads*  
```shell
kubectl apply -f services/Gitea/application-gitea.yaml
```

##### 5.2) [Argus]()    <small>(release management)</small>
```shell
kubectl apply -f services/Argus/application-argus.yaml
```
##### 5.3) [Authelia]()    <small>(single sign-on))</small>
```shell
kubectl apply -f services/Authelia/application-authelia.yaml
```
##### 5.4) [Vaultwarden](https://github.com/dani-garcia/vaultwarden)    <small>(password manager)</small>
```shell
kubectl apply -f services/Vaultwarden/application-vaultwarden.yaml
```
##### 5.5) [DDclient](https://github.com/linuxserver/docker-ddclient)	<small>(dynamic dns)</small>
```shell
kubectl apply -f services/DDclient/application-ddclient.yaml
```
##### 5.6) [Gotify](https://gotify.net/)    <small>(notifications)</small>
```shell
kubectl apply -f services/Gotify/application-gotify.yaml
```
##### 5.7) [Webtop](#)    <small>(remote desktop)</small>
```shell
kubectl apply -f services/Webtop/application-webtop.yaml
```
##### 5.8) [Lighttpd](https://www.lighttpd.net/)    <small>(webserver)</small>
```shell
kubectl apply -f services/Lighttpd/application-lighttpd.yaml
```
##### 5.9) PVR toolsuite    <small>(automated media management)</small>
*API-keys whitelisted in ingressroutes*:  
```yaml
spec:
  routes:
  - match: Host(`<fqdn>`) && (Headers(`X-Api-Key`, `<secret>`) || Query(`apikey`, `<secret>`))
    [...]
```
###### 5.9.1) [Jellyfin](#)    <small>(media library)</small>
```shell
kubectl apply -f services/PVR/Jellyfin/application-jellyfin.yaml
```
###### 5.9.2) [Jellyseerr](https://sonarr.tv/)    <small>(media requests management)</small>
```shell
kubectl apply -f services/PVR/Jellyseerr/application-jellyseerr.yaml
```
###### 5.9.3) [Prowlarr](https://github.com/Prowlarr/Prowlarr)    <small>(indexer management)</small>
```shell
kubectl apply -f services/PVR/Prowlarr/application-prowlarr.yaml
```
###### 5.9.4) [Radarr](https://radarr.video/)    <small>(movie management)</small>
```shell
kubectl apply -f services/PVR/Radarr/application-radarr.yaml
```
###### 5.9.5) [SABnzbd](https://sabnzbd.org/)    <small>(download client)</small>
```shell
kubectl apply -f services/PVR/SABnzbd/application-sabnzbd.yaml
```
###### 5.9.6) [Sonarr](https://sonarr.tv/)    <small>(tv management)</small>
```shell
kubectl apply -f services/PVR/Sonarr/application-sonarr.yaml
```

### 6) Miscellaneous
*Various notes/useful links*

* Replacement for [not-yet-deprecated](https://github.com/kubernetes/kubectl/issues/151) `kubectl get all -A`:


      kubectl get $(kubectl api-resources --verbs=list -o name | paste -sd, -) --ignore-not-found --all-namespaces
* `DaemonSet` to configure nodes' **sysctl** `fs.inotify.max-user-watches`:


      kubectl apply -f system/InotifyMaxWatchers/daemonSet-InotifyMaxWatchers.yml
* Debug DNS lookups within the cluster:


      kubectl run -it --rm dnsutils --restart=Never --image=gcr.io/kubernetes-e2e-test-images/dnsutils -- nslookup [-debug] [fqdn]
  or

      kubectl run -it --rm busybox --restart=Never --image=busybox:1.28 -- nslookup api.github.com [-debug] [fqdn]
* Memory-leak liveness probe:


      livenessProbe:
        exec:
          command:
            - sh
            - -c
            - test $(cat /proc/1/smaps | grep -i pss |  awk '{Total+=$2} END {print int(Total/1024)}') -le <limit>