328 lines
9.4 KiB
Markdown
328 lines
9.4 KiB
Markdown
# GitOps repository
|
|
|
|
## Hypervisor
|
|
|
|
### 1) Harvester Hyperconverged Infrastructure
|
|
[...]
|
|
|
|
Configure Harvester HCI nodes through cloud-init (requires node reboot):
|
|
```shell
|
|
kubectl apply -f system/Harvester/cloudinit-disable-nic-offloading.yaml
|
|
```
|
|
## Downstream cluster(s)
|
|
|
|
Cluster configuration for RKE2 cluster using kube-vip with Traefik ingress controller:
|
|
```yaml
|
|
apiVersion: provisioning.cattle.io/v1
|
|
kind: Cluster
|
|
spec:
|
|
[...]
|
|
rkeConfig:
|
|
chartValues:
|
|
harvester-cloud-provider:
|
|
[...]
|
|
kube-vip:
|
|
env:
|
|
svc_election: 'true'
|
|
[...]
|
|
rke2-traefik:
|
|
service:
|
|
[...]
|
|
spec:
|
|
externalTrafficPolicy: Local
|
|
```
|
|
|
|
Traefik advanced configuration:
|
|
```yaml
|
|
additionalArguments:
|
|
- --providers.file.directory=/etc/traefik/dynamic
|
|
- --providers.file.watch=true
|
|
- --entryPoints.websecure.transport.respondingTimeouts.readTimeout=300s
|
|
certificatesResolvers:
|
|
default:
|
|
acme:
|
|
dnsChallenge:
|
|
propagation:
|
|
delayBeforeChecks: 5m0s
|
|
provider: cloudflare
|
|
resolvers:
|
|
- 1.1.1.1:53
|
|
- 1.0.0.1:53
|
|
email: <omitted>
|
|
storage: /data/acme.json
|
|
deployment:
|
|
initContainers:
|
|
- command:
|
|
- sh
|
|
- -c
|
|
- touch /data/acme.json; chown 65532 /data/acme.json; chmod -v 600 /data/acme.json
|
|
image: busybox:latest
|
|
name: volume-permissions
|
|
securityContext:
|
|
runAsGroup: 0
|
|
runAsNonRoot: false
|
|
runAsUser: 0
|
|
volumeMounts:
|
|
- mountPath: /data
|
|
name: traefik-data
|
|
kind: Deployment
|
|
env:
|
|
- name: CF_API_EMAIL
|
|
valueFrom:
|
|
secretKeyRef:
|
|
key: CF_API_EMAIL
|
|
name: traefik-cloudflare
|
|
- name: CF_API_KEY
|
|
valueFrom:
|
|
secretKeyRef:
|
|
key: CF_API_KEY
|
|
name: traefik-cloudflare
|
|
extraObjects:
|
|
- apiVersion: v1
|
|
data:
|
|
config.yml: |
|
|
http:
|
|
middlewares:
|
|
2fa-authentication:
|
|
forwardAuth:
|
|
address: "https://auth.spamasaurus.com/api/verify?rd=https://auth.spamasaurus.com/"
|
|
trustForwardHeader: true
|
|
security-headers:
|
|
headers:
|
|
forceSTSHeader: true
|
|
stsSeconds: 315360000
|
|
stsIncludeSubdomains: true
|
|
stsPreload: true
|
|
tls:
|
|
options:
|
|
defaults:
|
|
minVersion: VersionTLS12
|
|
sniStrict: false
|
|
curvePreferences:
|
|
- secp521r1
|
|
- secp384r1
|
|
- secp256r1
|
|
cipherSuites:
|
|
- TLS_AES_128_GCM_SHA256
|
|
- TLS_AES_256_GCM_SHA384
|
|
- TLS_CHACHA20_POLY1305_SHA256
|
|
- TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
|
|
- TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
|
|
- TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305
|
|
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
|
|
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
|
|
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305
|
|
- TLS_FALLBACK_SCSV
|
|
kind: ConfigMap
|
|
metadata:
|
|
name: traefik-file-provider
|
|
namespace: kube-system
|
|
ingressRoute:
|
|
dashboard:
|
|
enabled: true
|
|
entryPoints:
|
|
- websecure
|
|
matchRule: Host(`ingress.lab.spamasaurus.com`)
|
|
middlewares:
|
|
- name: 2fa-authentication@file
|
|
- name: security-headers@file
|
|
logs:
|
|
general:
|
|
level: INFO
|
|
persistence:
|
|
enabled: true
|
|
name: traefik-data
|
|
ports:
|
|
web:
|
|
http:
|
|
redirections:
|
|
entryPoint:
|
|
permanent: true
|
|
scheme: https
|
|
to: websecure
|
|
websecure:
|
|
forwardedHeaders:
|
|
insecure: true
|
|
http:
|
|
tls:
|
|
certResolver: default
|
|
domains:
|
|
- main: '*.pvr.spamasaurus.com'
|
|
- main: '*.lab.spamasaurus.com'
|
|
- main: '*.spamasaurus.com'
|
|
sans:
|
|
- spamasaurus.com
|
|
- main: '*.bessems.com'
|
|
sans:
|
|
- bessems.com
|
|
- main: '*.bessems.eu'
|
|
sans:
|
|
- bessems.eu
|
|
- main: '*.gabaldon.eu'
|
|
sans:
|
|
- gabaldon.eu
|
|
- main: '*.gabaldon.nl'
|
|
sans:
|
|
- gabaldon.nl
|
|
- main: '*.itch.fyi'
|
|
sans:
|
|
- itch.fyi
|
|
options: defaults@file
|
|
providers:
|
|
kubernetesCRD:
|
|
ingressClass: ""
|
|
service:
|
|
annotations:
|
|
cloudprovider.harvesterhci.io/ip-pool: <ippoolname>
|
|
cloudprovider.harvesterhci.io/ipam: pool
|
|
spec:
|
|
externalTrafficPolicy: Local
|
|
type: LoadBalancer
|
|
volumes:
|
|
- mountPath: /etc/traefik/dynamic
|
|
name: traefik-file-provider
|
|
type: configMap
|
|
```
|
|
|
|
### 2) Persistent storage
|
|
|
|
#### 2.1) CSI plugin for SMB (CIFS):
|
|
```shell
|
|
kubectl apply -f storage/csi-driver-smb/application-csi-driver-smb.yaml
|
|
```
|
|
|
|
#### 2.2) Harvester CSI plugin
|
|
See [Harvester CSI Driver](https://docs.harvesterhci.io/v1.5/rancher/csi-driver)
|
|
|
|
### 3) GitOps
|
|
##### 3.1) Install Helm Chart
|
|
See [ArgoCD](https://argo-cd.readthedocs.io/en/stable/getting_started/#getting-started):
|
|
```shell
|
|
helm repo add argo https://argoproj.github.io/argo-helm
|
|
helm repo update
|
|
helm install argo-cd -n argo-cd --create-namespace argo/argo-cd --values system/ArgoCD/chart-values.yml
|
|
```
|
|
Retrieve initial password:
|
|
```shell
|
|
kubectl get secret -n argocd argocd-initial-admin-secret -oyaml | yq e '.data.password | @base64d'
|
|
```
|
|
Login with username `admin` and the initial password, browse to `User Info` and `Update Password`.
|
|
|
|
##### 3.1) Adopt through GitOps
|
|
```shell
|
|
kubectl apply -f system/ArgoCD/application-argo-cd.yaml
|
|
```
|
|
|
|
### 4) Secret management
|
|
*Prereq*: latest `kubeseal` [release](https://github.com/bitnami-labs/sealed-secrets/releases)
|
|
```shell
|
|
kubectl apply -f system/SealedSecrets/application-sealed-secrets-controller.yaml
|
|
```
|
|
|
|
Retrieve public/private keys (*store these on a **secure** location!*):
|
|
```shell
|
|
kubectl get secret -n kube-system -l sealedsecrets.bitnami.com/sealed-secrets-key -o yaml > BitnamiSealedSecrets.masterkey.yml
|
|
```
|
|
|
|
Restoring public/private keys:
|
|
```shell
|
|
kubectl apply -f BitnamiSealedSecrets.masterkey.yml
|
|
kubectl rollout restart deployment -n kube-system sealed-secrets-controller
|
|
```
|
|
|
|
### 5) Services
|
|
##### 5.1) [Gitea](https://gitea.io/) <small>(git repository)</small>
|
|
*Required for all other workloads*
|
|
```shell
|
|
kubectl apply -f services/Gitea/application-gitea.yaml
|
|
```
|
|
|
|
##### 5.2) [Argus]() <small>(release management)</small>
|
|
```shell
|
|
kubectl apply -f services/Argus/application-argus.yaml
|
|
```
|
|
##### 5.3) [Authelia]() <small>(single sign-on))</small>
|
|
```shell
|
|
kubectl apply -f services/Authelia/application-authelia.yaml
|
|
```
|
|
##### 5.4) [Vaultwarden](https://github.com/dani-garcia/vaultwarden) <small>(password manager)</small>
|
|
```shell
|
|
kubectl apply -f services/Vaultwarden/application-vaultwarden.yaml
|
|
```
|
|
##### 5.5) [DDclient](https://github.com/linuxserver/docker-ddclient) <small>(dynamic dns)</small>
|
|
```shell
|
|
kubectl apply -f services/DDclient/application-ddclient.yaml
|
|
```
|
|
##### 5.6) [Gotify](https://gotify.net/) <small>(notifications)</small>
|
|
```shell
|
|
kubectl apply -f services/Gotify/application-gotify.yaml
|
|
```
|
|
##### 5.7) [Webtop](#) <small>(remote desktop)</small>
|
|
```shell
|
|
kubectl apply -f services/Webtop/application-webtop.yaml
|
|
```
|
|
##### 5.8) [Lighttpd](https://www.lighttpd.net/) <small>(webserver)</small>
|
|
```shell
|
|
kubectl apply -f services/Lighttpd/application-lighttpd.yaml
|
|
```
|
|
##### 5.9) PVR toolsuite <small>(automated media management)</small>
|
|
*API-keys whitelisted in ingressroutes*:
|
|
```yaml
|
|
spec:
|
|
routes:
|
|
- match: Host(`<fqdn>`) && (Headers(`X-Api-Key`, `<secret>`) || Query(`apikey`, `<secret>`))
|
|
[...]
|
|
```
|
|
###### 5.9.1) [Jellyfin](#) <small>(media library)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/Jellyfin/application-jellyfin.yaml
|
|
```
|
|
###### 5.9.2) [Jellyseerr](https://sonarr.tv/) <small>(media requests management)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/Jellyseerr/application-jellyseerr.yaml
|
|
```
|
|
###### 5.9.3) [Prowlarr](https://github.com/Prowlarr/Prowlarr) <small>(indexer management)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/Prowlarr/application-prowlarr.yaml
|
|
```
|
|
###### 5.9.4) [Radarr](https://radarr.video/) <small>(movie management)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/Radarr/application-radarr.yaml
|
|
```
|
|
###### 5.9.5) [SABnzbd](https://sabnzbd.org/) <small>(download client)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/SABnzbd/application-sabnzbd.yaml
|
|
```
|
|
###### 5.9.6) [Sonarr](https://sonarr.tv/) <small>(tv management)</small>
|
|
```shell
|
|
kubectl apply -f services/PVR/Sonarr/application-sonarr.yaml
|
|
```
|
|
|
|
### 6) Miscellaneous
|
|
*Various notes/useful links*
|
|
|
|
* Replacement for [not-yet-deprecated](https://github.com/kubernetes/kubectl/issues/151) `kubectl get all -A`:
|
|
|
|
|
|
kubectl get $(kubectl api-resources --verbs=list -o name | paste -sd, -) --ignore-not-found --all-namespaces
|
|
* `DaemonSet` to configure nodes' **sysctl** `fs.inotify.max-user-watches`:
|
|
|
|
|
|
kubectl apply -f system/InotifyMaxWatchers/daemonSet-InotifyMaxWatchers.yml
|
|
* Debug DNS lookups within the cluster:
|
|
|
|
|
|
kubectl run -it --rm dnsutils --restart=Never --image=gcr.io/kubernetes-e2e-test-images/dnsutils -- nslookup [-debug] [fqdn]
|
|
or
|
|
|
|
kubectl run -it --rm busybox --restart=Never --image=busybox:1.28 -- nslookup api.github.com [-debug] [fqdn]
|
|
* Memory-leak liveness probe:
|
|
|
|
|
|
livenessProbe:
|
|
exec:
|
|
command:
|
|
- sh
|
|
- -c
|
|
- test $(cat /proc/1/smaps | grep -i pss | awk '{Total+=$2} END {print int(Total/1024)}') -le <limit>
|