diff --git a/site/content/posts/2023-08-09-v0.25.0-impersonation-proxy-with-external-certs.md b/site/content/posts/2023-08-09-v0.25.0-impersonation-proxy-with-external-certs.md index a51db654..726efe42 100644 --- a/site/content/posts/2023-08-09-v0.25.0-impersonation-proxy-with-external-certs.md +++ b/site/content/posts/2023-08-09-v0.25.0-impersonation-proxy-with-external-certs.md @@ -3,6 +3,9 @@ title: "Pinniped v0.25.0: With External Certificate Management for the Impersona slug: v0-25-0-external-cert-mgmt-impersonation-proxy date: 2023-08-09 author: Joshua T. Casey +authors: +- Joshua T. Casey +- Benjamin A. Petersen image: https://images.unsplash.com/photo-1618075254460-429d47b887c7?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2148&q=80 excerpt: "With v0.25.0 you get external certificate management for the impersonation proxy, easier scheduling of the kube-cert-agent, and more" tags: ['Joshua T. Casey','Ryan Richard', 'Benjamin Petersen', 'release', 'kubernetes', 'pki', 'pinniped', 'tls', 'mtls', 'kind', 'contour', 'cert-manager'] @@ -11,7 +14,10 @@ tags: ['Joshua T. Casey','Ryan Richard', 'Benjamin Petersen', 'release', 'kubern ![Friendly seal](https://images.unsplash.com/photo-1618075254460-429d47b887c7?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2148&q=80) *Photo by [karlheinz_eckhardt Eckhardt](https://unsplash.com/@karlheinz_eckhardt) on [Unsplash](https://unsplash.com/s/photos/seal)* -With Pinniped v0.25.0 you get the ability to configure an externally-generated certificate for Pinnniped Concierge's impersonation proxy to serve TLS. +With Pinniped v0.25.0 you get the ability to configure an externally-generated certificate for Pinnniped Concierge's impersonation proxy to serve TLS. The +impersonation proxy is a component within Pinniped that allows the project to support many types of clusters, such as +[Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/), [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) +and [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/overview/kubernetes-on-azure). To read more on this feature, and the design decisions behind it, see the [proposal](https://github.com/vmware-tanzu/pinniped/tree/main/proposals/1547_impersonation-proxy-external-certs). To read more about the impersonation proxy, see the [docs](https://pinniped.dev/docs/reference/supported-clusters/#background). @@ -19,11 +25,11 @@ To read more about the impersonation proxy, see the [docs](https://pinniped.dev/ To see the feature in practice on a local kind cluster, follow these instructions. This will perform mTLS between your local client (kubectl and the pinniped CLI) and the impersonation proxy. -The setup: Using a kind cluster, Contour as an ingress to the impersonation proxy, and `cert-manager` to generate a TLS serving cert. +The setup: We will be using a kind cluster, Contour as an ingress to the impersonation proxy, and `cert-manager` to generate a TLS serving cert. ```shell Docker desktop v1.20.1 -Kind v0.20.0 +Kind v0.20.0 Contour v1.25.2 Pinniped v0.25.0 pinniped CLI v0.25.0 (https://pinniped.dev/docs/howto/install-cli/) @@ -34,13 +40,16 @@ Set up kind to run with Contour, using the example kind cluster configuration fi ```shell $ wget https://raw.githubusercontent.com/projectcontour/contour/main/examples/kind/kind-expose-port.yaml +# the --kubeconfig flag on the "create cluster" command will automatically export the kubeconfig file for us $ kind create cluster \ --config kind-expose-port.yaml \ --name kind-with-contour \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Install Contour (see https://projectcontour.io/getting-started/ for more details). +Now we will install Contour (see https://projectcontour.io/getting-started/ for more details). Contour provides our kind +cluster with an Ingress Controller. We will later deploy a Contour HTTPProxy in order to create DNS that we can +use to access the Impersonation Proxy. ```shell # From https://projectcontour.io/getting-started/ @@ -54,7 +63,9 @@ $ kubectl get pods \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Install Pinniped’s local-user-authenticator and add some sample users (see https://pinniped.dev/docs/tutorials/concierge-only-demo/ for more details). +Pinniped's local-user-authenticator will act as a dummy Identity Provider for our example. This resource is not for production +use, but is sufficient for our needs to exercise the new feature of the impersonation proxy. Install Pinniped’s local-user-authenticator +and add some sample users (see https://pinniped.dev/docs/tutorials/concierge-only-demo/ for more details). ```shell # Install Pinniped's local-user-authenticator @@ -77,7 +88,8 @@ $ kubectl get secret local-user-authenticator-tls-serving-certificate \ | tee local-user-authenticator-ca.pem.b64 ``` -Install Pinniped’s Concierge: +In this example, we are only interacting with the Pinniped's Concierge. The Supervisor is not in use as we are not interacting +with a real external OIDC Identity Provider. Install Pinniped's Concierge: ```shell $ kubectl apply \ @@ -89,7 +101,8 @@ $ kubectl apply \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Install `cert-manager`: +To handle X.509 certificate management for us, we will install cert-manager. For the purposes of this exercise, we will use `cert-manager` +to generate our CA certificates as well as our TLS serving certificates. Install `cert-manager`: ```shell $ kubectl apply \ @@ -97,11 +110,29 @@ $ kubectl apply \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Configure a `cert-manager` certificate for the impersonation proxy to serve TLS. +For this demonstration we will be using `cert-manager` to simulate our own Public Key Infrastructure (PKI). +We will create the appropriate CA certificates and TLS serving certificates for the impersonation proxy to serve TLS. +For more information about using `cert-manager` to achieve this, see the [cert-manager docs](https://cert-manager.io/docs/configuration/selfsigned/#bootstrapping-ca-issuers). -Note that this section bootstraps a CA `Issuer` used to issue leaf certificates that can be used to serve TLS. -For more information on this, see the [cert-manager docs](https://cert-manager.io/docs/configuration/selfsigned/#bootstrapping-ca-issuers). -The `Certificate` with name `impersonation-serving-cert` will generate the leaf certificate used by the impersonation proxy to serve TLS. +In summary, we will do the following: + +- Create two `ClusterIssuer` resources, one named `selfsigned-cluster-issuer` and another named `my-ca-issuer`. +- The `ClusterIssuer` named `my-ca-issuer` will be used to create several `Certificat`e resources. First, we will create + the `Certificate` called `my-selfsigned-ca` (which will reference a `Secret` named `self-signed-ca-for-kind-testing` where + the actual certificate data will be stored). +- We will later retrieve the `Secret` called `self-signed-ca-for-kind-testing` so that we can add the CA to the Pinniped Concierge's + `CredentialIssuer` resource so that it can be advertised and used to verify TLS serving certificates. +- Then, we will create the `ClusterIssuer` called `my-ca-issuer`. We will reference the `Certificate` called `my-selfsigned-ca` via + it's `Secret` named `self-signed-ca-for-kind-testing`. This will allow us to use the CA to sign TLS serving certificates. +- Then, we will use the `ClusterIssuer` called `my-ca-issuer` to generate a `Certificate` that will be a TLS serving certificate + called `impersonation-serving-cert`. As before, the actual certificate data will be stored in a Kubernetes `Secret` which we + will name `impersonation-proxy-tls-serving-cert`. +- Finally, we will update the Pinniped Concierge's `CredentiaIissuer` resource to use the TLS serving certificate stored in the + `Secret` called `impersonation-proxy-tls-serving-cert`. + +If all goes well, the Impersonation Proxy endpoints will be served with a TLS serving certificate that can be validated by the +CA certificate that generated it. That's a lot! Fortunately, the majority of the work is done painlessly via the following +simple commands: ```shell $ cat << EOF > self-signed-cert.yaml @@ -150,7 +181,6 @@ metadata: namespace: pinniped-concierge spec: secretName: impersonation-proxy-tls-serving-cert - duration: 2160h # 90d renewBefore: 360h # 15d subject: @@ -177,7 +207,8 @@ $ kubectl apply \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Download the root (self-signed) CA's certificate so that it can be advertised as the CA bundle for the Concierge impersonation proxy: +Download the root (self-signed) CA's certificate. We will be adding it to the Pinniped Concierge's `CredentialIssuer` resource +in order to configure the impersonation proxy to advertise the certificate as its CA. ```shell $ kubectl get secret self-signed-ca-for-kind-testing \ @@ -189,13 +220,23 @@ $ kubectl get secret self-signed-ca-for-kind-testing \ # Tip: Put the contents of self-signed-ca-for-kind-testing.pem.b64 into your copy buffer for a later step! ``` -Now update the `CredentialIssuer` to use the impersonation proxy (which is disabled on kind by default): +The `CredentialIssuer` resource called `pinniped-concierge-config` already exists. We need to edit it. +Kind clusters have no need to use the impersonation proxy by default (it is designed for public cloud providers), +so we will make several changes to this resource: + +- Set the `spec.impersonationProxy.mode: enabled` +- Set the `spec.impersonationProxy.tls.certificateAuthorityData` to match the certificate named `my-ca-issuer` which + stores its certificate data in the `Secret` called `self-signed-ca-for-kind-testing` (which we previously recorded + in the file `self-signed-ca-for-kind-testing.pem.b64`) ```shell $ kubectl edit credentialissuer pinniped-concierge-config \ --kubeconfig kind-with-contour.kubeconfig.yaml -# Make sure that the spec has the following values: -... +``` + +Make sure that the spec has the following values: + +```yaml spec: impersonationProxy: externalEndpoint: impersonation-proxy-mtls.local @@ -205,14 +246,20 @@ $ kubectl edit credentialissuer pinniped-concierge-config \ tls: certificateAuthorityData: # paste the contents of the file self-signed-ca-for-kind-testing.pem.b64 secretName: impersonation-proxy-tls-serving-cert -... -# Now save and close the text editor +``` +Then save and close the text editor. Once saved, get the resource again and verify that the contents are correct: + +```bash # Confirm that the CredentialIssuer looks as expected $ kubectl get credentialissuers pinniped-concierge-config \ --output yaml \ --kubeconfig kind-with-contour.kubeconfig.yaml -... +``` + +Ensuring the following: + +```yaml spec: impersonationProxy: externalEndpoint: impersonation-proxy-mtls.local @@ -224,14 +271,39 @@ $ kubectl get credentialissuers pinniped-concierge-config \ tls: certificateAuthorityData: LS0tLUJFR0l.......... secretName: impersonation-proxy-tls-serving-cert -... + status: + strategies: + # this strategy should be automatically updated with the configured + # spec.tls.certificateAuthorityData from the previous step + - frontend: + impersonationProxyInfo: + certificateAuthorityData: LS0tLUJFR0l.......... +``` +In the `CredentialIssuer` `status.strategies` there should be a `frontend` strategy with a `impersonationProxyInfo.certificateAuthorityData` +value that matches that of the configured `spec.tls.certificateAuthorityData`. This is how the CredentialIssuer advertises +its CA bundle. + +Next, we review our `Service` configuration. + +```shell # Confirm that the ClusterIP service for the impersonation proxy was automatically created (may take a minute) $ kubectl get service pinniped-concierge-impersonation-proxy-cluster-ip \ --namespace pinniped-concierge \ --output yaml \ --kubeconfig kind-with-contour.kubeconfig.yaml +``` +Configure a webhook authenticator to tell Concierge to validate static tokens using the installed `local-user-authenticator`. +When we installed the Pinniped `local-user-authenticator`, we created a service called `local-user-authenticator` in the +`local-user-authenticator` namespace. We previously retrieved the Secret named `local-user-authenticator-tls-serving-certificate` +so that we could use it to configure this `WebhookAuthenticator` to use that certificate. Note that we did not generate this +certificate via `cert-manager`, this is still a self-signed certificate created by Pinniped. + +The `endpoint` here is referenced via Kubernetes DNS in the format `..svc` targeting the `/authenticate` +endpoint of the `local-user-authenticator`. We will be using https, if course. + +```yaml # Configure a webhook authenticator to tell Concierge to validate static tokens using the installed local-user-authenticator $ cat << EOF > concierge.webhookauthenticator.yaml apiVersion: authentication.concierge.pinniped.dev/v1alpha1 @@ -284,13 +356,24 @@ $ kubectl apply \ --kubeconfig kind-with-contour.kubeconfig.yaml ``` -Now generate the Pinniped kubeconfig so that you can perform mTLS with the impersonation proxy. +Now to generate the Pinniped kubeconfig so that you can perform mTLS with the impersonation proxy. -Note that using a static-token does embed those credentials into your kubeconfig. -Never use `local-user-authenticator` in production. +Since we are interacting with a kind cluster, we will need to ensure HTTP requests are routed to the cluster. +In this example, we will edit the `/etc/hosts` file to resolve the `impersonation-proxy-mtls.local` to `localhost` via `127.0.0.1`. ```shell -# add 127.0.0.1 impersonation-proxy-mtls.local to your /etc/hosts! +## +# Host Database +127.0.0.1 impersonation-proxy-mtls.local +``` + +Note that using a static-token does embed those credentials into your kubeconfig. This is not suitable for a production +deployment. As we said before, we are using `local-user-authenticator` as a simple Identity Provider for illustrative purposes +only. In a real production use case you would not employ the `--static-token` flag which would ensure credentials are not +embedded in your kubeconfig, an important security feature. Never use `local-user-authenticator` in production. + +```shell +# be sure you added 127.0.0.1 impersonation-proxy-mtls.local to your /etc/hosts! $ pinniped get kubeconfig \ --static-token "pinny:password123" \ --concierge-authenticator-type webhook \ @@ -324,7 +407,8 @@ NAMESPACE NAME pinniped-concierge pinniped-concierge-f4c78b674-bt6zl 1/1 Running 0 3h36m ``` -Congratulations, you have successfully performed mTLS authentication between your local client (kubectl, using the pinniped CLI) and the impersonation proxy inside the cluster. +Congratulations, you have successfully performed mTLS authentication between your local client (kubectl, using the pinniped CLI) +and the impersonation proxy inside the cluster. To verify that your username and groups are visible to Kubernetes, run the `pinniped whoami` command. @@ -333,5 +417,8 @@ pinniped whoami \ --kubeconfig pinniped-kubeconfig.yaml ``` -Now, to verify that the generated kubeconfig `pinniped-kubeconfig.yaml` has the contents of `self-signed-ca-for-kind-testing.pem.b64` as the contained CA bundle for the cluster, -simply cat out both files to compare. \ No newline at end of file +Finally, verify the expected outcome: + +- View the CA embedded in your kubeconfig file: `cat pinniped-kubeconfig.yaml | yq ".clusters[0].cluster.certificate-authority-data"` +- View the CA provided to the impersonation proxy: `kubectl get CredentialIssuer pinniped-concierge-config -o jsonpath="{.status.strategies[1].frontend.impersonationProxyInfo.certificateAuthorityData}"` +- View the CA we stored in our local PEM file: `cat self-signed-ca-for-kind-testing.pem.b64`