blog: impersonation-proxy post updates

site/content/posts/2023-08-09-v0.25.0-impersonation-proxy-with-external-certs.md

@@ -3,6 +3,9 @@ title: "Pinniped v0.25.0: With External Certificate Management for the Impersona
 slug: v0-25-0-external-cert-mgmt-impersonation-proxy
 date: 2023-08-09
 author: Joshua T. Casey
+authors: 
+- Joshua T. Casey
+- Benjamin A. Petersen
 image: https://images.unsplash.com/photo-1618075254460-429d47b887c7?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2148&q=80
 excerpt: "With v0.25.0 you get external certificate management for the impersonation proxy, easier scheduling of the kube-cert-agent, and more"
 tags: ['Joshua T. Casey','Ryan Richard', 'Benjamin Petersen', 'release', 'kubernetes', 'pki', 'pinniped', 'tls', 'mtls', 'kind', 'contour', 'cert-manager']
@@ -11,7 +14,10 @@ tags: ['Joshua T. Casey','Ryan Richard', 'Benjamin Petersen', 'release', 'kubern
 ![Friendly seal](https://images.unsplash.com/photo-1618075254460-429d47b887c7?ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D&auto=format&fit=crop&w=2148&q=80)
 *Photo by [karlheinz_eckhardt Eckhardt](https://unsplash.com/@karlheinz_eckhardt) on [Unsplash](https://unsplash.com/s/photos/seal)*
 
-With Pinniped v0.25.0 you get the ability to configure an externally-generated certificate for Pinnniped Concierge's impersonation proxy to serve TLS.
+With Pinniped v0.25.0 you get the ability to configure an externally-generated certificate for Pinnniped Concierge's impersonation proxy to serve TLS.  The
+impersonation proxy is a component within Pinniped that allows the project to support many types of clusters, such as 
+[Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/), [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine)
+and [Azure Kubernetes Service (AKS)](https://azure.microsoft.com/en-us/overview/kubernetes-on-azure).
 
 To read more on this feature, and the design decisions behind it, see the [proposal](https://github.com/vmware-tanzu/pinniped/tree/main/proposals/1547_impersonation-proxy-external-certs).
 To read more about the impersonation proxy, see the [docs](https://pinniped.dev/docs/reference/supported-clusters/#background).
@@ -19,11 +25,11 @@ To read more about the impersonation proxy, see the [docs](https://pinniped.dev/
 To see the feature in practice on a local kind cluster, follow these instructions.
 This will perform mTLS between your local client (kubectl and the pinniped CLI) and the impersonation proxy.
 
-The setup: Using a kind cluster, Contour as an ingress to the impersonation proxy, and `cert-manager` to generate a TLS serving cert.
+The setup: We will be using a kind cluster, Contour as an ingress to the impersonation proxy, and `cert-manager` to generate a TLS serving cert.
 
 ```shell
 Docker desktop v1.20.1
-Kind v0.20.0
+Kind v0.20.0 
 Contour v1.25.2
 Pinniped v0.25.0
 pinniped CLI v0.25.0 (https://pinniped.dev/docs/howto/install-cli/)
@@ -34,13 +40,16 @@ Set up kind to run with Contour, using the example kind cluster configuration fi
 
 ```shell
 $ wget https://raw.githubusercontent.com/projectcontour/contour/main/examples/kind/kind-expose-port.yaml
+# the --kubeconfig flag on the "create cluster" command will automatically export the kubeconfig file for us
 $ kind create cluster \
     --config kind-expose-port.yaml \
     --name kind-with-contour \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Install Contour (see https://projectcontour.io/getting-started/ for more details).
+Now we will install Contour (see https://projectcontour.io/getting-started/ for more details).  Contour provides our kind 
+cluster with an Ingress Controller. We will later deploy a Contour HTTPProxy in order to create DNS that we can 
+use to access the Impersonation Proxy.
 
 ```shell
 # From https://projectcontour.io/getting-started/
@@ -54,7 +63,9 @@ $ kubectl get pods \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Install Pinniped’s local-user-authenticator and add some sample users (see https://pinniped.dev/docs/tutorials/concierge-only-demo/ for more details).
+Pinniped's local-user-authenticator will act as a dummy Identity Provider for our example. This resource is not for production
+use, but is sufficient for our needs to exercise the new feature of the impersonation proxy. Install Pinniped’s local-user-authenticator 
+and add some sample users (see https://pinniped.dev/docs/tutorials/concierge-only-demo/ for more details).
 
 ```shell
 # Install Pinniped's local-user-authenticator
@@ -77,7 +88,8 @@ $ kubectl get secret local-user-authenticator-tls-serving-certificate \
     | tee local-user-authenticator-ca.pem.b64
 ```
 
-Install Pinniped’s Concierge:
+In this example, we are only interacting with the Pinniped's Concierge. The Supervisor is not in use as we are not interacting 
+with a real external OIDC Identity Provider. Install Pinniped's Concierge:
 
 ```shell
 $ kubectl apply \
@@ -89,7 +101,8 @@ $ kubectl apply \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Install `cert-manager`:
+To handle X.509 certificate management for us, we will install cert-manager. For the purposes of this exercise, we will use `cert-manager`
+to generate our CA certificates as well as our TLS serving certificates. Install `cert-manager`:
 
 ```shell
 $ kubectl apply \
@@ -97,11 +110,29 @@ $ kubectl apply \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Configure a `cert-manager` certificate for the impersonation proxy to serve TLS.
+For this demonstration we will be using `cert-manager` to simulate our own Public Key Infrastructure (PKI).
+We will create the appropriate CA certificates and TLS serving certificates for the impersonation proxy to serve TLS.
+For more information about using `cert-manager` to achieve this, see the [cert-manager docs](https://cert-manager.io/docs/configuration/selfsigned/#bootstrapping-ca-issuers).
 
-Note that this section bootstraps a CA `Issuer` used to issue leaf certificates that can be used to serve TLS.
+In summary, we will do the following:
-For more information on this, see the [cert-manager docs](https://cert-manager.io/docs/configuration/selfsigned/#bootstrapping-ca-issuers).
+
-The `Certificate` with name `impersonation-serving-cert` will generate the leaf certificate used by the impersonation proxy to serve TLS.
+- Create two `ClusterIssuer` resources, one named `selfsigned-cluster-issuer` and another named `my-ca-issuer`.
+- The `ClusterIssuer` named `my-ca-issuer` will be used to create several `Certificat`e resources. First, we will create 
+  the `Certificate` called `my-selfsigned-ca` (which will reference a `Secret` named `self-signed-ca-for-kind-testing` where
+  the actual certificate data will be stored).
+- We will later retrieve the `Secret` called `self-signed-ca-for-kind-testing` so that we can add the CA to the Pinniped Concierge's 
+  `CredentialIssuer` resource so that it can be advertised and used to verify TLS serving certificates.
+- Then, we will create the `ClusterIssuer` called `my-ca-issuer`. We will reference the `Certificate` called `my-selfsigned-ca` via 
+  it's `Secret` named `self-signed-ca-for-kind-testing`. This will allow us to use the CA to sign TLS serving certificates.
+- Then, we will use the `ClusterIssuer` called `my-ca-issuer` to generate a `Certificate` that will be a TLS serving certificate 
+  called `impersonation-serving-cert`. As before, the actual certificate data will be stored in a Kubernetes `Secret` which we 
+  will name `impersonation-proxy-tls-serving-cert`.
+- Finally, we will update the Pinniped Concierge's `CredentiaIissuer` resource to use the TLS serving certificate stored in the 
+  `Secret` called `impersonation-proxy-tls-serving-cert`.
+
+If all goes well, the Impersonation Proxy endpoints will be served with a TLS serving certificate that can be validated by the 
+CA certificate that generated it.  That's a lot!  Fortunately, the majority of the work is done painlessly via the following 
+simple commands:
 
 ```shell
 $ cat << EOF > self-signed-cert.yaml
@@ -150,7 +181,6 @@ metadata:
   namespace: pinniped-concierge
 spec:
   secretName: impersonation-proxy-tls-serving-cert
-
   duration: 2160h # 90d
   renewBefore: 360h # 15d
   subject:
@@ -177,7 +207,8 @@ $ kubectl apply \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Download the root (self-signed) CA's certificate so that it can be advertised as the CA bundle for the Concierge impersonation proxy:
+Download the root (self-signed) CA's certificate. We will be adding it to the Pinniped Concierge's `CredentialIssuer` resource 
+in order to configure the impersonation proxy to advertise the certificate as its CA.
 
 ```shell
 $ kubectl get secret self-signed-ca-for-kind-testing \
@@ -189,13 +220,23 @@ $ kubectl get secret self-signed-ca-for-kind-testing \
 # Tip: Put the contents of self-signed-ca-for-kind-testing.pem.b64 into your copy buffer for a later step!
 ```
 
-Now update the `CredentialIssuer` to use the impersonation proxy (which is disabled on kind by default):
+The `CredentialIssuer` resource called `pinniped-concierge-config` already exists.  We need to edit it.
+Kind clusters have no need to use the impersonation proxy by default (it is designed for public cloud providers),
+so we will make several changes to this resource:
+
+- Set the `spec.impersonationProxy.mode: enabled`
+- Set the `spec.impersonationProxy.tls.certificateAuthorityData` to match the certificate named `my-ca-issuer` which
+  stores its certificate data in the `Secret` called `self-signed-ca-for-kind-testing` (which we previously recorded 
+  in the file `self-signed-ca-for-kind-testing.pem.b64`)
 
 ```shell
 $ kubectl edit credentialissuer pinniped-concierge-config \
     --kubeconfig kind-with-contour.kubeconfig.yaml
-# Make sure that the spec has the following values:
+```
-...
+
+Make sure that the spec has the following values:
+
+```yaml
   spec:
     impersonationProxy:
       externalEndpoint: impersonation-proxy-mtls.local
@@ -205,14 +246,20 @@ $ kubectl edit credentialissuer pinniped-concierge-config \
       tls:
         certificateAuthorityData: # paste the contents of the file self-signed-ca-for-kind-testing.pem.b64
         secretName: impersonation-proxy-tls-serving-cert
-...
+```
-# Now save and close the text editor
 
+Then save and close the text editor. Once saved, get the resource again and verify that the contents are correct: 
+
+```bash
 # Confirm that the CredentialIssuer looks as expected
 $ kubectl get credentialissuers pinniped-concierge-config \
     --output yaml \
     --kubeconfig kind-with-contour.kubeconfig.yaml
-...
+```
+
+Ensuring the following: 
+
+```yaml
   spec:
     impersonationProxy:
       externalEndpoint: impersonation-proxy-mtls.local
@@ -224,14 +271,39 @@ $ kubectl get credentialissuers pinniped-concierge-config \
       tls:
         certificateAuthorityData: LS0tLUJFR0l..........
         secretName: impersonation-proxy-tls-serving-cert
-...
+  status:
+    strategies:
+      # this strategy should be automatically updated with the configured 
+      # spec.tls.certificateAuthorityData from the previous step
+      - frontend:
+      impersonationProxyInfo:
+        certificateAuthorityData: LS0tLUJFR0l..........
+```
 
+In the `CredentialIssuer` `status.strategies` there should be a `frontend` strategy with a `impersonationProxyInfo.certificateAuthorityData`
+value that matches that of the configured `spec.tls.certificateAuthorityData`.  This is how the CredentialIssuer advertises
+its CA bundle.
+
+Next, we review our `Service` configuration.
+
+```shell
 # Confirm that the ClusterIP service for the impersonation proxy was automatically created (may take a minute)
 $ kubectl get service pinniped-concierge-impersonation-proxy-cluster-ip \
     --namespace pinniped-concierge \
     --output yaml \
     --kubeconfig kind-with-contour.kubeconfig.yaml
+```
 
+Configure a webhook authenticator to tell Concierge to validate static tokens using the installed `local-user-authenticator`. 
+When we installed the Pinniped `local-user-authenticator`, we created a service called `local-user-authenticator` in the 
+`local-user-authenticator` namespace. We previously retrieved the Secret named `local-user-authenticator-tls-serving-certificate` 
+so that we could use it to configure this `WebhookAuthenticator` to use that certificate. Note that we did not generate this 
+certificate via `cert-manager`, this is still a self-signed certificate created by Pinniped.
+
+The `endpoint` here is referenced via Kubernetes DNS in the format `<namespace>.<service-name>.svc` targeting the `/authenticate` 
+endpoint of the `local-user-authenticator`. We will be using https, if course.
+
+```yaml
 # Configure a webhook authenticator to tell Concierge to validate static tokens using the installed local-user-authenticator
 $ cat << EOF > concierge.webhookauthenticator.yaml
 apiVersion: authentication.concierge.pinniped.dev/v1alpha1
@@ -284,13 +356,24 @@ $ kubectl apply \
     --kubeconfig kind-with-contour.kubeconfig.yaml
 ```
 
-Now generate the Pinniped kubeconfig so that you can perform mTLS with the impersonation proxy.
+Now to generate the Pinniped kubeconfig so that you can perform mTLS with the impersonation proxy.
 
-Note that using a static-token does embed those credentials into your kubeconfig.
+Since we are interacting with a kind cluster, we will need to ensure HTTP requests are routed to the cluster. 
-Never use `local-user-authenticator` in production.
+In this example, we will edit the `/etc/hosts` file to resolve the `impersonation-proxy-mtls.local` to `localhost` via `127.0.0.1`.
 
 ```shell
-# add 127.0.0.1 impersonation-proxy-mtls.local to your /etc/hosts!
+##
+# Host Database
+127.0.0.1 impersonation-proxy-mtls.local
+```
+
+Note that using a static-token does embed those credentials into your kubeconfig. This is not suitable for a production 
+deployment. As we said before, we are using `local-user-authenticator` as a simple Identity Provider for illustrative purposes 
+only. In a real production use case you would not employ the `--static-token` flag which would ensure credentials are not 
+embedded in your kubeconfig, an important security feature.  Never use `local-user-authenticator` in production.
+
+```shell
+# be sure you added 127.0.0.1 impersonation-proxy-mtls.local to your /etc/hosts!
 $ pinniped get kubeconfig \
     --static-token "pinny:password123" \
     --concierge-authenticator-type webhook \
@@ -324,7 +407,8 @@ NAMESPACE                  NAME
 pinniped-concierge         pinniped-concierge-f4c78b674-bt6zl                        1/1     Running     0              3h36m
 ```
 
-Congratulations, you have successfully performed mTLS authentication between your local client (kubectl, using the pinniped CLI) and the impersonation proxy inside the cluster.
+Congratulations, you have successfully performed mTLS authentication between your local client (kubectl, using the pinniped CLI) 
+and the impersonation proxy inside the cluster.
 
 To verify that your username and groups are visible to Kubernetes, run the `pinniped whoami` command.
 
@@ -333,5 +417,8 @@ pinniped whoami \
     --kubeconfig pinniped-kubeconfig.yaml
 ```
 
-Now, to verify that the generated kubeconfig `pinniped-kubeconfig.yaml` has the contents of `self-signed-ca-for-kind-testing.pem.b64` as the contained CA bundle for the cluster,
+Finally, verify the expected outcome:
-simply cat out both files to compare.
+
+- View the CA embedded in your kubeconfig file: `cat pinniped-kubeconfig.yaml | yq ".clusters[0].cluster.certificate-authority-data"`
+- View the CA provided to the impersonation proxy: `kubectl get CredentialIssuer pinniped-concierge-config -o jsonpath="{.status.strategies[1].frontend.impersonationProxyInfo.certificateAuthorityData}"` 
+- View the CA we stored in our local PEM file: `cat self-signed-ca-for-kind-testing.pem.b64`