test/integration: increase confidence that a cert has rotated

It looks like requests to our aggregated API service on GKE vacillate
between success and failure until they reach a converged successful
state. I think this has to do with our pods updating the API serving
cert at different times. If only one pod updates its serving cert to
the correct value, then it should respond with success. However, the
other pod would respond with failure. Depending on the load balancing
algorithm that GKE uses to send traffic to pods in a service, we could
end up with a success that we interpret as "all pods have rotated
their certs" when it really just means "at least one pod has rotated
its certs."

Signed-off-by: Andrew Keesler <akeesler@vmware.com>

test/integration/api_serving_certs_test.go

@@ -131,7 +131,10 @@ func TestAPIServingCertificateAutoCreationAndRotation(t *testing.T) {
 			// Check that we can still make requests to the aggregated API through the kube API server,
 			// because the kube API server uses these certs when proxying requests to the aggregated API server,
 			// so this is effectively checking that the aggregated API server is using these new certs.
+			// We ensure that 10 straight requests succeed so that we filter out false positives where a single
+			// pod has rotated their cert, but not the other ones sitting behind the service.
 			aggregatedAPIWorking := func() bool {
+				for i := 0; i < 10; i++ {
 					_, err = pinnipedClient.PinnipedV1alpha1().CredentialRequests().Create(ctx, &v1alpha1.CredentialRequest{
 						TypeMeta:   metav1.TypeMeta{},
 						ObjectMeta: metav1.ObjectMeta{},
@@ -140,6 +143,10 @@ func TestAPIServingCertificateAutoCreationAndRotation(t *testing.T) {
 							Token: &v1alpha1.CredentialRequestTokenCredential{Value: "not a good token"},
 						},
 					}, metav1.CreateOptions{})
+					if err != nil {
+						break
+					}
+				}
 				// Should have got a success response with an error message inside it complaining about the token value.
 				return err == nil
 			}