Skip to main content

Kubernetes | Mutating Webhook Failure: x509 Unknown Authority

Updated

Overview

This article addresses a failure in the Kubernetes Agent Injector where client workload pods fail to inject the Aembit Agent Proxy. This is typically caused by a TLS trust mismatch between the Kubernetes API Server and the Agent Injector's certificate.

Relates To

  • Deployment Type: Kubernetes

  • Component: aembit-agent-injector (Mutating Admission Webhook)

  • Dependency: cert-manager (if used for certificate management)

Cause (Symptom)

You should use this guide if your Agent Injector container logs (kubectl logs <agent_injector_pod> -c aembit-agent-injector -n aembit) show the following signature:

Internal error occurred: failed calling webhook "aembit-agent-injector.aembit.aembit.io": failed to call webhook: Post "...": x509: certificate signed by unknown authority

This symptom indicates that the Kubernetes API server does not trust the certificate presented by the Agent Injector because the caBundle in the webhook configuration is stale or mismatched.

Solution

Diagnostic Verification

To resolve this, you must verify that the Certificate Authority (CA) trusted by the Webhook matches the CA issuing the Injector's Secret.

1. Verify the Webhook's Trusted CA Run this to see which CA the Kubernetes API server is currently using to verify the Injector:

kubectl get mutatingwebhookconfiguration aembit-agent-injector.<namespace>.aembit.io \
    -o jsonpath='{$.webhooks[0].clientConfig.caBundle}' \
    | openssl enc -d -base64 -A \
    | openssl x509 -noout -subject

ROSA / OpenShift:

oc get mutatingwebhookconfiguration aembit-agent-injector.aembit.aembit.io \
    -o jsonpath='{$.webhooks[0].clientConfig.caBundle}' \
    | openssl enc -d -base64 -A \
    | openssl x509 -noout -subject

  • Success Criteria: subject=CN=<your expected cluster CA>

  • Failure Signal: If you see an unexpected CN, the cert-manager ca-injector has likely not updated this bundle.

2. Identify the Secret in Use Confirm which secret the Agent Injector pod is actually mounting:

kubectl -n <namespace> get pod -l aembit.io/component=aembit-agent-injector \
    -o jsonpath='{$.items[0].spec.volumes[0].secret.secretName}'

ROSA / OpenShift:

oc -n aembit get pod -l aembit.io/component=aembit-agent-injector \
    -o jsonpath='{$.items[0].spec.volumes[0].secret.secretName}'

  • Validation: This must match the <certificate secret name> used in your Helm configuration.

3. Verify the Secret's Issuer Finally, check the issuer of the certificate stored in that secret:

kubectl -n <namespace> get secret <certificate secret name> \
    -o jsonpath="{\$.data['ca\.crt']}" \
    | openssl enc -d -base64 -A \
    | openssl x509 -noout -issuer

ROSA / OpenShift:

oc -n aembit get secret <secret_name> \
    -o jsonpath="{\$.data['ca\.crt']}" \
    | openssl enc -d -base64 -A \
    | openssl x509 -noout -issuer

Evaluate the Outcome

Compare the Subject from Step 1 with the Issuer from Step 3.

  • If they match: The certificate chain is valid. If the error persists, check for network-level interception.

  • If they do NOT match: The Webhook is looking for the wrong CA. This is common when using an outdated Helm chart that lacks the required webhookAnnotations or Helm values are inaccurate.

Next Step: Proceed to Kubernetes | Upgrading Helm chart for Agent Injector TLS cert-manager administration to fix the synchronization.

 

 

 

 

Powered by Zendesk