Overview

To troubleshoot issues or unexpected behavior with your Edge Components in Kubernetes, examine the key areas detailed below. The guidance will help you confirm your setup and diagnose common issues.

If problems persist or questions arise, submit a support request to Aembit Support (refer to "How-To-Submit-a-Support-Request").

Common Error Codes

Edge Components - Agent Controller & Agent Injector

Health

• Are Agent Controller and Agent injector healthy and running?
• Are Liveness Probe and Readiness Probe reporting any failures?

$ kubectl get pods -n aembit -o wide

NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aembit-agent-controller-d884df854-k2ztp 1/1 Running 2 (3d10h ago) 6d1h 10.1.0.202 docker-desktop <none> <none>
aembit-agent-injector-6b88dbf9d5-4gjt6 1/1 Running 0 6d1h 10.1.0.203 docker-desktop <none> <none>

$ kubectl describe pods -n aembit

Liveness:   http-get http://:80/health delay=5s timeout=2s period=5s #success=1 #failure=3

Readiness:  http-get https://:443/health delay=5s timeout=2s period=5s #success=1 #failure=3

Agent Controller registration to Aembit Cloud

Configuration & Registration to Aembit Cloud

• Has the Agent Controller been configured with expected values for Device Code or Trust Provider Attestation?
• Validate
  • Agent Controller ID
    • Device or Trust Provider
  • Trust Provider
    • Review data values per Trust Provider documentation

kubectl describe pods -n aembit

 Environment:
    AEMBIT_STACK_DOMAIN:          useast2.aembit.io
    AEMBIT_TENANT_ID:             <TenantID>
    AEMBIT_AGENT_CONTROLLER_ID:   <ControllerID>
    AEMBIT_MANAGED_TLS_HOSTNAME:  aembit-agent-controller.aembit
    AEMBIT_METRICS_ENABLED:       true
    AEMBIT_LOG_LEVEL:             <log_level>

Registration - [Tenant UI]

Your Aembit tenant captures comprehensive details about the state of Agent Controllers. 

Edge Component - Agent Controllers - TLS Status

In this section, review the current state of your Agent Controller. The following reference will help you validate the health of your Agent Controller.

• • • Blue - Aembit managed TLS Certificate validated.
    • Red - Unhealthy
    • Green - Uptime Status and operational
    • No Color - TLS not enabled

Reporting -> Audit Logs

Audit logs provide detailed information about ongoing activity in your Aembit Edge Components. We can narrow our search criteria to review Agent Controller registration and attestation by reviewing the metadata details found in the event output. The primary questions we want to address are outlined below.

• Agent Controller successfully register to Aembit?
• Agent Controller attestation successful?

Edge Components - Agent Controller - Trust Provider (Attestation)

• Validate Trust Provider fields for accuracy per our documentation, https://docs.aembit.io/features/trust-providers/add-trust-provider

Diagnose Network Connectivity

• Validate your Agent Controller POD can successfully connect to your Aembit tenant. 

kubectl -n <NAMESPACE> exec -it <AGENT-CONTROLLER-POD> -c aembit-agent-controller -- /bin/bash -c \
'apt update && apt install -y bind9-host openssl && host \
-v TENANTID.ec.us-east-2.useast2.aembit.io && echo Q | \
openssl s_client -showcerts -connect TENANTID.ec.us-east-2.useast2.aembit.io:443'

Client Workload POD

Edge Components - Agent Proxy and Agent Sidecar containers 

Container Image Details

Aembit's official container images are designed for easy deployment via Helm charts for Kubernetes and Terraform modules for ECS. The information below provides details if you proceed to configure your own deployment path.

Container: aembit_agent_proxy
User ID: 65534

Container: aembit_sidecar_init
User ID: 26248

Client Workload UID: 65534

Container: aembit_agent_proxy

Aembit Managed TLS

Client Workload POD(s) will attempt to retrieve your Aembit tenant Root CA certificate to decrypt and manage traffic between your Client and Server Workloads.

Agent Proxy Container

• Were we able to successfully download Aembit Root CA Certificate?
• Is Aembit Root CA Certificate located in `Aembit-Agent-Proxy' container?

1. kubectl -n <namespace> exec -it <Client-Workload-POD> -c aembit-agent-proxy -- bash
2. awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "cert-" c ".pem"}' < /etc/ssl/certs/ca-certificates.crt
3. for cert in cert-*.pem; do openssl x509 -in "$cert" -subject -noout; done | grep "Aembit"

subject=CN = Aembit Tenant <TENANT_ID> Root CA, O = Aembit Inc, C = US, emailAddress = support@aembit.io

Client Workload 

Annotation 

Validate the Client Workload POD has included the expected Aembit annotation

template:
  metadata:
    annotations:
      aembit.io/agent-inject: "enabled"

Elastic Kubernetes Service (EKS)

Fargate

Client Workload - Explicit Steering

• Explicit Steering Mode ONLY
  • Annotate your Client Workloads with the following Agent Proxy option

    • aembit.io/steering-mode: explicit

Deployment

• Fargate Profile and Edge Component namespace must match

Connectivity

• Public internet access required to communicate with Aembit Cloud services.

Logging

Log Level

• To enable logging for your Edge Components, please refer to the following Knowledge Base article, https://support.aembit.io/hc/en-us/articles/34796856875156-How-To-Enable-logging-for-Edge-Components. We recommend setting the log level to 'DEBUG' for richer insights and information. 
• When working with Aembit Support we recommend 'DEBUG' log level for investigation.