Overview
Aembit Agent Proxy is the Edge Component that facilitates the access policy requests on behalf of your workloads. In order to begin serving your requests, the Agent Proxy must first register to an Agent Controller. While these events are simple in design, the distributed, micro-service architecture employed by most enterprises adds layers of complexity and introduces configuration challenges and constraints.
We want to provide you with a foundation for troubleshooting Agent Proxy registration along with common error signatures.
Errors
Error Signature | Environment |
ERROR aembit_agent_proxy [5] - Registration failed. Will attempt to register again. Error The agent is not registered. |
VM, K8s, Serverless |
Error: error sending request for url (https://aembit-agent-controller.aembit/api/token); client error (Connect); tcp connect error: Connection refused (os error 111); Connection refused (os error 111) |
VM, K8s, Serverless |
kubelet Liveness probe failed: HTTP probe failed with statuscode: 401 | K8s, Serverless - ECS Fargate |
Troubleshooting & Investigation
Logging
Enabling verbose logging (Debug) for investigative details and granular information. See the following Knowledge base article configuring Aembit Agent Proxy logging.
Network & Connectivity
Firewall, DNS, and Network architecture are instrumental in Aembit Agent Proxy communication and usage. We recommend reviewing your infrastructure and the associated ingress/egress channels for successful deployment of Edge Components. Running the following command(s) will help validate the connectivity to Aembit Agent Controller from your Aembit Agent Proxy environment.
- Telnet or Curl to Aembit Agent Controller
-
VM
telnet <Agent Controller Host> <Agent Controller port: 5000 or 5443>
curl -v http/s:<Agent Controller Host>
K8s
telnet aembit-agent-controller.aembit 443
curl -v https://aembit-agent-controller.aembit
-
Configuration
Deployment of your Aembit Agent Proxy is dependent on your environment selection. We recommend reviewing your deployment options for accuracy.
VM
Aembit Agent Proxy registration uses an environment variable at install. Confirm the values populated for your Aembit Agent Controller hostname and port along with your other configuration options at install.
AEMBIT_AGENT_CONTROLLER=http://<hostname>:<port>
$ sudo cat /etc/systemd/system/aembit_agent_proxy.service
Kubernetes
Agent Controller
Aembit Agent Proxy and Aembit Agent Controller are deployed through Aembit's Helm Chart. Validate your deployment by reviewing the details displayed using the following commands.
$kubectl describe $(kubectl get pod -l aembit.io/component=agent-controller -n aembit -o name) -n aembit
- Validate
- State
- Container Details
- Environment Settings
Client Workload POD
Agent Proxy Container
$ kubectl describe pod <client_workload_pod>
$ kubectl get pod <client_workload_pod> -o yaml
E.g.
$ kubectl describe $(kubectl get pod -l app=aembit-quickstart-client -n aembit-quickstart -o name) -n aembit-quickstart
- Validate
- State
- Container Details
- Environment Settings
Client-Workload Annotation
Aembit Agent Proxy needs to be injected as a sidecar container into your Client-Workload POD. Validate if the following is found in your YAML configuration
Annotations: aembit.io/agent-inject: enabled
Aembit Agent Controller
Health and Status
Aembit Agent Proxy will continuously attempt to register with the Aembit Agent Controller until successful. Validating the status and health of your Aembit Agent Controller may help ensure successful registration of Aembit Agent Proxy.
VM
$ sudo systemctl status aembit_agent_controller.service
Kubernetes
$ kubectl get pods -n aembit -o wide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
aembit-agent-controller-5c4fb6b55-ksdlv 1/1 Running 0 21h 10.1.0.134 docker-desktop <none> <none>
aembit-agent-injector-56645cddc-ptfcz 1/1 Running 0 21h 10.1.0.133 docker-desktop <none> <none>
[In Tenant UI] Reporting -> Audit Logs
Aembit audit log events capture usage details that can add visibility into Agent Controller and various other features. Here, we can review activity and confirm if there are indications of Agent Controller issues or health.
E.g.
Activity - agent controller attestation
"outcome": { "reason": "AgentControllerAttestation", "result": "Failure" }, "trustProvider": { "reason": "InvalidSignature" }, "severity": "Alert",