Skip to main content

Troubleshooting

Kernel version 4.14​

The standard Upwind Sensor uses the BPF CO-RE concept, but it is not available on version 4.14 of the Linux kernel. In order to install the Upwind Sensor on instances with kernel version 4.14, use the "bcc" flavor of the sensor image. Images are tagged with a version, e.g 0.48.2, and the bcc image has the -bcc suffix added, e.g 0.48.2-bcc.

Cluster Manager Egress Issues​

The cluster manager needs to be deployed in a subnet that has egress capabilities. Ensure that the cluster manager gets deployed into a private subnet that is connected to a NAT gateway.

In the case the cluster manager was created with incorrect network configuration the service can be updated via Terraform or CloudFormation with the correct subnets or be fixed manually via the AWS CLI.

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECR_CLUSTER_NAME> \
  --network-configuration file://updated-network-config.json

Where the contents of the file looks like this:

{
  "awsvpcConfiguration": {
    "subnets": [
      "<specify new subnet ID>"
    ],
    "securityGroups": [],
    "assignPublicIp": "DISABLED"
  }
}

We can also do this using the shorthand syntax:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECR_CLUSTER_NAME> \
  --network-configuration "awsvpcConfiguration={subnets=[<specify new subnet ID>]}"

After updating the network configuration the task needs to redeployed:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECR_CLUSTER_NAME> \
  --force-new-deployment

Cluster Manager Startup Logs​

The cluster manager is configured to use awsvpc network mode and the network can take some time to be set up properly, even while the cluster manager process is starting up. Some errors may be logged during startup until the Docker network for the task is fully set up.

Cluster Manager or Sensors Fail To Start due to Missing Secret Immediately After Install​

The cluster manager and sensor tasks needs a secret which is created by a lambda that is created as part of the CloudFormation stack or Terraform module. Immediately after installing, the ECS Service may be created before the lambda has run the first time, and the cluster manager task will fail to start. The task will be recreated by the ECS Service a short time later and the secret should have been created by then.

If the cluster manager or sensor tasks persistently fail to start, check the status of the UpwindRegistryCredentials-<YOUR_ECR_CLUSTER_NAME> Lambda function and inspect logs from the lambda. Double-check that the credentials you provided to the Terraform or CloudFormation are correct.

Sensor to Cluster Manager Communication​

The sensors that run on the host network must have network connectivity to the cluster manager that runs with awsvpc network mode. If the sensors are not able to communicate with the cluster manager, you may see errors like the following:

Error while dialing: dial tcp 172.29.218.199:8082: i/o timeout"

By default, the cluster manager is deployed without any security group associated with it. This means that AWS will assign the default security group for the VPC to it, and if the default security group does not allow inbound traffic on ports 8080 and 8082 the sensors will not be able to communicate with the cluster manager. The security group for the cluster manager can be set with the ClusterManagerSecurityGroup parameter of the CloudFormation stack or the security_groups_cluster_manager variable of the Terraform module.

In case a security group needs to be specified the group will need to allow for ingress from hosts in the cluster to port 8080 and 8082.

Destroying and Recreating Fails on Conflicting Secret Manager Secrets​

When removing the sensor installation and a secret has been created then the secret will get scheduled for deletion instead of being deleted right away. In order to reinstall the sensor before the scheduled deletion happens, this secret needs to be forcefully removed.

  • Delete the secret of the sensor credentials immediately:
aws secretsmanager delete-secret \
  --secret-id upwind-sensor-credentials-<YOUR_ECR_CLUSTER_NAME> \
  --force-delete-without-recovery
  • Delete the secret of the registry credentials immediately:
aws secretsmanager delete-secret \
  --secret-id upwind-registry-credentials-<YOUR_ECR_CLUSTER_NAME> \
  --force-delete-without-recovery