Skip to main content

Troubleshooting

This topic provides information to help troubleshoot issues with your Upwind components installation.

Cluster Manager Egress Issues​

The Cluster Manager needs to be deployed in a subnet that has egress capabilities. Ensure that the Cluster Manager gets deployed into a private subnet that is connected to a NAT gateway.

In the case the Cluster Manager was created with incorrect network configuration the service can be updated via Terraform or CloudFormation with the correct subnets or be fixed manually via the AWS CLI.

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --network-configuration file://updated-network-config.json

Where the contents of the file looks like this:

{
  "awsvpcConfiguration": {
    "subnets": [
      "<specify new subnet ID>"
    ],
    "securityGroups": [],
    "assignPublicIp": "DISABLED"
  }
}

We can also do this using the shorthand syntax:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --network-configuration "awsvpcConfiguration={subnets=[<specify new subnet ID>]}"

After updating the network configuration the task needs to redeployed:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --force-new-deployment

Tracer to Cluster Manager Communication​

The tracers in your tasks need network connectivity to the Cluster Manager. If the tracers cannot communicate with it, you may see errors like the following:

Error while dialing: dial tcp 172.29.218.199:8082: i/o timeout"

The Cluster Manager gets a security group that allows inbound traffic on port 8082 from the VPC CIDR block. The Terraform module always creates this security group (cluster-manager-traffic-*) and attaches any security groups set in the security_groups_cluster_manager variable alongside it. The CloudFormation stack creates it from the VPCCIDRBlock parameter, unless you set the ClusterManagerSecurityGroup parameter, which replaces it.

If the tracers still cannot reach the Cluster Manager, check that the tasks in the cluster have addresses within the allowed CIDR block, or that any security group you provided allows ingress on port 8082 from the tasks in the cluster.

Cluster Manager Fails to Start due to Missing Secret Immediately After Install​

The Cluster Manager needs a secret which is created by a lambda that is created as part of the CloudFormation stack or Terraform module. Immediately after installing, the ECS Service may be created before the lambda has run the first time, and the Cluster Manager task will fail to start. The task will be recreated by the ECS Service a short time later and the secret should have been created by then.

If the Cluster Manager tasks persistently fail to start, check the status of the registry credentials Lambda function and inspect the logs. Double-check that the credentials you provided to the Terraform or CloudFormation are correct. The Terraform module names the function UWRegCred-<YOUR_ECS_CLUSTER_NAME>, using at most the first 53 characters of the cluster name. In a CloudFormation stack, it is the UpwindRegistryCredentialsLoaderLambda resource.

Destroying and Recreating Fails on Conflicting Secret Manager Secrets​

This applies to CloudFormation installations. When you delete the CloudFormation stack, the secrets it created are scheduled for deletion instead of being deleted right away. To reinstall the sensor with CloudFormation before the scheduled deletion happens, force-delete these secrets. The Terraform module deletes its secrets immediately, so this does not affect Terraform installations.

  • Delete the secret of the sensor credentials immediately:
aws secretsmanager delete-secret \
  --secret-id upwind-sensor-<YOUR_ECS_CLUSTER_NAME> \
  --force-delete-without-recovery
  • Delete the secret of the registry credentials immediately:
aws secretsmanager delete-secret \
  --secret-id upwind-registry-credentials-<YOUR_ECS_CLUSTER_NAME> \
  --force-delete-without-recovery