Skip to main content

Troubleshooting

This topic provides information to help troubleshoot issues with your Upwind components installation.

Cluster Manager egress issues​

The Cluster Manager needs to be deployed in a subnet that has egress capabilities. Ensure that the Cluster Manager gets deployed into a private subnet that is connected to a NAT gateway.

In case the Cluster Manager was created with an incorrect network configuration the service can be updated via Terraform by specifying the correct subnets, or be fixed manually via the AWS CLI:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --network-configuration file://updated-network-config.json

Where the contents of the file looks like this:

{
  "awsvpcConfiguration": {
    "subnets": [
      "<specify new subnet ID>"
    ],
    "securityGroups": [],
    "assignPublicIp": "DISABLED"
  }
}

We can also do this using the shorthand syntax:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --network-configuration "awsvpcConfiguration={subnets=[<specify new subnet ID>]}"

After updating the network configuration the task needs to redeployed:

aws ecs update-service \
  --service upwind-cluster-manager \
  --cluster <YOUR_ECS_CLUSTER_NAME> \
  --force-new-deployment

Cluster Manager fails to start due to missing secret immediately after install​

The Cluster Manager needs a secret which is created by a Lambda function that is created as part of the Terraform module. Immediately after installing, the ECS Service may be created before the lambda has run for the first time, and the Cluster Manager task will fail to start. The task will be recreated by the ECS Service a short time later and the secret should have been created by then.

If the Cluster Manager tasks persistently fail to start, check the status of the UWRegCred-<YOUR_ECS_CLUSTER_NAME> Lambda function and inspect the logs. Double-check that the credentials you provided in Terraform are correct.

I do not see the Lambda function I wish to instrument when running upwindctl​

upwindctl will only show Lambda functions that can be instrumented. For a Lambda function to be instrumentable via upwindctl the following prerequisites must be met:

  • If the Lambda function should communicate with an Upwind Cluster Manager the function must be connected to a VPC
  • The Lambda function is not containerized. To instrument containerized functions you must build the Upwind Lambda Tracer image into your function image
  • The Lambda function is using a supported runtime
  • The Lambda function should not have any conflicting environment variables set. The environment variables are AWS_LAMBDA_EXEC_WRAPPER and LD_PRELOAD

You can run the following command to see Lambda functions that cannot be instrumented via upwindctl, along with an indication of what prerequisite is not met:

upwindctl lambda list --instrumentable=false