Skip to main content

Troubleshooting

This page lists common problems with the aws-ecs-runtime module. Each entry gives the symptom, the cause, and the fix.

Tasks fail with "can't find the specified secret value"​

Symptom​

The cluster manager or sensor task does not start. The ECS service events show:

ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to get registry auth from asm: ... ResourceNotFoundException: Secrets Manager can't find the specified secret value for staging label: AWSCURRENT

Cause​

The registry credentials Lambda function could not get a registry token, so the registry secret has no value. The most common reasons are wrong client credentials or a wrong upwind.region.

Right after the first install, the ECS service can also start before the Lambda function has run. ECS retries the task, and the error goes away once the secret exists.

Fix​

  1. Read the logs of the Lambda function. The function is named UWRegCred-<cluster-name>, and the log group is /aws/lambda/UWRegCred-<cluster-name>. The module uses only the first 53 characters of the cluster name.

    aws logs tail /aws/lambda/UWRegCred-<cluster-name> --since 1h

  2. If the log shows Encountered exception: HTTP Error 401: Unauthorized, the Upwind authentication endpoint rejected the credentials. Check that the client ID and client secret are correct. Check that the Upwind region matches your organization. See Credentials and secrets for where the credentials are set.

  3. Apply the corrected configuration.

  4. Invoke the function once so the secret gets a value. Otherwise wait for the hourly schedule. Terraform invokes the function during apply only when it first creates the function.

    aws lambda invoke \
      --function-name UWRegCred-<cluster-name> \
      --payload '{}' \
      --cli-binary-format raw-in-base64-out \
      /dev/stdout

  5. Start a new deployment of the failing service.

    aws ecs update-service \
      --cluster <cluster-name> \
      --service upwind-cluster-manager \
      --force-new-deployment

If you use a shared registry secret, only the owner deployment, the one that leaves credentials.registry_secret_arn unset, creates the Lambda function. Check the logs of that function, not of the other clusters.

Not authorized to perform iam:PassRole​

Symptom​

Terraform fails with an error like the following when it registers a task definition, creates the Lambda function, or creates the daemon CloudFormation stack:

not authorized to perform: iam:PassRole on resource: arn:aws:iam::111122223333:role/upwind-...

Cause​

You use pre-created IAM roles. AWS requires the identity that runs Terraform to have iam:PassRole on every role it gives to a service.

Fix​

Give the pipeline's role iam:PassRole on the pre-created roles. See The pipeline needs iam:PassRole for the policy and the pass_role_policy_json output.

The cluster manager is not reachable​

Symptom​

The sensors or the tracer cannot connect to the cluster manager. The logs show errors like:

Error while dialing: dial tcp 172.29.218.199:8444: i/o timeout

Cause​

The cluster manager listens on port 8082 (gRPC) and port 8444 (TLS gRPC, used by the sensor and scanner). A security group, a missing route, or a wrong address blocks the connection. The module creates a security group named cluster-manager-traffic-* that allows these ports from the VPC CIDR block, and it attaches any security groups you provide next to it.

Fix​

  1. Check that the cluster manager service is running. See Test connectivity.

  2. Check that the clients can reach the cluster manager on ports 8082 and 8444. The instances or tasks must have addresses inside the allowed CIDR block, or a security group you provided must allow ingress on these ports from them.

  3. Check how clients find the cluster manager. By default the module registers it in a private DNS namespace through service discovery. Clients must be in the VPC that is associated with the namespace. If you set cluster_manager.service_discovery.use_internal_lb, clients connect to the internal load balancer, which listens on ports 8082 and 8444, instead. Check that the load balancer and its target group are healthy.

  4. For workloads in another VPC or account, see PrivateLink.

The cluster manager cannot reach Upwind​

Symptom​

The cluster manager task starts but cannot connect to Upwind, or it never becomes healthy.

Cause​

The cluster manager needs outbound HTTPS access to the Upwind endpoints. A subnet without a route to a NAT gateway or internet gateway blocks it.

Fix​

Change subnets in your module configuration to private subnets with a NAT gateway route, or add the route, then run terraform apply. For the hosts to allow, see Upwind region. If your network uses a proxy, see Proxy.