Multi-Cluster Mode

When you run multiple ECS clusters, you can deploy a single cluster manager and point the sensors and scanners in the other clusters at it. A cluster dedicated to running the cluster manager, with no workloads of its own, is called a utility cluster.

Every cluster uses the same aws-ecs-runtime module, with a different cluster_manager block. Multi-cluster mode is not available with CloudFormation.

warning

If you already monitor AWS Lambda, do not reuse the upwind-utility-cluster-* cluster from the AWS Lambda guide. Its cluster manager exposes only port 8082, which is what the Lambda Tracer uses. ECS sensors and scanners need 8444.

Deploy the utility cluster

Disable the node daemons so the cluster runs the cluster manager and nothing else:

module "upwind_cluster_manager" {
  source = "https://get.upwind.io/terraform/modules/aws-ecs-runtime/aws-ecs-runtime-1.5.1.tar.gz"

  create           = true
  ecs_cluster_name = "upwind-utility"
  vpc_id           = var.vpc_id
  subnets          = toset(var.subnet_ids)

  credentials = {
    client_id     = var.upwind_client_id
    client_secret = var.upwind_client_secret
  }

  sensor  = { enabled = false }
  scanner = { enabled = false }

  cluster_manager = {
    create = true

    service_discovery = {
      use_internal_lb = true
    }
  }
}

• use_internal_lb puts the cluster manager behind an internal load balancer listening on 8082 and 8444. Omit it to use AWS Cloud Map service discovery instead, which only resolves inside a single VPC. Either way, the cluster_manager_discovery_endpoint output holds the address that the other clusters use: the load balancer DNS name with the internal load balancer, or the Cloud Map name otherwise. See Service discovery.
• You do not need to set multi_cluster. The cluster manager can describe tasks in any cluster, and the setting is deprecated with no effect.

Connect additional clusters

Use the same module with the cluster manager disabled and pointed at the shared one. Set cluster_manager.create to false and cluster_manager.domain to the utility cluster's cluster_manager_discovery_endpoint output. Do not use cluster_manager_lb_dns_name here: it is null without the internal load balancer, and a null domain with create = false silently means direct mode instead of an external cluster manager. See External cluster manager.

module "upwind_sensor" {
  source   = "https://get.upwind.io/terraform/modules/aws-ecs-runtime/aws-ecs-runtime-1.5.1.tar.gz"
  for_each = toset(["production-api", "production-workers"])

  create           = true
  ecs_cluster_name = each.value

  credentials = {
    secret_arn          = module.upwind_cluster_manager.sensor_secret_arn
    registry_secret_arn = module.upwind_cluster_manager.registry_credentials_secret_arn
  }

  cluster_manager = {
    create = false
    domain = module.upwind_cluster_manager.cluster_manager_discovery_endpoint
  }
}

The utility cluster owns the registry secret and the Lambda function that refreshes it hourly, because it leaves credentials.registry_secret_arn unset. Every other cluster passes that secret's ARN, as the example shows, and creates no Lambda function of its own. See Shared registry secret.

The sensor and scanner run as ECS daemon services, so each cluster needs registered EC2 container instances. One task is placed on every instance. Allow those instances to reach the cluster manager's security group on ports 8082 and 8444. The module exposes that group as the cluster_manager_security_group_id output.

If a cluster uses ECS Managed Instances, also set managed_instances_capacity_provider_names to the names of its capacity providers. The daemon services skip managed instances, which are covered by a separate managed daemon, and leaving this empty deploys nothing to them. See Managed instances.

Important

Set domain to a hostname only, with no port and no scheme. The sensor and scanner append the port themselves, so a port here produces a target like host:8444:8444 and every sensor fails with too many colons in address, even though the cluster manager is healthy.

To share a cluster manager across VPCs, keep use_internal_lb enabled, connect the VPCs with peering or Transit Gateway, and allow 8082 and 8444 from the other CIDR ranges. By default only the utility cluster's own VPC CIDR is allowed.