PrivateLink access to the Upwind cluster manager

PrivateLink lets ECS services in other VPCs or accounts reach the cluster manager's gRPC endpoints without VPC peering, a Transit Gateway, route table changes, or non-overlapping CIDR requirements. It exposes the cluster manager's internal load balancer, whether the cluster manager runs on EC2 or Fargate.

info

PrivateLink is unidirectional. Traffic flows from the consumer to the cluster manager only. Traffic in the other direction needs a separate endpoint service that points the opposite way.

Enabling it (provider side)

PrivateLink requires the cluster manager to sit behind an internal load balancer, which the aws-ecs-runtime Terraform module builds when you set cluster_manager.service_discovery.use_internal_lb = true. That load balancer listens on 8082 and 8444, and the endpoint service exposes it to other accounts. If you leave use_internal_lb unset, the module uses AWS Cloud Map service discovery instead. Cloud Map resolves only inside the cluster manager's own VPC, so an endpoint service would have nothing to attach to. See Service discovery. The cluster_manager.privatelink block is listed in the cluster manager settings.

Prerequisites

The endpoint service attaches to a load balancer the module owns. The module fails the plan unless this configuration is the one that deploys the cluster manager:

Setting	Required value	Why
`cluster_manager.create`	`true` (the default)	The module has to create the cluster manager for there to be a load balancer to expose.
`cluster_manager.domain`	`""` (the default)	A `domain` points at an external cluster manager, so there is nothing to expose.
`cluster_manager.service_discovery.use_internal_lb`	`true`	Creates the internal Network Load Balancer (NLB) that the endpoint service fronts.
`cluster_manager.privatelink.allowed_principals`	At least one ARN	The module rejects an empty list.

These are requirements of PrivateLink. Sharing a cluster manager between clusters in the same VPC needs none of them. Multi-cluster mode works over Cloud Map for any cluster whose tasks run in the cluster manager's VPC. The internal load balancer, and PrivateLink on top of it, takes it beyond that VPC.

When you combine the two, set PrivateLink in the utility cluster's configuration, the one that creates the cluster manager. The additional clusters set create = false and a domain, so they cannot enable it.

The aws-ecs-runtime module requires AWS provider hashicorp/aws >= 6.0. That is enough for everything on this page.

module "aws_ecs_runtime" {
  source = "https://get.upwind.io/terraform/modules/aws-ecs-runtime/aws-ecs-runtime-1.5.1.tar.gz"

  # ...

  cluster_manager = {
    service_discovery = {
      use_internal_lb = true
    }
    privatelink = {
      allowed_principals = ["arn:aws:iam::222222222222:root"]
    }
  }
}

Enabling PrivateLink forces cross-zone load balancing on the NLB. It overrides service_discovery.enable_cross_zone_load_balancing. Without cross-zone load balancing, a consumer's endpoint in one AZ could reach only the cluster manager tasks in that same AZ.

`allowed_principals`

Required, with at least one entry. These are the only supported ARN forms:

Form	ARN
AWS account (all principals in it)	`arn:aws:iam::{account_id}:root`
Role	`arn:aws:iam::{account_id}:role/{role_name}`
User	`arn:aws:iam::{account_id}:user/{user_name}`
All principals in all AWS accounts	`*`

Organization and OU ARNs are not supported. There is no way to scope access to an AWS Organization. To cover several accounts, list each account's root ARN individually.

Principal ARNs with path components are not supported. This rules out IAM Identity Center roles such as arn:aws:iam::123456789012:role/aws-reserved/sso.amazonaws.com/AWSReservedSSO_Admin. Use the account root ARN instead.

Use account root ARNs (arn:aws:iam::{account_id}:root) by default. They work regardless of how the consuming account manages its roles.

warning

Take care with *. From AWS's own documentation: "If you grant everyone permission to access the endpoint service and configure the endpoint service to accept all requests, your load balancer will be public even if it has no public IP address." The endpoint service name is hard to guess, but it is not an access control. Never combine allowed_principals = ["*"] with acceptance_required = false. The module fails the plan if you try.

`acceptance_required`

Defaults to true. Each new connection waits for a manual accept. You see the pending request in the console, which confirms the provider side works, and nothing connects just because it presents an allowed principal.

`private_dns_name`

Optional, and two-phase. You must own the domain and publish a TXT record for AWS to verify. That cannot finish within a single terraform apply. Expect verification_state = "pendingVerification" after the first apply, until the TXT record is published and AWS re-checks it. Leave this unset unless you need a private DNS name for the endpoint service.

Consumer side (you implement this)

The module creates only the provider side of the connection, the endpoint service. It does not create the consumer-side interface endpoint, its security group, or any Route 53 records. Those live in a different AWS account and a different Terraform state file. Work through this checklist when you build them:

Get the endpoint service name from the provider side. The module returns it as the cluster_manager_endpoint_service_name output (format com.amazonaws.vpce.<region>.vpce-svc-0abc…). Pass it as service_name on the consumer's aws_vpc_endpoint. The worked example below exposes it as a root output on the provider side.
Match AZ IDs, not AZ names. Create the interface endpoint only in subnets whose AZ IDs appear in the module's cluster_manager_endpoint_service_az_ids output. AZ names (e.g. us-east-1a) map to different physical zones in different accounts, so matching by name can place the endpoint in the wrong physical AZ. Cross-check with aws ec2 describe-availability-zones in the consumer account. This is the most common onboarding failure.
Open the endpoint ENI's security group inbound. The interface endpoint's network interfaces need inbound rules that allow 8082 and 8444 from your client security group. By default the endpoint uses the consumer VPC's default security group, which usually does not allow this.
Accept the connection request on the provider side, unless acceptance_required = false was set.
Prefer the zonal DNS name. Each interface endpoint exposes zonal DNS names alongside a regional one. The zonal name that matches the client's own AZ keeps traffic in that AZ and reduces cross-AZ data transfer cost.

note

TLS does not restrict the hostname you dial. The load balancer passes TCP straight through on both ports and holds no certificate. The cluster manager presents a self-signed certificate that clients do not hostname-verify, because an ECS deployment reaches it under many different names. The endpoint's DNS name works as it is, with no certificate changes on either side.

Worked example

Two Terraform configurations in two accounts, wired together by the provider's outputs. They cannot be a single apply, because they run in different accounts and different state files.

note

The consumer configuration in step 2 is the aws-ecs-runtime-privatelink-consumer example from the module repository. It uses plain AWS resources and no Upwind module. Add an aws provider block for the endpoint service's region.

Step 1. Provider side

In the account that runs the cluster manager. use_internal_lb is required. The privatelink block publishes the endpoint service.

provider/main.tf
module "upwind" {
  source = "https://get.upwind.io/terraform/modules/aws-ecs-runtime/aws-ecs-runtime-1.5.1.tar.gz"

  vpc_id           = var.vpc_id
  subnets          = var.subnet_ids
  ecs_cluster_name = var.ecs_cluster_name

  credentials = {
    client_id     = var.upwind_client_id
    client_secret = var.upwind_client_secret
  }

  cluster_manager = {
    service_discovery = {
      use_internal_lb = true
    }

    privatelink = {
      # Account root ARNs of the consuming accounts. Organization ARNs, OU
      # ARNs, and role ARNs containing a path are rejected by AWS.
      allowed_principals = ["arn:aws:iam::222222222222:root"]
    }
  }
}

output "endpoint_service_name" {
  value = module.upwind.cluster_manager_endpoint_service_name
}

# AZ IDs, not names. Step 2 needs these.
output "endpoint_service_az_ids" {
  value = module.upwind.cluster_manager_endpoint_service_az_ids
}

After apply, read the two values the consumer needs:

terraform output endpoint_service_name     # com.amazonaws.vpce.<region>.vpce-svc-0abc…
terraform output endpoint_service_az_ids   # ["use1-az2", "use1-az4"]

Step 2. Consumer side

In the account whose workloads need to reach the cluster manager, using the two values from step 1 as input.

The configuration checks AZ IDs for you. A subnet in the wrong zone fails the plan with a message that names the subnet, its AZ ID, and what the covered AZ IDs are called in this account. The covered_az_names_in_this_account output gives you the same translation before you pick subnets.

consumer/main.tf
variable "endpoint_service_name" {
  description = "The provider account's `cluster_manager_endpoint_service_name` output, e.g. com.amazonaws.vpce.us-east-1.vpce-svc-0abc123."
  type        = string
}

variable "endpoint_service_az_ids" {
  description = "The provider account's `cluster_manager_endpoint_service_az_ids` output. These are AZ IDs (use1-az2), not AZ names (us-east-1a)."
  type        = list(string)
}

variable "vpc_id" {
  description = "Consumer VPC that will hold the interface endpoint."
  type        = string
}

variable "subnet_ids" {
  description = "Consumer subnets for the endpoint network interfaces. At most one per availability zone, and each must sit in an AZ the endpoint service covers."
  type        = list(string)
}

variable "client_security_group_ids" {
  description = "Security groups of the workloads that will talk to the cluster manager. Inbound rules on 8082 and 8444 are created from each of these."
  type        = list(string)
}

# ---------------------------------------------------------------------------
# Availability zone checking.
#
# AZ names do not mean the same thing in different AWS accounts: us-east-1a in
# the provider account and us-east-1a here are usually different physical
# zones. The provider therefore publishes AZ *IDs*, and the consumer has to
# translate them into local subnet choices.
#
# Placing the endpoint in an uncovered zone is the most common way this setup
# fails. AWS rejects it, but its error names only the subnet and not what to
# use instead. The preconditions below fail the plan with the covered AZ IDs
# and what they are called in this account.
# ---------------------------------------------------------------------------

data "aws_availability_zones" "this" {
  state = "available"
}

data "aws_subnet" "endpoint" {
  for_each = toset(var.subnet_ids)
  id       = each.value
}

locals {
  # The endpoint service's AZ IDs, expressed as this account's AZ names.
  covered_az_names = [
    for az_id in var.endpoint_service_az_ids :
    data.aws_availability_zones.this.names[
      index(data.aws_availability_zones.this.zone_ids, az_id)
    ]
  ]

  chosen_az_ids = { for id, s in data.aws_subnet.endpoint : id => s.availability_zone_id }

  uncovered_subnets = [
    for id, az_id in local.chosen_az_ids : "${id} (${az_id})"
    if !contains(var.endpoint_service_az_ids, az_id)
  ]
}

# ---------------------------------------------------------------------------
# Security group for the endpoint network interfaces.
#
# This is the step most often missed. The interface endpoint's ENIs need
# INBOUND rules on the cluster manager ports. Omit this and the endpoint falls
# back to the VPC default security group, which usually does not allow the
# traffic; the symptom is a connection that simply hangs.
#
# The provider side disables security group enforcement for PrivateLink
# traffic on its own load balancer, because the source address it would see is
# this account's ENI IP. Access control there is allowed_principals plus
# acceptance_required. This security group is the consumer-side half.
# ---------------------------------------------------------------------------

resource "aws_security_group" "endpoint" {
  name_prefix = "upwind-cluster-manager-endpoint-"
  description = "Upwind cluster manager PrivateLink endpoint ENIs"
  vpc_id      = var.vpc_id

  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_vpc_security_group_ingress_rule" "grpc" {
  for_each = toset(var.client_security_group_ids)

  security_group_id            = aws_security_group.endpoint.id
  referenced_security_group_id = each.value
  from_port                    = 8082
  to_port                      = 8082
  ip_protocol                  = "tcp"
  description                  = "plaintext gRPC from Upwind workloads"
}

resource "aws_vpc_security_group_ingress_rule" "grpcs" {
  for_each = toset(var.client_security_group_ids)

  security_group_id            = aws_security_group.endpoint.id
  referenced_security_group_id = each.value
  from_port                    = 8444
  to_port                      = 8444
  ip_protocol                  = "tcp"
  description                  = "gRPC over TLS from Upwind workloads"
}

# ---------------------------------------------------------------------------
# The interface endpoint.
# ---------------------------------------------------------------------------

resource "aws_vpc_endpoint" "cluster_manager" {
  vpc_id             = var.vpc_id
  service_name       = var.endpoint_service_name
  vpc_endpoint_type  = "Interface"
  subnet_ids         = var.subnet_ids
  security_group_ids = [aws_security_group.endpoint.id]

  # The endpoint service publishes no private DNS name by default, so there is
  # nothing for this to attach to. Point clients at the endpoint's own DNS
  # name instead -- see the outputs.
  private_dns_enabled = false

  lifecycle {
    precondition {
      condition     = length(local.uncovered_subnets) == 0
      error_message = <<-EOT
        These subnets are in availability zones the endpoint service does not cover:
        ${join(", ", local.uncovered_subnets)}

        The service covers AZ IDs: ${join(", ", var.endpoint_service_az_ids)}
        In this account those AZ IDs are: ${join(", ", local.covered_az_names)}

        Pick subnets in those zones. Do not match on AZ name across accounts --
        the same AZ name maps to different physical zones in different accounts.
      EOT
    }

    precondition {
      condition     = length(distinct(values(local.chosen_az_ids))) == length(var.subnet_ids)
      error_message = "subnet_ids must contain at most one subnet per availability zone."
    }
  }
}

output "endpoint_dns_name" {
  description = "Regional DNS name of the interface endpoint. This is the cluster manager address your workloads should use, e.g. UPWIND_TRACER_API_HOST = \"<this>:8082\"."
  value       = [for e in aws_vpc_endpoint.cluster_manager.dns_entry : e.dns_name][0]
}

output "endpoint_dns_names_all" {
  description = "Every DNS name the endpoint exposes: the regional name plus one zonal name per availability zone. Resolving the zonal name matching a client's own AZ keeps traffic in-zone and avoids cross-AZ data transfer cost."
  value       = [for e in aws_vpc_endpoint.cluster_manager.dns_entry : e.dns_name]
}

output "endpoint_security_group_id" {
  description = "Security group attached to the endpoint network interfaces."
  value       = aws_security_group.endpoint.id
}

output "covered_az_names_in_this_account" {
  description = "The endpoint service's AZ IDs translated into this account's AZ names. Useful when choosing subnet_ids by hand."
  value       = local.covered_az_names
}

Step 3. Point your workloads at the endpoint

The endpoint_dns_name output is the cluster manager address. How you use it depends on whether the consumer account deploys its workloads with aws-ecs-runtime.

If it does, point the module at the endpoint and let it set the environment variables. This is the same as connecting an additional cluster in multi-cluster mode, with the endpoint DNS name in place of the load balancer's. See External cluster manager.

cluster_manager = {
  create = false
  domain = var.endpoint_dns_name # the consumer output from step 2
}

If you set the variables yourself, there are two, and each uses its own port. Give both the endpoint's DNS name on its own:

Workload	Variable	Reaches
Sensor and scanner	`UPWIND_CLUSTER_AGENT_DOMAIN`	`8444`, gRPC over TLS
Tracer	`UPWIND_TRACER_API_HOST`	`8082`, plaintext gRPC

UPWIND_CLUSTER_AGENT_DOMAIN=<endpoint_dns_name>
UPWIND_TRACER_API_HOST=<endpoint_dns_name>

Important

The two differ on whether a port is allowed. UPWIND_TRACER_API_HOST takes an optional host:port and falls back to 8082 when you leave the port off. UPWIND_CLUSTER_AGENT_DOMAIN takes a hostname only. The sensor and scanner append the port themselves, so a port here produces a target like host:8444:8444, and every sensor fails with too many colons in address even though the cluster manager is healthy.

The sensor and scanner reach 8444 over grpcs. The cluster manager's certificate is self-signed and not hostname-verified, so the endpoint's DNS name works as it is.

note

If a client runs in a known AZ, use the matching zonal name from endpoint_dns_names_all instead of the regional one. It keeps traffic in that zone and avoids cross-AZ data transfer cost.

Debugging

Healthy targets alongside failing client connections is the signature symptom of a security group that blocks PrivateLink traffic on the inbound side. Health checks are subject to the load balancer's outbound rules but not its inbound rules, so targets stay healthy while real client connections fail. Blocked flows show up only in the SecurityGroupBlockedFlowCount_Inbound CloudWatch metric.

The module sets enforce_security_group_inbound_rules_on_private_link_traffic = "off" on the NLB when PrivateLink is enabled, to avoid this failure. Access control on the provider side is allowed_principals plus acceptance_required.

note

Adding an AZ to the NLB does not automatically enable it on the endpoint service. Enabling a new AZ on the endpoint service is a separate step. AWS recommends waiting after you enable an AZ before you rely on it in automation.

Enabling it (provider side)​

Prerequisites​

allowed_principals​

acceptance_required​

private_dns_name​

Consumer side (you implement this)​

Worked example​

Step 1. Provider side​

Step 2. Consumer side​

Step 3. Point your workloads at the endpoint​

Debugging​

Enabling it (provider side)

Prerequisites

`allowed_principals`

`acceptance_required`

`private_dns_name`

Consumer side (you implement this)

Worked example

Step 1. Provider side

Step 2. Consumer side

Step 3. Point your workloads at the endpoint

Debugging