Skip to main content

Migrate to the Runtime Terraform Module

Overview​

This page shows how to move an existing Upwind ECS deployment from the aws-ecs-sensor or aws-ecs-fargate module to the aws-ecs-runtime module. You keep the same cluster and the same Terraform state. Terraform moved blocks tell Terraform that the renamed resources are the same resources, so it does not destroy and rebuild everything.

Pick the tab that matches the module you use today. The choice applies to every tab on this page.

The steps are:

  1. Translate your module inputs to the new variables.
  2. Add a moved.tf file.
  3. Run terraform plan and compare it with the expected plan.
  4. Apply and verify.
  5. Delete moved.tf.

What the migration costs​

The EC2 path is not a zero-downtime migration. Terraform recreates the cluster manager service, and the sensor and scanner tasks restart. Plan a maintenance window. The Fargate path has no cluster manager downtime.

ChangeEffect
The task execution role is replaced. Its name prefix changes from upwind-sensor- to upwind-task-execution-, and Terraform cannot rename a role in place.The sensor, scanner and cluster manager task definitions get new revisions.
The cluster manager service now sets its launch type to EC2 explicitly.Terraform destroys and recreates the cluster manager service.
The sensor and scanner services gain a placement constraint, attribute:ecs.capability.mi !exists.ECS replaces their tasks.

In the validated run, with one EC2 instance:

  • Cluster manager. About 2 minutes with no cluster manager running. Terraform deletes the old service, which takes about 77 seconds to drain, then creates the new one and waits for its task to start.
  • Scanner and sensor. About 4 to 5 seconds each with no task running. ECS stops the old task before it starts the new one.

Nothing else is destroyed or replaced. The secrets, the EventBridge rule, the CloudWatch log group, service discovery, and the cluster manager security group with its rules are untouched, and the registry credentials Lambda function is only updated in place (its environment variable is marked sensitive).

Step 1: Translate your variables​

The module groups the flat variables of the old modules into objects: credentials, upwind, sensor, scanner, cluster_manager, tracer, logging and proxy. The values keep their shapes.

aws-ecs-sensoraws-ecs-runtime
upwind_client_idcredentials.client_id
upwind_client_secretcredentials.client_secret
secret_arncredentials.secret_arn
registry_secret_arncredentials.registry_secret_arn. The runtime module rejects it together with credentials.registry_auth_enabled = false.
registry_credentialscredentials.registry_auth_enabled. If you set it to false, the plan destroys the registry secret, the Lambda function, its IAM role and role attachment, and its EventBridge resources. See the stop box in step 3.
create_secretRemoved. The module infers it. See the note below.
upwind_domainNot needed. Leave it out.
upwind_regionupwind.region
region (deprecated)upwind.region. If you set both region and upwind_region, the old module used region.
create_scannerscanner.enabled
image_sensorsensor.image. The scanner uses it too, as before.
resources_sensorsensor.resources
sensor_extra_envsensor.extra_env
docker_security_optionssensor.docker_security_options. The scanner uses it too, as before.
resources_scannerscanner.resources
scanner_extra_envscanner.extra_env
scanner_tmpfs_sizescanner.tmpfs_size
image_cluster_managercluster_manager.image
resources_cluster_managercluster_manager.resources
cluster_manager_extra_envcluster_manager.extra_env
cluster_manager_replicascluster_manager.replicas
cluster_manager_service_discoverycluster_manager.service_discovery
security_groups_cluster_managercluster_manager.security_groups
multi_clusterNot needed. cluster_manager.multi_cluster is deprecated and has no effect.
cloudwatch_log_grouplogging.cloudwatch_log_group
log_configurationlogging.log_configuration
proxy_configurationproxy
create, tags, ecs_cluster_name, subnets, vpc_id, lambda_archive_output_pathUnchanged
The module infers create_secret

Set credentials.client_id and credentials.client_secret to have the module create the secret. Set credentials.secret_arn to use an existing secret. Setting both, or neither, fails validation. The old create_secret = false with secret_arn becomes only credentials.secret_arn.

Here is an example of the change.

Before:

module "upwind_integration_aws_ecs_cluster" {
  source = "https://get.upwind.io/terraform/modules/aws-ecs-sensor/aws-ecs-sensor-<version>.tar.gz"

  ecs_cluster_name     = "my-cluster"
  vpc_id               = "vpc-0123456789abcdef0"
  subnets              = ["subnet-0123456789abcdef0", "subnet-0fedcba9876543210"]
  upwind_client_id     = var.upwind_client_id
  upwind_client_secret = var.upwind_client_secret
  create_scanner       = true
}

After:

module "upwind_integration_aws_ecs_cluster" {
  source = "https://get.upwind.io/terraform/modules/aws-ecs-runtime/aws-ecs-runtime-1.5.1.tar.gz"

  ecs_cluster_name = "my-cluster"
  vpc_id           = "vpc-0123456789abcdef0"
  subnets          = ["subnet-0123456789abcdef0", "subnet-0fedcba9876543210"]

  credentials = {
    client_id     = var.upwind_client_id
    client_secret = var.upwind_client_secret
  }

  scanner = {
    enabled = true
  }
}

The cluster manager launch type stays at its default, EC2.

Keep the module label the same

Every moved block on this page assumes the module is still called upwind_integration_aws_ecs_cluster. If yours has a different label, use it in every address. Do not rename the module in the same change.

The aws-ecs-runtime module needs the AWS provider >= 6.0. After you swap the module, run terraform init -upgrade.

Step 2: Add moved.tf​

The runtime module renames six resources. Add this file next to your module block. It needs no direct access to the state, so it also works in Terraform Cloud, Atlantis and other CI workflows.

moved.tf
moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_iam_role.upwind_sensor[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_iam_role.task_execution[0]
}

moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_iam_role_policy.sensor_credentials[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_iam_role_policy.task_execution_credentials[0]
}

moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_iam_role_policy_attachment.upwind_sensor_task_execution[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_iam_role_policy_attachment.task_execution[0]
}

moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_iam_role.upwind_cluster_manager_task[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_iam_role.cluster_manager_task[0]
}

moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_ecs_task_definition.upwind_cluster_manager[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_ecs_task_definition.cluster_manager[0]
}

moved {
  from = module.upwind_integration_aws_ecs_cluster.aws_ecs_service.upwind_cluster_manager[0]
  to   = module.upwind_integration_aws_ecs_cluster.aws_ecs_service.cluster_manager[0]
}

The [0] index is optional in moved blocks. Without it, Terraform moves every instance of the resource. The Fargate tab leaves it out.

Do not target module.iam

The runtime module keeps its IAM resources in a nested iam module. Write the to addresses exactly as above, without module.iam[0]. The runtime module has its own moved blocks that carry the IAM resources into the nested module. If you target module.iam[0] yourself, terraform plan fails with Ambiguous move statements.

Four other resources already have the same name in both modules: aws_iam_role_policy.describe_tasks, aws_iam_role.lambda, aws_iam_role_policy.manage_registry_credentials and aws_iam_role_policy_attachment.lambda_execution. They need no moved block. The last three exist only when the module creates the registry credentials Lambda function, so they are absent if registry auth is off.

Fallback: terraform state mv​

If you prefer to change the state directly, run these commands instead of adding moved.tf. Back up the state first.

terraform state pull > backup.tfstate

M=module.upwind_integration_aws_ecs_cluster

terraform state mv "${M}.aws_iam_role.upwind_sensor[0]" \
                   "${M}.aws_iam_role.task_execution[0]"

terraform state mv "${M}.aws_iam_role_policy.sensor_credentials[0]" \
                   "${M}.aws_iam_role_policy.task_execution_credentials[0]"

terraform state mv "${M}.aws_iam_role_policy_attachment.upwind_sensor_task_execution[0]" \
                   "${M}.aws_iam_role_policy_attachment.task_execution[0]"

terraform state mv "${M}.aws_iam_role.upwind_cluster_manager_task[0]" \
                   "${M}.aws_iam_role.cluster_manager_task[0]"

terraform state mv "${M}.aws_ecs_task_definition.upwind_cluster_manager[0]" \
                   "${M}.aws_ecs_task_definition.cluster_manager[0]"

terraform state mv "${M}.aws_ecs_service.upwind_cluster_manager[0]" \
                   "${M}.aws_ecs_service.cluster_manager[0]"

The [0] index is required with terraform state mv. Every resource is created with count, so state addresses always carry an index. In zsh, keep the braces in ${M}. Without them, zsh reads $M.name[0] as an array subscript and builds a wrong address.

Step 3: Plan and compare​

terraform init -upgrade
terraform plan

For the example configuration on this page, the plan ends with this line:

Plan: 7 to add, 5 to change, 7 to destroy.

Your counts can differ, because your inputs change which resources exist. For example, create_scanner = false removes the scanner task definition and service, and secret_arn removes the credentials secret. The safe check is that the plan contains only the kinds of actions in the table below, and does not destroy or replace anything in the stop box.

Addresses are relative to module.upwind_integration_aws_ecs_cluster.

AddressActionWhy
module.iam[0].aws_iam_role.task_executionReplaceThe name prefix changes. Terraform shows it as moved from aws_iam_role.upwind_sensor[0].
module.iam[0].aws_iam_role_policy.task_execution_credentialsReplaceThe role changes. The policy content does not.
module.iam[0].aws_iam_role_policy_attachment.task_executionReplaceThe role changes.
aws_ecs_task_definition.cluster_manager[0]ReplaceNew execution role, explicit EC2 compatibility, task-level CPU and memory, and the module's cluster manager image.
aws_ecs_task_definition.sensor[0]ReplaceNew execution role.
aws_ecs_task_definition.scanner[0]ReplaceNew execution role.
aws_ecs_service.cluster_manager[0]ReplaceThe launch type is now set to EC2, and the capacity provider strategy is dropped.
aws_ecs_service.sensor[0]Update in placeNew task definition and the new placement constraint.
aws_ecs_service.scanner[0]Update in placeNew task definition and the new placement constraint.
module.iam[0].aws_iam_role_policy.describe_tasks[0]Update in placeThe Resource changes from the tasks of your cluster to *.
module.iam[0].aws_iam_role_policy.manage_registry_credentials[0]Update in placeThe plan marks the policy as sensitive. The value does not change.
aws_lambda_function.registry_credentials_loader[0]Update in placeThe plan marks an environment variable as sensitive. The value does not change.

The 7 adds and the 7 destroys are the same 7 replacements: the task execution role, its policy and its attachment, the three task definitions, and the cluster manager service. The 5 changes are the in-place updates in the table.

Terraform also reports three resources as moved with no change: module.iam[0].aws_iam_role.cluster_manager_task[0], module.iam[0].aws_iam_role.lambda[0] and module.iam[0].aws_iam_role_policy_attachment.lambda_execution[0].

These resources are untouched:

  • archive_file.lambda[0]
  • aws_cloudwatch_event_rule.regenerate_registry_credentials[0]
  • aws_cloudwatch_event_target.regenerate_registry_credentials_target[0]
  • aws_cloudwatch_log_group.logs[0]
  • aws_lambda_invocation.generate_registry_credentials[0]
  • aws_lambda_permission.allow_eventbridge[0]
  • aws_secretsmanager_secret.this[0]
  • aws_secretsmanager_secret.upwind_registry_credentials[0]
  • aws_secretsmanager_secret_version.this[0]
  • aws_security_group.cluster_manager_traffic[0]
  • aws_service_discovery_private_dns_namespace.this[0]
  • aws_service_discovery_service.this[0]
  • aws_vpc_security_group_egress_rule.cluster_manager_allow_traffic_ipv4[0]
  • aws_vpc_security_group_ingress_rule.cluster_manager_allow_grpc_ipv4[0]
  • aws_vpc_security_group_ingress_rule.cluster_manager_allow_tls_grpc_ipv4[0]
Stop if the plan destroys any of these

Do not apply if the plan destroys or replaces a secret, the log group, a service discovery resource, the Lambda function, or the security group or one of its rules. A variable probably did not translate. Go back to step 1.

One exception: if you set registry_credentials = false, the plan also destroys the registry secret, the Lambda function, the Lambda IAM role (module.iam[0].aws_iam_role.lambda[0]) and its attachment (module.iam[0].aws_iam_role_policy_attachment.lambda_execution[0]), the EventBridge rule, target, permission and invocation, and the manage_registry_credentials policy. This is expected and safe. The module does not create them when credentials.registry_auth_enabled is false, and those images need no registry authentication.

Step 4: Apply and verify​

terraform apply

When the apply finishes, run the plan again. It must report No changes. Your infrastructure matches the configuration.

terraform plan

Check that running equals desired for all three services. The cluster manager takes a few minutes after the apply, because Terraform recreates it.

aws ecs describe-services \
  --cluster my-cluster \
  --services upwind-sensor upwind-scanner upwind-cluster-manager \
  --query 'services[].{name:serviceName,desired:desiredCount,running:runningCount}' \
  --output table

For further checks, see Test connectivity.

Step 5: Clean up​

When the plan is clean, delete moved.tf. The blocks do nothing once the state has moved, and removing the file leaves the plan clean.

If several people apply this configuration, or other branches still use the old module, keep moved.tf until everyone has applied. Then remove it.

Features to enable afterwards​

The runtime module has options that the old modules do not. None are on by default. Enable them in a separate apply after the migration is verified, so that the migration plan stays comparable with the expected plan.

FeatureHow to enable
Internal load balancer for the cluster manager, instead of service discovery DNScluster_manager.service_discovery.use_internal_lb = true
Sensor and scanner daemons on ECS managed instancesmanaged_instances_capacity_provider_names = ["..."]
Use an external cluster managercluster_manager.create = false and cluster_manager.domain
PrivateLink for cross-account accessSee PrivateLink

Appendix: address map​

Addresses are relative to module.upwind_integration_aws_ecs_cluster. These are the six resources the moved blocks rename.

Old addressNew address
aws_iam_role.upwind_sensor[0]aws_iam_role.task_execution[0]
aws_iam_role_policy.sensor_credentials[0]aws_iam_role_policy.task_execution_credentials[0]
aws_iam_role_policy_attachment.upwind_sensor_task_execution[0]aws_iam_role_policy_attachment.task_execution[0]
aws_iam_role.upwind_cluster_manager_task[0]aws_iam_role.cluster_manager_task[0]
aws_ecs_task_definition.upwind_cluster_manager[0]aws_ecs_task_definition.cluster_manager[0]
aws_ecs_service.upwind_cluster_manager[0]aws_ecs_service.cluster_manager[0]

The runtime module's own moved blocks then move the four IAM resources above into module.iam[0]. They also move aws_iam_role_policy.describe_tasks, aws_iam_role.lambda, aws_iam_role_policy.manage_registry_credentials and aws_iam_role_policy_attachment.lambda_execution there. Those four keep their names and need no moved block from you.

Keep credentials out of your code

Do not write client_id and client_secret into Terraform files that you commit. Pass them as variables, for example with TF_VAR_upwind_client_id, or read them from a secrets manager. If you already have the credentials in AWS Secrets Manager, set credentials.secret_arn instead.