Instrument Lambda functions

The Upwind Lambda Tracer is an AWS Lambda extension that you can add to your Lambda functions. It is distributed as a Lambda layer.

The instrumentation process involves making the following changes to the Lambda function configuration:

Adding the Upwind Lambda Tracer layer
Setting the AWS_LAMBDA_EXEC_WRAPPER environment variable
Modifying the function's execution role to grant permission to read an Upwind client credentials secret from AWS Secrets Manager (not required if Lambda functions are configured to communicate with an Upwind Cluster Manager)

Infrastructure as code (Terraform, CloudFormation, CDK)

The upwindctl CLI updates live Lambda configuration through the AWS API. If you already manage Lambda with infrastructure as code, using upwindctl as the primary way to change production functions can introduce configuration drift (your templates no longer match what is deployed). Prefer declaring the Upwind Lambda Tracer layer and environment variables in your IaC instead. Open the .zip archive tab below, then follow Manual instrumentation for Terraform and IaC.

Prerequisites

The Lambda function should use a supported runtime
If the Lambda function should communicate directly with the Upwind platform: The Lambda function must have access to credentials to authenticate with the Upwind platform
If the Lambda function should communicate with an Upwind Cluster Manager: The Lambda function must be connected to a VPC with an Upwind Cluster Manager running in the VPC

Supported runtimes

The Upwind Lambda Tracer supports Lambda functions using any of the supported AWS Lambda runtimes, with the exception of OS-only runtimes.

Supported runtimes are:

Node.js
Python
Java
.NET
Ruby

Upwind client credentials

info

If you are using upwindctl to instrument Lambda functions these steps will be done automatically by supplying the --auth-client-id and --auth-client-secret flags.

Lambda functions that communicate directly with the Upwind platform need to have access to Upwind client credentials for authentication. The credentials should be stored in an AWS Secrets Manager secret, and the Lambda function's execution role should have permission to read the secret.

Storing Upwind client credentials in AWS Secrets Manager

Upwind client credentials should be stored in an AWS Secrets Manager secret. The secret should have the following format:

{"ClientID":"my-client-id","ClientSecret":"my-client-secret"}

To generate new Upwind client credentials you can go to the Upwind console and generate new credentials of type Sensor.

Granting Lambda functions permission to read AWS Secrets Manager secret

The Lambda function's execution role needs to have permission to read the Upwind client credentials secret from AWS Secrets Manager. You can add an in-line policy to the function's execution role:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "<SECRET_ARN>"
      ]
    }
  ]
}

You can find the Lambda function's execution role in the AWS console, by going to Configuration -> Permissions on the Lambda function's console page.

Instrumentation

The instrumentation method differs based on if the Lambda function is deployed as a .zip archive or container.

.zip archive
Container

Instrument with `upwindctl`

You can use the upwindctl tool to easily instrument .zip archive Lambda functions.

info

The upwindctl tool will use your current AWS CLI profile to interact with the AWS Lambda API and instrument functions. You can supply the --aws-profile and/or --aws-region flags to specify a different profile and region.

Install `upwindctl`

curl -s https://get.upwind.io/upwindctl.sh | sh

To add upwindctl to your PATH, append this line to your shell config:

. "$HOME/.upwindctl/env"

Instrument Lambda functions

info

To configure Lambda functions to communicate with an Upwind Cluster Manager you can provide the --report-to-cluster-manager flag to the lambda list and lambda instrument commands.

Run the following command to see a list of Lambda functions that can be instrumented:

upwindctl lambda list

To instrument a specific function by name:

upwindctl lambda instrument --function-name my-lambda-function

You can provide the --auth-client-id and --auth-client-secret flags to the instrument command to create or update an Upwind client credentials secret in AWS Secrets Manager:

upwindctl lambda instrument --function-name my-lambda-function --auth-client-id $UPWIND_CLIENT_ID --auth-client-secret $UPWIND_CLIENT_SECRET

info

upwindctl will look for Upwind client credentials in an AWS Secrets Manager secret named upwind-lambda-credentials. You can change this by providing a different secret name to the --auth-secrets-manager-secret flag. If your AWS CLI user does not have permission to interact with AWS Secrets Manager you can provide the ARN of an existing secret to the --auth-secrets-manager-secret flag.

To configure Lambda functions to report to a specific Upwind region (e.g. eu or me), you can provide the --upwind-region flag:

upwindctl lambda instrument --function-name my-lambda-function --upwind-region eu

To operate on multiple Lambda functions you can use the --bulk flag, a confirmation prompt will appear before taking any action:

upwindctl lambda instrument --bulk

Note that the upwindctl tool offers a number of options, such as filtering functions by VPC ID or tags:

upwindctl lambda instrument --bulk --vpc-id vpc-XXX
upwindctl lambda instrument --bulk --tags Key=some-tag,Values=some-value

To see a list of available configuration options you can run e.g.:

upwindctl lambda instrument -h

Uninstrument Lambda functions

To remove the Upwind Lambda Tracer from Lambda functions you can use the corresponding uninstrument command, such as:

upwindctl lambda uninstrument --function-name my-lambda-function

Manual instrumentation for Terraform and IaC

If you manage .zip archive Lambda functions with Terraform, CloudFormation, the AWS CDK, or similar tools, apply the same configuration changes that upwindctl would make: attach the Upwind Lambda Tracer layer, set AWS_LAMBDA_EXEC_WRAPPER, and (when not using an Upwind Cluster Manager) grant access to the Upwind client credentials secret and set UPWIND_AUTH_SECRETS_MANAGER_SECRET.

Run the following command to get the latest Upwind Lambda Tracer layer ARN:

upwindctl lambda show-latest-versions --aws-region $LAMBDA_FUNCTION_REGION

This prints the ARN of the latest Upwind Lambda Tracer layer for amd64 (X86_64) and arm64. The ARNs are region-specific; use the same AWS Region as the function, and pick the line that matches the function's architecture . You can pass --aws-region explicitly if needed.

Add the layer to the Lambda function configuration by ARN.
Set the AWS_LAMBDA_EXEC_WRAPPER environment variable to the lambda function configuration:

AWS_LAMBDA_EXEC_WRAPPER=/opt/bin/upwind-tracer.sh

If the Lambda function communicates directly with the Upwind platform (not through an Upwind Cluster Manager), set UPWIND_AUTH_SECRETS_MANAGER_SECRET to the name or ARN of the secret that holds Upwind client credentials, and ensure the execution role can call secretsmanager:GetSecretValue on that secret. See Upwind client credentials.

UPWIND_AUTH_SECRETS_MANAGER_SECRET=<SECRET_NAME_OR_ARN>

If the function should use a specific Upwind region (for example eu or me), set UPWIND_REGION accordingly.

UPWIND_REGION=eu

Terraform example (.zip archive Lambda)

# --------------------------------------------------------------------------
# IAM role
# --------------------------------------------------------------------------
data "aws_iam_policy_document" "lambda_assume" {
  statement {
    actions = ["sts:AssumeRole"]
    principals {
      type        = "Service"
      identifiers = ["lambda.amazonaws.com"]
    }
  }
}

resource "aws_iam_role" "lambda" {
  name               = "my-lambda-function-role"
  assume_role_policy = data.aws_iam_policy_document.lambda_assume.json
}

resource "aws_iam_role_policy_attachment" "lambda_basic_execution" {
  role       = aws_iam_role.lambda.name
  policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
}

# Grant Lambda function permission to read the Upwind client credentials secret
# Required if the Lambda function should report directly to the Upwind platform, and not via the Upwind Cluster Manager
data "aws_iam_policy_document" "upwind_credentials_secret_read" {
  statement {
    actions   = ["secretsmanager:GetSecretValue"]
    resources = ["<SECRET_ARN>"]
  }
}

resource "aws_iam_role_policy" "upwind_credentials_secret_read" {
  name   = "my-lambda-function-upwind-lambda-credentials-read"
  role   = aws_iam_role.lambda.id
  policy = data.aws_iam_policy_document.upwind_credentials_secret_read.json
}

# --------------------------------------------------------------------------
# Lambda function
# --------------------------------------------------------------------------
data "archive_file" "lambda_zip" {
  type        = "zip"
  source_file = "${path.module}/main.py"
  output_path = "${path.module}/lambda.zip"
}

resource "aws_lambda_function" "this" {
  function_name    = "my-lambda-function"
  role             = aws_iam_role.lambda.arn
  handler          = "main.handler"
  runtime          = "python3.12"
  timeout          = 60
  memory_size      = 256
  filename         = data.archive_file.lambda_zip.output_path
  source_code_hash = data.archive_file.lambda_zip.output_base64sha256
  architectures    = ["arm64"]

  # Add Upwind Lambda Tracer layer by ARN
  # You can run 'upwindctl lambda show-latest-versions --aws-region $LAMBDA_FUNCTION_REGION' to find the latest ARN
  layers = ["arn:aws:lambda:us-east-1:693339160499:layer:upwind-lambda-tracer-arm64:8"]

  environment {
    variables = {
      # AWS_LAMBDA_EXEC_WRAPPER enables the Upwind Lambda Tracer extension
      AWS_LAMBDA_EXEC_WRAPPER = "/opt/bin/upwind-tracer.sh"

      # UPWIND_AUTH_SECRETS_MANAGER_SECRET should be the name or ARN of the AWS Secrets Manager Secret containing Upwind client credentials
      # Required if the Lambda function should report directly to the Upwind platform, and not via the Upwind Cluster Manager
      UPWIND_AUTH_SECRETS_MANAGER_SECRET = "<SECRET_NAME_OR_ARN>"

      # UPWIND_REPORT_TO_CLUSTER_MANAGER set to false configures the Lambda function to report directly to the Upwind platform
      UPWIND_REPORT_TO_CLUSTER_MANAGER = "false"
    }
  }

  logging_config {
    log_format = "JSON"
  }
}

To instrument a containerized Lambda function you need to build the Upwind Lambda Tracer extension into the function's container image.

Add the Upwind Lambda Tracer to your Dockerfile

This involves (a) adding a build stage, (b) copying the Upwind Lambda Tracer extension into the image, and (c) setting the AWS_LAMBDA_EXEC_WRAPPER environment variable.

Below is a simple example of a Dockerfile with these additions:

# (a) Add the Upwind Lambda Tracer image as a build stage
FROM public.ecr.aws/upwindsecurity/images/lambda-tracer:0.4.3 AS upwind-lambda-tracer

# Use an official AWS Lambda base image, in this example Python 3.12
FROM public.ecr.aws/lambda/python:3.12

# (b) Copy Upwind Lambda Tracer extension into image
COPY --from=upwind-lambda-tracer /opt /opt

# (c) Set the AWS_LAMBDA_EXEC_WRAPPER environment variable
# This can be done directly in the Dockerfile or via function configuration
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/bin/upwind-tracer.sh

# The rest is regular Lambda function image build
COPY requirements.txt ${LAMBDA_TASK_ROOT}/
RUN pip install -r requirements.txt
COPY app.py ${LAMBDA_TASK_ROOT}/
CMD [ "app.handler" ]

Supplying Upwind client credentials

For Lambda functions that communicate directly with the Upwind platform instead of reporting through an Upwind Cluster Manager, make sure that:

Upwind client credentials are present in an AWS Secrets Manager secret
The function's execution role has permission to read the secret

See Upwind client credentials for more details.

You must then set the UPWIND_AUTH_SECRETS_MANAGER_SECRET environment variable in the Lambda function configuration:

UPWIND_AUTH_SECRETS_MANAGER_SECRET=<SECRET_NAME_OR_ARN>

If the Lambda function should communicate with a specific Upwind region (e.g. eu or me), set the UPWIND_REGION environment variable in the Lambda function configuration, e.g.:

UPWIND_REGION=eu

Build and push image

After creating the image it needs to be pushed to a container registry like Amazon ECR.

Update Lambda function configuration

Update the Lambda function configuration to use the new container image.

Code signing

If your AWS Lambda functions are configured to use code signing you must add the Upwind signing profile ARN to the functions' code signing configurations .

The Upwind signing profile ARN is as follows:

arn:aws:signer:us-east-1:693339160499:/signing-profiles/upwindLambda/W1E0q9YiIE

Prerequisites​

Supported runtimes​

Upwind client credentials​

Storing Upwind client credentials in AWS Secrets Manager​

Granting Lambda functions permission to read AWS Secrets Manager secret​

Instrumentation​

Instrument with upwindctl​

Install upwindctl​

Instrument Lambda functions​

Uninstrument Lambda functions​

Manual instrumentation for Terraform and IaC​

Add the Upwind Lambda Tracer to your Dockerfile​

Supplying Upwind client credentials​

Build and push image​

Update Lambda function configuration​

Code signing​

Prerequisites

Supported runtimes

Upwind client credentials

Storing Upwind client credentials in AWS Secrets Manager

Granting Lambda functions permission to read AWS Secrets Manager secret

Instrumentation

Instrument with `upwindctl`

Install `upwindctl`

Instrument Lambda functions

Uninstrument Lambda functions

Manual instrumentation for Terraform and IaC

Add the Upwind Lambda Tracer to your Dockerfile

Supplying Upwind client credentials

Build and push image

Update Lambda function configuration

Code signing