S3

Overview

The Upwind S3 integration writes a copy of the reports the Upwind Cluster Manager sends to Upwind into an S3 bucket you own, so you can retain, query, or forward the raw data with your own tooling.

The integration is a Cluster Manager feature and is not specific to any cloud provider, so it works from any cluster the Cluster Manager runs in and it can target either AWS S3 or an S3-compatible object storage service.

How It Works

As the Cluster Manager reports to Upwind, it also gzips each report you have configured to copy and writes it to your bucket as a JSON object. Uploads are additive: enabling this integration does not change what the Cluster Manager sends to Upwind, and you choose which report types are copied.

Each Cluster Manager pod uploads independently, and each pod verifies that the bucket exists and is reachable when it starts. If that check fails - a wrong bucket name, missing permissions, or credentials the AWS SDK cannot resolve - the pod logs the reason and uploads stay disabled until it restarts.

Objects are written to keys of the following format:

<PREFIX>/<REPORT TYPE>/<CLUSTER>/YYYY/MM/DD/HH/MM/<NODE ID>.<UNIQUE SUFFIX>.json.gz

Segment	Description
<PREFIX>	Value of --s3-upload-prefix. Omitted when the flag is not set.
<REPORT TYPE>	The report type of the object, as configured in --s3-bucket-reports.
<CLUSTER>	Cluster name, or the cluster ID when no name is available.
YYYY/MM/DD/HH/MM	Time at which the Cluster Manager wrote the object.
<NODE ID>	ID of the node the report came from.
<UNIQUE SUFFIX>	Random string that keeps each key unique.

note

Copying reports to S3 adds work to the Cluster Manager. It marshals, gzips, and uploads a second copy of every report you configure, so enabling this integration raises Cluster Manager CPU and memory usage and adds outbound traffic from the cluster to the bucket, which your cloud provider may bill as data transfer. The increase scales with the number of report types you enable and with the volume of reports your cluster produces, so review the Cluster Manager resource requests and limits before enabling this on a large cluster.

Prerequisites

• A bucket on AWS S3, or on a service that exposes an S3-compatible API.
• Network egress from the Cluster Manager pods to the bucket endpoint.
• Credentials the Cluster Manager can use to write to the bucket. The Cluster Manager resolves credentials through the standard AWS SDK credential chain, and Step 1 covers the mechanisms for each environment.

Integration

The examples below set values on the upwind-operator chart, which passes everything under clusterAgent.values through to the Cluster Manager. If you install the upwind-cluster-agent chart directly, use the same keys without the clusterAgent.values prefix.

Step 1. Grant the Cluster Manager Access to the Bucket

Whichever mechanism you use, the identity writing to the bucket needs to list the bucket and put objects into it. The bucket listing permission is what the Cluster Manager uses to verify access at startup.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "s3:ListBucket",
            "Resource": "arn:aws:s3:::BUCKETNAME"
        },
        {
            "Effect": "Allow",
            "Action": "s3:PutObject",
            "Resource": "arn:aws:s3:::BUCKETNAME/*"
        }
    ]
}

Select the tab that matches where your cluster runs.

Amazon EKS

On EKS, Upwind recommends EKS Pod Identities.

Follow the AWS EKS Pod Identities Guide .

note

Use the Kubernetes service account upwind-cluster-agent in the upwind namespace, and associate it with an IAM role carrying the policy above.

As an alternative, you can use IAM roles for service accounts and annotate the Cluster Manager service account with the role to assume:

clusterAgent:
  values:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNTID:role/ROLENAME

Self-hosted OIDC

Outside EKS, the Cluster Manager can assume an AWS IAM role with its Kubernetes service account token through sts:AssumeRoleWithWebIdentity. This keeps static credentials out of the cluster, and AWS never needs inbound access to your API server.

Prepare the cluster and AWS account first:

• Configure the API server with a stable, publicly resolvable HTTPS issuer (--service-account-issuer) and service account signing keys, so the TokenRequest API can issue tokens for that issuer.

• Publish the cluster's discovery documents, /.well-known/openid-configuration and /openid/v1/jwks, at the issuer URL over HTTPS on port 443 with a publicly trusted certificate. Only public keys leave the cluster: AWS STS reads the discovery documents from that URL and never connects to the API server. See Service account issuer discovery .

• Create an IAM OIDC identity provider for the issuer URL with the audience sts.amazonaws.com. See Create an OpenID Connect identity provider in IAM .

• Create an IAM role that carries the policy above and trusts the Cluster Manager service account:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Federated": "arn:aws:iam::ACCOUNTID:oidc-provider/ISSUERHOST/ISSUERPATH"
              },
              "Action": "sts:AssumeRoleWithWebIdentity",
              "Condition": {
                  "StringEquals": {
                      "ISSUERHOST/ISSUERPATH:sub": "system:serviceaccount:upwind:upwind-cluster-agent",
                      "ISSUERHOST/ISSUERPATH:aud": "sts.amazonaws.com"
                  }
              }
          }
      ]
  }

The Cluster Manager pod then needs a service account token issued for the sts.amazonaws.com audience, and the AWS environment variables that point at it. You can either let a webhook inject both, or project the token yourself.

Option A: inject with a pod identity webhook

Deploy amazon-eks-pod-identity-webhook in the cluster and annotate the Cluster Manager service account. The webhook projects the token and sets AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE on the pod, so the Upwind configuration is the same annotation used on EKS:

clusterAgent:
  values:
    serviceAccount:
      annotations:
        eks.amazonaws.com/role-arn: arn:aws:iam::ACCOUNTID:role/ROLENAME

Option B: project the token directly

Mount a projected service account token on the Cluster Manager and point the AWS environment variables at it:

clusterAgent:
  values:
    extraVolumes:
      - name: aws-sts-token
        projected:
          sources:
            - serviceAccountToken:
                audience: sts.amazonaws.com
                expirationSeconds: 3600
                path: token
    extraVolumeMounts:
      - name: aws-sts-token
        mountPath: /var/run/secrets/aws
        readOnly: true
    env:
      - name: AWS_ROLE_ARN
        value: arn:aws:iam::ACCOUNTID:role/ROLENAME
      - name: AWS_WEB_IDENTITY_TOKEN_FILE
        value: /var/run/secrets/aws/token
      - name: AWS_REGION
        value: BUCKETREGION

Access keys

S3-compatible services such as MinIO and Ceph validate requests against their own access key pairs rather than against AWS STS, so access keys are the only option when the bucket is not on AWS S3. They also work for AWS S3 in clusters that cannot run either of the other mechanisms, though short-lived credentials are preferable there.

Store the key pair in a Kubernetes Secret:

kubectl create secret generic --namespace upwind upwind-s3-credentials \
        --from-literal=accessKeyId=$ACCESS_KEY_ID \
        --from-literal=secretAccessKey=$SECRET_ACCESS_KEY

Then reference it from the Cluster Manager environment:

clusterAgent:
  values:
    env:
      - name: AWS_ACCESS_KEY_ID
        valueFrom:
          secretKeyRef:
            name: upwind-s3-credentials
            key: accessKeyId
      - name: AWS_SECRET_ACCESS_KEY
        valueFrom:
          secretKeyRef:
            name: upwind-s3-credentials
            key: secretAccessKey
      - name: AWS_REGION
        value: BUCKETREGION

Set AWS_REGION to the region of the bucket when the bucket is on AWS S3.

To keep the credentials out of the pod specification, store them in an AWS credentials file instead:

[default]
aws_access_key_id = ACCESSKEYID
aws_secret_access_key = SECRETACCESSKEY

kubectl create secret generic --namespace upwind upwind-s3-credentials-file \
        --from-file=credentials=./credentials

Mount that Secret on the Cluster Manager and point the AWS SDK at the file:

clusterAgent:
  values:
    extraVolumes:
      - name: aws-credentials
        secret:
          secretName: upwind-s3-credentials-file
    extraVolumeMounts:
      - name: aws-credentials
        mountPath: /var/run/secrets/aws
        readOnly: true
    env:
      - name: AWS_SHARED_CREDENTIALS_FILE
        value: /var/run/secrets/aws/credentials

Step 2. Configure the Bucket and Reports

Set the bucket, the report types to copy, and an optional key prefix on the Cluster Manager:

clusterAgent:
  values:
    extraArgs:
      - --s3-bucket-name=BUCKETNAME
      - --s3-bucket-reports=process,process_events,aggregated_network
      - --s3-upload-prefix=PREFIX

Flag	Required	Description
--s3-bucket-name	✅	Name of the bucket reports are uploaded to.
--s3-bucket-reports	✅	Comma-separated list of report types to upload. See Report Types.
--s3-upload-prefix		Prefix for every object key the Cluster Manager writes.
--s3-compatible-endpoint		Base endpoint of an S3-compatible service. Omit this flag when the bucket is on AWS S3.

If either the bucket name or the report list is missing, the Cluster Manager logs that S3 upload is being skipped and continues reporting to Upwind only.

For a bucket on an S3-compatible service, add the endpoint as well. The Cluster Manager addresses the bucket in path style when this flag is set.

clusterAgent:
  values:
    extraArgs:
      - --s3-bucket-name=BUCKETNAME
      - --s3-bucket-reports=process,process_events,aggregated_network
      - --s3-compatible-endpoint=https://minio.example.com

note

--s3-compatible-endpoint requires Cluster Manager v0.60.0 or later.

Step 3. Verify the Uploads

Check the Cluster Manager logs for upload activity:

kubectl logs -n upwind -l app.kubernetes.io/name=upwind-cluster-agent | grep -i s3

A working configuration logs Uploading report to S3 bucket with the bucket, prefix, and report type. Objects appear in the bucket under the keys described in How It Works.

If you scrape Cluster Manager metrics, the cluster_agent_uploader_outbound_s3_bytes_total counter reports how many bytes have been written to the bucket, broken down by report_type. Use it to measure the added egress before enabling more report types.

Report Types

Configure any of the following report types in --s3-bucket-reports. The Cluster Manager only writes objects for report types it actually produces, which depends on the features enabled in your installation.

Report type	Contents
process	Process lifecycle events (start/exit).
process_events	Syscall-level process events.
process_analysis	Results of process analysis.
running_processes	Snapshot of running processes.
aggregated_network	Network flows aggregated across the cluster by the Cluster Manager.
posture_k8s	Kubernetes posture findings for the cluster. EKS clusters report these as posture.
image_scan, layer_scan, container_scan, host_scan, node_scan	Vulnerability and package scan results. The type depends on what was scanned.
secrets	Secret scan results.
apisec_catalog	API catalog entries discovered by API security.
function_events	In-use workload events.

Troubleshooting

Log message	Cause
no S3 bucket specified, skipping S3 upload	--s3-bucket-name is not set.
S3 bucket configured but no reports configured	--s3-bucket-reports is empty.
unable to create an S3 client	The AWS SDK could not build a client. Check the endpoint value when using an S3-compatible service.
unable to access S3 bucket	The bucket does not exist, the identity lacks s3:ListBucket, no credentials could be resolved, or no region is set.

If reports are not appearing for a report type you configured, confirm the name matches the Report Types table exactly. An unrecognized name is accepted without an error and simply never matches a report.

Restart the Cluster Manager after fixing credentials or bucket permissions. The bucket access check runs only at startup, so a pod that failed the check does not retry on its own.

For further assistance, contact support@upwind.io.