Scan Jobs

The scan jobs model allows you to reduce the scanner memory footprint because the scanner consumes resources only when it needs to perform a scan, unlike the daemonset model where scanner sidecar is reserved resources throughout its lifecycle. The scan jobs are managed by the Cluster Manager which initiates scans when pods with fresh images are encountered in the cluster, and to refresh existing SBOM findings, typically every 24 hrs.

Enabling Scan Jobs

To enable the Scan Jobs deployment model for the Scanner, upgrade the Upwind Operator chart to disable the scanner as a sidecar container in the Sensor Daemonset and activate management of the jobs from the Cluster Manager. Ensure you have updated to the latest upwind-operator chart before enabling Scan Jobs.

agent:
  values:
    scanAgent:
      enabled: false
clusterAgent:
  values:
    scanJob:
      enabled: true

Refer to the Configuration Library for additional configuration.

Tuning Scan Jobs

The Cluster Manager decides when and where scan jobs run. Tune it with the following options, set as environment variables at clusterAgent.values.env or as args at clusterAgent.values.extraArgs.

Environment Variable	Arg	Description	Default
UPWIND_IMAGE_SCAN_JOBS_THRESHOLD	--image-scan-jobs-threshold	Scan jobs will not be scheduled if the number of active jobs exceeds this limit.	5
UPWIND_IMAGE_SCAN_PENDING_JOBS_THRESHOLD	--image-scan-pending-jobs-threshold	Scan jobs will not be scheduled if the number of pending jobs exceeds this limit, even if active jobs do not exceed image-scan-jobs-threshold.	5
UPWIND_RUN_SCANNER_WITH_TOLERATIONS	--run-scanner-with-tolerations	Copy the taints of the nodes that have the image into the scan job's tolerations. See Tolerations.	true
UPWIND_IMAGE_SCAN_JOB_TAINT_EXCEPTIONS	--image-scan-job-taint-exceptions	Additional node taints that stop a node's taints from being copied into scan job tolerations, e.g. key=value:NoSchedule. See Tolerations.
UPWIND_SCANNER_MEM_INCREASE	--scanner-mem-increase	Scan job memory increase factor for retries in case of OOMKill.	1.5
UPWIND_SCANNER_MEM_LIMIT	--scanner-mem-limit	Ceiling for scan job memory on retries, as a percentage of the memory capacity of the largest node in the cluster.	75
UPWIND_SCAN_JOB_DELETE_GRACE_PERIOD	--scan-job-delete-grace-period	How long a scan job may stay pending before it is deleted and resubmitted.	15m
UPWIND_SCAN_REPROCESSING_INTERVAL	--scan-reprocessing-interval	Interval to reprocess all pods for scanning, if not scanned already.	4h

Durations such as --scan-reprocessing-interval and --scan-job-delete-grace-period take a unit, e.g. 30m or 4h. A plain number fails to parse as an arg, and as an environment variable it is read as nanoseconds.

clusterAgent:
  values:
    extraArgs:
      - --image-scan-jobs-threshold=10
      - --scan-reprocessing-interval=8h

Cluster Manager and Scanner Settings

The scan jobs run a different container from the Cluster Manager, and the chart has separate settings for each:

Value	Applies to
clusterAgent.values.extraArgs	The Cluster Manager only. Scan job tuning args from the table above go here.
clusterAgent.values.scanJob.scanner.extraArgs	The scan job containers only.
clusterAgent.values.scanJob.scanner.env	The scan job containers only.
clusterAgent.values.env	The Cluster Manager and the scan job containers.

The same applies to placement: the top-level clusterAgent.values.nodeSelector, affinity and tolerations place the Cluster Manager pod, not the scan jobs. Scan jobs are placed with the scanJob.scanner.* values described below.

Node Placement

Each scan job prefers a node that already has the image, so it can be scanned without pulling it. On top of that you can restrict where scan jobs run with these values:

• scanJob.scanner.nodeSelector: available from cluster manager chart 0.24.0. Applied together with the default affinity, so you don't need to restate the os/arch terms.
• scanJob.scanner.affinity: defaults to requiring a Linux node on amd64 or arm64. Helm replaces lists rather than merging them, so a custom affinity must include these terms again.
• scanJob.scanner.tolerations: static tolerations added to every scan job, in addition to the ones copied from nodes (see Tolerations).

clusterAgent:
  values:
    scanJob:
      enabled: true
      scanner:
        nodeSelector:
          workload-type: general
        affinity:
          nodeAffinity:
            requiredDuringSchedulingIgnoredDuringExecution:
              nodeSelectorTerms:
                - matchExpressions:
                    - key: kubernetes.io/os
                      operator: In
                      values:
                        - linux
                    - key: kubernetes.io/arch
                      operator: In
                      values:
                        - amd64
                        - arm64
                    - key: topology.kubernetes.io/zone
                      operator: In
                      values:
                        - us-east-1a

From chart 0.24.0, scan jobs do not run on control-plane nodes (node-role.kubernetes.io/control-plane or node-role.kubernetes.io/master), including with a custom affinity. On clusters where every node is a control-plane node, such as single-node clusters, set scanJob.scanner.runOnControlPlane: true or no scan job can be scheduled.

Tolerations

By default (--run-scanner-with-tolerations=true) the Cluster Manager copies the taints of the nodes that have the image into the scan job's tolerations, so images on tainted nodes can be scanned where they already are.

A node's taints are not copied if it has any of the taint exceptions. The exceptions always include the built-in node-condition taints, such as node.kubernetes.io/not-ready, node.kubernetes.io/unreachable, node.kubernetes.io/memory-pressure and node.kubernetes.io/disk-pressure. --image-scan-job-taint-exceptions adds to this list, it does not replace it. Exceptions are written as key[=value]:Effect but match on key and effect only, the value is ignored.

For example, to keep scan jobs off nodes tainted for GPU workloads:

clusterAgent:
  values:
    extraArgs:
      - --image-scan-job-taint-exceptions=nvidia.com/gpu:NoSchedule

To stop copying taints altogether, set --run-scanner-with-tolerations=false. Scan jobs then only get the tolerations in scanJob.scanner.tolerations, and images that only exist on tainted nodes are scanned elsewhere.

Image Pulls

When a scan job runs on a node that doesn't have the image, the scanner has to pull it. This happens when the nodes that have the image are excluded by nodeSelector, affinity, a taint exception or turning off tolerations, or when those nodes are full. Pulls from public registries work, but pulls from private registries fail if the node doesn't have credentials for them. If you restrict scan job placement, make sure the allowed nodes can pull the images you want scanned.

Resources and Retries

The chart sets the following defaults for scan jobs:

Value	Default
scanJob.scanner.resources.requests.cpu	50m
scanJob.scanner.resources.limits.cpu	500m
scanJob.scanner.resources.requests.memory	50Mi
scanJob.scanner.resources.limits.memory	500Mi
scanJob.scanner.activeDeadlineSeconds	3600
backoffLimit (fixed)	0

Lowering the CPU limit makes scans slower. A scan that runs past activeDeadlineSeconds fails with DeadlineExceeded and is not retried, so raise the deadline if you lower CPU.

Only two kinds of scan jobs are retried:

• OOMKilled jobs are resubmitted with more memory. The new request is the previous limit, and the new limit is the previous limit multiplied by --scanner-mem-increase, e.g. 500Mi, 750Mi, 1125Mi and so on. Retries stop once the next limit would exceed --scanner-mem-limit percent of the largest node's memory capacity. The ceiling is based on the largest node in the cluster even if nodeSelector or affinity only allows smaller nodes, in which case a retried job can stay pending.
• Pending jobs that are still pending after --scan-job-delete-grace-period are deleted and resubmitted, up to 4 times.

All other failures are not retried. Pending jobs also count towards --image-scan-pending-jobs-threshold, so jobs that can't be scheduled hold back new scans until they are cleaned up.