Skip to main content

Troubleshooting

This topic provides information to help troubleshoot issues with your Upwind cluster components installation.

note

The commands on this page assume the Upwind Operator Helm release is named upwind-operator. To check the name of your release, run helm list -n upwind, and substitute it in the commands if it differs.

Update Client Credentials​

To update the upwind-operator chart with a new client credentials, generate the new client ID and client secret in the console. Then update them the same way you provided them at install:

Replace the Secret and restart the operator so it picks up the new credentials:

 export UPWIND_CLIENT_ID="<redacted>"
 export UPWIND_CLIENT_SECRET="<redacted>"

kubectl create secret generic upwind-secret -n upwind \
  --from-literal=clientId="$UPWIND_CLIENT_ID" \
  --from-literal=clientSecret="$UPWIND_CLIENT_SECRET" \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n upwind rollout restart deployment/upwind-operator

The upwind-operator pod is not running or crashing​

Check the state of the upwind-operator helm chart​

Run the following command from your terminal:

helm status upwind-operator \
  --namespace upwind

Confirm that the upwind-operator chart STATUS is "deployed".

Check the logs of the upwind-operator pod​

Inspect logs from the upwind-operator pod.

kubectl -n upwind logs --selector=app.kubernetes.io/name=upwind-operator | grep "ERROR"

You may encounter common issues with the operator such as:

  1. Errors related to token fetching
  2. Errors related to the cloud metadata service (IMDSv2)

The Kubernetes cluster is not appearing in the Upwind Runtime Map​

Confirm that the Sensor and Clusteragent custom resources have been installed​

kubectl -n upwind get agent upwind
kubectl -n upwind get clusteragent upwind

The status of both custom resources should be "Installed".

Confirm that the Upwind Sensor DaemonSet is running​

Check the status of the upwind-agent DaemonSet and the pods within the DaemonSet.

kubectl -n upwind get daemonset upwind-agent
kubectl -n upwind get pods --selector=app.kubernetes.io/name=upwind-agent
kubectl get nodes

Confirm that all pods in the DaemonSet are running and that the number of pods matches the number of nodes in the cluster, and that all containers in the pods are healthy.

Confirm that the Upwind Cluster Manager is running​

Check the status of the upwind-cluster-agent deployment and the pod in the deployment.

kubectl -n upwind get deployment upwind-cluster-agent
kubectl -n upwind get pods --selector=app.kubernetes.io/name=upwind-cluster-agent

The Sensor Cannot Access debugfs​

The Sensor reads kernel trace events from debugfs (/sys/kernel/debug) or tracefs (/sys/kernel/tracing) to attach its eBPF programs. The upwind-agent Helm chart mounts both paths from the node.

Some operating systems, such as Talos Linux, do not mount debugfs and expose only tracefs. upwind-agent chart versions earlier than 0.25.2 mount only debugfs, so the Sensor cannot attach its eBPF programs on these nodes. Upgrade to a chart version that mounts tracefs:

  • Helm Operator or Terraform: if agent.version is not set, the Operator upgrades the upwind-agent chart automatically and no action is needed. If you pinned agent.version, set it to 0.25.2 or later.
  • Helm Operatorless: upgrade the upwind-agent release to chart version 0.25.2 or later.
  • Helm Umbrella Chart: upgrade the upwind release to chart version 0.7.26 or later.

If you previously added a tracefs volume through extraVolumes, remove it. upwind-agent chart 0.25.2 and later already define a tracefs volume, and Kubernetes rejects the Sensor DaemonSet when the volume name and mount path are duplicated.

  1. Save your current values. Replace <RELEASE_NAME> with the release from the table below:

    helm get values <RELEASE_NAME> -n upwind -o yaml > values.yaml

  2. In values.yaml, remove only the entries named tracefs from volumes and sensorMounts under the key for your install method. Keep any other extra volumes.

    Install method<RELEASE_NAME><CHART_NAME>Key
    Helm Operator or Terraformupwind-operatorupwind-operatoragent.values.extraVolumes
    Helm Operatorlessupwind-agentupwind-agentextraVolumes
    Helm Umbrella Chartupwindupwindupwind-operator.agent.values.extraVolumes

  3. Re-apply your values:

    helm upgrade <RELEASE_NAME> upwind/<CHART_NAME> \
      --namespace upwind \
      --values values.yaml

    values.yaml can include your client credentials, so delete it when you are done.

If you install with Terraform, remove the tracefs entries from the Helm values in your Terraform configuration and run terraform apply instead, so Terraform does not add them back.

Upgrade​

The Upwind Operator will occasionally need to be upgraded to take advantage of the latest features of the Upwind platform.

helm repo update upwind
helm upgrade upwind-operator upwind/upwind-operator \
  --namespace upwind \
  --reset-then-reuse-values

Reinstall Troubleshooting​

If you run into issues when reinstalling Upwind's Sensor after an initial uninstall, there are several things you can do to troubleshoot.

In general, Upwind's Sensor uninstall steps should be safe to follow from any cluster state, and if the commands run successfully or return errors such as Error: uninstall: Release not loaded: upwind-operator: release: not found for the helm uninstall or Error from server (NotFound): agents.components.upwind.io "upwind" not found when deleting the custom resources you should be able to reinstall without any issues.

If you have completed these steps and are still having trouble reinstalling the Upwind Sensor, you may have failed to delete custom resources before uninstalling. Please make sure to delete any custom resources and then proceed to the reinstall steps.

Uninstall​

In the event that you need to uninstall the Upwind cluster components from a cluster, it is recommended to first delete the custom resources (CRs) created by Upwind and then uninstall the helm chart to avoid any issues with Kubernetes finalizers and CRs not being removed.

  1. Delete the custom resources.
kubectl -n upwind delete agent upwind
kubectl -n upwind delete clusteragent upwind
  1. Confirm the custom resources have been deleted. Both of these commands should not return any resources.
kubectl -n upwind get agents
kubectl -n upwind get clusteragents
  1. Uninstall the upwind-operator helm chart.
helm uninstall upwind-operator \
  --namespace upwind
  1. Delete the upwind namespace.
kubectl delete namespace upwind

If you experience any issues, please contact us at our 24/7 console chat support for a quick resolution. Open a Chat .

Firewall Configuration​

If you are deploying the Sensor, Cluster Manager, and/or Operator to a cluster or host with a firewall that restricts outbound connections to the public internet, ensure that firewall rules are configured to allow HTTPS communication on port 443. These rules should permit access to the hosts specified in the Network access section.