Troubleshooting

Overview

If you are experiencing issues with the Upwind Sensor, follow the steps below to troubleshoot:

Check the Service Status

1. Check the status of the Upwind Sensor service by running the following command:

systemctl status upwind-agent.service

This will display information about the service, including whether it is currently running or not.

2. If the service is not running, you can start it by running the following command:

systemctl start upwind-agent.service

If the service fails to start, check the logs (as described below) for error messages that may indicate the cause of the issue.

Check the Logs

1. To view the Upwind Sensor logs, run the following command:

journalctl -u upwind-agent.service

This will display the logs for the Upwind Sensor service, including any error messages that may be related to the issue you are experiencing.

2. If you need to view only the most recent logs, you can run the following command:

journalctl -u upwind-agent.service -n 100

This will display the 100 most recent log entries.

If the sensor has been configured to log to a file, see Logging to a File, the logs are in that file rather than the journal.

Update Client Credentials

1. To update the client credentials, edit the /etc/upwind/agent.yaml file and modify the auth-client-id and auth-client-secret variables:

[...]
auth-client-id: <CLIENT_ID>
auth-client-secret: <CLIENT_SECRET>
[...]

Then, restart the appropriate Upwind Sensor services by running:

systemctl restart upwind-agent-hostconfig.service

and

systemctl restart upwind-agent.service

Check Firewall and Networking

The Sensor, Scanner, hostconfig, and auto-update services connect to Upwind only over outbound HTTPS on port 443. No inbound traffic to the host is needed.

If the host is behind a firewall, proxy, or security group that restricts outbound traffic, allow access to the Global domains and the domains for your region listed in Network access. If you use a proxy, refer to Proxy Configuration.

High Disk IO or CPU Usage

Sustained disk IOPS, high iowait, or steady CPU usage on a host is most likely caused by the Scanner, not the Sensor. Scans are expected to use resources and can take several hours on large hosts.

1. Check whether a scan is running:

systemctl status upwind-agent-scanner.service

2. If the Scanner is the cause, you can:

• Lower its resource limits with the UPWIND_SCANNER_* variables. The UPWIND_AGENT_* variables do not affect the Scanner. Refer to Resource Limits.
• Exclude data directories that don't need scanning. Refer to Choosing What to Exclude.
• Disable the Scanner and cover the host with Cloud Scanner instead. Refer to When to Use Cloud Scanner Instead.

Contact Support

If you are still having issues with the Upwind Sensor, you can contact Upwind support via email at support@upwind.io or through the Chat button in the Upwind Management Console.

Provide as much detail as possible about the issue you are experiencing, including any relevant logs or error messages. Upwind support will work with you to diagnose and resolve the issue as quickly as possible.

Uninstalling the Upwind Sensor

If you need to uninstall the Upwind Sensor for any reason, use the method that matches how you installed it. The /etc/upwind configuration directory contains your client secret, so remove it too unless you plan to reinstall the Sensor with the same configuration.

Automatic Installer

Run the following command to uninstall the Upwind Sensor:

curl -fsSL https://get.upwind.io/sensor.sh | UPWIND_UNINSTALL_PURGE=true bash -s uninstall

This command removes the Upwind Sensor services and binaries. UPWIND_UNINSTALL_PURGE=true also deletes /etc/upwind. Without it, the uninstall leaves /etc/upwind on the host.

Debian Package

sudo systemctl disable --now upwind-agent-hostconfig.service
sudo apt purge upwind-agent
sudo rm -rf /etc/upwind

The package does not stop the hostconfig service when it is removed, so stop it first. apt purge removes /etc/upwind/agent.yaml, while apt remove keeps it. Deleting /etc/upwind removes the files the package did not install, such as agent-hostconfig.yaml.

RPM Package

sudo systemctl disable --now upwind-agent-hostconfig.service
sudo dnf remove upwind-agent

On hosts without dnf, use sudo yum remove upwind-agent. The package does not stop the hostconfig service when it is removed, so stop it first. Removing the package deletes /etc/upwind.

If you encounter any issues during the uninstallation process, please contact Upwind support for assistance.

Disabling File Monitoring

On hosts with a high volume of file IO operations (e.g. database servers, log forwarders, CI servers, etc.), file monitoring can have performance impacts. The file monitoring features of the sensor can be disabled by setting flags on the sensor. Set the following flags to false following the instructions in Configuring the Upwind Sensor.

Flag	Description	Default
file-permission-events	Enable the security_file_permission kprobe	"true"
path-truncate-events	Enable the security_path_truncate kprobe	"true"
file-truncate-events	Enable the security_file_truncate kprobe (supported from kernel 6.2)	"true"

There is no fixed threshold for what counts as a high volume of file IO. File monitoring adds a small amount of latency to each file operation, and whether that affects a workload depends on how sensitive it is to IO latency. Watch for workloads that are IO bound or that constantly read, write, and truncate files - for example, message brokers, databases, object stores, and log-heavy services. If you see performance impacts on these hosts after installing the sensor, disable file monitoring as described above.

Kafka Brokers

Kafka brokers continuously write to and truncate log segment files, and generate a high volume of socket activity. For dedicated Kafka brokers, add the following to /etc/upwind/agent.yaml:

file-permission-events: 0
path-truncate-events: 0
file-truncate-events: 0
exclusions: "**/var/lib/kafka/** **/tmp/kafka-logs/**"

The exclusions flag excludes the Kafka data directories from scanner path scans. Replace the paths with the data directories configured in your broker's log.dirs setting. For details, refer to Scanner Exclusions.

If you continue to see performance impacts, you can also disable the socket probes:

bpf-sock-sendmsg-probe: 0
bpf-sock6-sendmsg-probe: 0
bpf-sock-recvmsg-probe: 0
bpf-sock6-recvmsg-probe: 0

warning

Kafka brokers are often IO bound, and the scanner performs a large amount of IO. Even with the Kafka data directories excluded and the scanner running at a reduced IO priority, scans can still affect broker performance. Consider this before running the scanner on Kafka brokers. You can adjust the scanner's IO priority with UPWIND_SCANNER_IOWEIGHT, or disable periodic scanning with UPWIND_SCANNER_ENABLED - refer to Configuring the Upwind Sensor.