Troubleshooting

Overview

This guide helps you resolve common issues you might encounter while using Upwind with Azure. Below are various sections addressing specific problems, with step-by-step instructions to guide you through resolving them.

Common Issues

Insufficient Spot VM Quota

If you encounter issues with Spot VM deployment:

Check your current quota:

az vm list-usage --location <region> --query "[?name.value=='lowPriorityCores']" --output table

Request quota increase through Azure Portal if needed:
- Go to Azure Portal → Subscriptions → Select your subscription
- Click Settings and Usage + quotas
- Find Spot vCPUs and request an increase
Consider using a different region if quota increase is not immediately possible

Authentication Issues

If you encounter authentication problems:

Verify Azure CLI authentication:
```
az login
az account list
```
Ensure you have Owner privileges in the Azure tenant
Check if the service principal has all required permissions

Root Tenant Management Group Access Issues

If you receive errors about the root tenant management group not being found or accessible:

Verify Elevated Access Status

Check if elevated access is active:
- Ensure you have successfully elevated access (Global Administrator) or been assigned Owner role at root scope
- Wait 1-2 minutes after elevating access before proceeding
- Verify you can see the root management group in Azure Portal under Management Groups
Validate tenant permissions:
- Confirm you're signed in to the correct Azure AD tenant
- Verify your account has the required permissions (Global Administrator or Owner at root scope)
- If using a service principal, ensure it has been granted appropriate permissions at the tenant level

Test management group access:

az login --tenant <your-tenant-id>
az account show
az rest --method get --url "https://management.azure.com/providers/Microsoft.Management/managementGroups?api-version=2020-05-01"

Running Terraform on Windows

If you are running Terraform on Windows you will encounter issues with the local-exec part of the resource null_resource.register_app_service_provider. The issue is that it uses /bin/bash as the shell, which is not available on Windows.

If you have added configured your azurerm provider as the UI and these docs suggest, with "Microsoft.App" in the resource_providers_to_register list, you will encounter this issue.

provider "azurerm" {
  # For detailed instructions on configuring the Azure provider, please refer to:
  # https://registry.terraform.io/providers/hashicorp/azurerm/latest/docs
  subscription_id = local.azure_orchestrator_subscription_id
  features {
    resource_group {
      prevent_deletion_if_contains_resources = false
    }
    key_vault {
      recover_soft_deleted_keys    = true
      recover_soft_deleted_secrets = true
    }
  }

  # Register App Service Provider for Container App support
  # Used by Upwind Cloud Scanners that are deployed automatically after onboarding
  resource_providers_to_register = ["Microsoft.App"]
}

Then you can safely add the variable skip_app_service_provider_registration = true to the Upwind module

module "upwind_integration_azure_onboarding" {
    source = "upwindsecurity/onboarding/azurerm//modules/tenant"

    ...

    skip_app_service_provider_registration = true

    ...
}

Authorization Errors During Terraform Operations

If you encounter authorization errors during terraform apply or other Terraform operations:

Verify sufficient permissions:
- Confirm your account has Owner privileges at the tenant level
- Check that elevated access is still active (it may have expired)
- Ensure you're authenticated with the correct account: az account show
Service principal troubleshooting:
- If using a service principal, verify it has the required permissions
- Check that the service principal secret hasn't expired
- Ensure the service principal is assigned to the correct scope
API access validation:
- Verify that the required Azure Resource Manager APIs are accessible
- Check for any conditional access policies that might be blocking the requests
- Ensure your IP address is not blocked by network policies

Re-authentication procedure:

az logout
az login
terraform plan  # Verify permissions before applying
terraform apply

Narrow Storage Blob Data Reader access

By default, the Cloud Scanner worker managed identity will get Storage Blob Data Reader permissions in the whole scope of the onboarding (Tenant, management groups, subscriptions). The purpose of that permissions is to enable the Cloud Scanner workers to access storage accounts where Azure Function App source code is stored.

Avoiding the role assignment

You can avoid that role assignment by setting the following variable. It comes at the cost of not being able to scan Azure Function Apps where source code is stored in storage accounts.

disable_function_scanning = true

Narrowing the scope

You can narrow the scope of the role assignment by setting the function_storage_accounts variable to Azure resource IDs of storage accounts where Azure Function App source code is stored.

function_storage_accounts = [
    "subscriptions/your-subscription-id/resourceGroups/your-resource-group/providers/Microsoft.Storage/storageAccounts/your-storage-account"
]

How to get the Azure resource ID of a storage account

The following script should help you identify storage accounts used by Azure Function Apps. It's confirmed to work on Mac OS, with GNU bash, version 5.2.37(1)-release.

You need Azure CLI (az) to be installed and authenticated.

#!/bin/bash

# Script to identify storage accounts used by Azure Function Apps
# This helps determine which storage accounts need "Storage Blob Data Reader" access
# for Upwind CloudScanner to scan Function App code

set -e

# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color

echo -e "${BLUE}================================================${NC}"
echo -e "${BLUE}Function App Storage Account Discovery${NC}"
echo -e "${BLUE}================================================${NC}"
echo ""

# Check if Azure CLI is installed
if ! command -v az &> /dev/null; then
    echo -e "${RED}Error: Azure CLI is not installed${NC}"
    echo "Please install it from: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli"
    exit 1
fi

# Check if logged in
if ! az account show &> /dev/null; then
    echo -e "${RED}Error: Not logged in to Azure CLI${NC}"
    echo "Please run: az login"
    exit 1
fi

# Get current subscription
CURRENT_SUB=$(az account show --query id -o tsv)
CURRENT_SUB_NAME=$(az account show --query name -o tsv)

echo -e "${GREEN}Current Subscription:${NC} $CURRENT_SUB_NAME ($CURRENT_SUB)"
echo ""

# Allow user to specify scope
echo "Select scope for discovery:"
echo "1) Current subscription only"
echo "2) All accessible subscriptions"
echo "3) Specific management group"
read -p "Enter choice (1-3): " SCOPE_CHOICE

SUBSCRIPTIONS=()

case $SCOPE_CHOICE in
    1)
        SUBSCRIPTIONS=("$CURRENT_SUB")
        ;;
    2)
        echo -e "${YELLOW}Fetching all accessible subscriptions...${NC}"
        while IFS= read -r sub; do
            SUBSCRIPTIONS+=("$sub")
        done < <(az account list --query "[].id" -o tsv)
        echo -e "${GREEN}Found ${#SUBSCRIPTIONS[@]} subscriptions${NC}"
        ;;
    3)
        read -p "Enter management group ID: " MG_ID
        echo -e "${YELLOW}Fetching subscriptions in management group $MG_ID...${NC}"
        while IFS= read -r sub; do
            SUBSCRIPTIONS+=("$sub")
        done < <(az account management-group show --name "$MG_ID" --expand --recurse --query "children[?type=='Microsoft.Management/managementGroups/subscriptions'].name" -o tsv)
        if [ ${#SUBSCRIPTIONS[@]} -eq 0 ]; then
            echo -e "${RED}No subscriptions found in management group${NC}"
            exit 1
        fi
        echo -e "${GREEN}Found ${#SUBSCRIPTIONS[@]} subscriptions${NC}"
        ;;
    *)
        echo -e "${RED}Invalid choice${NC}"
        exit 1
        ;;
esac

echo ""
echo -e "${YELLOW}Scanning for Function Apps and their storage accounts...${NC}"
echo ""

# Simple arrays to store results (bash 3.2 compatible)
STORAGE_ACCOUNTS=()  # Will store unique storage account resource IDs
TOTAL_FUNCTION_APPS=0

# Iterate through subscriptions
for SUB_ID in "${SUBSCRIPTIONS[@]}"; do
    SUB_NAME=$(az account show --subscription "$SUB_ID" --query name -o tsv 2>/dev/null || echo "Unknown")

    echo -e "${BLUE}Checking subscription: ${NC}$SUB_NAME ($SUB_ID)"

    # Get all function apps in the subscription
    FUNCTION_APP_LIST=$(az functionapp list --subscription "$SUB_ID" --query "[].{name:name, resourceGroup:resourceGroup, id:id}" -o json 2>/dev/null || echo "[]")

    FUNCTION_APP_COUNT=$(echo "$FUNCTION_APP_LIST" | jq -r '. | length')

    if [ "$FUNCTION_APP_COUNT" -eq 0 ]; then
        echo -e "  ${YELLOW}No Function Apps found${NC}"
        continue
    fi

    echo -e "  ${GREEN}Found $FUNCTION_APP_COUNT Function App(s)${NC}"
    TOTAL_FUNCTION_APPS=$((TOTAL_FUNCTION_APPS + FUNCTION_APP_COUNT))

    # Process each function app - avoid subshell to preserve array modifications
    FUNCTION_APP_INDICES=$(echo "$FUNCTION_APP_LIST" | jq -r 'to_entries | .[] | .key')

    for INDEX in $FUNCTION_APP_INDICES; do
        FUNCTION_APP_NAME=$(echo "$FUNCTION_APP_LIST" | jq -r ".[$INDEX].name")
        FUNCTION_APP_RG=$(echo "$FUNCTION_APP_LIST" | jq -r ".[$INDEX].resourceGroup")
        FUNCTION_APP_ID=$(echo "$FUNCTION_APP_LIST" | jq -r ".[$INDEX].id")

        STORAGE_ACCOUNT_NAME=""

        # Method 1: Try to get from functionAppConfig.deployment.storage.value (newer flex consumption model - blob URL)
        FUNCTION_APP_CONFIG=$(az functionapp show \
            --name "$FUNCTION_APP_NAME" \
            --resource-group "$FUNCTION_APP_RG" \
            --subscription "$SUB_ID" \
            --query "properties.functionAppConfig.deployment.storage.value" \
            -o tsv 2>/dev/null || echo "")

        if [ -n "$FUNCTION_APP_CONFIG" ] && [ "$FUNCTION_APP_CONFIG" != "null" ]; then
            # Extract storage account name from blob URL
            # Format: https://<storage-account>.blob.core.windows.net/...
            STORAGE_ACCOUNT_NAME=$(echo "$FUNCTION_APP_CONFIG" | sed -n 's|https://\([^.]*\)\.blob\.core\.windows\.net.*|\1|p')
        fi

        # Method 2: Try to get from functionAppConfig.deployment.storage.authentication.storageAccountConnectionStringName
        if [ -z "$STORAGE_ACCOUNT_NAME" ]; then
            CONNECTION_STRING_NAME=$(az functionapp show \
                --name "$FUNCTION_APP_NAME" \
                --resource-group "$FUNCTION_APP_RG" \
                --subscription "$SUB_ID" \
                --query "properties.functionAppConfig.deployment.storage.authentication.storageAccountConnectionStringName" \
                -o tsv 2>/dev/null || echo "")

            if [ -n "$CONNECTION_STRING_NAME" ] && [ "$CONNECTION_STRING_NAME" != "null" ]; then
                # Get the connection string from app settings
                STORAGE_CONNECTION=$(az functionapp config appsettings list \
                    --name "$FUNCTION_APP_NAME" \
                    --resource-group "$FUNCTION_APP_RG" \
                    --subscription "$SUB_ID" \
                    --query "[?name=='$CONNECTION_STRING_NAME'].value" \
                    -o tsv 2>/dev/null || echo "")

                if [ -n "$STORAGE_CONNECTION" ]; then
                    # Extract storage account name from connection string
                    STORAGE_ACCOUNT_NAME=$(echo "$STORAGE_CONNECTION" | sed -n 's/.*AccountName=\([^;]*\).*/\1/p')

                    # Try alternative format (resource ID)
                    if [ -z "$STORAGE_ACCOUNT_NAME" ]; then
                        if [[ "$STORAGE_CONNECTION" =~ /subscriptions/([^/]+)/resourceGroups/([^/]+)/providers/Microsoft.Storage/storageAccounts/([^/]+) ]]; then
                            STORAGE_ACCOUNT_NAME="${BASH_REMATCH[3]}"
                        fi
                    fi
                fi
            fi
        fi

        # Method 3: Try traditional AzureWebJobsStorage app setting
        if [ -z "$STORAGE_ACCOUNT_NAME" ]; then
            STORAGE_CONNECTION=$(az functionapp config appsettings list \
                --name "$FUNCTION_APP_NAME" \
                --resource-group "$FUNCTION_APP_RG" \
                --subscription "$SUB_ID" \
                --query "[?name=='AzureWebJobsStorage'].value" \
                -o tsv 2>/dev/null || echo "")

            if [ -n "$STORAGE_CONNECTION" ]; then
                # Extract storage account name from connection string
                # Format: DefaultEndpointsProtocol=https;AccountName=<name>;...
                STORAGE_ACCOUNT_NAME=$(echo "$STORAGE_CONNECTION" | sed -n 's/.*AccountName=\([^;]*\).*/\1/p')

                # Try alternative format (resource ID)
                if [ -z "$STORAGE_ACCOUNT_NAME" ]; then
                    if [[ "$STORAGE_CONNECTION" =~ /subscriptions/([^/]+)/resourceGroups/([^/]+)/providers/Microsoft.Storage/storageAccounts/([^/]+) ]]; then
                        STORAGE_ACCOUNT_NAME="${BASH_REMATCH[3]}"
                    fi
                fi
            fi
        fi

        if [ -z "$STORAGE_ACCOUNT_NAME" ]; then
            echo -e "    ${YELLOW}⚠ $FUNCTION_APP_NAME: No storage account configured${NC}"
            continue
        fi

        # Get the full resource ID of the storage account
        STORAGE_ACCOUNT_ID=$(az storage account list \
            --subscription "$SUB_ID" \
            --query "[?name=='$STORAGE_ACCOUNT_NAME'].id" \
            -o tsv 2>/dev/null | head -n 1)

        if [ -z "$STORAGE_ACCOUNT_ID" ]; then
            # Try to find in other subscriptions if not found
            for OTHER_SUB in "${SUBSCRIPTIONS[@]}"; do
                STORAGE_ACCOUNT_ID=$(az storage account list \
                    --subscription "$OTHER_SUB" \
                    --query "[?name=='$STORAGE_ACCOUNT_NAME'].id" \
                    -o tsv 2>/dev/null | head -n 1)
                if [ -n "$STORAGE_ACCOUNT_ID" ]; then
                    break
                fi
            done
        fi

        if [ -n "$STORAGE_ACCOUNT_ID" ]; then
            echo -e "    ${GREEN}✓ $FUNCTION_APP_NAME → $STORAGE_ACCOUNT_NAME${NC}"

            # Check if storage account is already in the array (avoid duplicates)
            ALREADY_EXISTS=0
            for existing in "${STORAGE_ACCOUNTS[@]}"; do
                if [ "$existing" = "$STORAGE_ACCOUNT_ID" ]; then
                    ALREADY_EXISTS=1
                    break
                fi
            done

            if [ $ALREADY_EXISTS -eq 0 ]; then
                STORAGE_ACCOUNTS+=("$STORAGE_ACCOUNT_ID")
            fi
        else
            echo -e "    ${RED}✗ $FUNCTION_APP_NAME: Storage account '$STORAGE_ACCOUNT_NAME' not found${NC}"
        fi
    done
done

echo ""
echo -e "${BLUE}================================================${NC}"
echo -e "${BLUE}Summary${NC}"
echo -e "${BLUE}================================================${NC}"
echo ""
echo -e "${GREEN}Total Function Apps found: $TOTAL_FUNCTION_APPS${NC}"
echo -e "${GREEN}Unique Storage Accounts: ${#STORAGE_ACCOUNTS[@]}${NC}"
echo ""

if [ ${#STORAGE_ACCOUNTS[@]} -eq 0 ]; then
    echo -e "${YELLOW}No storage accounts found. This could mean:${NC}"
    echo -e "${YELLOW}  - No Function Apps are deployed${NC}"
    echo -e "${YELLOW}  - Function Apps use managed identities instead of connection strings${NC}"
    echo -e "${YELLOW}  - Storage accounts are in different subscriptions not included in scan${NC}"
    exit 0
fi

echo -e "${BLUE}Storage Account Resource IDs:${NC}"
echo ""
for STORAGE_ID in "${STORAGE_ACCOUNTS[@]}"; do
    echo "  $STORAGE_ID"
done

echo ""
echo -e "${BLUE}================================================${NC}"
echo -e "${BLUE}Terraform Configuration${NC}"
echo -e "${BLUE}================================================${NC}"
echo ""
echo "Add this to your Terraform configuration:"
echo ""
echo -e "${GREEN}variable \"function_storage_accounts\" {${NC}"
echo -e "${GREEN}  description = \"List of storage account resource IDs used by Function Apps for code storage\"${NC}"
echo -e "${GREEN}  type        = list(string)${NC}"
echo -e "${GREEN}  default     = [${NC}"
for STORAGE_ID in "${STORAGE_ACCOUNTS[@]}"; do
    echo -e "${GREEN}    \"$STORAGE_ID\",${NC}"
done
echo -e "${GREEN}  ]${NC}"
echo -e "${GREEN}}${NC}"

echo ""
echo -e "${BLUE}================================================${NC}"
echo -e "${BLUE}Azure CLI Commands${NC}"
echo -e "${BLUE}================================================${NC}"
echo ""
echo "To manually grant Storage Blob Data Reader access:"
echo ""
for STORAGE_ID in "${STORAGE_ACCOUNTS[@]}"; do
    echo "az role assignment create \\"
    echo "  --role \"Storage Blob Data Reader\" \\"
    echo "  --assignee-object-id <CLOUDSCANNER_IDENTITY_PRINCIPAL_ID> \\"
    echo "  --scope \"$STORAGE_ID\""
    echo ""
done

# Save to file
OUTPUT_FILE="function-storage-accounts-$(date +%Y%m%d-%H%M%S).txt"
{
    echo "# Function App Storage Accounts"
    echo "# Generated: $(date)"
    echo ""
    echo "# Storage Account Resource IDs:"
    for STORAGE_ID in "${STORAGE_ACCOUNTS[@]}"; do
        echo "$STORAGE_ID"
    done
} > "$OUTPUT_FILE"

echo -e "${GREEN}Results saved to: $OUTPUT_FILE${NC}"

Client Secret Management

When a client secret expires or needs to be updated:

In Azure Console

Navigate to App Registration:
- Access the Azure application linked with Upwind
- Go to "Certificates & Secrets" in the application settings
Generate New Client Secret:
- Click on "New client secret"
- Set the expiration to at least 12 months
- Note down the new client secret
Alternatively, use Azure CLI:
```
az ad app credential reset --id [Application (client) ID]
```
The "password" field in the output is your new client secret.

In Upwind Management Console

Access Upwind Console:
- Visit Upwind Console - Settings -> Organizations and accounts.
- Select the relevant subscription ID from the "Account ID" column.
Update Client Secret:
- Click the edit button next to the relevant subscription
- Choose "Update Azure client secret"
- Enter the new client secret and click "Test account connectivity"

Support

If you encounter any issues not covered in this guide:

Check the Azure Activity Log for any error messages
Review RBAC assignments and permissions
Contact Upwind support with:
- Detailed error messages
- Azure subscription ID
- Resource group name
- Relevant log entries

Overview​

Common Issues​

Insufficient Spot VM Quota​

Authentication Issues​

Root Tenant Management Group Access Issues​

Verify Elevated Access Status​

Running Terraform on Windows​

Authorization Errors During Terraform Operations​

Permission-Related Issues​

Narrow Storage Blob Data Reader access​

Avoiding the role assignment​

Narrowing the scope​

Client Secret Management​

In Azure Console​

In Upwind Management Console​

Support​