Complete guide for deploying LastFMReaderv3 to Azure Container Instances (ACI) with production-ready security and observability.
- Overview
- Prerequisites
- Quick Start
- Step-by-Step Deployment
- Azure Key Vault Integration
- Managed Identity Setup
- Networking Configuration
- Persistent Storage
- Logging and Monitoring
- Structured Logging Format
- Scaling Considerations
- Cost Optimization
- Troubleshooting
Azure Container Instances (ACI) provides serverless container execution without managing infrastructure. This guide covers production deployment with:
- ✅ Managed Identity authentication (no connection strings)
- ✅ Azure Key Vault for secrets management
- ✅ Log Analytics workspace integration
- ✅ Custom metrics and structured logging
- ✅ Automated deployment scripts
- ✅ Cost-effective resource allocation
Architecture:
┌─────────────────────────────────────────────────┐
│ Azure Container Instances │
│ │
│ ┌──────────────────────────────────────────┐ │
│ │ lastfm-sync Container │ │
│ │ • Managed Identity enabled │ │
│ │ • Fetches secrets from Key Vault │ │
│ │ • Writes to Azure Blob Storage │ │
│ │ • Sends logs to Log Analytics │ │
│ └──────────────────────────────────────────┘ │
│ │
└─────────────────────────────────────────────────┘
│ │ │
↓ ↓ ↓
┌──────────┐ ┌─────────────┐ ┌──────────────────┐
│ Key Vault│ │Blob Storage │ │ Log Analytics │
│ (secrets)│ │ (output) │ │ (monitoring) │
└──────────┘ └─────────────┘ └──────────────────┘
- Active Azure subscription
- Contributor or Owner role on resource group
Install Azure CLI 2.50.0+:
# macOS
brew install azure-cli
# Linux
curl -sL https://aka.ms/InstallAzureCLIDeb | sudo bash
# Windows
# Download from https://aka.ms/installazurecliwindows
# Verify installation
az --version# Login to Azure
az login
# Set default subscription (if you have multiple)
az account set --subscription "your-subscription-id"
# Verify current subscription
az account showYour account needs these permissions:
- Microsoft.ContainerInstance/containerGroups/write: Create container instances
- Microsoft.ManagedIdentity/userAssignedIdentities/assign/action: Assign managed identity
- Microsoft.Storage/storageAccounts/read: Read storage account
- Microsoft.KeyVault/vaults/secrets/read: Read Key Vault secrets (via managed identity)
Copy and edit the deployment parameters:
cp azure/aci-params.json.example azure/aci-params.json
nano azure/aci-params.jsonEdit values:
resourceGroup: Your Azure resource group namelocation: Azure region (e.g.,eastus,westus2)containerName: Unique name for container instancestorageAccount: Your Azure Storage account namekeyVaultName: Your Azure Key Vault name
# Set your Last.fm API key in Key Vault
az keyvault secret set \
--vault-name your-keyvault \
--name lastfm-api-key \
--value "your-lastfm-api-key"# Run deployment script
./azure/deploy-aci.sh -p azure/aci-params.json
# Or deploy manually (see below)# Create resource group
az group create \
--name lastfm-rg \
--location eastus
# Verify
az group show --name lastfm-rg# Create storage account
az storage account create \
--name lastfmstorage$(date +%s | tail -c 6) \
--resource-group lastfm-rg \
--location eastus \
--sku Standard_LRS \
--kind StorageV2
# Create container
az storage container create \
--name scrobbles \
--account-name lastfmstorage123456 \
--auth-mode login# Create Key Vault
az keyvault create \
--name lastfm-kv-$(date +%s | tail -c 6) \
--resource-group lastfm-rg \
--location eastus \
--enable-rbac-authorization
# Store Last.fm API key
az keyvault secret set \
--vault-name lastfm-kv-123456 \
--name lastfm-api-key \
--value "your-api-key-here"# Create user-assigned managed identity
az identity create \
--name lastfm-identity \
--resource-group lastfm-rg \
--location eastus
# Get identity details
IDENTITY_ID=$(az identity show \
--name lastfm-identity \
--resource-group lastfm-rg \
--query id -o tsv)
PRINCIPAL_ID=$(az identity show \
--name lastfm-identity \
--resource-group lastfm-rg \
--query principalId -o tsv)
echo "Identity ID: $IDENTITY_ID"
echo "Principal ID: $PRINCIPAL_ID"# Storage Blob Data Contributor role
az role assignment create \
--assignee $PRINCIPAL_ID \
--role "Storage Blob Data Contributor" \
--scope "/subscriptions/$(az account show --query id -o tsv)/resourceGroups/lastfm-rg/providers/Microsoft.Storage/storageAccounts/lastfmstorage123456"
# Key Vault Secrets User role
az role assignment create \
--assignee $PRINCIPAL_ID \
--role "Key Vault Secrets User" \
--scope "/subscriptions/$(az account show --query id -o tsv)/resourceGroups/lastfm-rg/providers/Microsoft.KeyVault/vaults/lastfm-kv-123456"# Create workspace
az monitor log-analytics workspace create \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--location eastus
# Get workspace credentials
WORKSPACE_ID=$(az monitor log-analytics workspace show \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--query customerId -o tsv)
WORKSPACE_KEY=$(az monitor log-analytics workspace get-shared-keys \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--query primarySharedKey -o tsv)
echo "Workspace ID: $WORKSPACE_ID"
echo "Workspace Key: $WORKSPACE_KEY"# Fetch secret from Key Vault
LASTFM_API_KEY=$(az keyvault secret show \
--vault-name lastfm-kv-123456 \
--name lastfm-api-key \
--query value -o tsv)
# Deploy container with managed identity
az container create \
--resource-group lastfm-rg \
--name lastfm-sync-alice \
--image ghcr.io/lastfm-reader/lastfm-sync:latest \
--cpu 0.5 \
--memory 0.5 \
--restart-policy Never \
--assign-identity $IDENTITY_ID \
--environment-variables \
LASTFM_API_KEY="$LASTFM_API_KEY" \
AZURE_STORAGE_ACCOUNT=lastfmstorage123456 \
--command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth mi" \
--log-analytics-workspace $WORKSPACE_ID \
--log-analytics-workspace-key $WORKSPACE_KEY
# Verify deployment
az container show \
--resource-group lastfm-rg \
--name lastfm-sync-alice \
--query "{FQDN:ipAddress.fqdn,ProvisioningState:provisioningState}" --output table- ✅ Centralized secret management
- ✅ Automatic secret rotation support
- ✅ Access logging and auditing
- ✅ No secrets in environment variables or code
# Set secret
az keyvault secret set \
--vault-name your-keyvault \
--name lastfm-api-key \
--value "your-secret-value"
# Get secret (for deployment)
az keyvault secret show \
--vault-name your-keyvault \
--name lastfm-api-key \
--query value -o tsv
# List secrets
az keyvault secret list \
--vault-name your-keyvault \
--query "[].name" -o table
# Rotate secret (update value)
az keyvault secret set \
--vault-name your-keyvault \
--name lastfm-api-key \
--value "new-secret-value"Option 1: Fetch at deployment time (recommended for one-off jobs)
LASTFM_API_KEY=$(az keyvault secret show --vault-name kv --name lastfm-api-key --query value -o tsv)
az container create ... --environment-variables LASTFM_API_KEY="$LASTFM_API_KEY"Option 2: Use managed identity to fetch at runtime (requires code changes)
// Modify application to fetch from Key Vault using DefaultAzureCredential
// Not implemented in current version - future enhancementManaged Identity eliminates the need for connection strings and access keys.
| Feature | User-Assigned | System-Assigned |
|---|---|---|
| Lifecycle | Independent of resource | Tied to resource |
| Reusability | Can be shared across resources | One per resource |
| Setup | More complex | Simpler |
| Recommended | ✅ Production | Development |
# Create identity
az identity create \
--name lastfm-identity \
--resource-group lastfm-rg
# Get ID and Principal ID
IDENTITY_ID=$(az identity show --name lastfm-identity --resource-group lastfm-rg --query id -o tsv)
PRINCIPAL_ID=$(az identity show --name lastfm-identity --resource-group lastfm-rg --query principalId -o tsv)# Storage Blob Data Contributor (read/write blobs)
az role assignment create \
--assignee $PRINCIPAL_ID \
--role "Storage Blob Data Contributor" \
--scope /subscriptions/{subscription-id}/resourceGroups/lastfm-rg
# Key Vault Secrets User (read secrets)
az role assignment create \
--assignee $PRINCIPAL_ID \
--role "Key Vault Secrets User" \
--scope /subscriptions/{subscription-id}/resourceGroups/lastfm-rg/providers/Microsoft.KeyVault/vaults/lastfm-kvaz container create \
--name lastfm-sync \
--assign-identity $IDENTITY_ID \
--environment-variables AZURE_STORAGE_ACCOUNT=lastfmstorage \
--command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth mi"For private deployments:
# Create VNet and subnet
az network vnet create \
--resource-group lastfm-rg \
--name lastfm-vnet \
--address-prefix 10.0.0.0/16 \
--subnet-name container-subnet \
--subnet-prefix 10.0.1.0/24
# Deploy container in VNet
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--vnet lastfm-vnet \
--subnet container-subnetConnect to storage and Key Vault via private endpoints:
# Create private endpoint for storage
az network private-endpoint create \
--name storage-pe \
--resource-group lastfm-rg \
--vnet-name lastfm-vnet \
--subnet container-subnet \
--private-connection-resource-id "/subscriptions/{sub-id}/resourceGroups/lastfm-rg/providers/Microsoft.Storage/storageAccounts/lastfmstorage" \
--group-id blob \
--connection-name storage-connectionRestrict outbound traffic:
az network nsg create \
--resource-group lastfm-rg \
--name lastfm-nsg
az network nsg rule create \
--resource-group lastfm-rg \
--nsg-name lastfm-nsg \
--name allow-https-outbound \
--priority 100 \
--direction Outbound \
--access Allow \
--protocol Tcp \
--destination-port-ranges 443While Managed Identity is recommended for production, the tool supports multiple authentication methods:
Use connection string for development or when managed identity is not available:
# Get connection string
CONN_STR=$(az storage account show-connection-string \
--name lastfmstorage \
--resource-group lastfm-rg \
--query connectionString -o tsv)
# Deploy with connection string
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--environment-variables \
LASTFM_API_KEY="$LASTFM_API_KEY" \
AZURE_STORAGE_CONNECTION_STRING="$CONN_STR" \
--command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth connstr"Use storage account key directly:
# Get storage account key
STORAGE_KEY=$(az storage account keys list \
--resource-group lastfm-rg \
--account-name lastfmstorage \
--query "[0].value" -o tsv)
# Deploy with account key
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--environment-variables \
LASTFM_API_KEY="$LASTFM_API_KEY" \
AZURE_STORAGE_ACCOUNT=lastfmstorage \
AZURE_STORAGE_ACCOUNT_KEY="$STORAGE_KEY" \
--command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth key"Use SAS token for time-limited access:
# Generate SAS token (valid for 24 hours)
SAS_TOKEN=$(az storage container generate-sas \
--account-name lastfmstorage \
--name scrobbles \
--permissions rwdl \
--expiry $(date -u -d "24 hours" '+%Y-%m-%dT%H:%MZ') \
--output tsv)
# Construct container URL with SAS token
CONTAINER_URL="https://lastfmstorage.blob.core.windows.net/scrobbles?$SAS_TOKEN"
# Deploy with SAS token
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--environment-variables \
LASTFM_API_KEY="$LASTFM_API_KEY" \
--command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container-url \"$CONTAINER_URL\" --azure-auth sas"Security Note: For production deployments, prefer Managed Identity (
--azure-auth mior--azure-auth default) over connection strings, account keys, or SAS tokens. These credential-based methods should only be used for development or when Managed Identity is not available.
# Create file share
az storage share create \
--name lastfm-data \
--account-name lastfmstorage
# Get storage account key
STORAGE_KEY=$(az storage account keys list \
--resource-group lastfm-rg \
--account-name lastfmstorage \
--query "[0].value" -o tsv)
# Mount file share in container
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--azure-file-volume-account-name lastfmstorage \
--azure-file-volume-account-key $STORAGE_KEY \
--azure-file-volume-share-name lastfm-data \
--azure-file-volume-mount-path /data \
--command-line "/app/lastfm-sync fetch --user alice --output local"View logs in Azure Portal:
- Navigate to Container Instances
- Select your container
- Click "Containers" in left menu
- Click "Logs" tab
- View real-time output
Download logs:
az container logs \
--resource-group lastfm-rg \
--name lastfm-sync-alice \
--followLog Analytics provides advanced querying and alerting capabilities.
Create workspace (if not exists):
az monitor log-analytics workspace create \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--location eastusGet workspace credentials:
WORKSPACE_ID=$(az monitor log-analytics workspace show \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--query customerId -o tsv)
WORKSPACE_KEY=$(az monitor log-analytics workspace get-shared-keys \
--resource-group lastfm-rg \
--workspace-name lastfm-logs \
--query primarySharedKey -o tsv)Deploy container with Log Analytics:
az container create \
--resource-group lastfm-rg \
--name lastfm-sync \
--image lastfm-sync:latest \
--log-analytics-workspace $WORKSPACE_ID \
--log-analytics-workspace-key $WORKSPACE_KEY \
...Query logs in Log Analytics:
// All logs from lastfm-sync container
ContainerInstanceLog_CL
| where ContainerGroup_s == "lastfm-sync"
| project TimeGenerated, Message
| order by TimeGenerated desc
// Errors only
ContainerInstanceLog_CL
| where ContainerGroup_s == "lastfm-sync"
| where Message contains "error" or Message contains "Error"
| project TimeGenerated, Message
| order by TimeGenerated desc
// Fetch operations
ContainerInstanceLog_CL
| where ContainerGroup_s == "lastfm-sync"
| where Message contains "fetch.page"
| project TimeGenerated, Message
| order by TimeGenerated descUnderstanding container exit codes:
| Exit Code | Meaning | Action |
|---|---|---|
| 0 | Success | Normal completion |
| 1 | General error | Check logs for error details |
| 2 | Misuse of shell command | Verify command syntax |
| 125 | Container failed to run | Check image and entrypoint |
| 126 | Command cannot execute | Check permissions |
| 127 | Command not found | Verify binary path |
| 128+n | Fatal error signal n | Signal interrupted (e.g., 137 = SIGKILL) |
Check exit code:
az container show \
--resource-group lastfm-rg \
--name lastfm-sync \
--query "containers[0].instanceView.currentState.exitCode" -o tsvLastFMReaderv3 can export custom metrics for monitoring:
Metrics Available:
fetch_duration_seconds: Total fetch durationfetch_pages_total: Number of pages fetchedfetch_scrobbles_total: Number of scrobbles fetchedapi_calls_total: Total API calls madeapi_errors_total: Number of API errors
Configure Application Insights (future enhancement):
az monitor app-insights component create \
--app lastfm-insights \
--location eastus \
--resource-group lastfm-rg \
--application-type otherLastFMReaderv3 uses structured JSON logging for Log Analytics integration.
All log entries must include:
| Field | Type | Description | Example |
|---|---|---|---|
timestamp |
ISO 8601 | UTC timestamp | 2026-01-06T14:30:22Z |
level |
string | Log level | info, debug, error |
message |
string | Human-readable message | fetch.page.complete |
context |
object | Structured context data | See below |
Additional fields for enriched logging:
| Field | Type | Description |
|---|---|---|
user |
string | Last.fm username |
duration_ms |
integer | Operation duration in milliseconds |
api_calls |
integer | Number of API calls made |
error_details |
string | Error message and stack trace |
page |
integer | Current page number |
total_pages |
integer | Total pages to fetch |
Successful fetch:
{
"timestamp": "2026-01-06T14:30:22Z",
"level": "info",
"message": "fetch.page.complete",
"context": {
"user": "alice",
"page": 5,
"total_pages": 10,
"scrobbles_fetched": 200,
"duration_ms": 1250
}
}API error:
{
"timestamp": "2026-01-06T14:30:25Z",
"level": "error",
"message": "fetch.api.error",
"context": {
"user": "alice",
"error_details": "rate limited: 429 Too Many Requests",
"retry_after": 60,
"api_calls": 3
}
}Watermark update:
{
"timestamp": "2026-01-06T14:30:30Z",
"level": "info",
"message": "watermark.update.complete",
"context": {
"user": "alice",
"max_uts": 1704067200,
"duration_ms": 50
}
}// Parse JSON logs
ContainerInstanceLog_CL
| extend LogEntry = parse_json(Message)
| extend
Level = tostring(LogEntry.level),
LogMessage = tostring(LogEntry.message),
User = tostring(LogEntry.context.user),
Duration = toint(LogEntry.context.duration_ms)
| where Level == "error"
| project TimeGenerated, User, LogMessage, Duration
| order by TimeGenerated desc
// Calculate average fetch duration
ContainerInstanceLog_CL
| extend LogEntry = parse_json(Message)
| where tostring(LogEntry.message) == "fetch.page.complete"
| extend Duration = toint(LogEntry.context.duration_ms)
| summarize AvgDuration = avg(Duration), Count = count() by bin(TimeGenerated, 1h)
| render timechartRun multiple container instances for different users:
# Deploy for user alice
az container create --name lastfm-sync-alice ...
# Deploy for user bob
az container create --name lastfm-sync-bob ...
# Deploy for user charlie
az container create --name lastfm-sync-charlie ...Use Azure Logic Apps or Azure Functions to trigger container creation:
# Create container group with restart policy
az container create \
--resource-group lastfm-rg \
--name lastfm-sync-daily \
--image lastfm-sync:latest \
--restart-policy OnFailure \
--schedule "0 0 * * *" # Daily at midnight (not directly supported - use Logic Apps)Alternative: Azure Logic Apps
- Create Logic App with recurrence trigger (daily)
- Add "Run Command" action:
az container create ... - Add "Delete Container" action after completion
Optimize CPU and memory based on workload:
| User Size | Scrobbles | CPU | Memory | Cost/Month |
|---|---|---|---|---|
| Small | < 10K | 0.5 | 0.5 GB | ~$15 |
| Medium | 10K-100K | 1.0 | 1.0 GB | ~$30 |
| Large | > 100K | 2.0 | 2.0 GB | ~$60 |
az container create \
--cpu 1.0 \
--memory 1.0 \
...ACI charges only for execution time (per second):
Pricing (as of 2026, US East):
- CPU: $0.0000125/vCPU/second (~$0.045/vCPU/hour)
- Memory: $0.0000014/GB/second (~$0.005/GB/hour)
Example:
- Container: 0.5 vCPU, 0.5 GB RAM
- Runtime: 10 minutes daily
- Cost: 0.5 × $0.0000125 × 600 × 30 = $0.11/month (CPU)
- Cost: 0.5 × $0.0000014 × 600 × 30 = $0.01/month (Memory)
- Total: ~$0.12/month
- Use restart policy "Never" for one-off jobs
- Delete container after completion:
az container delete --name lastfm-sync --resource-group lastfm-rg --yes
- Use spot instances (if available)
- Optimize fetch frequency (weekly vs daily)
- Use incremental sync (only fetch new scrobbles)
Symptom: failed to get token: [credential authentication failed]
Diagnostic Commands:
# Verify managed identity is assigned
az container show \
--resource-group lastfm-rg \
--name lastfm-sync \
--query identity
# Check role assignments
az role assignment list \
--assignee $PRINCIPAL_ID \
--query "[].{Role:roleDefinitionName, Scope:scope}" -o table
# Test identity access to storage
az storage blob list \
--account-name lastfmstorage \
--container-name scrobbles \
--auth-mode login \
--only-show-errorsSolutions:
- Verify managed identity is assigned to container
- Check role assignments include "Storage Blob Data Contributor"
- Wait 5-10 minutes for role propagation
- Ensure storage account allows managed identity access
Alternative: Use connection string temporarily:
az container create \
--environment-variables \
AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=..." \
--command-line "... --azure-auth connstr"Symptom: LASTFM_API_KEY is required
Diagnostic Commands:
# Verify secret exists in Key Vault
az keyvault secret show \
--vault-name lastfm-kv \
--name lastfm-api-key
# Check container environment variables
az container show \
--resource-group lastfm-rg \
--name lastfm-sync \
--query "containers[0].environmentVariables" -o tableSolutions:
- Verify secret name matches exactly (case-sensitive)
- Check Key Vault access policy or RBAC roles
- Ensure secret value is fetched before deployment:
LASTFM_API_KEY=$(az keyvault secret show --vault-name kv --name lastfm-api-key --query value -o tsv) - Pass as environment variable:
--environment-variables LASTFM_API_KEY="$LASTFM_API_KEY"
Symptom: Quota exceeded for container group count
Diagnostic Commands:
# Check current usage
az container list \
--resource-group lastfm-rg \
--query "length(@)"
# Check subscription limits
az vm list-usage \
--location eastus \
--query "[?name.value=='containerGroups'].{Name:name.localizedValue, Current:currentValue, Limit:limit}" -o tableSolutions:
- Delete unused container instances:
az container delete --name old-container --resource-group lastfm-rg --yes
- Request quota increase (Azure Portal → Support)
- Deploy to different region with available capacity
- Use Container Apps or AKS for higher scale
Symptom: connection timeout or name resolution failed
Diagnostic Commands:
# Check container network profile
az container show \
--resource-group lastfm-rg \
--name lastfm-sync \
--query "networkProfile"
# Test DNS resolution (from local machine)
nslookup api.last.fm
nslookup lastfmstorage.blob.core.windows.net
# Check NSG rules (if using VNet)
az network nsg rule list \
--resource-group lastfm-rg \
--nsg-name lastfm-nsg \
--query "[].{Name:name, Priority:priority, Direction:direction, Access:access}" -o tableSolutions:
- Verify outbound internet access (port 443 required)
- Check NSG rules allow HTTPS to:
api.last.fm(Last.fm API)*.blob.core.windows.net(Azure Storage)*.vault.azure.net(Key Vault)
- If using private endpoints, verify DNS resolution
- Check firewall rules on storage account
- Temporary workaround: Deploy without VNet:
az container create --name lastfm-sync --image lastfm-sync:latest ... # (without --vnet or --subnet flags)
- Configuration Reference - Environment variables and settings
- Docker Documentation - Building and running containers locally
- Security Best Practices - Secure deployment patterns
- Troubleshooting Guide - Common issues and solutions
- Azure Container Instances Documentation
- Azure Managed Identities
- Azure Key Vault
- Log Analytics Query Language
Last Updated: 2026-01-06
Feature: 002-containerization-documentation