Update changelog, README, and documentation to include console progress bar and title normalization features

wesback · wesback · commit 51dec3eb2067 · 2026-01-07T13:36:24.000+01:00
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -7,6 +7,47 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added - Feature 005: Console Progress Bar
+
+- **Progress Bar Implementation** (`internal/progress/`)
+  - Real-time progress tracking with visual feedback using `github.com/schollz/progressbar/v3`
+  - ETA calculation based on actual throughput
+  - Speed metrics (scrobbles per second)
+  - Current/total scrobble counts with percentage
+  - Automatic terminal width detection
+  - Thread-safe progress updates for concurrent operations
+  - Configurable via `--no-progress` flag or `LASTFM_NO_PROGRESS` environment variable
+  - Graceful degradation for non-interactive terminals (CI/CD pipelines)
+
+- **Progress Reporting**
+  - Factory pattern for creating progress bars vs no-op implementations
+  - `ProgressReporter` interface for flexible progress tracking
+  - Progress state tracking (completed, in-progress, pending)
+  - Terminal capability detection using `golang.org/x/term`
+  - Comprehensive test coverage including terminal emulation tests
+
+### Added - Feature 004: Normalized Title Field
+
+- **Title Normalization** (`internal/normalize/`)
+  - Automatic removal of track title annotations for better data quality
+  - Pattern-based removal of:
+    - Remaster/Remastered annotations (e.g., "2009 Remaster", "Remastered 2015")
+    - Live performance markers (e.g., "Live at Wembley", "Live 1969")
+    - Version/edit labels (e.g., "Radio Edit", "Extended Version")
+    - Date/year markers in parentheses or brackets
+    - Remix labels (e.g., "Dave's Remix")
+    - Featuring/collaboration markers (e.g., "feat. Artist", "with Orchestra")
+  - Thread-safe global feature flag for enabling/disabling normalization
+  - Preserves original title in `track` field while adding clean `normalized_title` field
+  - Configurable minimum length protection to avoid over-aggressive cleaning
+  - DEBUG-level logging for title changes (when logger provided)
+  - YAML-based configuration for customizable patterns (`internal/normalize/patterns.go`)
+
+- **Scrobble Model Enhancement** (`internal/models/scrobble.go`)
+  - Added `normalized_title` field to all NDJSON output
+  - Automatic normalization in `NewScrobble` constructor
+  - Original title preserved for audit trail and debugging
+
 ### Added - Feature 002: Containerization & Documentation
 
 - **Configuration Documentation** (`docs/configuration.md`)
@@ -86,6 +127,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
     - DefaultAzureCredential (recommended for Azure VMs/AKS)
     - Managed Identity (workload identity)
     - Connection string
+    - Account key (SharedKeyCredential)
     - SAS token
   - Time-partitioned blob paths for efficient data organization
   - Separate watermark blob storage for state management
diff --git a/README.md b/README.md
@@ -12,9 +12,11 @@
 ✅ **Rate Limit Compliant**: 3 QPS throttling with Retry-After header support  
 ✅ **Production Ready**: Exponential backoff, timeout handling, structured logging  
 ✅ **Dry-Run Mode**: Test configurations without consuming API quota  
-✅ **Azure Integration**: DefaultAzureCredential, managed identity, SAS tokens, connection strings  
+✅ **Azure Integration**: DefaultAzureCredential, managed identity, SAS tokens, connection strings, account keys  
 ✅ **Secret Redaction**: Automatic credential masking in logs  
-✅ **NDJSON Output**: Newline-delimited JSON for easy streaming and processing
+✅ **NDJSON Output**: Newline-delimited JSON for easy streaming and processing  
+✅ **Console Progress Bar**: Real-time progress tracking with ETA, speed, and scrobble count  
+✅ **Title Normalization**: Automatic removal of annotations (Live, Remastered, Featuring) for better data quality
 
 ## Quick Start
 
@@ -148,8 +150,9 @@ Flags:
   Azure Options:
       --azure-container string   Azure container name
       --azure-prefix string      Azure blob prefix (default: "lastfm/")
-      --azure-auth string        Auth method: default, mi, connstr, sas (default: "default")
+      --azure-auth string        Auth method: default, mi, connstr, key, sas (default: "default")
       --azure-account string     Azure storage account name
+      --azure-account-key string Azure storage account key (for key auth)
       --azure-container-url string  Full container URL (for SAS)
       
   Watermark Options:
@@ -162,6 +165,7 @@ Flags:
       
   Other Options:
       --dry-run                  Preview only, no API calls or writes
+      --no-progress              Disable progress bar (useful for CI/CD)
       --log-level string         Log level: info or debug (default: "info")
   -h, --help                     Help for fetch
 ```
@@ -231,6 +235,17 @@ lastfm-sync fetch \
   --azure-auth mi
 ```
 
+**Using account key:**
+```bash
+lastfm-sync fetch \
+  --user alice \
+  --output azure \
+  --azure-container scrobbles \
+  --azure-account mystorageaccount \
+  --azure-account-key "your-account-key" \
+  --azure-auth key
+```
+
 ### Advanced Usage
 
 **Debug logging:**
diff --git a/docs/azure-deployment.md b/docs/azure-deployment.md
@@ -459,6 +459,85 @@ az network nsg rule create \
 
 ---
 
+## Alternative Authentication Methods
+
+While Managed Identity is recommended for production, the tool supports multiple authentication methods:
+
+### Connection String Authentication
+
+Use connection string for development or when managed identity is not available:
+
+```bash
+# Get connection string
+CONN_STR=$(az storage account show-connection-string \
+  --name lastfmstorage \
+  --resource-group lastfm-rg \
+  --query connectionString -o tsv)
+
+# Deploy with connection string
+az container create \
+  --resource-group lastfm-rg \
+  --name lastfm-sync \
+  --image lastfm-sync:latest \
+  --environment-variables \
+    LASTFM_API_KEY="$LASTFM_API_KEY" \
+    AZURE_STORAGE_CONNECTION_STRING="$CONN_STR" \
+  --command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth connstr"
+```
+
+### Account Key Authentication
+
+Use storage account key directly:
+
+```bash
+# Get storage account key
+STORAGE_KEY=$(az storage account keys list \
+  --resource-group lastfm-rg \
+  --account-name lastfmstorage \
+  --query "[0].value" -o tsv)
+
+# Deploy with account key
+az container create \
+  --resource-group lastfm-rg \
+  --name lastfm-sync \
+  --image lastfm-sync:latest \
+  --environment-variables \
+    LASTFM_API_KEY="$LASTFM_API_KEY" \
+    AZURE_STORAGE_ACCOUNT=lastfmstorage \
+    AZURE_STORAGE_ACCOUNT_KEY="$STORAGE_KEY" \
+  --command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container scrobbles --azure-auth key"
+```
+
+### SAS Token Authentication
+
+Use SAS token for time-limited access:
+
+```bash
+# Generate SAS token (valid for 24 hours)
+SAS_TOKEN=$(az storage container generate-sas \
+  --account-name lastfmstorage \
+  --name scrobbles \
+  --permissions rwdl \
+  --expiry $(date -u -d "24 hours" '+%Y-%m-%dT%H:%MZ') \
+  --output tsv)
+
+# Construct container URL with SAS token
+CONTAINER_URL="https://lastfmstorage.blob.core.windows.net/scrobbles?$SAS_TOKEN"
+
+# Deploy with SAS token
+az container create \
+  --resource-group lastfm-rg \
+  --name lastfm-sync \
+  --image lastfm-sync:latest \
+  --environment-variables \
+    LASTFM_API_KEY="$LASTFM_API_KEY" \
+  --command-line "/app/lastfm-sync fetch --user alice --output azure --azure-container-url \"$CONTAINER_URL\" --azure-auth sas"
+```
+
+> **Security Note**: For production deployments, prefer Managed Identity (`--azure-auth mi` or `--azure-auth default`) over connection strings, account keys, or SAS tokens. These credential-based methods should only be used for development or when Managed Identity is not available.
+
+---
+
 ## Persistent Storage
 
 ### Azure File Share (for local output mode)
diff --git a/docs/configuration.md b/docs/configuration.md
@@ -46,17 +46,19 @@ For more complex setups, see the [Environment Variables](#environment-variables)
 | `LASTFM_LOG_LEVEL` | string | `info` | Logging level. Options: `info`, `debug`. |
 | `LASTFM_STATE` | string | `~/.lastfm` | Directory for storing state (watermarks, local output files). |
 | `LASTFM_CONFIG` | string | none | Path to config file. If not set, searches `./`, `~/.lastfm/`, `/etc/lastfm/` for `config.yaml`. |
+| `LASTFM_NO_PROGRESS` | boolean | `false` | Disable console progress bar. Useful for CI/CD pipelines or when redirecting output. |
 
 ### Azure Storage Variables
 
 Required only when using `--output azure`:
 
 | Variable | Type | Default | Description |
 |----------|------|---------|-------------|
-| `AZURE_STORAGE_ACCOUNT` | string | none | Azure Storage account name. Used with `--azure-auth default` or `--azure-auth mi`. |
+| `AZURE_STORAGE_ACCOUNT` | string | none | Azure Storage account name. Used with `--azure-auth default`, `--azure-auth mi`, or `--azure-auth key`. |
 | `AZURE_STORAGE_CONNECTION_STRING` | string | none | **Sensitive**. Full Azure Storage connection string. Used with `--azure-auth connstr`. |
+| `AZURE_STORAGE_ACCOUNT_KEY` | string | none | **Sensitive**. Azure Storage account key. Used with `--azure-auth key`. |
 
-> **Security Note**: Never commit `AZURE_STORAGE_CONNECTION_STRING` to version control. Use environment variables or Azure Key Vault.
+> **Security Note**: Never commit `AZURE_STORAGE_CONNECTION_STRING` or `AZURE_STORAGE_ACCOUNT_KEY` to version control. Use environment variables or Azure Key Vault.
 
 ---
 
@@ -95,16 +97,18 @@ Required when `--output azure`:
 | Flag | Type | Default | Description |
 |------|------|---------|-------------|
 | `--azure-container` | string | none | **Required**. Azure Blob Storage container name. |
-| `--azure-account` | string | `$AZURE_STORAGE_ACCOUNT` | Storage account name (alternative to connection string). |
-| `--azure-prefix` | string | `lastfm/` | Blob prefix (folder path). Blobs written to `{prefix}{user}/{year}/{month}/{timestamp}.ndjson`. |
-| `--azure-auth` | string | `default` | Authentication method. Options: `default` (DefaultAzureCredential), `mi` (Managed Identity), `connstr` (Connection String), `sas` (SAS Token). |
+| `--azure-account` | string | `$AZURE_STORAGE_ACCOUNT` | Storage account name (required for `default`, `mi`, and `key` auth). |
+| `--azure-prefix` | string | `lastfm/` | Blob prefix (folder path). Blobs written to `{prefix}dt=YYYY-MM-DD/{user}-YYYYMMDD-HHMMSS.ndjson`. |
+| `--azure-auth` | string | `default` | Authentication method. Options: `default`, `mi`, `connstr`, `key`, `sas`. |
+| `--azure-account-key` | string | none | **Sensitive**. Storage account key (required for `--azure-auth key`). |
 | `--azure-container-url` | string | none | Full container URL with SAS token (for `--azure-auth sas`). |
 
 **Azure Authentication Methods**:
 
 - **`default`** (recommended for Azure VMs/AKS): Uses Azure SDK's DefaultAzureCredential chain (tries Managed Identity, Azure CLI, Environment, etc.).
 - **`mi`**: Explicitly uses Managed Identity (Azure VMs, AKS, Container Instances with identity).
 - **`connstr`**: Uses connection string from `AZURE_STORAGE_CONNECTION_STRING` environment variable.
+- **`key`**: Uses storage account key from `--azure-account-key` flag or `AZURE_STORAGE_ACCOUNT_KEY` environment variable. Requires `--azure-account`.
 - **`sas`**: Uses SAS token embedded in `--azure-container-url`.
 
 #### Watermark Storage
@@ -126,6 +130,7 @@ Required when `--output azure`:
 | Flag | Type | Default | Description |
 |------|------|---------|-------------|
 | `--log-level` | string | `info` | Logging verbosity. Options: `info`, `debug`. |
+| `--no-progress` | boolean | `false` | Disable console progress bar. Useful for CI/CD pipelines or when redirecting output. |
 | `--dry-run` | boolean | `false` | Preview mode. Fetches data but doesn't write to storage or update watermarks. |
 
 ---
@@ -292,6 +297,20 @@ lastfm-sync fetch --user alice \
   --azure-auth mi
 ```
 
+### Azure with Account Key
+
+```bash
+export LASTFM_API_KEY="your-api-key"
+export AZURE_STORAGE_ACCOUNT_KEY="your-account-key"
+lastfm-sync fetch --user alice \
+  --output azure \
+  --azure-container scrobbles \
+  --azure-account mystorageaccount \
+  --azure-auth key
+# Or pass key directly via flag:
+# --azure-account-key "your-account-key"
+```
+
 ### Time Range Fetch
 
 ```bash
@@ -320,6 +339,43 @@ lastfm-sync fetch --user alice --log-level debug
 
 ---
 
+## Output Format
+
+### NDJSON Structure
+
+Each scrobble is written as a single line of JSON with the following fields:
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `username` | string | Last.fm username |
+| `artist` | string | Artist name |
+| `track` | string | Original track name from Last.fm (may include annotations) |
+| `normalized_title` | string | Clean track title with annotations removed (Live, Remastered, featuring, etc.) |
+| `album` | string | Album name |
+| `uts` | int64 | Unix timestamp (seconds since epoch) |
+| `local_time` | string | Human-readable UTC timestamp (RFC3339 format) |
+| `mbid` | string | MusicBrainz ID (omitted if null) |
+| `source` | string | Data source (always "lastfm") |
+| `ingested_at` | string | UTC timestamp when record was created (RFC3339) |
+| `raw` | object | Original Last.fm API response for debugging |
+
+**Example:**
+```json
+{"username":"alice","artist":"The Beatles","track":"Come Together - Remastered","normalized_title":"Come Together","album":"Abbey Road","uts":1704067200,"local_time":"2024-01-01T00:00:00Z","source":"lastfm","ingested_at":"2024-01-06T14:30:22Z","raw":{}}
+```
+
+**Note on normalized_title**: This field is automatically generated by removing common annotations like:
+- Remaster/Remastered annotations (e.g., "2009 Remaster", "Remastered 2015")
+- Live performance markers (e.g., "Live at Wembley", "Live 1969")
+- Version/edit labels (e.g., "Radio Edit", "Extended Version")
+- Date/year markers in parentheses or brackets
+- Remix labels (e.g., "Dave's Remix")
+- Featuring/collaboration markers (e.g., "feat. Artist", "with Orchestra")
+
+This normalization improves data quality for aggregation and matching while preserving the original title in the `track` field.
+
+---
+
 ## Security Best Practices
 
 1. **Never commit secrets to version control**:
