Add Normalize Command for Re-normalizing Scrobble Files

feat: Add normalize command with Azure Blob Storage support

Author: wesback

.github/copilot-instructions.md

@@ -7,6 +7,7 @@ Auto-generated from all feature plans. Last updated: 2026-01-06
 - N/A (UI component only) (005-console-progress-bar)
 - Go 1.24.0+ (alpine-based Docker build) (002-containerization-documentation)
 - Go 1.24.0+ + `crypto/sha256`, `bufio.Scanner`, `encoding/json` (006-scrobble-dedup-merge)
+- Local filesystem and Azure Blob Storage (via existing `writer` abstraction) (007-normalize-command)
 
 ## Project Structure
 
@@ -50,10 +51,9 @@ Go 1.24.0+ (006-scrobble-dedup-merge): Follow standard Go conventions
   - Table-driven tests for strategy variations
 
 ## Recent Changes
+- 007-normalize-command: Added Go 1.24.0+
 - 006-scrobble-dedup-merge: Added merge command for deduplicating and merging multiple NDJSON scrobble files. Uses in-memory hash map with SHA256 keys. Supports 4 deduplication strategies (default/strict/relaxed/mbid) and 3 conflict resolution modes (completeness/first/last). Includes checkpointing for resume capability. Performance targets: ≥10K scrobbles/sec, <500MB for 1M records. Reuses existing internal/writer, internal/progress, internal/models packages.
 - 005-console-progress-bar: Added Go 1.24.0+ + `github.com/schollz/progressbar/v3`, `golang.org/x/term`
-- 004-normalized-title-field: Adding `normalized_title` field to remove annotations (Live, Remastered, featuring, etc.) from track titles for better matching and grouping. Uses internal/normalize package with gopkg.in/yaml.v3 for configuration. DEBUG logging when titles modified.
-- 002-containerization-documentation: Added [if applicable, e.g., PostgreSQL, CoreData, files or N/A]
 
 <!-- MANUAL ADDITIONS START -->
 If you notice any systemic issues please add the needed requirements to this file or to the constitution if that is more appropriate.

.specify/specs/001-normalize-command/spec.md

@@ -0,0 +1,109 @@
+# Feature Specification: Normalize Command
+
+**Feature Branch**: `001-normalize-command`  
+**Created**: 2026-01-08  
+**Status**: Draft  
+**Input**: User description: "Add a new normalize command to process all JSON files for a specified user and update the normalized_title field by reapplying normalization logic to the track field"
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Re-normalize User's Scrobble Files (Priority: P1)
+
+A data administrator needs to update the normalized_title field for all existing scrobble files after improvements or fixes have been made to the normalization logic. They want to ensure all historical data uses the latest normalization rules without having to re-fetch data from Last.fm.
+
+**Why this priority**: Core functionality - enables retroactive application of normalization improvements to existing data, which is the primary purpose of the command.
+
+**Independent Test**: Can be fully tested by running the normalize command on a user's existing files and verifying that normalized_title fields are updated correctly according to current normalization rules, delivering immediate value of consistent data normalization.
+
+**Acceptance Scenarios**:
+
+1. **Given** a user has 100 JSON files with scrobbles in local storage, **When** the administrator runs `./app normalize --user john_doe`, **Then** all 100 files are processed and normalized_title fields are updated based on current normalization rules
+2. **Given** a user has scrobble files in Azure Blob Storage, **When** the administrator runs `./app normalize --user jane_doe --azure-account myaccount --azure-container scrobbles`, **Then** all files in Azure storage are processed and updated with new normalized_title values
+3. **Given** some files already have correct normalized_title values, **When** the normalize command runs, **Then** only files with changed normalized_title values are updated, unchanged files are left as-is
+4. **Given** a file contains scrobbles where track field is "Track #1 - Some Title", **When** normalization is applied, **Then** normalized_title is updated to "track 1 some title" (lowercased, special characters removed)
+
+---
+
+### User Story 2 - Preview Changes Before Applying (Priority: P2)
+
+A data administrator wants to see what changes would be made to normalized_title fields before actually modifying the files, to verify the normalization logic is working as expected and to estimate impact.
+
+**Why this priority**: Important safety feature - allows verification before making bulk changes to data files.
+
+**Independent Test**: Can be fully tested by running normalize command with --dry-run flag and confirming that preview output is shown but no files are modified, providing immediate value of safe verification.
+
+**Acceptance Scenarios**:
+
+1. **Given** a user has 50 files needing normalization updates, **When** the administrator runs `./app normalize --user john_doe --dry-run`, **Then** the system displays which files would be updated showing current and new normalized_title values, but does not write any changes
+2. **Given** files are in Azure storage, **When** the administrator runs normalize with --dry-run and Azure flags, **Then** preview is shown without modifying Azure storage
+3. **Given** dry-run mode is active, **When** processing completes, **Then** the summary clearly indicates "Dry-run mode: No changes written to storage"
+
+---
+
+### User Story 3 - Monitor Progress and Review Results (Priority: P3)
+
+A data administrator processing hundreds of files wants to see real-time progress during processing and a comprehensive summary afterward to understand what was changed and identify any issues.
+
+**Why this priority**: Enhances user experience - provides visibility and confidence during long-running operations.
+
+**Independent Test**: Can be fully tested by running normalize on a large dataset and verifying progress indicators appear during execution and comprehensive summary is shown at completion.
+
+**Acceptance Scenarios**:
+
+1. **Given** processing 200 files, **When** the normalize command runs, **Then** progress is displayed showing which file is currently being processed
+2. **Given** processing completes successfully, **When** the command finishes, **Then** a summary shows total files processed, number updated, number unchanged, and any errors encountered
+3. **Given** 5 files fail to parse during processing, **When** the command completes, **Then** the error count is 5 and processing continues for remaining files
+4. **Given** processing a mix of files with and without changes, **When** the summary is displayed, **Then** it accurately categorizes files as "updated" or "unchanged"
+
+---
+
+### Edge Cases
+
+- What happens when a file cannot be parsed (malformed JSON)?
+- What happens when a file is missing the track field?
+- What happens when no files exist for the specified user?
+- What happens when normalized_title already matches the newly calculated value?
+- What happens when storage permissions prevent reading or writing files?
+- What happens when the user specifies both local and Azure flags (conflicting storage targets)?
+- What happens when Azure credentials are invalid or the container doesn't exist?
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: System MUST provide a `normalize` command that accepts `--user <username>` as a required argument
+- **FR-002**: System MUST support local storage mode when no Azure arguments are provided
+- **FR-003**: System MUST support Azure Blob Storage mode when Azure account and container arguments are provided
+- **FR-004**: System MUST locate all JSON/NDJSON files for the specified user in the determined storage location
+- **FR-005**: System MUST read each file, extract the `track` field, and apply existing normalization logic to generate a new `normalized_title` value
+- **FR-006**: System MUST update only the `normalized_title` field in each scrobble record, preserving all other fields unchanged
+- **FR-007**: System MUST write updated files back to the same storage location (local or Azure) unless dry-run mode is active
+- **FR-008**: System MUST support a `--dry-run` flag that shows what would change without modifying any files
+- **FR-009**: System MUST display real-time progress showing which file is currently being processed
+- **FR-010**: System MUST generate a summary report showing total files processed, number updated, number unchanged, and error count
+- **FR-011**: System MUST continue processing remaining files when individual files fail to parse or process
+- **FR-012**: System MUST report all errors encountered during processing in the summary
+- **FR-013**: System MUST clearly indicate in output when dry-run mode is active and no changes are written
+- **FR-014**: System MUST use the same Azure configuration pattern and argument names as existing fetch and merge commands
+- **FR-015**: System MUST use the same storage abstraction layer as existing commands for consistency
+- **FR-016**: System MUST handle files that already have correct normalized_title values by skipping updates for those files
+- **FR-017**: System MUST display both current and new normalized_title values during dry-run mode for files that would change
+- **FR-018**: System MUST validate that required user argument is provided and error appropriately if missing
+- **FR-019**: System MUST validate Azure configuration when Azure mode is used and error appropriately if incomplete or invalid
+
+### Key Entities
+
+- **Scrobble File**: Represents a JSON/NDJSON file containing scrobble records for a user, stored in either local filesystem or Azure Blob Storage
+- **Scrobble Record**: Individual listening event containing fields including track (original title) and normalized_title (processed title)
+- **Storage Location**: Either local filesystem or Azure Blob Storage container, determined by command-line arguments provided
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: Administrator can process all files for a user in under 5 seconds per 1000 files
+- **SC-002**: System correctly identifies and updates 100% of files where normalized_title differs from newly calculated value
+- **SC-003**: Zero data loss - all fields except normalized_title remain unchanged after processing
+- **SC-004**: Dry-run mode produces accurate preview - 100% match between preview and actual changes when run without --dry-run
+- **SC-005**: System continues processing and completes successfully even when up to 10% of files encounter parsing errors
+- **SC-006**: Summary report provides complete accounting - sum of updated, unchanged, and error counts equals total files processed

.specify/specs/007-normalize-command/checklists/requirements.md

@@ -0,0 +1,79 @@
+# Specification Quality Checklist: Normalize Command
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-01-08
+**Feature**: [../spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Validation Results
+
+### Content Quality Assessment
+✅ **PASS**: Specification contains no implementation details about Go, command structure, or storage implementations. Focuses entirely on what the feature does from user perspective.
+
+✅ **PASS**: All content emphasizes user value (data consistency, verification, visibility) and business needs (retroactive normalization, safe bulk operations).
+
+✅ **PASS**: Language is accessible to non-technical stakeholders - describes scenarios in terms of data files, users, and business outcomes.
+
+✅ **PASS**: All mandatory sections (User Scenarios & Testing, Requirements, Success Criteria) are complete with detailed content.
+
+### Requirement Completeness Assessment
+✅ **PASS**: No [NEEDS CLARIFICATION] markers present - all requirements are fully specified with reasonable defaults assumed.
+
+✅ **PASS**: All 19 functional requirements are testable and unambiguous. Each FR specifies a concrete capability or behavior that can be verified.
+
+✅ **PASS**: All 6 success criteria include specific metrics (5 seconds per 1000 files, 100% accuracy, zero data loss, etc.).
+
+✅ **PASS**: Success criteria avoid implementation details - metrics focus on user-observable outcomes like processing speed and accuracy, not internal mechanisms.
+
+✅ **PASS**: Each user story includes 3-4 detailed acceptance scenarios in Given/When/Then format covering main flows and variations.
+
+✅ **PASS**: Edge cases section identifies 7 specific boundary conditions and error scenarios.
+
+✅ **PASS**: Scope is clearly bounded - command operates on existing files, updates only normalized_title field, supports local and Azure storage.
+
+✅ **PASS**: Assumptions are implicit but reasonable (reuse existing normalization logic, same storage patterns as fetch/merge, existing user files).
+
+### Feature Readiness Assessment
+✅ **PASS**: All 19 functional requirements map to acceptance scenarios in user stories (FR-001 to FR-019 covered).
+
+✅ **PASS**: Three prioritized user stories cover core functionality (P1), safety features (P2), and user experience (P3).
+
+✅ **PASS**: Success criteria align with feature goals - processing speed, accuracy, data integrity, error handling.
+
+✅ **PASS**: No implementation leakage - specification describes behavior and outcomes without prescribing technical solutions.
+
+## Summary
+
+**Status**: ✅ READY FOR PLANNING
+
+All checklist items passed validation. The specification is complete, unambiguous, and focused on user value. No implementation details are present. The feature is ready to proceed to `/speckit.plan` or `/speckit.clarify` phases.
+
+## Notes
+
+- Specification assumes reuse of existing normalization logic from internal/normalize package (reasonable assumption based on project context)
+- Azure storage integration assumes same patterns as fetch/merge commands (explicitly stated in requirements)
+- No clarifications needed - all requirements have reasonable defaults based on existing application patterns

.specify/specs/007-normalize-command/contracts/cli-interface.md

@@ -0,0 +1,169 @@
+# CLI Contract: Normalize Command
+
+**Feature**: 007-normalize-command  
+**Date**: 2026-01-08  
+**Type**: Command-Line Interface
+
+## Command Signature
+
+```bash
+lastfm-sync normalize --user <username> [options]
+```
+
+## Required Arguments
+
+| Argument | Type | Description | Validation |
+|----------|------|-------------|------------|
+| `--user` | string | Username whose files to process | MUST be non-empty (FR-018) |
+
+## Optional Arguments - Storage Selection
+
+### Local Storage (default)
+No additional arguments required. Uses current working directory or configured base path.
+
+### Azure Blob Storage
+When ANY Azure argument is provided, Azure mode is activated (FR-003).
+
+| Argument | Type | Description | Validation |
+|----------|------|-------------|------------|
+| `--azure-account` | string | Azure storage account name | Required when Azure mode active (FR-019) |
+| `--azure-container` | string | Azure container name | Required when Azure mode active (FR-019) |
+| `--azure-prefix` | string | Blob prefix for file discovery | Optional, default: "" |
+| `--azure-auth` | string | Authentication method: "key", "sas", "default" | Optional, default: "default" |
+| `--azure-account-key` | string | Storage account key (if auth=key) | Required when auth=key |
+| `--azure-sas-token` | string | SAS token (if auth=sas) | Required when auth=sas |
+
+**Validation**: Azure mode requires `--azure-account` AND `--azure-container` at minimum (FR-019).
+
+## Optional Arguments - Behavior
+
+| Argument | Type | Description | Default |
+|----------|------|-------------|---------|
+| `--dry-run` | boolean | Preview changes without writing | false |
+| `--log-level` | string | Logging level: debug, info, warn, error | info |
+
+## Output Format
+
+### Progress Display (stdout)
+
+Per-file progress (FR-009):
+```
+Processing files for user: {username}
+Storage: {Local: /path/to/data | Azure: account/container}
+
+Processing: episode_001.ndjson [1/150]
+  Current: "Track #1 - Some Title"
+  New:     "track 1 some title"
+  Status:  {Updated | No change needed}
+
+Processing: episode_002.ndjson [2/150]
+  Status:  No change needed
+
+...
+```
+
+### Summary Report (stdout)
+
+```
+Summary:
+  Total files: 150
+  Updated: 45
+  Unchanged: 105
+  Errors: 0
+  Duration: 2.3s
+
+{if --dry-run}
+Dry-run mode: No changes written to storage
+{/if}
+
+{if errors}
+Errors encountered:
+  - username_042.ndjson: parse error
+  - username_099.ndjson: permission denied
+{/if}
+```
+
+### Error Output (stderr)
+
+Individual file errors logged at DEBUG level, summary errors at INFO level.
+
+## Exit Codes
+
+| Code | Meaning | Scenario |
+|------|---------|----------|
+| 0 | Success | All files processed (some may have errors but processing completed) |
+| 1 | General error | Unexpected error during command execution |
+| 2 | Validation error | Missing required arguments or invalid configuration (FR-018, FR-019) |
+
+**Note**: File-level errors (parse errors, missing fields) do NOT cause non-zero exit code per FR-011 (continue processing). Only configuration/validation errors exit early.
+
+## Examples
+
+### Local Storage
+
+```bash
+# Basic usage - normalize all files for user
+./lastfm-sync normalize --user john_doe
+
+# Dry-run preview
+./lastfm-sync normalize --user john_doe --dry-run
+
+# With debug logging
+./lastfm-sync normalize --user john_doe --log-level debug
+```
+
+### Azure Storage
+
+```bash
+# Azure with default authentication (managed identity/Azure CLI)
+./lastfm-sync normalize --user jane_doe \
+  --azure-account myaccount \
+  --azure-container scrobbles
+
+# Azure with account key
+./lastfm-sync normalize --user jane_doe \
+  --azure-account myaccount \
+  --azure-container scrobbles \
+  --azure-auth key \
+  --azure-account-key "abc123..."
+
+# Azure dry-run
+./lastfm-sync normalize --user jane_doe \
+  --azure-account myaccount \
+  --azure-container scrobbles \
+  --dry-run
+```
+
+## File Discovery Pattern
+
+Matching pattern: `{username}_*.ndjson` (FR-004)
+
+### Local Storage
+- Search in configured base directory
+- Example: If base is `/data`, search `/data/john_doe_*.ndjson`
+- Recursive search NOT performed (flat directory expected)
+
+### Azure Storage
+- List blobs with prefix `{azure-prefix}{username}_`
+- Example: If prefix is `lastfm/`, search blobs like `lastfm/john_doe_*.ndjson`
+- Blob name extraction: Use blob name as displayed filename
+
+## Alignment with Existing Commands
+
+**Follows patterns from**:
+- `fetch` command: Azure argument structure, authentication modes
+- `merge` command: Progress display, NDJSON processing, summary format
+
+**Consistency**:
+- Same Azure flag names and behavior (FR-014)
+- Same logging configuration
+- Same progress library usage
+- Same error handling patterns
+
+## Non-Functional Contracts
+
+- **Performance**: Process ≥1000 files in under 5 seconds (SC-001)
+- **Memory**: Streaming processing, O(1) memory per file
+- **Reliability**: Continue on individual file errors (FR-011, SC-005)
+- **Safety**: Dry-run produces accurate preview (SC-004)
+- **Idempotency**: Running multiple times produces same result (normalized_title stabilizes)