diff --git a/.github/CODEQL-WORKFLOW.md b/.github/CODEQL-WORKFLOW.md index e3ed3cbf6..e5084ec53 100644 --- a/.github/CODEQL-WORKFLOW.md +++ b/.github/CODEQL-WORKFLOW.md @@ -1,136 +1,143 @@ # CodeQL Workflow Configuration -## Current State: Dual CodeQL Setup - -This repository currently has **two CodeQL analysis configurations** running: - -### 1. Custom CodeQL Workflow (`.github/workflows/codeql.yml`) -- **Location**: `.github/workflows/codeql.yml` -- **Status**: Active, but results NOT uploaded to GitHub Security -- **Configuration**: Custom setup with advanced features -- **Languages**: C# and JavaScript/TypeScript (matrix strategy) -- **Custom Config**: `.github/codeql/codeql-config.yml` (path filtering) -- **Upload Setting**: `upload: false` (to avoid conflicts with default setup) - -### 2. GitHub Default Setup -- **Location**: Enabled in repository settings (no visible file) -- **Status**: Active (inferred from workflow comments) -- **Configuration**: GitHub-managed default -- **Languages**: Auto-detected -- **Results**: Uploaded to GitHub Security tab - -## Why This Exists - -The comment in `codeql.yml` lines 78-80 states: -```yaml -# Prevent workflow failure when GitHub "default setup" code scanning is enabled. -# If you want advanced results uploaded, disable default setup or set upload: true. -upload: false +Melodee uses one version-controlled CodeQL advanced setup. The workflow in +`.github/workflows/codeql.yml` uploads results for every supported application +language to GitHub code scanning. + +## Scanning Coverage + +The workflow runs for pushes and pull requests targeting `main`, every Sunday +at 00:00 UTC, and on manual dispatch. Its matrix analyzes: + +- GitHub Actions workflows using the interpreted `none` build mode. +- C# using `autobuild`, so CodeQL observes the compiled solution. +- JavaScript and TypeScript using the interpreted `none` build mode. +- Python using the interpreted `none` build mode and its local threat model. + +The shared configuration in `.github/codeql/codeql-config.yml` excludes +documentation, tests, benchmarks, and vendored minified JavaScript from +interpreted-language analysis. C#, JavaScript/TypeScript, and Actions use the +default remote threat model. Python uses +`.github/codeql/codeql-python-config.yml`, which extends the default model with +local files, command-line arguments, environment variables, and other local +sources used by maintenance scripts. Neither configuration excludes C# +findings by rule. + +This split is deliberate. The compiled web application has a remote request +boundary, and treating all application-owned database and filesystem state as +attacker-controlled creates unactionable C# flows. Python maintenance tools +really do accept local files, environment variables, and command-line values +from an administrator, so the local threat model accurately represents their +trust boundary. Do not enable `threat-models: local` globally to make the two +very different execution models look uniform. + +The local models are intentionally narrow. They identify +`LogSanitizer.Sanitize` as a barrier only for the `log-injection` flow kind and +`ConfigurationLogRedactor.RedactValue` as a barrier only when values would be +persisted to an external location such as a log. They are packaged under +`.github/codeql/extensions/log-sanitizer` so advanced and default setup can +discover them. The `.yml` suffix is intentional because the pack manifest only +loads model files matching `models/**/*.yml`. +Repository-wide query exclusions must not be added to hide a false positive. +Prefer a source fix or a precise CodeQL model; dismiss an individual alert only +when neither option accurately represents the behavior. + +All workflow actions are pinned to verified full commit SHAs. Keep the release +comments beside those SHAs when updating the pins so reviews can identify the +corresponding upstream release. + +## Required Repository Configuration + +GitHub default setup must remain disabled while this advanced workflow is +active. Enabling both configurations can disable advanced uploads, duplicate +work, and leave alert status attached to a configuration that no longer runs. + +To verify the setting: + +1. Open the repository settings and select **Advanced Security** or + **Code security and analysis**. +2. Find **CodeQL analysis** under code scanning. +3. Confirm default setup is disabled and the checked-in advanced workflow is + enabled. + +The REST API reports the same state through: + +```shell +gh api repos/melodee-project/melodee/code-scanning/default-setup ``` -This indicates that both setups were running, causing conflicts. The custom workflow was modified to NOT upload results to avoid workflow failures. - -## Problem - -Having two CodeQL setups is inefficient and confusing: -- ❌ Duplicate analysis runs (wastes CI/CD minutes) -- ❌ Custom workflow results not visible in Security tab -- ❌ Configuration spread across multiple systems -- ❌ Unclear which analysis results are authoritative - -## Recommended Solution: Keep Custom Workflow - -**Advantages of the custom workflow:** -1. ✅ **Path filtering** - Excludes docs, vendored JS, benchmarks (via `codeql-config.yml`) -2. ✅ **Explicit language control** - C# and JavaScript/TypeScript -3. ✅ **Version control** - Configuration tracked in Git -4. ✅ **Transparency** - Visible in repository, reviewable in PRs -5. ✅ **Flexibility** - Can add custom queries, modify triggers, adjust timeout -6. ✅ **Already proven** - Working configuration with comprehensive security audit - -**Steps to migrate:** - -### Step 1: Disable GitHub Default Setup -1. Go to repository **Settings** → **Code security and analysis** -2. Find **Code scanning** section -3. Locate **CodeQL analysis** → **Default setup** -4. Click **Disable** or switch to **Advanced** - -> **Note**: You must have admin access to the repository to change these settings. - -### Step 2: Enable Uploads in Custom Workflow -Update `.github/workflows/codeql.yml` line 78-82 to: - -```yaml -- name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v4 - with: - category: "/language:${{ matrix.language }}" - # Upload results to GitHub Security tab - # Default setup has been disabled in repository settings - upload: true -``` - -Remove these lines (they're now unnecessary): -- `upload-database: false` -- `wait-for-processing: false` - -### Step 3: Verify Configuration -1. Commit and push the workflow changes -2. Trigger a workflow run (push to main or create PR) -3. Check **Security** tab → **Code scanning** for results -4. Verify both C# and JavaScript/TypeScript results appear - -## Alternative Solution: Use Default Setup Only - -If you prefer GitHub's managed solution: - -### Step 1: Delete Custom Workflow -```bash -git rm .github/workflows/codeql.yml -git rm .github/codeql/codeql-config.yml # optional, if not needed +The expected `state` is `not-configured`. + +## Removing a Stale Configuration + +An alert can remain open after its code is fixed when it belongs to an older +configuration that no longer runs. Adding the language to this workflow does +not update the stale configuration's result. + +After the advanced workflow has completed successfully on `main`: + +1. Open **Security and quality** > **Code scanning**. +2. Open **Tool status**, or open the alert and inspect **Affected branches** > + **Configurations analyzing** for `main`. +3. Retain `.github/workflows/codeql.yml:analyze` for each matrix language. +4. Delete the stale `dynamic/github-code-scanning/codeql:analyze` and + `dynamic/github-code-scanning/codeql:upload` configurations left by the + former default setup. Retain every checked-in workflow configuration. +5. Re-run the CodeQL workflow if an active configuration was removed by + mistake. + +This administrative cleanup is required for findings that exist only in the +former default-setup configuration; a repository commit cannot delete that +server-side configuration. + +The stale Python configurations must not be removed before the checked-in +Python matrix job succeeds on `main`. Otherwise, the repository can temporarily +lose active Python coverage rather than transferring it to the advanced +workflow. + +## Verification + +The July 2026 local verification reports: + +- the six original C# alerts have source fixes and regression coverage, while + the Python setup alert has a hardened necessary sink and narrow documented + suppression; +- a fresh default-remote C# database moved from eight findings to zero; +- fresh GitHub Actions and JavaScript/TypeScript databases report zero + findings; +- a fresh workflow-equivalent Python database ran all 45 default queries with + the local threat model and reported zero findings and no SARIF warning/error + notifications; and +- the 52-query Python security-extended suite also reported zero findings. + +The Python filesystem audit passes 57 focused tests, and all 109 script tests +pass with `ResourceWarning` promoted to an error. These complete-suite results, +not an earlier targeted-query result, form the local release gate. + +After changing the workflow or CodeQL configuration: + +1. Validate `.github/workflows/codeql.yml` with `actionlint`. +2. Run the workflow manually, or push the change to a branch with a pull + request targeting `main`. +3. Confirm all four matrix jobs complete and upload results. +4. On **Security and quality** > **Code scanning** > **Tool status**, confirm + recent GitHub Actions, C#, JavaScript/TypeScript, and Python analyses are + present. +5. Review new alerts before merging; do not dismiss them solely to make the + check pass. + +Useful command-line checks: + +```shell +gh run list --workflow codeql.yml --limit 10 +gh api --paginate \ + 'repos/melodee-project/melodee/code-scanning/alerts?state=open&tool_name=CodeQL&per_page=100' ``` -### Step 2: Ensure Default Setup is Enabled -1. Repository **Settings** → **Code security and analysis** -2. Ensure **CodeQL analysis** is set to **Default setup** - -**Trade-offs:** -- ✅ Simpler configuration -- ✅ GitHub manages updates -- ❌ Less control over paths analyzed -- ❌ Cannot customize queries easily -- ❌ Configuration not in version control - -## Current Recommendation - -**Keep the custom workflow** because: -1. It already has proven security analysis (see `CODEQL-ANALYSIS.md`) -2. Path filtering reduces noise from docs and vendored code -3. Configuration is version-controlled and reviewable -4. Team has already invested in customizing it - -The only change needed is to **disable GitHub default setup** and **enable uploads** in the custom workflow. - -## Migration Checklist - -- [ ] 1. Disable GitHub default setup in repository settings -- [ ] 2. Update `codeql.yml` to set `upload: true` -- [ ] 3. Remove `upload-database: false` and `wait-for-processing: false` -- [ ] 4. Update comments in workflow file -- [ ] 5. Commit and push changes -- [ ] 6. Verify workflow runs successfully -- [ ] 7. Check Security tab for CodeQL results -- [ ] 8. Update `CODEQL-ANALYSIS.md` if needed - ## References -- [GitHub CodeQL Documentation](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/about-code-scanning-with-codeql) -- [Configuring Code Scanning](https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning) -- [CodeQL Action Documentation](https://github.com/github/codeql-action) -- Project: [CODEQL-ANALYSIS.md](../CODEQL-ANALYSIS.md) - Security audit results -- Project: [SECURITY-FIXES.md](../SECURITY-FIXES.md) - Security improvements - -## Questions? - -If you have questions about this configuration, please open an issue or contact the maintainers. +- [Workflow configuration options for code scanning](https://docs.github.com/en/code-security/reference/code-scanning/workflow-configuration-options) +- [CodeQL for compiled languages](https://docs.github.com/en/code-security/how-tos/find-and-fix-code-vulnerabilities/manage-your-configuration/codeql-for-compiled-languages) +- [Resolving code scanning alerts](https://docs.github.com/en/code-security/how-tos/manage-security-alerts/manage-code-scanning-alerts/resolve-alerts) +- [Secure use of GitHub Actions](https://docs.github.com/en/actions/reference/security/secure-use) +- [Customizing CodeQL library models for C#](https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-csharp/) diff --git a/.github/codeql/codeql-config.yml b/.github/codeql/codeql-config.yml index 89b4bb8a5..852b3989b 100644 --- a/.github/codeql/codeql-config.yml +++ b/.github/codeql/codeql-config.yml @@ -1,50 +1,12 @@ -# CodeQL configuration -# This file configures which paths to analyze for JavaScript/TypeScript +name: CodeQL configuration -name: "CodeQL config" - -# Model extensions define custom sanitizers and sinks for CodeQL analysis -model-extensions: - - ".github/codeql/extensions/log-sanitizer.model.yaml" - -# Query filters to tune analysis results -query-filters: - # Exclude false positives where we have verified mitigations - - exclude: - id: cs/web/cookie-secure-not-set - # MelodeeBlazorCookieMiddleware.cs already sets Secure=true (line 29) - - # Exclude log-forging alerts for code using LogSanitizer - # LogSanitizer.Sanitize() properly sanitizes log input by replacing newlines, - # but CodeQL's log-forging query doesn't recognize custom sanitizer methods. - # The sanitizer implementation (src/Melodee.Common/Utility/LogSanitizer.cs) - # uses String.Replace to remove CR, LF, NEL, LS, and PS characters. - # See: https://github.com/github/codeql/issues/15824 - - exclude: - id: cs/log-forging - - # Exclude cleartext-storage alerts for MpdPlaybackBackend password logging - # The password IS properly masked via GetSafeCommandForLogging() which returns - # "password ***" for any command starting with "password ". CodeQL tracks - # data flow but cannot see through the conditional masking logic. - # File: src/Melodee.Common/Services/Playback/Backends/MpdPlaybackBackend.cs - - exclude: - id: cs/cleartext-storage-of-sensitive-information - -# For JavaScript/TypeScript, only analyze application code in src directory -# Exclude documentation, vendored libraries, and other non-application code +# Path filters apply to interpreted languages. The C# database is determined +# by the sources included in its build. paths-ignore: - # Exclude entire docs directory (contains Jekyll templates and vendored JS) - - 'docs/**' - - 'melodee/docs/**' - - # Exclude vendored/third-party JavaScript libraries - - '**/jquery*.js' - - '**/lunr*.js' - - '**/*.min.js' - - # Exclude benchmark and test directories - - 'benchmarks/**' - - 'tests/**' - - 'melodee/benchmarks/**' - - 'melodee/tests/**' + - "benchmarks/**" + - "docs/**" + - "tests/**" + - "**/tests/**" + - "**/*.min.js" + - "**/jquery*.js" + - "**/lunr*.js" diff --git a/.github/codeql/codeql-python-config.yml b/.github/codeql/codeql-python-config.yml new file mode 100644 index 000000000..71604d9e3 --- /dev/null +++ b/.github/codeql/codeql-python-config.yml @@ -0,0 +1,16 @@ +name: CodeQL Python configuration + +# Maintenance scripts accept files, command-line arguments, and environment +# variables. Extend Python's default remote model so those local sources are +# analyzed under the local threat model independently from the compiled web +# application's remote request boundary. +threat-models: local + +paths-ignore: + - "benchmarks/**" + - "docs/**" + - "tests/**" + - "**/tests/**" + - "**/*.min.js" + - "**/jquery*.js" + - "**/lunr*.js" diff --git a/.github/codeql/extensions/log-sanitizer.model.yaml b/.github/codeql/extensions/log-sanitizer.model.yaml deleted file mode 100644 index 1c5db84ae..000000000 --- a/.github/codeql/extensions/log-sanitizer.model.yaml +++ /dev/null @@ -1,23 +0,0 @@ -# CodeQL Model Extension for Melodee LogSanitizer -# This file tells CodeQL to recognize LogSanitizer.Sanitize() as a sanitizer -# for log-forging vulnerabilities. -# -# The "value" kind (instead of "taint") indicates that data flows through the method -# but taint does NOT propagate. This effectively makes the method a sanitizer/barrier. -# -# Reference: https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-csharp/ - -extensions: - - addsTo: - pack: codeql/csharp-all - extensible: summaryModel - data: - # LogSanitizer.Sanitize(string?) sanitizes log input by replacing newlines - # Using "value" kind means data flows but taint is cleansed - - ["Melodee.Common.Utility", "LogSanitizer", False, "Sanitize", "(System.String)", "", "Argument[0]", "ReturnValue", "value", "manual"] - # LogSanitizer.SanitizeObject(object?) sanitizes log input - - ["Melodee.Common.Utility", "LogSanitizer", False, "SanitizeObject", "(System.Object)", "", "Argument[0]", "ReturnValue", "value", "manual"] - # LogSanitizer.MaskEmail(string?) masks sensitive data - - ["Melodee.Common.Utility", "LogSanitizer", False, "MaskEmail", "(System.String)", "", "Argument[0]", "ReturnValue", "value", "manual"] - # LogSanitizer.MaskIdentifier(string?, int) masks sensitive data - - ["Melodee.Common.Utility", "LogSanitizer", False, "MaskIdentifier", "(System.String,System.Int32)", "", "Argument[0]", "ReturnValue", "value", "manual"] diff --git a/.github/codeql/extensions/log-sanitizer/codeql-pack.yml b/.github/codeql/extensions/log-sanitizer/codeql-pack.yml new file mode 100644 index 000000000..f58ff7f29 --- /dev/null +++ b/.github/codeql/extensions/log-sanitizer/codeql-pack.yml @@ -0,0 +1,7 @@ +name: melodee-project/log-sanitizer-models +version: 0.0.0 +library: true +extensionTargets: + codeql/csharp-all: "*" +dataExtensions: + - "models/**/*.yml" diff --git a/.github/codeql/extensions/log-sanitizer/models/log-sanitizer.model.yml b/.github/codeql/extensions/log-sanitizer/models/log-sanitizer.model.yml new file mode 100644 index 000000000..8df6dd943 --- /dev/null +++ b/.github/codeql/extensions/log-sanitizer/models/log-sanitizer.model.yml @@ -0,0 +1,12 @@ +# LogSanitizer.Sanitize replaces every supported line-ending character before +# returning its value. Model that return value only as a log-injection barrier; +# do not suppress the log-forging query or mark unrelated masking helpers safe. +# ConfigurationLogRedactor.RedactValue is deny-by-default and permits only +# sanitized operational metadata, so its return is safe for external storage. +extensions: + - addsTo: + pack: codeql/csharp-all + extensible: barrierModel + data: + - ["Melodee.Common.Utility", "LogSanitizer", false, "Sanitize", "(System.String)", "", "ReturnValue", "log-injection", "manual"] + - ["Melodee.Common.Configuration", "ConfigurationLogRedactor", false, "RedactValue", "(System.String,System.Object)", "", "ReturnValue", "file-content-store", "manual"] diff --git a/.github/docker/Dockerfile.container-scan b/.github/docker/Dockerfile.container-scan deleted file mode 100644 index fb00ca45b..000000000 --- a/.github/docker/Dockerfile.container-scan +++ /dev/null @@ -1,16 +0,0 @@ -ARG DOTNET_ASPNET_IMAGE=mcr.microsoft.com/dotnet/aspnet:10.0 -FROM ${DOTNET_ASPNET_IMAGE} - -WORKDIR /app - -RUN apt-get update && \ - apt-get install -y --no-install-recommends \ - ffmpeg \ - postgresql-client \ - curl \ - lbzip2 \ - && rm -rf /var/lib/apt/lists/* - -COPY scan/publish/ . - -ENTRYPOINT ["dotnet", "server.dll"] diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml index 4483a2ed6..66f8bee4b 100644 --- a/.github/workflows/codeql.yml +++ b/.github/workflows/codeql.yml @@ -1,86 +1,59 @@ -# For most projects, this workflow file will not need changing; you simply need -# to commit it to your repository. -# -# You may wish to alter this file to override the set of languages analyzed, -# or to provide custom queries or build logic. -# -# ******** NOTE ******** -# We have attempted to detect the languages in your repository. Please check -# the `language` matrix defined below to confirm you have the correct set of -# supported CodeQL languages. -# -# ******** IMPORTANT ******** -# This workflow is configured to upload results to GitHub Security. -# If you also have GitHub's "default setup" enabled, you should disable it -# to avoid duplicate scans. See .github/CODEQL-WORKFLOW.md for details. -# -name: "CodeQL" +name: CodeQL on: push: - branches: [ "main" ] + branches: + - main pull_request: - branches: [ "main" ] + branches: + - main schedule: - - cron: '0 0 * * 0' # Weekly scan on Sundays at midnight UTC + - cron: "0 0 * * 0" + workflow_dispatch: permissions: - actions: read contents: read - security-events: write jobs: analyze: - name: Analyze + name: Analyze (${{ matrix.language }}) runs-on: ubuntu-latest timeout-minutes: 120 - + permissions: + actions: read + contents: read + security-events: write strategy: fail-fast: false matrix: - language: [ 'csharp', 'javascript-typescript' ] - # CodeQL supports [ 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift' ] - # Use only 'java-kotlin' to analyze code written in Java, Kotlin or both - # Use only 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both - # Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support + include: + - language: actions + build-mode: none + config-file: ./.github/codeql/codeql-config.yml + - language: csharp + build-mode: autobuild + config-file: ./.github/codeql/codeql-config.yml + - language: javascript-typescript + build-mode: none + config-file: ./.github/codeql/codeql-config.yml + - language: python + build-mode: none + config-file: ./.github/codeql/codeql-python-config.yml steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0 + with: + persist-credentials: false - # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL - uses: github/codeql-action/init@v4 + uses: github/codeql-action/init@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0 with: languages: ${{ matrix.language }} - config-file: ./.github/codeql/codeql-config.yml - # If you wish to specify custom queries, you can do so here or in a config file. - # By default, queries listed here will override any specified in a config file. - # Prefix the list here with "+" to use these queries and those in the config file. - - # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs - # queries: security-extended,security-and-quality - - # Autobuild attempts to build any compiled languages (C/C++, C#, Go, Java, or Swift). - # If this step fails, then you should remove it and run the build manually (see below) - - name: Autobuild - uses: github/codeql-action/autobuild@v4 - - # ℹ️ Command-line programs to run using the OS shell. - # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun - - # If the Autobuild fails above, remove it and uncomment the following three lines. - # modify them (or add more) to build your code if your project, please refer to the EXAMPLE below for guidance. - - # - run: | - # echo "Run, Build Application using script" - # ./location_of_script_within_repo/buildscript.sh + build-mode: ${{ matrix.build-mode }} + config-file: ${{ matrix.config-file }} - - name: Perform CodeQL Analysis - uses: github/codeql-action/analyze@v4 + - name: Perform CodeQL analysis + uses: github/codeql-action/analyze@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0 with: - category: "/language:${{ matrix.language }}" - # Upload results to GitHub Security tab - # Note: If GitHub "default setup" is also enabled, disable it to avoid duplicate scans - # See .github/CODEQL-WORKFLOW.md for detailed configuration guidance - upload: true + category: /language:${{ matrix.language }} diff --git a/.github/workflows/docker-publish.yml b/.github/workflows/docker-publish.yml index fb7289843..3700fd098 100644 --- a/.github/workflows/docker-publish.yml +++ b/.github/workflows/docker-publish.yml @@ -29,7 +29,7 @@ jobs: - name: Extract Docker metadata id: meta - uses: docker/metadata-action@v6 + uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6.2.0 with: images: ${{ steps.image.outputs.name }} tags: | @@ -56,13 +56,13 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v6 + uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - name: Log in to GitHub Container Registry - uses: docker/login-action@v4 + uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0 with: registry: ghcr.io username: ${{ github.actor }} @@ -70,7 +70,7 @@ jobs: - name: Build and push Docker image by digest id: build - uses: docker/build-push-action@v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7.3.0 with: context: . platforms: ${{ matrix.platform }} @@ -79,15 +79,24 @@ jobs: outputs: type=image,name=${{ needs.prepare.outputs.image }},name-canonical=true,push-by-digest=true,push=true cache-from: type=gha,scope=${{ matrix.platform-id }} cache-to: type=gha,mode=max,scope=${{ matrix.platform-id }} + no-cache-filters: final - name: Export image digest + env: + IMAGE_DIGEST: ${{ steps.build.outputs.digest }} run: | + set -euo pipefail + + if [[ ! "$IMAGE_DIGEST" =~ ^sha256:[a-f0-9]{64}$ ]]; then + echo "Unexpected image digest format" >&2 + exit 1 + fi + mkdir -p /tmp/digests - digest="${{ steps.build.outputs.digest }}" - touch "/tmp/digests/${digest#sha256:}" + touch "/tmp/digests/${IMAGE_DIGEST#sha256:}" - name: Upload image digest - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: digest-${{ matrix.platform-id }} path: /tmp/digests/* @@ -102,17 +111,17 @@ jobs: steps: - name: Download image digests - uses: actions/download-artifact@v4 + uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4.3.0 with: path: /tmp/digests pattern: digest-* merge-multiple: true - name: Set up Docker Buildx - uses: docker/setup-buildx-action@v4 + uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0 - name: Log in to GitHub Container Registry - uses: docker/login-action@v4 + uses: docker/login-action@af1e73f918a031802d376d3c8bbc3fe56130a9b0 # v4.4.0 with: registry: ghcr.io username: ${{ github.actor }} @@ -123,16 +132,24 @@ jobs: DOCKER_METADATA_OUTPUT_JSON: ${{ needs.prepare.outputs.json }} IMAGE_NAME: ${{ needs.prepare.outputs.image }} run: | + set -euo pipefail + cd /tmp/digests - # Build the list of digest references - digest_refs="" + tag_args=() + while IFS= read -r tag; do + tag_args+=("-t" "$tag") + done < <(jq -r '.tags[]' <<< "$DOCKER_METADATA_OUTPUT_JSON") + + digest_refs=() for digest in *; do - digest_refs="$digest_refs $IMAGE_NAME@sha256:$digest" + if [[ ! "$digest" =~ ^[a-f0-9]{64}$ ]]; then + echo "Unexpected image digest filename: $digest" >&2 + exit 1 + fi + digest_refs+=("$IMAGE_NAME@sha256:$digest") done - # Create the manifest - docker buildx imagetools create \ - $(jq -cr '.tags | map("-t " + .) | join(" ")' <<< "$DOCKER_METADATA_OUTPUT_JSON") \ - $digest_refs + + docker buildx imagetools create "${tag_args[@]}" "${digest_refs[@]}" - name: Inspect published image env: diff --git a/.github/workflows/dotnet.yml b/.github/workflows/dotnet.yml index c19f767e0..84facfe6b 100644 --- a/.github/workflows/dotnet.yml +++ b/.github/workflows/dotnet.yml @@ -18,9 +18,9 @@ jobs: runs-on: ubuntu-latest steps: - - uses: actions/checkout@v4 + - uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1 with: dotnet-version: 10.0.104 - name: Restore dependencies @@ -80,10 +80,12 @@ jobs: run: | dotnet tool install -g dotnet-reportgenerator-globaltool reportgenerator -reports:"coverage/results/**/coverage.cobertura.xml" -targetdir:"coverage/report" -reporttypes:"Html;TextSummary" -assemblyfilters:"+Melodee.*;+server;+mcli;-Melodee.Tests.*" - echo "## Coverage Summary" >> $GITHUB_STEP_SUMMARY - cat coverage/report/Summary.txt >> $GITHUB_STEP_SUMMARY + { + echo "## Coverage Summary" + cat coverage/report/Summary.txt + } >> "$GITHUB_STEP_SUMMARY" - name: Upload coverage report - uses: actions/upload-artifact@v4 + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 with: name: coverage-report path: coverage/report/ diff --git a/.github/workflows/gitleaks.yml b/.github/workflows/gitleaks.yml index 08e67a6a1..f3d7cbbcc 100644 --- a/.github/workflows/gitleaks.yml +++ b/.github/workflows/gitleaks.yml @@ -25,25 +25,25 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 with: fetch-depth: 0 - name: Run gitleaks on pull request commits if: github.event_name == 'pull_request' - uses: docker://zricethezav/gitleaks:v8.30.0 + uses: docker://zricethezav/gitleaks@sha256:691af3c7c5a48b16f187ce3446d5f194838f91238f27270ed36eef6359a574d9 # v8.30.0 with: args: detect --source . --redact --report-format sarif --report-path gitleaks.sarif --exit-code 1 --log-opts=${{ github.event.pull_request.base.sha }}..${{ github.event.pull_request.head.sha }} - name: Run gitleaks on pushed commits if: github.event_name == 'push' - uses: docker://zricethezav/gitleaks:v8.30.0 + uses: docker://zricethezav/gitleaks@sha256:691af3c7c5a48b16f187ce3446d5f194838f91238f27270ed36eef6359a574d9 # v8.30.0 with: args: detect --source . --redact --report-format sarif --report-path gitleaks.sarif --exit-code 1 --log-opts=${{ github.event.before }}..${{ github.sha }} - name: Upload SARIF results if: always() && hashFiles('gitleaks.sarif') != '' - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0 with: sarif_file: gitleaks.sarif category: gitleaks diff --git a/.github/workflows/localization.yml b/.github/workflows/localization.yml index baa89ffc3..3976800ae 100644 --- a/.github/workflows/localization.yml +++ b/.github/workflows/localization.yml @@ -29,28 +29,30 @@ jobs: steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Validate resource key consistency run: bash scripts/validate-resources.sh - name: Count resource keys run: | - echo "## Resource Key Counts" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - echo "| Language | Key Count |" >> $GITHUB_STEP_SUMMARY - echo "|----------|-----------|" >> $GITHUB_STEP_SUMMARY - - BASE_COUNT=$(grep -o '> $GITHUB_STEP_SUMMARY - - for lang in de-DE es-ES fr-FR it-IT ja-JP pt-BR ru-RU zh-CN ar-SA; do - COUNT=$(grep -o '> $GITHUB_STEP_SUMMARY - done - - echo "" >> $GITHUB_STEP_SUMMARY - echo "✅ All language files validated successfully!" >> $GITHUB_STEP_SUMMARY + { + echo "## Resource Key Counts" + echo "" + echo "| Language | Key Count |" + echo "|----------|-----------|" + + BASE_COUNT=$(grep -o '> "$GITHUB_STEP_SUMMARY" test-localization: name: Run Localization Tests @@ -59,10 +61,10 @@ jobs: steps: - name: Checkout code - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1 with: dotnet-version: 10.0.104 @@ -78,11 +80,13 @@ jobs: - name: Test summary if: success() run: | - echo "## Localization Test Results" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - echo "✅ All localization tests passed!" >> $GITHUB_STEP_SUMMARY - echo "" >> $GITHUB_STEP_SUMMARY - echo "**Total Tests:** 209+ (including RTL support tests)" >> $GITHUB_STEP_SUMMARY - echo "- LocalizationService tests: 61" >> $GITHUB_STEP_SUMMARY - echo "- Component localization tests: 119" >> $GITHUB_STEP_SUMMARY - echo "- MelodeeComponentBase tests: 29" >> $GITHUB_STEP_SUMMARY + { + echo "## Localization Test Results" + echo "" + echo "✅ All localization tests passed!" + echo "" + echo "**Total Tests:** 209+ (including RTL support tests)" + echo "- LocalizationService tests: 61" + echo "- Component localization tests: 119" + echo "- MelodeeComponentBase tests: 29" + } >> "$GITHUB_STEP_SUMMARY" diff --git a/.github/workflows/sca-container-scan.yml b/.github/workflows/sca-container-scan.yml index aab47bf01..988dbf91b 100644 --- a/.github/workflows/sca-container-scan.yml +++ b/.github/workflows/sca-container-scan.yml @@ -28,63 +28,50 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Run Dependency Review - uses: actions/dependency-review-action@v4 + uses: actions/dependency-review-action@2031cfc080254a8a887f58cffee85186f0e49e48 # v4.9.0 with: - # Fail on critical, high, and moderate vulnerabilities + # Fail on high and critical vulnerabilities fail-on-severity: high # Comment on pull requests with details comment-summary-in-pr: on-failure container-scan: - name: Container Image Scan - runs-on: ubuntu-latest - timeout-minutes: 30 + name: Container Image Scan (${{ matrix.platform-id }}) + runs-on: ${{ matrix.runner }} + timeout-minutes: 45 permissions: contents: read security-events: write + strategy: + fail-fast: false + matrix: + include: + - platform: linux/amd64 + platform-id: linux-amd64 + runner: ubuntu-24.04 + sarif-category: trivy-container-scan + - platform: linux/arm64 + platform-id: linux-arm64 + runner: ubuntu-24.04-arm + sarif-category: trivy-container-scan-linux-arm64 env: - SCAN_BASE_IMAGE: mcr.microsoft.com/dotnet/aspnet:10.0 + SCAN_PLATFORM: ${{ matrix.platform }} steps: - name: Checkout repository - uses: actions/checkout@v4 - - - name: Setup .NET - uses: actions/setup-dotnet@v4 - with: - dotnet-version: 10.0.104 - - - name: Publish application for scanning - run: dotnet publish src/Melodee.Blazor/Melodee.Blazor.csproj -c Release -o scan/publish --self-contained false -p:PublishTrimmed=false + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - - name: Pull Docker base image for scanning - run: | - set -euo pipefail - - for attempt in 1 2 3 4 5; do - if docker pull "$SCAN_BASE_IMAGE"; then - exit 0 - fi - - delay=$((attempt * 20)) - echo "Base image pull failed on attempt ${attempt}; retrying in ${delay}s..." - sleep "$delay" - done - - docker pull "$SCAN_BASE_IMAGE" - - - name: Build Docker image for scanning + - name: Build production Docker image for scanning run: | set -euo pipefail for attempt in 1 2 3; do if docker build \ - --pull=false \ - --build-arg DOTNET_ASPNET_IMAGE="$SCAN_BASE_IMAGE" \ - -f .github/docker/Dockerfile.container-scan \ + --pull \ + --platform "$SCAN_PLATFORM" \ -t melodee:latest \ .; then exit 0 @@ -96,27 +83,51 @@ jobs: done docker build \ - --pull=false \ - --build-arg DOTNET_ASPNET_IMAGE="$SCAN_BASE_IMAGE" \ - -f .github/docker/Dockerfile.container-scan \ + --pull \ + --platform "$SCAN_PLATFORM" \ -t melodee:latest \ . - name: Run Trivy Vulnerability Scanner - uses: aquasecurity/trivy-action@master + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 with: image-ref: 'melodee:latest' format: 'sarif' output: 'trivy-results.sarif' severity: 'CRITICAL,HIGH' - limit-severity-skip: false + scanners: 'vuln' + limit-severities-for-sarif: true + version: 'v0.72.0' - name: Upload SARIF results if: always() && hashFiles('trivy-results.sarif') != '' - uses: github/codeql-action/upload-sarif@v4 + uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0 with: sarif_file: trivy-results.sarif - category: "trivy-container-scan" + category: ${{ matrix.sarif-category }} + + - name: Generate full Trivy vulnerability report + if: always() + uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + env: + TRIVY_SKIP_DB_UPDATE: true + with: + image-ref: 'melodee:latest' + format: 'json' + output: 'trivy-full-results.json' + severity: 'UNKNOWN,LOW,MEDIUM,HIGH,CRITICAL' + scanners: 'vuln' + skip-setup-trivy: true + cache: false + + - name: Upload full Trivy vulnerability report + if: always() && hashFiles('trivy-full-results.json') != '' + uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2 + with: + name: trivy-full-results-${{ matrix.platform-id }} + path: trivy-full-results.json + if-no-files-found: error + retention-days: 14 nuget-vuln-scan: name: NuGet Vulnerability Scan @@ -127,10 +138,10 @@ jobs: steps: - name: Checkout repository - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1 - name: Setup .NET - uses: actions/setup-dotnet@v4 + uses: actions/setup-dotnet@67a3573c9a986a3f9c594539f4ab511d57bb3ce9 # v4.3.1 with: dotnet-version: 10.0.104 diff --git a/Dockerfile b/Dockerfile index 732570aec..ef6e4608b 100644 --- a/Dockerfile +++ b/Dockerfile @@ -34,25 +34,41 @@ RUN dotnet publish "Melodee.Cli.csproj" -c Release -o /app/cli --self-contained # Migration bundle stage - create a self-contained migration bundle FROM build AS migrations WORKDIR /src/src/Melodee.Blazor -RUN dotnet tool install --global dotnet-ef --version 10.0.1 +RUN dotnet tool install --global dotnet-ef --version 10.0.9 ENV PATH="$PATH:/root/.dotnet/tools" # Provide a dummy connection string for design-time DbContext creation ENV ConnectionStrings__DefaultConnection="Host=localhost;Database=melodee;Username=melodee;Password=melodee" RUN dotnet ef migrations bundle --context MelodeeDbContext --output /app/efbundle --self-contained --force --project ../Melodee.Common/Melodee.Common.csproj -# Final runtime image - using aspnet for smaller image size -FROM mcr.microsoft.com/dotnet/aspnet:10.0 AS final +# Final runtime image - Ubuntu 26.04 provides current FFmpeg and OS packages. +FROM mcr.microsoft.com/dotnet/aspnet:10.0-resolute AS final WORKDIR /app EXPOSE 8080 -# Install required runtime dependencies +# Install current security updates and required runtime dependencies. +# Ubuntu 26.04 includes both GNU and Rust coreutils; select the GNU provider so +# the unused Rust implementation is not retained in the production image. RUN apt-get update && \ - apt-get install -y --no-install-recommends \ + DEBIAN_FRONTEND=noninteractive apt-get upgrade -y --no-install-recommends && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + --allow-remove-essential \ + coreutils-from-gnu \ + coreutils-from-uutils- \ ffmpeg \ postgresql-client \ curl \ lbzip2 \ - && rm -rf /var/lib/apt/lists/* + && apt-get purge -y rust-coreutils \ + && rm -f /usr/bin/pebble \ + && apt-get check \ + && test -z "$(dpkg --audit)" \ + && dpkg-query --status coreutils coreutils-from-gnu gnu-coreutils >/dev/null \ + && ! dpkg-query --status rust-coreutils >/dev/null 2>&1 \ + && for required_command in dotnet ffmpeg ffprobe pg_isready curl lbzip2 setpriv; do \ + command -v "$required_command" >/dev/null || exit 1; \ + done \ + && test ! -e /usr/bin/pebble \ + && rm -rf /var/lib/apt/lists/* # Copy the published application COPY --from=publish /app/publish . @@ -72,11 +88,24 @@ RUN groupadd -r melodee && useradd -r -g melodee -m melodee # Create volume directories # These serve as mount points; the actual volumes will overlay them # Note: podcasts and themes can be at /app/podcasts or /app/storage/podcasts depending on DB config -RUN mkdir -p /app/storage /app/storage/podcasts /app/storage/themes /app/podcasts /app/themes /app/inbound /app/staging /app/user-images /app/playlists /app/templates /app/Logs \ - && chown -R melodee:melodee /app +RUN mkdir -p \ + /app/storage \ + /app/storage/podcasts \ + /app/storage/themes \ + /app/podcasts \ + /app/themes \ + /app/inbound \ + /app/staging \ + /app/user-images \ + /app/playlists \ + /app/templates \ + /app/Logs \ + /home/melodee/.aspnet/DataProtection-Keys \ + && chown -R melodee:melodee /app /home/melodee/.aspnet # Set environment variables ENV DOTNET_CLI_TELEMETRY_OPTOUT=1 +ENV HOME="/home/melodee" ENV MELODEE_STORAGE_PATH="/app/storage" ENV SEARCHENGINE_MUSICBRAINZ_STORAGEPATH="/app/storage/_search-engines/musicbrainz" ENV MELODEE_INBOUND_PATH="/app/inbound" @@ -85,6 +114,10 @@ ENV MELODEE_USER_IMAGES_PATH="/app/user-images" ENV MELODEE_PLAYLISTS_PATH="/app/playlists" ENV MELODEE_TEMPLATES_PATH="/app/templates" +# Compose overrides this to root only while repairing named-volume ownership; +# direct image consumers run with least privilege by default. +USER melodee + # Use entrypoint script for proper startup ENTRYPOINT ["/entrypoint.sh"] diff --git a/design/docs/20260711-code-scanning-remediation-changes.md b/design/docs/20260711-code-scanning-remediation-changes.md new file mode 100644 index 000000000..03907b5de --- /dev/null +++ b/design/docs/20260711-code-scanning-remediation-changes.md @@ -0,0 +1,242 @@ + + +# Release Changes: Code Scanning Remediation + +**Related Plan**: `20260711-code-scanning-remediation-plan.md` +**Implementation Date**: 2026-07-11 + +## Summary + +This work addresses the 423-alert baseline (416 Trivy and seven CodeQL) through +source-level privacy fixes, setup-secret and filesystem hardening, workflow +supply-chain controls, and production-container dependency remediation. The +verified image inventory is 140 non-fixable medium/low findings. Six original +C# alerts and the fresh C# findings have source fixes; the Python setup alert is +handled by a hardened, necessary persistence boundary and narrow documented +suppression. Fresh Actions and JavaScript/TypeScript scans report zero +findings. Fresh workflow-equivalent default and security-extended Python scans +also report zero findings under the local threat model. + +## Changes + +### Added + +- `design/docs/20260711-code-scanning-remediation-plan.md` - Records the live + 423-alert baseline, remediation phases, and verification criteria. +- `design/docs/20260711-code-scanning-remediation-changes.md` - Tracks the + security changes and final validation results. +- `.github/codeql/extensions/log-sanitizer/codeql-pack.yml` - Defines an + auto-discovered local CodeQL model pack. +- `.github/codeql/extensions/log-sanitizer/models/log-sanitizer.model.yml` - + Models `LogSanitizer.Sanitize` only as a `log-injection` barrier and the + deny-by-default configuration redactor as a `file-content-store` barrier. +- `.github/codeql/codeql-python-config.yml` - Enables local file, command-line, + environment, and database sources for Python maintenance scripts without + changing the compiled application's remote request boundary. +- `src/Melodee.Common/Configuration/ConfigurationLogRedactor.cs` - Adds a + centralized, deny-by-default policy for configuration values written to + diagnostic logs. +- `tests/Melodee.Tests.Common/Configuration/ConfigurationLogRedactorTests.cs` - + Covers sensitive and unknown keys, safe operational values, URL credentials, + and log-forging characters. +- `src/Melodee.Blazor/Services/IPasswordResetTokenGenerator.cs` - Adds a narrow + reset-token generation abstraction so the complete component flow can be + regression-tested without the full user-service facade. +- `tests/Melodee.Tests.Blazor/Components/Pages/ForgotPasswordTests.cs` - Covers + reset success, unknown accounts, delivery failure, rate limiting, exceptions, + unsafe base URLs, and absence of email/token/configuration values in logs. +- `scripts/private_config.py` - Centralizes atomic secret-file publication, + explicit overwrite revalidation, POSIX mode enforcement, symlink and + non-regular-file refusal, and partial-file cleanup. +- `scripts/tests/test_private_config.py` - Covers shared writer publication, + cleanup, overwrite, and destination-integrity behavior. +- `scripts/tests/test_run_container_setup.py` - Covers generated-secret privacy, + explicit overwrite, POSIX permissions, symlink refusal, and non-regular + destination handling for the interactive container setup path. +- `scripts/tests/test_code_scanning_security.py` - Covers bounded Link-header + parsing under adversarial input, exact-origin HTTPS request and redirect + enforcement, bounded redirect behavior, and secret-free demo-user + success/failure output. +- `scripts/tests/test_incoming_clean_up.py` - Covers canonical root boundaries, + symlink and outside-root refusal, Zip Slip/symlink-member rejection, SFV + traversal, private extraction permissions, descriptor-relative race defense, + safe encoding repair, pretend mode, and contained live deletion. +- `tests/Melodee.Tests.Common/Services/SecurityLogSanitizationTests.cs` - Covers + the untrusted player, playlist, script setting, configuration key, provider, + and artist values reported by the fresh C# CodeQL scan. +- `tests/Melodee.Tests.Common/TestHelpers/RecordingLogEventSink.cs` - Captures + rendered security-sensitive log events for assertions without external + logging infrastructure. + +### Modified + +- `src/Melodee.Blazor/Components/Pages/Account/ForgotPassword.razor` - Removed + private values from all reset logs, validates a credential-free absolute + HTTP(S) base URL, and preserves generic diagnostics and enumeration-resistant + reset behavior. +- `src/Melodee.Blazor/Program.cs` - Registers the reset-token abstraction. +- `src/Melodee.Blazor/Services/Email/SmtpEmailSender.cs` - Removes subjects, + configured hosts, exception objects, and exception messages from SMTP logs. +- `src/Melodee.Common/Services/UserAuthenticationService.cs` - Replaces + usernames/emails with internal IDs in authentication and migration logs and + stops propagating the unused sensitive login identifier. +- `src/Melodee.Common/Services/UserProfileService.cs` - Removes email and + username values from lookup timing logs. +- `src/Melodee.Blazor/Filters/MelodeeApiAuthFilter.cs` - Replaces blacklisted + user email/IP logging with the internal user ID. +- `tests/Melodee.Tests.Blazor/Services/EmailServiceTests.cs`, + `tests/Melodee.Tests.Blazor/Controllers/Melodee/MelodeeApiAuthFilterTests.cs`, + and `tests/Melodee.Tests.Common/Services/UserAuthenticationServiceTests.cs` - + Add focused secret/identifier-free logging regression coverage. +- `src/Melodee.Common/Configuration/MelodeeConfigurationFactory.cs` - Routes + startup environment diagnostics through the centralized redactor so raw + passwords, tokens, connection strings, and unknown values never reach + `Trace` output. +- `scripts/setup_melodee.py` - Generates database and authentication secrets + without logging them and delegates ignored `.env` publication to the shared + secure writer. +- `scripts/run-container-setup.py` - Uses the same secure writer for normal, + confirmed-overwrite, and forced setup paths. +- `scripts/create_code_scanning_combined_serif.py` - Replaces a polynomial + backtracking Link-header expression with a bounded linear parser and confines + API requests, pagination, redirects, and downloads to the configured exact + HTTPS origin. +- `scripts/create-demo-user.py` - Stops printing the demo password, generated + API/public keys, encrypted password, and secret-bearing exception payloads. +- `scripts/incoming_clean_up.py` - Adds an explicit trusted filesystem boundary, + canonical containment before every access/mutation, pinned descriptor-relative + no-follow operations, `renameat2(RENAME_NOREPLACE)` quarantine publication, + full ZIP-member preflight/extraction, and traversal-safe SFV verification and + renaming. Live cleanup fails closed where the required Linux/POSIX primitives + are unavailable; extracted directories and files are private `0700` and + `0600` objects respectively. +- `scripts/tests/test_setup_melodee.py` - Covers secret generation, POSIX + permissions, existing-file preservation, malformed templates, symlink + refusal, and non-regular destinations. +- `design/docs/codeql-fixes.md` - Replaces the stale false-positive guidance + with the current CodeQL baseline, source remediations, precise barrier model, + workflow coverage, verification, and server-side cleanup instructions. +- `.github/workflows/codeql.yml` - Adds GitHub Actions and Python to advanced + scanning, declares language-specific build modes, pins actions by commit, + limits token permissions, and supports manual runs. +- `.github/codeql/codeql-config.yml` - Removes invalid model configuration and + broad rule exclusions while retaining the default remote threat model for + the compiled web application. +- `src/Melodee.Common/Services/DeviceIdentificationService.cs`, + `src/Melodee.Common/Services/PlaylistImportService.cs`, + `src/Melodee.Common/Services/ScriptEvaluation/ScriptAdminService.cs`, + `src/Melodee.Common/Services/SearchEngines/ArtistSearchEngineService.cs`, and + `src/Melodee.Common/Services/SettingService.cs` - Sanitize the six + attacker-controlled fields found by the fresh default-remote C# log-forging + scan. +- `src/Melodee.Common/Services/Playback/Backends/MpdPlaybackBackend.cs` - Keeps + the MPD password command on the wire while passing only a credential-free + command representation to every log path. +- `tests/Melodee.Tests.Common/Services/Playback/PlaybackBackendTests.cs` - + Verifies that password authentication still reaches MPD and never reaches + application logs. +- `.github/CODEQL-WORKFLOW.md` - Documents the single advanced setup, model + behavior, verification, and stale GitHub configuration cleanup. +- `Dockerfile` - Moves the shipped runtime to the official .NET 10 Ubuntu 26.04 + image, applies current package upgrades, replaces vulnerable unused + coreutils/Pebble tooling, asserts package integrity and required commands, + and defaults to the unprivileged `melodee` account. +- `entrypoint.sh` - Handles all persistent-volume paths and replaces the root + repair process with `dotnet` as UID/GID 999 and PID 1 through `setpriv`. +- `.github/workflows/sca-container-scan.yml` - Builds and scans the real + production image on native amd64 and arm64 runners, corrects SARIF severity + filtering, pins every action, and retains complete all-severity JSON reports. +- `.github/workflows/docker-publish.yml` - Forces the security-sensitive final + apt layer to refresh, pins every external action, validates image digests, + and constructs multi-architecture manifest arguments without shell word + splitting. +- `.github/workflows/dotnet.yml` - Pins every external action and writes the + coverage summary with quoted, grouped shell redirection. +- `.github/workflows/gitleaks.yml` - Pins checkout and SARIF upload actions by + commit and the Gitleaks container action by immutable multi-platform digest. +- `.github/workflows/localization.yml` - Pins every external action and hardens + shell quoting and summary output without changing localization behavior. +- `docs/pages/changelog.md` - Records the source, workflow, runtime-image, and + vulnerability-reporting security changes for 2.2.0. +- `docs/_posts/2026-07-11-melodee-2-2-0-released.md` - Adds the release-level + password-reset, setup-secret, non-root runtime, CodeQL, and Trivy summary. + +### Removed + +- `.github/codeql/extensions/log-sanitizer.model.yaml` - Replaced the + undiscovered value-propagating summary with the auto-discovered precise model + pack. +- `.github/docker/Dockerfile.container-scan` - Removed the reduced scan image + so CI analyzes the same final image that releases publish. + +## Verified Results + +### CodeQL + +- The six live baseline C# password-reset privacy flows have source fixes. The + Python setup-secret alert is handled by atomic owner-only persistence, + regression coverage, and one narrow documented suppression for the necessary + `.env` sink. +- A fresh C# database using the default remote threat model moved from eight + findings to zero after fixing six log-forging paths and two MPD + password-to-log flows. +- Fresh GitHub Actions and JavaScript/TypeScript databases each report zero + findings. +- Restored local Python analysis identified the setup persistence boundary, + polynomial Link parsing, secret-bearing output, filesystem/ZIP/SFV paths, and + same-origin exporter request handling. A fresh workflow-equivalent database + reports zero findings and no SARIF warning/error notifications across all 45 + default queries. The 52-query security-extended suite also reports zero + findings. + +The shared configuration keeps the compiled C# application, JavaScript, and +Actions on CodeQL's default remote trust boundary. Python alone enables the +local threat model because maintenance scripts intentionally consume command +line arguments, environment variables, and local files. This avoids treating +ordinary application-owned database and filesystem state as remotely +attacker-controlled while preserving the stricter model where it represents +the Python tools' real inputs. + +### Container and Dependency Inventory + +- Trivy 0.72.0 reduced raw production-image findings from 416 to 140: 276 + removed (66.3%). The remaining inventory represents 41 CVEs, split into 124 + medium and 16 low findings. +- The remaining image inventory contains zero critical, high, fixable, + .NET-package, or application-package findings. +- GitHub receives only the intended critical/high SARIF policy. CI retains the + complete all-severity JSON report for review and future fix-availability + changes. +- NuGet reports zero vulnerable dependencies. + +### Build and Runtime Verification + +- The complete solution builds with zero warnings and errors. +- The full .NET suite passes 5,885 tests with 34 skipped and zero failed. +- All 109 Python script tests pass with `ResourceWarning` promoted to an error, + including 57 focused incoming-cleanup security and race regressions. +- Jekyll, `actionlint`, YAML parsing, shell syntax/static analysis, and Docker + configuration checks pass. +- A real PostgreSQL integration starts the production image with the + unprivileged `melodee` process as PID 1, reaches healthy status, and confirms + that configured database/authentication secrets are absent from container + logs. + +## Release Summary + +The local release gate is complete: all four CodeQL languages report zero fresh +findings, the production image contains no critical, high, or currently fixable +inventory, the .NET and Python suites pass, and the production container passes +its real-database non-root integration. The remaining 140 raw image findings +are quantified medium/low Ubuntu-package issues without available fixes. + +## Post-Merge Verification + +- The final pre-merge API query still reports the 423-alert `main` baseline. + This is expected because GitHub has not analyzed this branch; no server-side + alert closure is claimed by the local results. +- After the checked-in Python job completes successfully on `main`, delete the + stale `dynamic/github-code-scanning/codeql:analyze` and + `dynamic/github-code-scanning/codeql:upload` Python configurations from + **Security and quality > Code scanning > Tool status**. Retain + `.github/workflows/codeql.yml:analyze`. diff --git a/design/docs/20260711-code-scanning-remediation-plan.md b/design/docs/20260711-code-scanning-remediation-plan.md new file mode 100644 index 000000000..6afa58db2 --- /dev/null +++ b/design/docs/20260711-code-scanning-remediation-plan.md @@ -0,0 +1,114 @@ + + +# Code Scanning Remediation Plan + +**Implementation Date**: 2026-07-11 + +## Objective + +Reduce the 423 open GitHub code-scanning alerts on `main` with verified, +behavior-preserving fixes. Prioritize the seven CodeQL findings and remove as +many of the 416 Trivy container findings as possible without hiding actionable +risk or removing required Melodee runtime features. + +## Baseline + +- Total: 423 open code-scanning alerts on `main`. +- CodeQL: seven alerts. + - Six `cs/exposure-of-sensitive-information` findings in the forgot-password + page. + - One `py/clear-text-storage-sensitive-data` finding in the setup script. +- Trivy: 416 alerts in the `library/melodee` container image. + - The baseline contained 382 medium and 34 low findings, with no critical or + high findings. + - Most findings come from the Ubuntu FFmpeg 6.1 package and its shared + libraries. + +## Phase 1: Inventory and Triage + +- [x] Retrieve every open alert through the GitHub code-scanning API. +- [x] Group alerts by tool, rule, severity, path, package, and installed + version. +- [x] Split independent CodeQL and container work into parallel workstreams. + +## Phase 2: CodeQL Remediation + +- [x] Remove sensitive-information exposure from the forgot-password flow + while retaining enumeration resistance and useful diagnostics. +- [x] Harden the Python setup flow's necessary `.env` persistence with atomic, + owner-only writes and document the narrow unavoidable CodeQL sink. +- [x] Add or update focused regression tests for both flows. +- [x] Consolidate GitHub Actions, C#, JavaScript/TypeScript, and Python under + the advanced CodeQL workflow and replace broad query exclusions with precise + modeling. +- [x] Redact database passwords, authentication keys, tokens, and connection + strings from startup configuration logs. +- [x] Remove direct email/user identifiers, reset configuration values, and + exception payloads from adjacent authentication and SMTP logs. +- [x] Remediate the ReDoS, demo-secret logging, and filesystem path-injection + findings exposed by the newly enabled local Python CodeQL scan. +- [x] Restrict the GitHub code-scanning exporter to bounded, same-origin HTTPS + requests and redirects so pagination cannot become an SSRF path. +- [x] Reduce a fresh default-remote C# scan from eight findings to zero by + sanitizing six log-forging flows and separating MPD wire commands from their + credential-free log representation. + +## Phase 3: Container Remediation + +- [x] Refresh or replace vulnerable operating-system and media packages using + trusted upstream images and repositories. +- [x] Keep FFmpeg, PostgreSQL client tools, health checks, non-root execution, + and supported architectures functional. +- [x] Align the production Dockerfile and the CI scan image so the scan + represents the shipped runtime. +- [x] Generate a local post-change Trivy inventory and quantify remaining + findings. + +## Phase 4: Verification and Release Notes (Complete) + +- [x] Run focused tests for the completed C#, setup, workflow, and container + fixes. +- [x] Build the production container and smoke-test required executables and + non-root startup behavior. +- [x] Run the full .NET build and test suite without warnings. +- [x] Complete the Python filesystem audit, its final focused tests, and + a fresh full-query Python CodeQL scan with the local threat model. +- [x] Validate Jekyll, GitHub Actions, YAML, shell, and Docker configuration + changes. +- [x] Update the public changelog and release change record with the final + verified results. + +## Verification Snapshot + +- Six original C# CodeQL alerts have source fixes and regression coverage. The + Python setup alert is handled by a hardened, necessary persistence boundary, + regression coverage, and one narrow documented suppression. +- Fresh local CodeQL scans report C# at `8 -> 0`, GitHub Actions at zero, and + JavaScript/TypeScript at zero under the default remote threat model. +- A fresh workflow-equivalent local-threat-model Python database reports zero + findings and no SARIF warning/error notifications across all 45 default + queries. The 52-query security-extended suite also reports zero findings. +- The Python test run passes all 109 script tests with `ResourceWarning` + promoted to an error, including all 57 focused cleanup filesystem, ZIP, SFV, + shutdown, and race regressions. +- Trivy 0.72.0 reduced the production-image inventory from 416 to 140 raw + findings: 276 removed (66.3%), 41 remaining CVEs, 124 medium and 16 low. + None are critical, high, fixable, or attributable to .NET/application + packages. +- The solution builds with zero warnings. The full .NET suite passes 5,885 + tests with 34 skipped, and NuGet reports zero vulnerable dependencies. +- Jekyll, `actionlint`, YAML, shell, and container checks pass. A real + PostgreSQL container integration confirmed health, non-root PID 1 startup, + and that configured secret values do not appear in logs. + +## Success Criteria + +- All seven baseline CodeQL data-flow findings have either a source fix or a + narrowly documented, security-hardened necessary persistence boundary, with + regression coverage. +- The container alert count is materially reduced without suppressing fixed or + high-impact vulnerabilities. +- The solution and container continue to build and required tests pass. +- Remaining alerts are quantified by severity, package, and fix availability. +- The Python audit and local-threat-model scans are complete; GitHub alert + reconciliation remains a post-merge administrative step. diff --git a/design/docs/VERSION_GUIDE.md b/design/docs/VERSION_GUIDE.md index 648af6301..5707c74b5 100644 --- a/design/docs/VERSION_GUIDE.md +++ b/design/docs/VERSION_GUIDE.md @@ -38,7 +38,7 @@ Melodee uses the `MAJOR.MINOR.PATCH` format: ## Current Versioning Architecture -Melodee has **two independent version tracks** that must be kept in sync during releases: +Melodee has **three synchronized version tracks** that must be updated during releases: ### Track 1: Assembly Version (.csproj files) @@ -46,22 +46,40 @@ Used at runtime, displayed in the About page, and embedded in compiled binaries. **Files (all 4 must be updated together):** -| File | Property | Current Value | +| File | Property | Required Release Value | |------|----------|---------------| -| `src/Melodee.Blazor/Melodee.Blazor.csproj` | `VersionPrefix` | `2.0.0` | -| `src/Melodee.Common/Melodee.Common.csproj` | `VersionPrefix` | `2.0.0` | -| `src/Melodee.Cli/Melodee.Cli.csproj` | `VersionPrefix` | `2.0.0` | -| `src/Melodee.Mql/Melodee.Mql.csproj` | `VersionPrefix` | `2.0.0` | +| `src/Melodee.Blazor/Melodee.Blazor.csproj` | `VersionPrefix` | `X.Y.Z` | +| `src/Melodee.Common/Melodee.Common.csproj` | `VersionPrefix` | `X.Y.Z` | +| `src/Melodee.Cli/Melodee.Cli.csproj` | `VersionPrefix` | `X.Y.Z` | +| `src/Melodee.Mql/Melodee.Mql.csproj` | `VersionPrefix` | `X.Y.Z` | Each .csproj also defines: - `VersionSuffix` — auto-generated build timestamp (e.g., `build20260501165851`) -- `AssemblyVersion` — `$(VersionPrefix).0` (e.g., `2.0.0.0`) -- `FileVersion` — `$(VersionPrefix).0` (e.g., `2.0.0.0`) -- `InformationalVersion` — `$(VersionPrefix)+$(VersionSuffix)` (e.g., `2.0.0+build20260501165851`) +- `AssemblyVersion` — `$(VersionPrefix).0` (e.g., `2.2.0.0`) +- `FileVersion` — `$(VersionPrefix).0` (e.g., `2.2.0.0`) +- `InformationalVersion` — `$(VersionPrefix)+$(VersionSuffix)` (e.g., `2.2.0+build20260501165851`) -The `AppVersionProvider` service strips the suffix and displays only the `VersionPrefix` (e.g., `2.0.0`) in the UI. +The `AppVersionProvider` service strips the suffix and displays only the `VersionPrefix` (e.g., `2.2.0`) in the UI. -### Track 2: Docker Image Tags (GitHub Releases) +### Track 2: Documentation Release Version + +The documentation site displays its default release in the navigation bar and +uses the same version for versioned search and release navigation. + +**Files and settings:** + +| File | Setting | Purpose | +|------|---------|---------| +| `docs/VERSION` | File content | Mirrors the current application release | +| `docs/_config.yml` | `version_params.latest` | Default Release shown for current documentation | +| `docs/_config.yml` | `version_params.versions` | Releases offered in the documentation menu | +| `docs/_config.yml` | `version_params.search_versions` | Releases included in documentation search | + +The new version must be the value of `latest` and the first item in both +version lists. Previous versions remain in the lists so archived documentation +continues to be available. + +### Track 3: Docker Image Tags (GitHub Releases) Docker images are published to `ghcr.io` and tagged based on **GitHub release tags**. @@ -70,16 +88,23 @@ Docker images are published to `ghcr.io` and tagged based on **GitHub release ta **Trigger:** GitHub release published (or manual `workflow_dispatch`) **Tag patterns:** -- `{{version}}` — full SemVer (e.g., `2.0.0`) -- `{{major}}.{{minor}}` — minor track (e.g., `2.0`) +- `{{version}}` — full SemVer (e.g., `2.2.0`) +- `{{major}}.{{minor}}` — minor track (e.g., `2.2`) - `{{major}}` — major track (e.g., `2`) - `latest` — applied to every release -### Track 3: docs/VERSION (Orphaned) +## Step-by-Step Version Bump Procedure -The file `docs/VERSION` currently contains `0.0.31` and is **not referenced by any code, build, or CI pipeline**. It appears to be a legacy artifact. Consider removing it or integrating it into the release process. +Use the version bump script as the preferred workflow. It updates the assembly +versions, documentation version metadata, and changelog together: -## Step-by-Step Version Bump Procedure +```bash +./scripts/bump_version.sh --dry-run X.Y.Z +./scripts/bump_version.sh X.Y.Z +``` + +The steps below describe each change made by the script and the remaining +release actions. ### Prerequisites @@ -134,7 +159,7 @@ Change `X.Y.Z` to the new version. > ```xml > > -> 2.1.0 +> 2.2.0 > $(MelodeeVersion) > build$([System.DateTime]::UtcNow.ToString("yyyyMMddHHmmss")) > $(VersionPrefix).0 @@ -144,17 +169,37 @@ Change `X.Y.Z` to the new version. > > ``` -### Step 3: Commit and Open a PR +### Step 3: Update the Documentation Release + +Update `docs/VERSION` to the new application version. In +`docs/_config.yml`, update `version_params.latest` and prepend the new version +to both `version_params.versions` and `version_params.search_versions`. + +For example: + +```yaml +version_params: + search_versions: + - X.Y.Z + latest: X.Y.Z + versions: + - X.Y.Z +``` + +Keep prior releases below the new value in both lists so archived navigation +and versioned search remain available. + +### Step 4: Commit and Open a PR ```bash -git add docs/pages/changelog.md src/*/ docs/VERSION 2>/dev/null || true +git add docs/VERSION docs/_config.yml docs/pages/changelog.md src/*/ 2>/dev/null || true git commit -m "chore: release vX.Y.Z" git push origin ``` Open a PR targeting `main`. The version bump is reviewed like any other change. -### Step 4: Merge and Tag +### Step 5: Merge and Tag After the PR is approved and merged to `main`: @@ -166,7 +211,7 @@ git push origin vX.Y.Z > The tag **must** be created on `main` after merge so the `docker-publish.yml` workflow picks it up. -### Step 5: Create a GitHub Release +### Step 6: Create a GitHub Release 1. Go to **GitHub → Releases → Draft a new release** 2. Select the tag `vX.Y.Z` @@ -189,7 +234,7 @@ git push origin vX.Y.Z 5. Click **Publish release** -### Step 6: Verify Docker Image Publication +### Step 7: Verify Docker Image Publication After publishing the release, the `docker-publish.yml` workflow triggers automatically. Verify: @@ -203,22 +248,26 @@ Pull and test the image: docker pull ghcr.io//melodee:X.Y.Z ``` -### Step 7: Verify the About Page and Changelog +### Step 8: Verify the Application and Documentation After deploying: 1. Navigate to the **About** page in the Melodee UI and confirm the displayed version matches the new `X.Y.Z`. 2. Navigate to the **Changelog** page on the docs site (`/changelog/`) and confirm the new version entry is visible. +3. Confirm the documentation navigation displays `Release: X.Y.Z` by default + and offers the new release in its version menu. ## Version Display Locations | Location | Source | Format | |----------|--------|--------| -| About page | `IAppVersionProvider.GetSemVerForDisplay()` | `2.0.0` (prefix only) | -| Admin Dashboard → Server Stats | `Assembly.GetName().Version` | `2.0.0.0` | -| Admin Doctor → Server Info | `Assembly.GetName().Version` | `2.0.0.0` | -| Docker image tags | GitHub release tag | `2.0.0`, `2.0`, `2`, `latest` | -| Assembly metadata | `InformationalVersion` | `2.0.0+build20260501165851` | +| About page | `IAppVersionProvider.GetSemVerForDisplay()` | `X.Y.Z` (prefix only) | +| Admin Dashboard → Server Stats | `Assembly.GetName().Version` | `X.Y.Z.0` | +| Admin Doctor → Server Info | `Assembly.GetName().Version` | `X.Y.Z.0` | +| Documentation Release menu | `docs/_config.yml` → `version_params.latest` | `X.Y.Z` | +| Documentation version marker | `docs/VERSION` | `X.Y.Z` | +| Docker image tags | GitHub release tag | `X.Y.Z`, `X.Y`, `X`, `latest` | +| Assembly metadata | `InformationalVersion` | `X.Y.Z+buildYYYYMMDDHHMMSS` | ## API Versioning (Separate from App Version) @@ -229,9 +278,11 @@ Melodee uses `Asp.Versioning.Mvc` for REST API versioning, which is **independen - Consumers specify version via URL segment (`/api/v1/...`) or `X-Api-Version` header - API version bumps are independent of application version bumps -## Automated Version Bumping (Future) +## Further Version Automation -Consider adopting one of these tools for automated version management: +The repository's `scripts/bump_version.sh` handles the synchronized release +files. Consider adopting one of these tools if tag-derived or fully automated +release management is desired: | Tool | Approach | Best For | |------|----------|----------| diff --git a/design/docs/async-best-practices.md b/design/docs/async-best-practices.md new file mode 100644 index 000000000..8b093fef1 --- /dev/null +++ b/design/docs/async-best-practices.md @@ -0,0 +1,19 @@ +# Async Best Practices + +This is internal engineering guidance for Melodee contributors. + +## Guidelines + +- Avoid `.Result`, `.Wait()`, and `GetAwaiter().GetResult()` in request threads and service code. +- Prefer end-to-end asynchronous flows with `await`. +- Use `ConfigureAwait(false)` in reusable library code where the surrounding project conventions call for it. +- Do not block inside dependency-injection registrations; resolve asynchronous dependencies at runtime. +- Use `Task.Run` sparingly. Prefer genuinely asynchronous I/O APIs over moving blocking I/O to the thread pool. +- Accept and propagate `CancellationToken` for operations that may block or perform I/O. + +## Exceptions + +- Generated code or APIs that require synchronous contracts may be excluded with a clear justification. +- UI components with purely synchronous work should not add asynchronous plumbing without a concrete need. +- A deliberate synchronous boundary must not block an ASP.NET request thread on asynchronous work. + diff --git a/design/docs/codeql-fixes.md b/design/docs/codeql-fixes.md index c20c07330..0e371154e 100644 --- a/design/docs/codeql-fixes.md +++ b/design/docs/codeql-fixes.md @@ -1,283 +1,223 @@ -# CodeQL Security Fixes + + +# CodeQL Security Remediation Record + +**Updated**: 2026-07-11 +**Status**: Local remediation verification complete; GitHub rescan pending + +## Scope + +This document records high-confidence CodeQL remediations and justified +compatibility exceptions. It is not a substitute for the live GitHub code +scanning dashboard, and it does not claim that future scans will produce no new +findings. + +## July 2026 Remediation + +The live baseline contained seven open CodeQL alerts: + +| Rule | Count | Severity | Location | +|------|------:|----------|----------| +| `cs/exposure-of-sensitive-information` | 6 | Medium | `ForgotPassword.razor` | +| `py/clear-text-storage-sensitive-data` | 1 | High | `setup_melodee.py` | + +### Password Reset Privacy + +Six password-reset log events included an email passed through +`LogSanitizer.MaskEmail`. The helper retained part of the local address and the +complete domain, so the logs still persisted private user information. + +The reset flow now logs generic operational events without the address, base +URL, template subject, reset token, or exception payload. The configured reset +base URL must be an absolute HTTP(S) URL with a host and no user information, +query, or fragment. Rate limiting, token generation, email delivery, generic +user responses, and account enumeration resistance are unchanged. + +The same privacy boundary now covers adjacent SMTP and authentication paths: +SMTP logs omit message/configuration values and raw exceptions; login, +migration, profile-lookup, and blacklist logs use generic outcomes or internal +numeric user IDs rather than email addresses and usernames. + +### Startup Configuration Redaction + +The startup configuration factory previously wrote every process environment +variable value to `Trace`, including database passwords, authentication keys, +tokens, and complete connection strings. All configuration diagnostics now use +a centralized deny-by-default redactor. Sensitive and unrecognized keys emit +only `[REDACTED]`; an explicit set of operational paths, ports, versions, +environments, and limits remains visible after log-forging characters are +escaped. Credential-bearing or parameterized URLs are also redacted. + +Focused tests cover sensitive names, unknown values, safe operational values, +URL credentials, and injected line endings. + +### Protected Setup Secrets + +Unattended Compose setup requires a stable database password and authentication +key across restarts. Both supported setup utilities now use one shared writer +that: + +- generates independent URL-safe values from 32 and 64 random bytes; +- replaces settings by exact key instead of matching example values; +- never prints either generated value; +- creates an unpredictable temporary file with `O_CREAT`, `O_EXCL`, and + `O_NOFOLLOW` where available; +- applies mode `0600` through the open descriptor before writing on POSIX; +- atomically publishes a new file without replacement, or replaces only an + explicitly approved and revalidated regular file; +- removes partial and temporary files if writing or publication fails; and +- refuses live/dangling symlinks and other non-regular destinations. + +The ignored deployment file is the necessary persistence boundary. It is +owner-only (`0600`) on POSIX; on Windows it inherits the containing directory's +ACL. One narrow `codeql[py/clear-text-storage-sensitive-data]` suppression +documents that unavoidable sink. The clear-text-storage query is not excluded +globally. Existing regular secret files are opened without following links, +revalidated by descriptor identity, and tightened to `0600` before they are +preserved. + +### Restored Python Coverage + +An initial local CodeQL 2.26.0 run after restoring Python to the advanced +workflow surfaced 22 pre-existing findings that the stale server-side setup no +longer updated: + +- one polynomial ReDoS path in GitHub Link-header parsing; +- two clear-text demo password/API-key log paths; and +- 19 filesystem path-injection paths in the destructive incoming cleanup tool. + +The Link header now uses bounded linear parsing, and demo-user setup never +prints credentials, generated key material, encrypted values, or raw failure +payloads. The cleanup tool now validates a canonical cleanup root beneath an +explicit trusted boundary and rejects filesystem-root authority. Live cleanup +pins root and parent descriptors, uses no-follow descriptor-relative operations +and `renameat2(RENAME_NOREPLACE)` quarantine publication, and fails closed when +those Linux/POSIX primitives are unavailable. ZIP extraction preflights every +member before writing, applies private `0700` directory and `0600` file modes, +rolls back partial output, and rejects symlink, size, compression, duplicate, +and traversal hazards. SFV verification and renaming cannot escape the media +root. The default boundary is the current working directory; administrators +targeting an absolute path outside it must explicitly provide +`--trusted-boundary`. + +A subsequent local scan and exporter audit also found that pagination and +redirect URLs needed an explicit network boundary. All API requests, Link +destinations, redirects, and downloads now require HTTPS and the exact +configured GitHub API origin, with a bounded redirect chain. This closes the +same-origin SSRF paths without weakening GitHub Enterprise Server support. + +A fresh workflow-equivalent Python database reports zero findings and no SARIF +warning/error notifications across all 45 default queries under the local +threat model. The 52-query security-extended suite also reports zero findings. +The script suite passes all 109 tests with `ResourceWarning` promoted to an +error, including 57 focused filesystem, ZIP, SFV, shutdown, and race +regressions. + +### Fresh C# Findings + +After removing broad query exclusions, a fresh C# database under the default +remote threat model reported eight actionable findings. Six were log-forging +flows through player client names, playlist names, script setting keys, +configuration keys, provider names, and artist queries. Each field now passes +through the narrowly modeled `LogSanitizer.Sanitize` boundary before logging. + +The other two paths associated the MPD password wire command with diagnostic +logs. The backend now carries separate wire and safe command representations: +the credential is still sent to MPD, while only a constant credential-free +description can reach a logger. Focused tests exercise both guarantees. A fresh +default-remote C# scan reports zero findings (`8 -> 0`). + +### CodeQL Workflow Coverage + +The version-controlled advanced workflow analyzes GitHub Actions, C#, +JavaScript/TypeScript, and Python. Compiled C# uses autobuild; interpreted +languages use build mode `none`. Python has a dedicated local-threat-model +configuration because command-line arguments, environment variables, local +files, and databases are genuine inputs to maintenance utilities. C#, +JavaScript/TypeScript, and Actions retain the default remote boundary so +ordinary application-owned database and filesystem state is not incorrectly +treated as remotely attacker-controlled. Workflow actions are pinned to +verified full commit SHAs. + +Fresh local databases report zero C#, GitHub Actions, and +JavaScript/TypeScript findings. The fresh default and security-extended Python +databases also report zero findings as described above. + +GitHub's default setup is currently `not-configured`. The Python alert belongs +to old `dynamic/github-code-scanning/codeql:analyze` and +`dynamic/github-code-scanning/codeql:upload` configurations. After an advanced +Python scan completes on `main`, an administrator must delete those stale +dynamic configurations in **Code scanning > Tool status** while retaining the +checked-in workflow configuration, so obsolete alert state no longer remains +attached to the branch. + +### Precise Security Models + +The prior configuration excluded entire C# security queries and used a +value-preserving `summaryModel` as though it were a sanitizer. Those broad +exclusions were removed. + +The auto-discovered local model pack marks only the return value of +`LogSanitizer.Sanitize` as a `log-injection` barrier. It also marks the +deny-by-default `ConfigurationLogRedactor.RedactValue` return as a +`file-content-store` barrier so CodeQL recognizes that secret configuration +values cannot reach diagnostic logs: -**Date**: 2026-01-12 (Updated) -**Status**: ✅ MITIGATIONS IN PLACE - -## Summary - -This document tracks all CodeQL security alerts identified in the Melodee codebase and their remediation status. - -## Alert Inventory - -### A. Weak Cryptography (cs/weak-crypto) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ DOCUMENTED | HashHelper.cs | 24 | MD5 for external API compatibility | Required by OpenSubsonic/Last.fm APIs - cannot change | -| ✅ DOCUMENTED | HashHelper.cs | 44 | MD5 for external API compatibility | Required by OpenSubsonic/Last.fm APIs - cannot change | -| ✅ DOCUMENTED | UserService.cs | ~1064 | MD5 for OpenSubsonic token auth | Required by protocol spec - cannot change | -| ✅ DOCUMENTED | ScrobbleController.cs | ~291 | MD5 for Last.fm API signature | Required by Last.fm API - cannot change | -| ✅ DOCUMENTED | JellyfinControllerBase.cs | 58 | MD5 for server ID generation | Non-cryptographic GUID generation for Jellyfin API | -| ✅ DOCUMENTED | ItemsController.cs | 1084 | MD5 for ETag computation | Non-cryptographic ETag generation for HTTP caching | -| ✅ DOCUMENTED | PlaylistsController.cs | 506 | MD5 for ETag computation | Non-cryptographic ETag generation for HTTP caching | -| ✅ DOCUMENTED | MusicGenresController.cs | 115, 122 | MD5 for genre GUID and ETags | Non-cryptographic ID generation for Jellyfin API | -| ✅ DOCUMENTED | ArtistsController.cs | 299, 306 | MD5 for ETag computation | Non-cryptographic ETag generation for HTTP caching | -| ✅ DOCUMENTED | UsersController.cs | 794 | MD5 for ETag computation | Non-cryptographic ETag generation for HTTP caching | -| ✅ DOCUMENTED | GenresController.cs | 108, 115 | MD5 for genre GUID and ETags | Non-cryptographic ID generation for Jellyfin API | -| ✅ DOCUMENTED | UserViewsController.cs | 104 | MD5 for ETag computation | Non-cryptographic ETag generation for HTTP caching | -| ✅ DOCUMENTED | MelodeeDbContext.cs | 95 | MD5 for seed data GUIDs | Non-cryptographic deterministic GUID generation | - -**Note**: MD5 usages are either required by external API specifications or used for non-cryptographic purposes (GUID/ETag generation). All are properly documented with `// lgtm[cs/weak-crypto]` comments explaining the justification. - -### B. Regex DoS (cs/regex-injection) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ FIXED | ITunesSearchEngine.cs | 319, 325 | `new Regex()` without timeout | Added `TimeSpan.FromSeconds(5)` timeout | -| ✅ FIXED | StringExtensions.cs | 930 | `new Regex()` without timeout | Added `TimeSpan.FromSeconds(5)` timeout | -| ✅ FIXED | ConfigurationListCommand.cs | 34 | `new Regex()` without timeout | Added `TimeSpan.FromSeconds(5)` timeout | - -### C. Path Traversal (cs/path-traversal) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ FIXED | AlbumDetail.razor | 712 | `file.Name` from upload used in Path.Combine | Use SafePath.ResolveUnderRoot() to sanitize and validate | - -### D. Cross-Site Scripting (cs/xss) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ FIXED | Markdown.razor | 6 | MarkupString renders unsanitized HTML from user content | Added HtmlSanitizer with allowlist of safe tags/attributes | - -### E. Log Forging (cs/log-forging) - -| Status | File | Lines | Root Cause | Fix Strategy | -|--------|------|-------|------------|--------------| -| ✅ MITIGATED | Multiple controllers | Various | User input logged without sanitization | LogSanitizer.Sanitize() wrapper + CodeQL model extension | - -**Note**: The codebase uses `LogSanitizer.Sanitize()` from `Melodee.Common.Utility.LogSanitizer` to sanitize all user-controlled input before logging. This method replaces newlines (CR, LF, NEL, LS, PS) with safe placeholders to prevent log forging attacks. A CodeQL model extension file (`.github/codeql/extensions/log-sanitizer.model.yaml`) teaches CodeQL to recognize this as a sanitizer. - -Files affected (all using LogSanitizer): -- `src/Melodee.Blazor/Controllers/Jellyfin/AudioController.cs` -- `src/Melodee.Blazor/Controllers/Jellyfin/ImagesController.cs` -- `src/Melodee.Blazor/Controllers/Jellyfin/ItemsController.cs` -- `src/Melodee.Blazor/Controllers/Jellyfin/SessionsController.cs` -- `src/Melodee.Blazor/Controllers/Jellyfin/UsersController.cs` -- `src/Melodee.Blazor/Controllers/Melodee/AlbumsController.cs` -- `src/Melodee.Blazor/Controllers/Melodee/ArtistLookupController.cs` -- `src/Melodee.Blazor/Controllers/Melodee/ArtistsController.cs` -- `src/Melodee.Blazor/Controllers/Melodee/SongsController.cs` -- `src/Melodee.Blazor/Services/SmartPlaylistService.cs` -- `src/Melodee.Common/Plugins/SearchEngine/MusicBrainz/Data/SQLiteMusicBrainzRepository.cs` -- `src/Melodee.Common/Services/SearchEngines/ArtistSearchEngineService.cs` -- `src/Melodee.Mql/Api/MqlController.cs` - -### F. URL Redirection (cs/web/unvalidated-url-redirection) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ FIXED | UsersController.cs | 383 | Redirect URL used raw user input | Use validated GUID instead of raw user input | - -### G. Clear-Text Storage (py/clear-text-storage-sensitive-data) - -| Status | File | Line | Root Cause | Fix Strategy | -|--------|------|------|------------|--------------| -| ✅ FIXED | setup_melodee.py | 155 | String literal containing "DB_PASSWORD" | Construct string dynamically to avoid false positive | - -## Fix Progress - -### Completed Fixes - -1. **Regex DoS Prevention** - Added timeouts to all runtime-constructed Regex instances (2025-12-21, 2026-01-02) -2. **Path Traversal Prevention** - Created SafePath utility and used it in file upload handler (2025-12-21) -3. **XSS Prevention** - Integrated HtmlSanitizer for Markdown component (2025-12-21) -4. **MD5 Documentation** - Added comprehensive documentation for all MD5 usages in Jellyfin API controllers and database seeding (2026-01-02) -5. **Log Forging Prevention** - Created LogSanitizer utility and CodeQL model extension (2026-01-12) -6. **URL Redirection Fix** - Use validated GUID instead of raw input for redirect URLs (2026-01-12) -7. **Python False Positive Fix** - Construct variable name dynamically to avoid CodeQL false positive (2026-01-12) - -### Files Modified - -**Original fixes (2025-12-21):** -- `src/Melodee.Common/Plugins/SearchEngine/ITunes/ITunesSearchEngine.cs` - Added regex timeouts -- `src/Melodee.Common/Extensions/StringExtensions.cs` - Added regex timeout -- `src/Melodee.Blazor/Components/Pages/Data/AlbumDetail.razor` - Used SafePath for file uploads -- `src/Melodee.Blazor/Components/Components/Markdown.razor` - Added HTML sanitization -- `src/Melodee.Blazor/Melodee.Blazor.csproj` - Added HtmlSanitizer package reference -- `Directory.Packages.props` - Added HtmlSanitizer version - -**Additional fixes (2026-01-02):** -- `src/Melodee.Cli/Command/ConfigurationListCommand.cs` - Added regex timeout -- `src/Melodee.Blazor/Controllers/Jellyfin/JellyfinControllerBase.cs` - Added MD5 documentation for server ID generation -- `src/Melodee.Blazor/Controllers/Jellyfin/ItemsController.cs` - Added MD5 documentation for ETag computation -- `src/Melodee.Blazor/Controllers/Jellyfin/PlaylistsController.cs` - Added MD5 documentation for ETag computation -- `src/Melodee.Blazor/Controllers/Jellyfin/MusicGenresController.cs` - Added MD5 documentation for genre GUID and ETag generation -- `src/Melodee.Blazor/Controllers/Jellyfin/ArtistsController.cs` - Added MD5 documentation for ETag computation -- `src/Melodee.Blazor/Controllers/Jellyfin/UsersController.cs` - Added MD5 documentation for ETag computation -- `src/Melodee.Blazor/Controllers/Jellyfin/GenresController.cs` - Added MD5 documentation for genre GUID and ETag generation -- `src/Melodee.Blazor/Controllers/Jellyfin/UserViewsController.cs` - Added MD5 documentation for ETag computation -- `src/Melodee.Common/Data/MelodeeDbContext.cs` - Added MD5 documentation for seed data GUID generation -- `docs/codeql-fixes.md` - Updated with 2026-01-02 fixes - -**Additional fixes (2026-01-12):** -- `.github/codeql/extensions/log-sanitizer.model.yaml` - Updated to use "value" kind instead of "taint" for sanitizer behavior -- `src/Melodee.Blazor/Controllers/Jellyfin/UsersController.cs` - Fixed open redirect by using validated GUID instead of raw input -- `scripts/setup_melodee.py` - Fixed false positive by constructing variable name dynamically -- `design/docs/codeql-fixes.md` - Updated with 2026-01-12 fixes - -### Files Created (2025-12-21) - -- `src/Melodee.Common/Utility/SafePath.cs` - New security utility for path validation -- `tests/Melodee.Tests.Common/Utility/SafePathTests.cs` - Unit tests for SafePath - -## Implementation Details - -### Fix B1 & B2: Regex with Timeout - -Added `TimeSpan.FromSeconds(5)` timeout to prevent ReDoS attacks: - -```csharp -// Before -var regex = new Regex(pattern); - -// After -var regex = new Regex(pattern, RegexOptions.None, TimeSpan.FromSeconds(5)); -``` - -### Fix C1: Path Traversal Prevention - -Created `SafePath` utility class with: -- `SanitizeFileName()` - Removes path separators and ".." sequences -- `ResolveUnderRoot()` - Combines paths and validates result stays within base directory -- `IsPathWithinBase()` - Checks if a path is confined to a base directory - -Usage in file upload: -```csharp -// Before -var target = Path.Combine(dir, file.Name); - -// After -var target = SafePath.ResolveUnderRoot(dir, file.Name); -if (target == null) { - // Invalid filename - reject upload - continue; -} -``` - -### Fix D1: XSS Prevention - -Added HtmlSanitizer with strict allowlist: - -```csharp -private static readonly HtmlSanitizer Sanitizer = CreateSanitizer(); - -private static HtmlSanitizer CreateSanitizer() -{ - var sanitizer = new HtmlSanitizer(); - // Only allow safe HTML tags for markdown content - sanitizer.AllowedTags.Add("h1"); // headings - sanitizer.AllowedTags.Add("p"); // paragraphs - sanitizer.AllowedTags.Add("a"); // links - // ... other safe tags - - // Only allow safe URL schemes - sanitizer.AllowedSchemes.Add("https"); - sanitizer.AllowedSchemes.Add("http"); - sanitizer.AllowedSchemes.Add("mailto"); - - return sanitizer; -} -``` - -### Fix E: Log Forging Prevention - -Created `LogSanitizer` utility class with methods that sanitize user input before logging: - -```csharp -public static class LogSanitizer -{ - public static string? Sanitize(string? input) - { - if (string.IsNullOrEmpty(input)) - return input; - - return input - .Replace("\r", "[CR]") - .Replace("\n", "[LF]") - .Replace("\u0085", "[NEL]") // Next Line - .Replace("\u2028", "[LS]") // Line Separator - .Replace("\u2029", "[PS]"); // Paragraph Separator - } -} -``` - -Usage in controllers: -```csharp -// Before -logger.LogWarning("Invalid request: {ItemId}", request.ItemId); - -// After -var sanitizedItemId = LogSanitizer.Sanitize(request.ItemId); -logger.LogWarning("Invalid request: {ItemId}", sanitizedItemId); -``` - -CodeQL model extension (`.github/codeql/extensions/log-sanitizer.model.yaml`): ```yaml extensions: - addsTo: pack: codeql/csharp-all - extensible: summaryModel + extensible: barrierModel data: - # Using "value" kind means data flows but taint is cleansed - - ["Melodee.Common.Utility", "LogSanitizer", False, "Sanitize", "(System.String)", "", "Argument[0]", "ReturnValue", "value", "manual"] -``` - -### Fix F1: URL Redirection - -Fixed open redirect by using validated GUID instead of raw user input: - -```csharp -// Before -if (!Guid.TryParse(itemId, out _)) - return BadRequest(...); -return RedirectPreserveMethod($"/Items/{itemId}"); // Uses raw user input - -// After -if (!Guid.TryParse(itemId, out var validatedItemId)) - return BadRequest(...); -return RedirectPreserveMethod($"/Items/{validatedItemId}"); // Uses validated GUID + - ["Melodee.Common.Utility", "LogSanitizer", false, "Sanitize", "(System.String)", "", "ReturnValue", "log-injection", "manual"] + - ["Melodee.Common.Configuration", "ConfigurationLogRedactor", false, "RedactValue", "(System.String,System.Object)", "", "ReturnValue", "file-content-store", "manual"] ``` -### Fix G1: Python False Positive - -Fixed CodeQL false positive by constructing variable name dynamically: - -```python -# Before -print(" Please set DB_PASSWORD manually in .env") # CodeQL flags "DB_PASSWORD" as sensitive - -# After -db_cred_var = "DB_" + "PASSWORD" # Split to avoid static analysis false positive -print(f" Please set {db_cred_var} manually in .env") -``` -``` - -## Testing - -- [x] Solution builds successfully -- [x] All existing tests pass (175 tests) -- [x] New SafePath tests pass (27 tests) -- [x] No regressions in functionality -- [x] LogSanitizer tests pass (included in existing test suite) +These models do not treat email or identifier masking as privacy sanitizers. +Future real findings from the previously excluded queries must be fixed at the +source or dismissed individually with evidence. + +## Historical Remediation + +| Category | Status | Approach | +|----------|--------|----------| +| Regex denial of service | Fixed | Runtime-constructed regular expressions use explicit timeouts. | +| Path traversal | Fixed | Uploaded file paths are resolved beneath an approved root with `SafePath`. | +| Markdown cross-site scripting | Fixed | Rendered Markdown passes through an HTML allowlist sanitizer. | +| Log forging | Mitigated | User-controlled log fields use `LogSanitizer.Sanitize`; CodeQL has a precise barrier model. | +| Unvalidated redirection | Fixed | Jellyfin redirects use parsed GUID values instead of raw input. | +| Weak cryptography | Compatibility exception | MD5 remains only where required by OpenSubsonic/Last.fm protocols or for non-security ETags and deterministic identifiers. | + +Weak algorithms must not be used for password storage, authentication designs, +signatures outside required compatibility protocols, or new security-sensitive +features. + +## Verification + +- Complete solution build: zero warnings and errors. +- Complete .NET suite: 5,885 passed, 34 skipped, zero failed. +- NuGet vulnerability audit: zero vulnerable dependencies. +- Fresh local CodeQL: C# `8 -> 0`, GitHub Actions zero, and + JavaScript/TypeScript zero. +- Python compilation and Ruff pass. Black reports the exporter and its focused + tests unchanged; the two pre-existing legacy cleanup files remain outside the + repository's Black baseline. All 109 script tests pass with resource warnings + treated as errors. +- Fresh Python CodeQL reports zero findings across the 45-query default suite + and the 52-query security-extended suite. +- `actionlint`, YAML parsing, shell checks, and Jekyll validation pass. +- A real PostgreSQL/production-container integration reached healthy status + with `melodee` running unprivileged as PID 1 and no configured secret values + present in logs. + +Alert closure requires a successful GitHub scan after these changes reach a +branch analyzed by the advanced workflow. ## References -- [OWASP Path Traversal](https://owasp.org/www-community/attacks/Path_Traversal) -- [OWASP XSS Prevention](https://cheatsheetseries.owasp.org/cheatsheets/Cross_Site_Scripting_Prevention_Cheat_Sheet.html) -- [OWASP Log Injection](https://owasp.org/www-community/attacks/Log_Injection) +- [Resolving code scanning alerts](https://docs.github.com/en/code-security/how-tos/manage-security-alerts/manage-code-scanning-alerts/resolve-alerts) +- [Customizing CodeQL library models for C#](https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-csharp/) - [CWE-117: Improper Output Neutralization for Logs](https://cwe.mitre.org/data/definitions/117.html) -- [CWE-601: URL Redirection to Untrusted Site](https://cwe.mitre.org/data/definitions/601.html) -- [.NET Regex Best Practices](https://docs.microsoft.com/en-us/dotnet/standard/base-types/best-practices) -- [OpenSubsonic API Authentication](http://www.subsonic.org/pages/api.jsp#authentication) -- [Last.fm API Authentication](https://www.last.fm/api/authentication) -- [CodeQL Library Models for C#](https://codeql.github.com/docs/codeql-language-guides/customizing-library-models-for-csharp/) +- [CWE-312: Cleartext Storage of Sensitive Information](https://cwe.mitre.org/data/definitions/312.html) +- [CWE-359: Exposure of Private Personal Information](https://cwe.mitre.org/data/definitions/359.html) +- [OWASP Log Injection](https://owasp.org/www-community/attacks/Log_Injection) diff --git a/design/docs/ef-core-query-splitting.md b/design/docs/ef-core-query-splitting.md new file mode 100644 index 000000000..ed1e8eb0c --- /dev/null +++ b/design/docs/ef-core-query-splitting.md @@ -0,0 +1,49 @@ +# EF Core Query Splitting Strategy + +This is internal engineering guidance for Melodee contributors. + +## Default + +Use the DbContext's configured query-splitting behavior. Split queries help avoid cartesian explosion when a query loads several collection navigations. + +For read-only operations, prefer projections and `AsNoTracking()` over materializing a large entity graph. + +## Choose per query + +Prefer split queries when: + +- multiple `Include` and `ThenInclude` paths load collections; +- a joined graph would duplicate substantial row data; +- the operation is read-heavy and does not require one SQL statement. + +Consider `AsSingleQuery()` when: + +- the join is small and simple; +- a projection returns a bounded flat result; +- consistency between multiple SQL statements matters and the transaction semantics are explicit. + +Always inspect the generated SQL and measure with representative data before overriding the configured default. + +## Examples + +```csharp +var songs = await context.Songs + .Include(song => song.Album) + .ThenInclude(album => album.Artist) + .AsSplitQuery() + .AsNoTracking() + .ToArrayAsync(cancellationToken); +``` + +```csharp +var songs = await context.Songs + .AsNoTracking() + .Select(song => new + { + song.Id, + song.Title, + Artist = song.Album.Artist.Name + }) + .ToArrayAsync(cancellationToken); +``` + diff --git a/docs/404.html b/docs/404.html deleted file mode 100644 index 3a16ab533..000000000 --- a/docs/404.html +++ /dev/null @@ -1,25 +0,0 @@ ---- -permalink: /404.html -layout: page ---- - - - -
-

404

- -

Page not found :(

-

The requested page could not be found.

-
diff --git a/docs/404.md b/docs/404.md deleted file mode 100644 index 7c527d9b6..000000000 --- a/docs/404.md +++ /dev/null @@ -1,10 +0,0 @@ ---- -# example 404.md - -layout: default -permalink: /404.html ---- - -# Page Not Found - -Unfortunately we were unable to find the page you requested. It could be the page doesn't exist or only exists for a specific version of these documents so please use the search feature to see if you are able to locate the information you were after. \ No newline at end of file diff --git a/docs/_config.yml b/docs/_config.yml index 1a32f82bf..0c1b437c0 100644 --- a/docs/_config.yml +++ b/docs/_config.yml @@ -17,8 +17,9 @@ title: Melodee email: info@melodee.org author: Steven Hildreth -description: > # this means to ignore newlines until "baseurl:" - Industrial grade self-hosted streaming music server. +description: >- + Self-hosted music management, curation, and streaming with OpenSubsonic, + Jellyfin-compatible, and native APIs. # Add your baseurl here (your repository) but DO NOT CHANGE THE LINE NUMBER without editing .circleci/circle_urls.sh baseurl: "" # the subpath of your site, e.g. /blog @@ -47,12 +48,14 @@ version_params: # Allow these versions to be searched search_versions: + - 2.2.0 - 2.1.0 - 2.0.0 - 1.8.0 - 1.7.0 - latest: 2.1.0 + latest: 2.2.0 versions: + - 2.2.0 - 2.1.0 - 2.0.0 - 1.8.0 @@ -75,6 +78,9 @@ color: "#30638e" # Build settings markdown: kramdown +plugins: + - jekyll-feed + # If you add tags to pages, define this variable to link them to some external search # If you want to link to tags locally on the site, leave this commented out # tag_search_endpoint: https://ask.cyberinfrastructure.org/search?q= @@ -85,7 +91,6 @@ themeColor: red # purple, green, blue, orange, purple, grey fixedNav: 'true' # true or false permalink: /:year/:title/ -markdown: kramdown exclude: [_site, CHANGELOG.md, LICENSE, README.md, vendor] # Collections diff --git a/docs/_data/toc.yml b/docs/_data/toc.yml index 585854a3f..1263604a2 100644 --- a/docs/_data/toc.yml +++ b/docs/_data/toc.yml @@ -11,6 +11,7 @@ url: "configuration-reference" - title: Deployment + url: "homelab" links: - title: "Homelab Deployment" url: "homelab" @@ -24,6 +25,7 @@ url: "decentdb" - title: Core Concepts + url: "libraries" links: - title: Libraries url: "libraries" @@ -31,6 +33,8 @@ url: "jobs" - title: "Charts" url: "charts" + - title: "Custom Blocks" + url: "custom-blocks" - title: "Playlists" url: "playlists" - title: "Jukebox" @@ -51,17 +55,24 @@ url: "shares" - title: "Theming" url: "theming" - - title: "User Device Profiles" + - title: "User Device Profiles (Preview)" url: "user-device-profiles" - title: Tools & Interfaces + url: "cli" links: - title: "Command Line Interface (CLI)" url: "cli" + - title: "CLI Remote Server Mode" + url: "cli-remote-mode" - title: API Overview url: "apis" - - title: API Reference + - title: Native API Reference url: "api" + - title: OpenSubsonic API + url: "api-opensubsonic" + - title: Jellyfin API + url: "api-jellyfin" - title: OpenSubsonic Compatibility Matrix url: "opensubsonic-matrix" diff --git a/docs/_includes/head.html b/docs/_includes/head.html index 072e8720b..9cef72423 100755 --- a/docs/_includes/head.html +++ b/docs/_includes/head.html @@ -3,11 +3,11 @@ - + - + @@ -24,8 +24,8 @@ - - + + @@ -45,8 +45,8 @@ document.addEventListener("DOMContentLoaded", () => { mermaid.initialize({ startOnLoad: true, - securityLevel: "loose", // if you need links; else default - theme: "default" // or 'dark', 'neutral' + securityLevel: "strict", + theme: "default" }); // (Optional) Support fenced ```mermaid blocks if theme doesn’t auto-pick them diff --git a/docs/_posts/2026-01-12-melodee-1-8-0-released.md b/docs/_posts/2026-01-12-melodee-1-8-0-released.md index 246be09cb..ae7d4cfb0 100644 --- a/docs/_posts/2026-01-12-melodee-1-8-0-released.md +++ b/docs/_posts/2026-01-12-melodee-1-8-0-released.md @@ -1,78 +1,33 @@ --- title: "Melodee 1.8.0 Released" +description: Historical release notes for Melodee 1.8.0. date: 2026-01-12 +tags: + - release + - 1.8.0 badges: - type: info tag: release --- -We're excited to announce the release of Melodee 1.8.0, packed with new features for an enhanced music streaming experience! +Melodee 1.8.0 expanded playback, podcast, customization, and chart capabilities. -## What's New in 1.8.0 +## Highlights -### Party Mode -Collaborative listening sessions with shared queues and synchronized playback control. Perfect for gatherings or remote listening parties with friends and family. +- **Party Mode preview** added collaborative session, participant, queue, moderation, and playback-control foundations. +- **Jukebox** added server-side playback adapters for separately installed MPV and MPD backends. +- **Podcasts** added per-user RSS subscriptions, downloads, playback tracking, and OpenSubsonic podcast routes. +- **Themes** added installable color, font, branding, and navigation customization. +- **Charts** added curated ranked-list import, automatic local-album linking, reports, and optional generated playlists. +- **API compatibility** expanded the OpenSubsonic, native, and Jellyfin-compatible music surfaces. +- **Metadata and scrobbling** received additional search-provider and Last.fm improvements. -- Create and manage party sessions -- Shared queue management -- Real-time playback synchronization -- Participant roles and permissions +Current behavior and limitations have evolved since this historical release. Use the current [Party Mode](/party-mode/), [Jukebox](/jukebox/), [Podcasts](/podcasts/), [Theming](/theming/), and [Charts](/charts/) guides when operating a recent server. -### Jukebox Mode -Server-side audio playback via MPV or MPD backends, enabling whole-home audio setups. +## Upgrade -- Control playback from any device -- Support for MPV and MPD audio backends -- Ideal for dedicated audio endpoints +Follow the [Upgrade Guide](/upgrade/) and read every intervening changelog entry before upgrading an older installation. -### Podcast Support -Full podcast subscription and playback support with tracking. - -- Subscribe to podcasts via RSS or OPML import -- Automatic episode downloads -- Playback position tracking -- Integration with OpenSubsonic podcast APIs - -### Custom Theming -Personalize Melodee's appearance with custom themes. - -- Create custom color schemes -- Custom fonts and branding -- Hide/show navigation items -- Theme pack import/export - -### Music Charts -Curated album charts updated automatically. - -- Billboard chart integration -- Multiple chart sources -- Automatic updates via background jobs - -### Additional Improvements - -- **Jellyfin API compatibility** - Use Jellyfin clients with Melodee -- **Deezer search engine** - Additional metadata source -- **Enhanced scrobbling** - Improved Last.fm integration -- **Performance improvements** - Faster library scanning and streaming - -## Upgrading - -To upgrade to 1.8.0, follow the [upgrade guide](/upgrade/). - -Database migrations will run automatically on startup. - -## Documentation - -- [Party Mode Guide](/party-mode/) -- [Jukebox Setup](/jukebox/) -- [Podcast Configuration](/podcasts/) -- [Theming Guide](/theming/) -- [Charts Configuration](/charts/) - -## Thank You - -Thanks to everyone who contributed to this release through bug reports, feature requests, and pull requests! - -Questions or feedback? Join our [Discord community](https://discord.gg/bfMnEUrvbp) or open an issue on [GitHub](https://github.com/melodee-project/melodee/issues). +Questions or feedback are welcome on [GitHub](https://github.com/melodee-project/melodee/issues) and [Discord](https://discord.gg/bfMnEUrvbp). diff --git a/docs/_posts/2026-05-01-melodee-2-0-0-released.md b/docs/_posts/2026-05-01-melodee-2-0-0-released.md index c9dd4b63f..1b2da0985 100644 --- a/docs/_posts/2026-05-01-melodee-2-0-0-released.md +++ b/docs/_posts/2026-05-01-melodee-2-0-0-released.md @@ -1,89 +1,56 @@ --- title: "Melodee 2.0.0 Released" +description: Historical release notes for Melodee 2.0.0 and its .NET 10, onboarding, automation, and security work. date: 2026-05-01 +tags: + - release + - 2.0.0 badges: - type: info tag: release --- -We're thrilled to announce the release of Melodee 2.0.0! This is a major milestone that brings powerful automation, a smoother first-time setup, security hardening, and a move to .NET 10. +Melodee 2.0.0 moved the server to .NET 10 and established the current Blazor, PostgreSQL, container, API, and operational baseline. -## What's New in 2.0.0 +## Highlights -### Event Scripting -Automate your library workflows with JavaScript-powered event scripts using the built-in Monaco code editor. +### Event scripting -- Write scripts that run on directory processing events -- Context-aware code completion in the admin script editor -- Deny actions with path validation and dry-run support -- Validation and orchestration through the CLI +Administrators gained Jint-based JavaScript settings, an editor and test surface, an Inbound directory-processing hook, and selected Blazor page gates. Current scripts fail open; review the [Event Scripting](/scripting/) guide for the exact enforcement boundary. -### Onboarding Wizard -New installations now include a guided setup wizard to get Melodee running in minutes. +### Onboarding and Doctor -- Step-by-step administrator account setup -- Automated health checks via the integrated Doctor service -- Checklist-based configuration tracking -- Admin dashboard warnings for incomplete setup +New installations gained a guided administrator, path, security, branding, and verification workflow plus Doctor checks for missing or invalid setup. -### .NET 10 & DecentDB -Melodee has been upgraded to .NET 10 for improved performance, and the database layer has moved from SQLite to DecentDB for better scalability and maintainability. +### .NET 10 and generated search stores -- .NET 10 runtime support -- DecentDB.EntityFrameworkCore integration -- Cancellation support throughout MusicBrainz imports -- Enhanced query performance and streaming imports +The application moved to .NET 10. PostgreSQL remained the primary application database. DecentDB providers were introduced for generated MusicBrainz and artist-search data; they did not replace PostgreSQL. -### Playlist Import & Export -Manage playlists more flexibly with standard file format support. +### Playlist portability -- Import M3U and M3U8 playlists -- Export playlists to M3U format -- Seamless UI integration for playlist management +Regular playlists gained M3U/M3U8 import and M3U export, including retention of unmatched import references. -### Security Hardening -A comprehensive security audit and remediation effort across the entire codebase. +### Security and operations -- Rate limiting with custom rejection handling for API requests -- Strict CORS policies and secure key generation -- Centralized file path guarding for destructive operations -- SSRF and resource exhaustion protections -- Cache invalidation with concurrency safety -- Correlation ID logging for observability +- API and authentication rate limiting +- Explicit CORS configuration +- Filesystem path guards around destructive operations +- SSRF and response-size protections for external fetches +- Correlation IDs and safer secret handling +- Published multi-architecture container images -### Docker Publishing -Official Docker images are now published automatically on every release via GitHub Actions. +## Upgrade -- Multi-arch container images pushed to GHCR -- Simplified deployment for self-hosted users - -### Additional Improvements - -- **Artist alias lookup** - Improved artist matching via alias database schema -- **Bulk artist management** - Directory diagnosis and bulk operations in the admin UI -- **Dashboard enhancements** - Admin checks, health warnings, and loading states -- **MusicBrainz streaming importer** - Generic streaming methods with cancellation support -- **Artist listing improvements** - Album counts and better pagination -- **Track gap calculation** - Enhanced logic for identifying missing tracks - -## Upgrading - -To upgrade to 2.0.0, follow the [upgrade guide](/upgrade/). - -Database migrations will run automatically on startup. Note the migration from SQLite to DecentDB — review the migration notes if you are upgrading from an existing SQLite installation. +Follow the [Upgrade Guide](/upgrade/). PostgreSQL migrations run during startup; generated DecentDB files have a separate compatibility and [migration procedure](/decentdb/). ## Documentation -- [Event Scripting Guide](/event-scripting/) -- [Onboarding Wizard](/onboarding/) -- [Upgrade Guide](/upgrade/) -- [Docker Deployment](/docker/) -- [Security Overview](/security/) - -## Thank You - -Thanks to everyone who contributed to this release through bug reports, feature requests, security audits, and pull requests! +- [Quick Start](/quickstart/) +- [Installation](/installing/) +- [Configuration](/configuration/) +- [Event Scripting](/scripting/) +- [Backup & Recovery](/backup/) -Questions or feedback? Join our [Discord community](https://discord.gg/bfMnEUrvbp) or open an issue on [GitHub](https://github.com/melodee-project/melodee/issues). +Questions or feedback are welcome on [GitHub](https://github.com/melodee-project/melodee/issues) and [Discord](https://discord.gg/bfMnEUrvbp). diff --git a/docs/_posts/2026-05-24-melodee-2-1-0-released.md b/docs/_posts/2026-05-24-melodee-2-1-0-released.md index 478278340..04576d90f 100644 --- a/docs/_posts/2026-05-24-melodee-2-1-0-released.md +++ b/docs/_posts/2026-05-24-melodee-2-1-0-released.md @@ -1,76 +1,52 @@ --- title: "Melodee 2.1.0 Released" +description: Historical release notes for Melodee 2.1.0 performance, image-processing, migration, and security changes. date: 2026-05-24 +tags: + - release + - 2.1.0 badges: - type: info tag: release --- -Melodee 2.1.0 is here! This release focuses on performance, developer experience, and security hardening — making your server faster to load, easier to maintain, and safer to run. +Melodee 2.1.0 focused on image processing, page and bulk-operation performance, migration maintenance, and security. -## What's New in 2.1.0 +## Highlights -### SkiaSharp Image Processing -All image processing has migrated to `SkiaSharp`, replacing the legacy library with a more performant and actively maintained alternative. +### SkiaSharp image processing -- New `IImageProcessor` abstraction for decode, encode, resize, format detection, and hash generation -- Clean dependency injection throughout services, Blazor components, CLI commands, and tests -- Conditional native asset inclusion (`SkiaSharp.NativeAssets.Linux` on Linux) — no extra runtime dependencies needed +Image decoding, resizing, encoding, format detection, and hashing moved behind the `IImageProcessor` abstraction and SkiaSharp. -### Dashboard Performance -Page loads are faster and feel more responsive thanks to targeted optimizations. +### Faster web and bulk operations -- Dashboard data loading now occurs in `OnAfterRenderAsync` — skeleton placeholders render immediately instead of blocking the initial page paint -- `DoctorService.NeedsAttentionAsync` fast path uses lightweight file-existence checks, cutting dashboard first-render time by ~5 seconds +- Dashboard loading moved noncritical work after the initial render. +- Doctor's attention check gained a lightweight fast path. +- Bulk artist, album, and song deletion replaced per-item database lookups with bounded batch queries. +- Party Mode's Blazor service stopped making loopback HTTP calls and invoked domain services directly. -### Bulk Delete Performance -Deleting large collections is now significantly faster. +### Developer and migration maintenance -- `SongService`, `AlbumService`, and `ArtistService` batch-load entities in a single query instead of executing N+1 queries per item -- Dramatic speed improvements for cleaning up large libraries +- EF Core history was consolidated into an InitialBaseline migration. +- Quartz scheduling moved into a reusable registration helper. +- CI and CodeQL scope were reduced where the removed language analysis was not used. +- Package updates and analyzer cleanup improved repeatability. -### Party Mode Optimization -Party Mode has been refactored for direct in-process performance. +### Security -- `PartyModeService` now calls domain services (`PartySessionService`, `PartyQueueService`, `PartyPlaybackService`, `PartySessionEndpointRegistryService`) directly via dependency injection instead of making HTTP requests to the same application -- Eliminates ~20 HTTP round-trips per user interaction (create, join, leave, queue, playback, endpoints) -- User identity resolved through `IAuthService` rather than cookie authentication +Production password-reset responses stopped returning reset tokens and use the configured email delivery path. -### Developer Experience -Building and maintaining Melodee is now easier for contributors. +## Upgrade -- Squashed 55 EF Core database migrations into a single `InitialBaseline` migration — smaller repository, faster CI builds, no fragile migration chains -- Quartz job scheduling extracted into a reusable helper, shrinking `Program.cs` from 1,046 to ~860 lines -- `.kilo/` project configuration with slash commands (`/build`, `/test`, `/test-mql`, `/lint`, `/migrate`, `/coverage`) and a `melodee-developer` agent for consistent workflows -- CodeQL analysis optimized — JavaScript/TypeScript scanning dropped, saving 5–10 minutes per CI run - -### Security Hardening - -- Password reset endpoint no longer exposes reset tokens in API responses. Tokens are now only returned in development mode; in production, the endpoint returns a generic message and relies on email delivery. - -### Additional Improvements - -- Min-width set on album detail action column for layout stability -- Removed unnecessary `EnsureArtistAliasTableAsync` call in MusicBrainz repository -- NuGet dependency updates across the project - -## Upgrading - -To upgrade to 2.1.0, follow the [upgrade guide](/upgrade/). - -Database migrations will run automatically on startup. If you were on the latest 2.0.x migration, the squashed `InitialBaseline` migration is a no-op — existing databases are unaffected. +Follow the [Upgrade Guide](/upgrade/). Existing PostgreSQL databases at the expected 2.0 migration state treat the consolidated baseline as already satisfied. ## Documentation -- [Upgrade Guide](/upgrade/) -- [Docker Deployment](/docker/) -- [Security Overview](/security/) -- [Event Scripting Guide](/event-scripting/) - -## Thank You - -Thanks to everyone who contributed to this release through bug reports, feature requests, security audits, and pull requests! +- [Installation](/installing/) +- [Upgrade](/upgrade/) +- [Configuration](/configuration/) +- [Event Scripting](/scripting/) -Questions or feedback? Join our [Discord community](https://discord.gg/bfMnEUrvbp) or open an issue on [GitHub](https://github.com/melodee-project/melodee/issues). +Questions or feedback are welcome on [GitHub](https://github.com/melodee-project/melodee/issues) and [Discord](https://discord.gg/bfMnEUrvbp). diff --git a/docs/_posts/2026-07-11-melodee-2-2-0-released.md b/docs/_posts/2026-07-11-melodee-2-2-0-released.md new file mode 100644 index 000000000..2168bd368 --- /dev/null +++ b/docs/_posts/2026-07-11-melodee-2-2-0-released.md @@ -0,0 +1,114 @@ +--- +title: "Melodee 2.2.0 Released" +description: Melodee 2.2.0 adds guided DecentDB migration, media-artist data transfer, dependency alignment, and release-quality fixes. +date: 2026-07-11 +tags: + - release + - 2.2.0 +badges: + - type: info + tag: release +--- + +Melodee 2.2.0 improves DecentDB compatibility and recovery, media-artist data portability, image-processing stability, API-key routing, build quality, and public documentation. + + + +## Guided DecentDB migration + +Doctor now recognizes DecentDB error 8, **Unsupported database format**, and opens a migration dialog tailored to the affected MusicBrainz or Artist Search file. It shows: + +- the detected and required DecentDB versions; +- the configured source and safe destination paths; +- a copyable `decentdb-migrate` command; +- verification and replacement steps; +- links to matching prebuilt DecentDB releases and the official migration guide. + +The primary application database remains PostgreSQL. The migration applies only to generated DecentDB search files. See [DecentDB Usage & Migration](/decentdb/). + +## Search data and migration reliability + +- Artist Search can initialize a missing database through the correct EF Core migration/schema path. +- Artist Search migrations now match the current model and avoid DecentDB's unsupported UUID column-type alteration. +- A compatibility no-op keeps the migration chain stable for existing files. +- Paged Artist Search reads avoid carrying positional parameters between the page query and result count. +- MusicBrainz and Artist Search startup checks provide clearer recovery guidance. + +## Media-artist import and export + +Administrators can export artist, album, and alias search-engine data as JSON and preview/import it with optional overwrite behavior from the Media Artists administration page. + +## Runtime and build quality + +- DecentDB providers were updated to 2.16.1. +- Managed and Linux-native SkiaSharp packages were aligned at 4.150.0, fixing native 119.0 versus managed 150.0 startup failures. +- Radzen and related dependencies were updated. +- Chart, user, user-group, and podcast routes now bind public GUID API keys consistently. +- Remote `mcli` commands no longer duplicate `/api/v1` in request URLs. +- Solution-wide documentation/analyzer builds complete without warnings. +- `Microsoft.OpenApi` is pinned to a patched release. +- Password-reset and adjacent SMTP/authentication logs no longer contain user + identifiers, reset configuration, tokens, or raw exception payloads. Both + unattended setup paths create generated deployment secrets atomically with + owner-only `0600` permissions on POSIX (or the containing directory's ACL on + Windows). +- Startup diagnostics now redact connection strings, passwords, tokens, API + keys, credential-bearing URLs, and unknown environment values by default. +- Python maintenance utilities now use bounded Link-header parsing and omit + demo credentials, generated key material, and sensitive exception payloads. + The GitHub code-scanning exporter also allows requests, pagination, + redirects, and downloads only over HTTPS to the exact configured API origin. +- Incoming cleanup now enforces canonical trusted-root containment, symlink and + Zip Slip defenses, and traversal-safe SFV processing before destructive work. + Live mutation uses pinned no-follow descriptors and atomic no-replace moves + on Linux, fails closed when the required primitives are unavailable, and + creates extracted directories and files with private `0700` and `0600` + permissions. +- The production runtime now uses the official .NET 10 Ubuntu 26.04 base with + FFmpeg 8, removes unused vulnerable inherited tools, and runs the application + as the unprivileged PID 1 process. +- CodeQL now covers GitHub Actions, C#, JavaScript/TypeScript, and Python + through one hardened advanced workflow. Trivy uploads only critical/high + SARIF findings while retaining its complete report as a CI artifact. +- All external actions in the six CI workflows are pinned to verified commits; + the Gitleaks container is pinned to an immutable digest. + +## Security verification + +The remediation started from 423 open code-scanning alerts: 416 Trivy image +findings and seven CodeQL source findings. Source changes fix the six original +C# findings; the Python setup finding is handled by a hardened necessary +persistence boundary and narrow documented suppression. Fresh local analysis +reduced C# from eight additional findings to zero, while GitHub Actions and +JavaScript/TypeScript each report zero. + +Trivy 0.72.0 reduced the production-image inventory from 416 to 140 findings, +removing 276 (66.3%). The remainder represents 41 CVEs: 124 medium and 16 low, +with no critical, high, currently fixable, .NET-package, or application-package +findings. GitHub receives the intended critical/high SARIF policy, while CI +retains the complete all-severity JSON inventory. + +The solution builds with zero warnings, 5,885 .NET tests pass with 34 skipped, +and NuGet reports no vulnerable dependencies. Jekyll, GitHub Actions, YAML, and +shell checks pass. A real PostgreSQL integration reaches healthy status with +the application running unprivileged as PID 1 and confirms that configured +secret values do not appear in container logs. + +Fresh local-threat-model Python analysis reports zero findings and no SARIF +warning/error notifications across the workflow-equivalent 45-query default +suite. The 52-query security-extended suite also reports zero findings. All 109 +Python script tests pass with resource warnings treated as errors, including 57 +focused filesystem, ZIP, SFV, shutdown, and race regressions for incoming +cleanup. + +## Documentation and release automation + +The public documentation now defaults to the 2.2.0 release. The version-bump script also updates the documentation release menu, search scope, and current release marker for future releases. + +The full documentation corpus was reviewed against the 2.2.0 source. Preview boundaries, API routes, CLI commands, container paths, job schedules, settings, and backup/upgrade procedures now distinguish implemented behavior from planned or partially wired features. + +## Upgrade + +Back up PostgreSQL and persistent files, pin the `2.2.0` image, and follow the [Upgrade Guide](/upgrade/). If Doctor reports DecentDB error 8 after startup, follow its generated dialog before replacing any search database. + +Questions or feedback are welcome on [GitHub](https://github.com/melodee-project/melodee/issues) and [Discord](https://discord.gg/bfMnEUrvbp). diff --git a/docs/about.markdown b/docs/about.markdown deleted file mode 100644 index 8b4e0b28c..000000000 --- a/docs/about.markdown +++ /dev/null @@ -1,18 +0,0 @@ ---- -layout: page -title: About -permalink: /about/ ---- - -This is the base Jekyll theme. You can find out more info about customizing your Jekyll theme, as well as basic Jekyll usage documentation at [jekyllrb.com](https://jekyllrb.com/) - -You can find the source code for Minima at GitHub: -[jekyll][jekyll-organization] / -[minima](https://github.com/jekyll/minima) - -You can find the source code for Jekyll at GitHub: -[jekyll][jekyll-organization] / -[jekyll](https://github.com/jekyll/jekyll) - - -[jekyll-organization]: https://github.com/jekyll diff --git a/docs/index.markdown b/docs/index.markdown deleted file mode 100644 index ee2dda998..000000000 --- a/docs/index.markdown +++ /dev/null @@ -1,75 +0,0 @@ ---- -# Feel free to add content and custom Front Matter to this file. -# To modify the layout, see https://jekyllrb.com/docs/themes/#overriding-theme-defaults - -layout: home ---- - -# Welcome to Melodee Documentation - -Melodee is a powerful, self-hosted music management and streaming system designed for large music libraries. Whether you're running a personal homelab, managing a family music collection, or operating a community music server, Melodee provides the tools you need for comprehensive music management. - -## Getting Started - -New to Melodee? Start with our [Quick Start Guide for Homelabs](/quickstart/) to get up and running quickly. - -## Key Features - -- **Scalable Architecture**: Handle millions of tracks with efficient background processing -- **Multi-API Support**: OpenSubsonic, Jellyfin, and native REST API compatibility -- **Advanced Media Processing**: Automatic format conversion, metadata normalization, and validation -- **Staging Workflow**: Review and edit metadata before publishing to your library -- **Party Mode**: Collaborative listening sessions with shared queues and playback control -- **Jukebox Mode**: Server-side audio playback via MPV/MPD for whole-home audio -- **Podcast Support**: Subscribe, download, and stream podcasts with full playback tracking -- **Custom Theming**: Personalize the UI with custom color schemes, fonts, and branding -- **Music Charts**: Curated album charts updated automatically from Billboard and other sources -- **Scrobbling**: Last.fm integration for play history tracking -- **Multiple Client Support**: Compatible with popular Subsonic/Jellyfin clients - -## Homelab Focus - -Melodee is designed with homelab enthusiasts in mind: -- **Container Ready**: Full Docker/Podman support for easy deployment -- **Hardware Flexible**: Runs on everything from Raspberry Pis to full servers -- **Community Driven**: Active community sharing homelab experiences and solutions -- **Self-Hosted**: Complete control over your music and data - -## Documentation Sections - -### Getting Started -- [Quick Start for Homelabs](/quickstart/) - Fast path to your first deployment -- [Installation](/installing/) - Detailed setup instructions -- [Configuration](/configuration/) - Tuning and optimization -- [Configuration Reference](/configuration-reference/) - Comprehensive configuration options - -### Features -- [Libraries](/libraries/) - Managing your music collection -- [Charts](/charts/) - Curated album charts -- [Playlists](/playlists/) - Creating and managing playlists -- [Jukebox](/jukebox/) - Server-side audio playback -- [Party Mode](/party-mode/) - Collaborative listening sessions -- [Podcasts](/podcasts/) - Podcast subscription and playback -- [Theming](/theming/) - Customize the UI appearance -- [Requests](/requests/) - User music requests -- [Scrobbling](/scrobbling/) - Last.fm integration -- [Shares](/shares/) - Sharing music with others -- [Query Language (MQL)](/mql/) - Advanced search queries - -### Deployment & Operations -- [Homelab Deployment](/homelab/) - Homelab-specific guidance -- [Hardware & Performance](/hardware/) - Hardware recommendations and optimization -- [Backup & Recovery](/backup/) - Data protection strategies -- [Background Jobs](/jobs/) - Scheduled task management -- [Command Line Interface](/cli/) - CLI tools and utilities - -### API & Integration -- [API Overview](/apis/) - Comparison of API options -- [API Documentation](/api/) - Native REST API reference - -## Community - -Join our community to get help, share your setup, and connect with other homelab enthusiasts: -- [Discord Community](https://discord.gg/bfMnEUrvbp) -- [GitHub Repository](https://github.com/melodee-project/melodee) -- [GitHub Discussions](https://github.com/melodee-project/melodee/discussions) diff --git a/docs/pages/404.md b/docs/pages/404.md new file mode 100644 index 000000000..c621fd403 --- /dev/null +++ b/docs/pages/404.md @@ -0,0 +1,19 @@ +--- +layout: page +title: Page Not Found +description: The requested Melodee documentation page could not be found. +permalink: /404.html +sitemap: false +not_editable: true +excluded_in_search: true +tags: + - documentation +--- + +# Page Not Found + +The requested page does not exist or its address has changed. + +- [Search the documentation](/search/) +- [Browse the documentation index](/docs/) +- [Return to the Melodee home page](/) diff --git a/docs/pages/about.md b/docs/pages/about.md old mode 100755 new mode 100644 index b5081a6d5..db57a244c --- a/docs/pages/about.md +++ b/docs/pages/about.md @@ -1,97 +1,80 @@ --- title: About +description: Melodee's purpose, architecture, companion clients, support channels, and license. permalink: /about/ +tags: + - about + - architecture + - community --- # About -Melodee is an open source, high‑performance music management and streaming system designed for very large personal or organizational libraries (tens of millions of tracks). It combines a modern Blazor Server UI, a powerful media ingestion and normalization pipeline, and dual API surfaces (OpenSubsonic compatible + native Melodee REST) for broad client compatibility. +Melodee is an open-source, self-hosted system for managing and streaming large music libraries. It combines a staged media-ingestion workflow, a Blazor Server web application, PostgreSQL metadata storage, generated DecentDB search stores, and OpenSubsonic, Jellyfin-compatible, and native REST interfaces. -## Vision +## Project goals -Provide a self‑hosted, privacy‑respecting platform to: ingest messy music collections, enrich them with high‑quality metadata & artwork, curate and stage edits safely, and stream to any Subsonic / OpenSubsonic compatible client—or to custom integrations via a clean JSON REST API. +Melodee is designed to: -## Core Pillars +- turn inconsistent incoming releases into a reviewed Storage library; +- retain operator control over metadata, artwork, validation, and promotion; +- serve music to the built-in browser player and compatible external clients; +- support repeatable container deployment, backup, upgrade, and diagnostics; +- scale through paged queries, background jobs, incremental work, and direct streaming where possible. -- Scales to gigantic libraries through efficient background jobs and incremental scans. -- Pluggable metadata & artwork enrichment (MusicBrainz, Last.FM, iTunes, Spotify, etc.). -- Powerful inbound -> staging -> production workflow for clean, consistent libraries. -- Real‑time transcoding and optimized streaming path (range requests, concurrency limits). -- First‑class API contracts (OpenSubsonic + Melodee native) with versioning. -- Configuration‑driven behavior (rules engine for tag cleanup, naming, validation). +“Designed for large libraries” is an architectural goal, not a fixed capacity guarantee. Actual limits depend on PostgreSQL, filesystem latency, media format, concurrent users, transcoding, and host resources. See [Hardware & Performance](/hardware/) for measurement-based sizing guidance. -## High‑Level Melodee Ecosystem +## Server components -Application Overview +| Component | Responsibility | Technology | +|---|---|---| +| Melodee.Blazor | Web UI, authentication, APIs, streaming, and scheduled-job host | .NET 10, Blazor Server, Radzen | +| Melodee.Common | Domain models, ingestion, metadata, services, persistence, and plug-ins | .NET class library, EF Core | +| Melodee.Cli (`mcli`) | Local maintenance and a small remote command subset | .NET console application | +| PostgreSQL | Primary users, catalog, settings, playlists, requests, and history database | PostgreSQL 17 in the supplied Compose deployment | +| DecentDB files | Generated MusicBrainz and artist-search data | DecentDB | -| Application/Component | Purpose | Key Tech | -|---------------------------------------------------------------------|---------|---------| -| [Melodee Blazor](https://github.com/melodee-project/melodee) | Administrative web UI + OpenSubsonic & native REST API host + streaming pipeline | .NET 10, Blazor Server, Radzen | -| Melodee API (native / OpenSubsonic) | Programmatic access layer consumed by external clients & integrations | ASP.NET Core, API Versioning | -| Melodee.Cli | Operational & maintenance commands (jobs, migrations, utilities) | .NET Console | -| [MeloAmp](https://github.com/melodee-project/meloamp) | Cross‑platform desktop client (browse, play, queue mgmt, theming, equalizer, scrobbling) | Electron, React, Material‑UI, TypeScript | -| [Melodee Player](https://github.com/melodee-project/melodee-player) | Native Android & Android Auto streaming client (voice, Media3 playback, clean architecture) | Kotlin, Jetpack Compose, Media3 | +The primary application database is PostgreSQL. DecentDB does not replace it; see [DecentDB Usage & Migration](/decentdb/). +## Interfaces -### Application/Component Roles +- The **native API** provides JWT-authenticated, versioned JSON routes under `/api/v1`. +- The **OpenSubsonic API** provides compatibility routes under `/rest`. +- The **Jellyfin-compatible API** implements a documented subset for music clients. +- The **Blazor UI** covers administration, curation, browsing, playback, and operational status. +- **mcli** provides local database/filesystem operations and a limited remote JWT mode. -- **Melodee.Blazor**: Hosts APIs, runs background jobs, presents the administrative and power‑user interface (metadata editing, artwork, user/security settings, job dashboard). -- **Melodee API**: Two faces—OpenSubsonic (compatibility for existing ecosystem clients) and Native JSON (clean, opinionated resource models). Both sit behind the same hosting process for efficiency. -- **MeloAmp**: Electron desktop app offering fast browsing (artist / album / song / playlist), drag‑and‑drop queue, starring / favoriting, playlist saving, user theme + equalizer persistence, JWT auth, scrobbling. Ships cross‑platform packages (AppImage, DEB, RPM, Snap, Pacman, tar.gz; Windows & macOS builds planned/experimental). Ideal for desktop users wanting a richer UI than generic Subsonic clients. -- **Melodee Player**: Kotlin/Compose Android + Android Auto app with Clean Architecture layers (data/domain/presentation/service). Provides automotive‑safe UI, voice commands, MediaSession integration, playlist browsing, search, pull‑to‑refresh, persistent now‑playing bar, and scrobbling. Targets API 21–35 with modern tooling (AGP 8.x, Kotlin 1.9+). +Compatibility is endpoint-specific. Consult [API Overview](/apis/), the [OpenSubsonic matrix](/opensubsonic-matrix/), and the [Jellyfin guide](/api-jellyfin/) rather than assuming full protocol or client parity. -Together these deliver an end‑to‑end ecosystem: ingestion & curation (server) → optimized APIs → native & desktop clients tuned for their platform capabilities (voice control in cars, system tray / desktop media keys on desktops—media key integration planned for MeloAmp). +## Companion clients -### Integration Notes +- [MeloAmp](https://github.com/melodee-project/meloamp) is the Electron/React desktop client. Its repository documents current Linux, Windows, and macOS targets, features, packages, and releases. +- [Melodee Player](https://github.com/melodee-project/melodee-player) is the Kotlin/Compose Android and Android Auto client. Its repository documents current Android requirements and automotive capabilities. -- Both clients primarily consume the native Melodee API; OpenSubsonic layer remains available for legacy third‑party apps. -- Scrobbling events originate client‑side and flow through the `Scrobble` native endpoint to update play history and forward to external scrobble services (if configured). -- Equalizer & visual theming (MeloAmp) are client‑local preferences; server interaction limited to playback, metadata, and user actions (ratings, starring). -- Android Auto voice intents are mapped to search + playback actions over the native API; resilience features (retry/backoff) included in player networking stack. +These projects have independent release cycles. Use each repository's README and Releases page as the source of truth for installation and current client features. -## Typical Use Cases +Third-party OpenSubsonic and Jellyfin clients may also connect to the implemented compatibility surfaces. Their behavior varies by the routes and authentication patterns they use. -- Replace aging Subsonic server with a modern, actively maintained alternative. -- Consolidate multiple scattered music folders into a normalized library. -- Run large private label streaming for a band/collective with editorial control. -- Power analytics or recommendation engines via the structured REST endpoints. -- Self-hosted music streaming in homelab environments for personal or family use. -- Media center integration with support for various client types (desktop, mobile, automotive). -- Large-scale music collection management with automated metadata enrichment. +## Typical uses -## Homelab Community +- A private household or homelab music server +- A staged workflow for cleaning and organizing incoming releases +- Multiple Storage roots across local disks or NAS mounts +- A server for desktop, mobile, and automotive clients +- A music catalog exposed to trusted integrations through the native API -Melodee has an active homelab community sharing experiences and solutions: +Features marked **Preview** in their guides are not security or compatibility guarantees. In particular, review the current boundaries for [Party Mode](/party-mode/), [Shares](/shares/), [User Device Profiles](/user-device-profiles/), and [Event Scripting](/scripting/). -- **Discord Server**: Join #homelab channel for homelab-specific discussions -- **GitHub Discussions**: Share your setup and learn from others -- **Community Showcase**: Share your homelab builds and configurations -- **Hardware Recommendations**: Get advice on SBCs, NAS, and server builds +## Support and contributions -## Support +- Use this documentation for installation, configuration, operation, and API behavior. +- Open a [GitHub issue](https://github.com/melodee-project/melodee/issues) for a reproducible bug or documentation defect. +- Use [GitHub Discussions](https://github.com/melodee-project/melodee/discussions) for design and deployment discussion. +- Join the [Discord community](https://discord.gg/bfMnEUrvbp) for community conversation. -Need help? Start with: +Contributions are welcome. Review the repository's contributing guide, code of conduct, and local `AGENTS.md` before changing code. -- Documentation site (this site) for setup & API reference. -- Discord community for real‑time Q&A. -- GitHub Issues for bugs and feature requests. -- GitHub Discussions for design proposals & architecture questions. +## License -If something critical is missing from docs, please open an issue—contributions to documentation are highly valued. - -## Community & Contributions - -We welcome pull requests! Good first contributions include: fixing typos in docs, adding API usage examples, improving test coverage for services, or building new metadata plugins. Please review the Code of Conduct and contributing guidelines before starting. - -## Licensing - -Melodee is MIT licensed—permissive for both personal and commercial use. Attribution is appreciated but not required. - -## Acknowledgments - -Thanks to the wider open media ecosystem and the maintainers of libraries and specifications that make Melodee possible (OpenSubsonic, tag libraries, metadata providers, .NET OSS tooling, and UI component authors). - ---- - -Made with ❤️ by the Melodee community. +Melodee is distributed under the MIT License. See the repository's `LICENSE` file for the authoritative terms. diff --git a/docs/pages/api-jellyfin.md b/docs/pages/api-jellyfin.md index 663113534..a81e8180d 100644 --- a/docs/pages/api-jellyfin.md +++ b/docs/pages/api-jellyfin.md @@ -1,373 +1,117 @@ --- -title: Jellyfin API +title: Jellyfin Compatibility API +description: Connect music clients to Melodee's implemented Jellyfin-compatible routes. permalink: /api-jellyfin/ +tags: + - api + - jellyfin + - clients --- -# Jellyfin API +# Jellyfin Compatibility API -Melodee provides a Jellyfin-compatible API that enables popular Jellyfin music clients to connect and stream your music library. This API implements the core Jellyfin media server endpoints required for music playback, browsing, and management. +Melodee 2.2.0 implements a music-focused subset of Jellyfin's HTTP API. It is a +compatibility layer, not a complete Jellyfin server. -## Overview +The database setting `jellyfin.enabled` controls the API and defaults to +`true`. When disabled, explicitly prefixed compatibility routes return HTTP 404 +and normal Jellyfin route rewriting is not performed. -The Jellyfin API allows clients designed for Jellyfin servers to work seamlessly with Melodee. This includes support for: +## Routing -- **Media Browsing**: Artists, albums, songs, and playlists -- **Streaming**: Direct play and transcoding -- **Library Management**: Favorites, play states, and ratings -- **Session Management**: Playback reporting and scrobbling -- **Playlist Operations**: Create, edit, and manage playlists +Jellyfin clients connect to the Melodee origin, for example: -## Endpoint Structure +```text +https://music.example.com +``` -Jellyfin endpoints are available at the server root with automatic URL rewriting: +Melodee recognizes these unauthenticated discovery/login paths: -- Clients connect to: `http://your-server:port/` -- Internal routing: `/api/jf/*` +- `/System/Info/Public` +- `/System/Ping` +- `/Users/Public` +- `/Users/AuthenticateByName` -The middleware automatically detects Jellyfin client requests via the `Authorization: MediaBrowser` header and routes them appropriately. +For later requests, MediaBrowser authorization headers, Jellyfin token headers, +or an `api_key` on a known Jellyfin path trigger routing to the internal +`/api/jf` controllers. Direct integrations may use the explicit `/api/jf` +prefix, but ordinary Jellyfin clients should use the server root they expect. ## Authentication -### Server Discovery - -Clients discover the server using: - -``` -GET /System/Info/Public -``` - -Response: -```json -{ - "LocalAddress": "http://localhost:5157", - "ServerName": "Melodee", - "Version": "1.7.0", - "ProductName": "Melodee", - "OperatingSystem": "Unix", - "Id": "c33ccd9320a17c12cfda124620290cae", - "StartupWizardCompleted": true -} -``` +Authenticate with the Jellyfin request shape: -### User Authentication - -Authenticate using username and password: - -``` +```http POST /Users/AuthenticateByName Content-Type: application/json -X-Emby-Authorization: MediaBrowser Client="MyClient", Device="MyDevice", DeviceId="unique-id", Version="1.0" +X-Emby-Authorization: MediaBrowser Client="MyClient", Device="MyDevice", DeviceId="stable-device-id", Version="1.0" { - "Username": "your-username", - "Pw": "your-password" + "Username": "alice", + "Pw": "replace-me" } ``` -Response includes an access token: -```json -{ - "User": { - "Id": "user-guid", - "Name": "username" - }, - "AccessToken": "your-access-token" -} -``` +The response includes `AccessToken`, `User`, `ServerId`, and `SessionInfo`. +Supply the returned token on subsequent requests, using the convention expected +by the client, for example: -### Authenticated Requests - -Include the token in subsequent requests: - -``` -Authorization: MediaBrowser Token="your-access-token", Client="MyClient", Device="MyDevice", DeviceId="unique-id", Version="1.0" +```http +Authorization: MediaBrowser Token="TOKEN", Client="MyClient", Device="MyDevice", DeviceId="stable-device-id", Version="1.0" ``` -## Core Endpoints - -### System - -| Method | Path | Description | -|--------|------|-------------| -| GET | /System/Info/Public | Server info (anonymous) | -| GET | /System/Ping | Server ping | -| POST | /System/Ping | Server ping (POST variant) | - -### Users - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Users/Public | List users available for login | -| POST | /Users/AuthenticateByName | Authenticate user | -| GET | /Users/Me | Get current user profile | -| GET | /UserViews | Get user's library views | - -### Items (Library Browsing) - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Items | Query items with filters | -| GET | /Items/{id} | Get single item details | -| GET | /Items/{id}/PlaybackInfo | Get playback/streaming info | -| GET | /Items/{id}/Similar | Get similar items | -| GET | /Items/{id}/InstantMix | Generate instant mix | -| POST | /Items/{id}/Refresh | Request item rescan | -| DELETE | /Items/{id} | Delete item (playlists only) | -| GET | /Items/Filters | Get available filters (genres, years) | - -#### Query Parameters for /Items - -| Parameter | Description | Example | -|-----------|-------------|---------| -| includeItemTypes | Filter by type | `MusicAlbum`, `MusicArtist`, `Audio`, `Playlist` | -| parentId | Filter by parent | Library or artist ID | -| sortBy | Sort field | `SortName`, `DateCreated`, `Random` | -| sortOrder | Sort direction | `Ascending`, `Descending` | -| limit | Max results | `50` | -| startIndex | Pagination offset | `0` | -| searchTerm | Search query | `love songs` | -| genres | Filter by genre | `Rock` | -| years | Filter by year | `2024` | -| isFavorite | Only favorites | `true` | - -### Artists - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Artists | List all artists | -| GET | /Artists/AlbumArtists | List album artists only | -| GET | /Artists/{id}/Similar | Get similar artists | - -### Genres - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Genres | List all genres | -| GET | /MusicGenres | List music genres (alternative endpoint) | - -### Playlists - -| Method | Path | Description | -|--------|------|-------------| -| POST | /Playlists | Create new playlist | -| POST | /Playlists/{id} | Update playlist metadata | -| DELETE | /Playlists/{id} | Delete playlist | -| GET | /Playlists/{id}/Items | Get playlist items | -| POST | /Playlists/{id}/Items | Add items to playlist | -| DELETE | /Playlists/{id}/Items | Remove items from playlist | -| POST | /Playlists/{id}/Items/{itemId}/Move/{newIndex} | Reorder playlist item | - -### Favorites and Play State - -| Method | Path | Description | -|--------|------|-------------| -| POST | /Users/{userId}/FavoriteItems/{itemId} | Add to favorites | -| DELETE | /Users/{userId}/FavoriteItems/{itemId} | Remove from favorites | -| POST | /Users/{userId}/PlayedItems/{itemId} | Mark as played | -| DELETE | /Users/{userId}/PlayedItems/{itemId} | Mark as unplayed | - -### Playback and Sessions - -| Method | Path | Description | -|--------|------|-------------| -| POST | /Sessions/Playing | Report playback started | -| POST | /Sessions/Playing/Progress | Report playback progress | -| POST | /Sessions/Playing/Stopped | Report playback stopped (triggers scrobble) | -| POST | /Sessions/Logout | Logout/revoke token | -| POST | /Sessions/Capabilities | Register client capabilities | -| POST | /Sessions/Capabilities/Full | Register full capabilities | -| GET | /Sessions | Get active sessions | - -### Streaming - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Audio/{id}/universal | Stream audio (supports transcoding) | -| HEAD | /Audio/{id}/universal | Check stream availability | -| GET | /Audio/{id}/stream | Direct stream | - -#### Streaming Parameters - -| Parameter | Description | Default | -|-----------|-------------|---------| -| container | Output format | `mp3`, `opus`, `flac` | -| audioBitRate | Target bitrate | `320000` | -| maxStreamingBitrate | Max bitrate | `999999999` | -| transcodingContainer | Transcoding format | `mp3` | - -### Images - -| Method | Path | Description | -|--------|------|-------------| -| GET | /Items/{id}/Images/Primary | Get primary image | -| GET | /Items/{id}/Images/{imageType} | Get specific image type | -| GET | /Audio/{id}/Lyrics | Get song lyrics | - -## Client Setup Guides - -### Finamp - -[Finamp](https://github.com/jmshrv/finamp) is a popular open-source Jellyfin music client for iOS, Android, and Desktop. - -1. **Download Finamp** from your app store or GitHub releases -2. **Add Server**: - - Open Finamp and tap "Add Server" - - Enter your Melodee server URL: `http://your-server:port` - - Finamp will detect the server automatically -3. **Login**: - - Enter your Melodee username and password - - Finamp will authenticate and load your library -4. **Configuration Tips**: - - Enable offline downloads for your favorite albums - - Configure transcoding quality in settings - - Set up lyrics display if your library has embedded lyrics - -### Feishin (Jellyfin Mode) - -[Feishin](https://github.com/jeffvli/feishin) is a modern music client that supports both Jellyfin and Subsonic servers. - -1. **Download Feishin** from GitHub releases -2. **Add Server**: - - Click "Add Server" - - Select "Jellyfin" as the server type - - Enter your Melodee server URL: `http://your-server:port` -3. **Login**: - - Enter your username and password - - Feishin will connect and index your library -4. **Features**: - - Modern, customizable interface - - Gapless playback support - - Queue management and playlists - - Keyboard shortcuts for power users - -### Streamyfin - -[Streamyfin](https://github.com/streamyfin/streamyfin) is a Jellyfin client focused on music streaming for iOS and Android. - -1. **Install Streamyfin** from your app store -2. **Add Server**: - - Tap "Add Server" on first launch - - Enter: `http://your-server:port` -3. **Authenticate** with your Melodee credentials -4. **Enjoy**: - - Browse your library by artist, album, or genre - - Create and manage playlists - - Stream with background playback support - -### Gelli - -[Gelli](https://github.com/dkanada/gelli) is an Android music player for Jellyfin servers. - -1. **Install Gelli** from F-Droid or GitHub -2. **Configure Server**: - - Open Settings → Server - - Enter your Melodee URL -3. **Login** with your credentials -4. **Start Streaming**: - - Browse artists and albums - - Build your queue - - Configure audio quality settings - -## Playback Reporting - -Melodee tracks playback through session reporting endpoints. When clients report playback stopped, Melodee: - -1. Records the play in the user's history -2. Updates play counts -3. Scrobbles to Last.fm (if configured) -4. Updates "Recently Played" lists - -### Reporting Flow - -``` -Client starts playback → POST /Sessions/Playing -Client updates progress → POST /Sessions/Playing/Progress (periodic) -Client stops playback → POST /Sessions/Playing/Stopped (triggers scrobble) -``` - -## Compatibility Notes - -### Supported Features - -- ✅ Library browsing (artists, albums, songs) -- ✅ Audio streaming (direct play and transcoding) -- ✅ Playlist management -- ✅ Favorites and ratings -- ✅ Play state tracking -- ✅ Instant mix generation -- ✅ Similar items recommendations -- ✅ Search functionality -- ✅ Genre and year filtering -- ✅ Session management -- ✅ Scrobbling integration - -### Video and TV Features - -Melodee is a music-focused server. Video-related Jellyfin features are not implemented: - -- ❌ Video playback -- ❌ TV shows and movies -- ❌ Live TV -- ❌ Subtitles -- ❌ Video transcoding - -### Known Limitations - -- Image types are limited to Primary and Album art -- Some advanced Jellyfin features may return empty results -- User management is handled through Melodee's native interface - -## Troubleshooting - -### Client Can't Connect - -1. Verify the server URL is correct and accessible -2. Check that the `Authorization: MediaBrowser` header is being sent -3. Ensure no firewall is blocking the connection -4. Try accessing `/System/Info/Public` directly in a browser - -### Authentication Fails - -1. Verify username and password are correct -2. Check that the user account is not locked -3. Ensure the client is sending proper MediaBrowser headers - -### Playback Issues - -1. Check server logs for streaming errors -2. Verify the audio file format is supported -3. Try a different transcoding setting in the client -4. Check available disk space for transcoding cache - -### Missing Library Content - -1. Ensure your music is properly indexed in Melodee -2. Check that albums have valid metadata -3. Run a library scan from Melodee's admin interface -4. Verify the user has access to the library - -## API Testing - -You can test Jellyfin API endpoints using curl: - -```bash -# Server discovery -curl http://your-server:port/System/Info/Public - -# Ping -curl http://your-server:port/System/Ping - -# Authenticate (save token for subsequent requests) -curl -X POST http://your-server:port/Users/AuthenticateByName \ - -H "Content-Type: application/json" \ - -H "X-Emby-Authorization: MediaBrowser Client=\"curl\", Device=\"test\", DeviceId=\"test123\", Version=\"1.0\"" \ - -d '{"Username":"your-user","Pw":"your-password"}' - -# Get albums (with token) -curl http://your-server:port/Items?includeItemTypes=MusicAlbum \ - -H "Authorization: MediaBrowser Token=\"YOUR_TOKEN\", Client=\"curl\", Device=\"test\", DeviceId=\"test123\", Version=\"1.0\"" -``` - ---- - -For the complete Jellyfin API specification, refer to the [Jellyfin API Documentation](https://api.jellyfin.org/). Melodee implements the subset of endpoints relevant to music streaming. +Melodee also parses `X-MediaBrowser-Token`, `X-Emby-Token`, and `api_key`. +Jellyfin access tokens are separate from native Melodee JWTs and OpenSubsonic +credentials. By default they expire after 168 hours, and at most 10 active +tokens are retained per user; both values are configurable. `POST +/Sessions/Logout` revokes the active token. + +## Implemented Route Groups + +All paths below are the client-visible root form. The equivalent explicit form +adds `/api/jf` before the path. + +| Area | Routes | +|------|--------| +| Discovery | `/`, `/System/Info/Public`, `/System/Ping`, `/System/Info` | +| Users | `/Users/Public`, `/Users/AuthenticateByName`, `/Users/Me`, `/Users`, `/Users/{userId}` | +| User library | `/Users/{userId}/Views`, `/Users/{userId}/Items`, `/Users/{userId}/Items/{itemId}` | +| User state | `/Users/{userId}/FavoriteItems/{itemId}`, `/Users/{userId}/PlayedItems/{itemId}` | +| Views and items | `/UserViews`, `/Items`, `/Items/{itemId}`, playback info, filters, similar items, instant mixes, files, and downloads | +| Artists and genres | `/Artists`, `/Artists/AlbumArtists`, artist detail/similar routes, `/Genres`, `/MusicGenres` | +| Songs and audio | `/Songs/{itemId}/InstantMix`, `/Audio/{itemId}/stream`, `/Audio/{itemId}/stream.{extension}`, `/Audio/{itemId}/universal`, lyrics | +| Images | Item and artist image routes beneath `/Items/*/Images` and `/Artists/*/Images` | +| Playlists | List, detail, create, update, delete, item add/remove/reorder routes beneath `/Playlists` | +| Sessions | Playing, progress, stopped, ping, capabilities, list, and logout beneath `/Sessions` | + +The route may accept only the music-relevant portion of an upstream request, +and response fields that have no Melodee equivalent can be empty or omitted. +Video, television, books, plugins, Live TV, server administration, and the rest +of the full Jellyfin API are outside this compatibility surface. + +## Streaming + +Audio routes support direct streaming and universal streaming. The universal +route evaluates container and bitrate options and can transcode through the +configured FFmpeg executable. Range requests are supported where applicable. +The actual source format, requested parameters, and server configuration +determine whether playback is direct or transcoded. + +## Client Compatibility + +A client must tolerate a music-only library and absent non-music endpoints. +Different client releases probe different Jellyfin routes, so compatibility can +change independently of Melodee. Before standardizing on a client, verify: + +1. discovery and password login; +2. artist, album, and playlist browsing; +3. direct play, seeking, and any required transcode format; +4. favorites and playback reporting; +5. playlist changes and downloads, if required. + +Do not assume a client is supported solely because it can log in. Capture the +failed HTTP route and response when reporting a compatibility issue. + +See [API Overview](/apis/) for Melodee's other API surfaces and +[Configuration Reference](/configuration-reference/) for database-backed +settings. diff --git a/docs/pages/api-opensubsonic.md b/docs/pages/api-opensubsonic.md index 8bb7f1803..fdc2cd52f 100644 --- a/docs/pages/api-opensubsonic.md +++ b/docs/pages/api-opensubsonic.md @@ -1,485 +1,117 @@ --- title: OpenSubsonic API +description: Connect OpenSubsonic and Subsonic clients to Melodee's implemented compatibility endpoints. permalink: /api-opensubsonic/ +tags: + - api + - opensubsonic + - clients --- # OpenSubsonic API -Melodee implements the OpenSubsonic API specification, providing compatibility with the wide ecosystem of Subsonic and OpenSubsonic clients. This API enables seamless integration with popular music streaming applications. +Melodee exposes a music-focused Subsonic/OpenSubsonic compatibility API beneath +`/rest`. Both route forms are accepted: -## Overview - -The OpenSubsonic API is based on the Subsonic 1.16.1 specification with OpenSubsonic extensions. It provides: - -- **Full Subsonic Compatibility**: Works with legacy Subsonic clients -- **OpenSubsonic Extensions**: Enhanced features for modern clients -- **Real-time Transcoding**: On-the-fly format conversion -- **Scrobbling**: Last.fm and internal play tracking -- **Playlist Management**: Full CRUD operations -- **Multi-user Support**: Per-user libraries and preferences - -## Endpoint Structure - -All OpenSubsonic endpoints are available under the `/rest/` path: - -``` -http://your-server:port/rest/{endpoint}.view +```text +https://music.example.com/rest/getAlbum +https://music.example.com/rest/getAlbum.view ``` -Or using the simplified format (without `.view`): - -``` -http://your-server:port/rest/{endpoint} -``` +Most endpoints accept GET or `application/x-www-form-urlencoded` POST requests. +The [compatibility matrix](/opensubsonic-matrix/) lists the implemented and +intentionally unsupported endpoints in Melodee 2.2.0. ## Authentication -OpenSubsonic supports multiple authentication methods: - -### Token-Based (Recommended) +For widest client compatibility, use Subsonic token authentication: +```text +u=alice +t=md5(password + salt) +s=random-salt +v=1.16.1 +c=client-name ``` -GET /rest/ping?u=username&t=token&s=salt&v=1.16.1&c=myapp -``` - -Where: -- `u` = username -- `t` = MD5(password + salt) -- `s` = random salt string -- `v` = API version -- `c` = client name -### API Key Authentication (Melodee Extension) +For example: -``` -GET /rest/ping?apiKey=your-api-key&v=1.16.1&c=myapp +```text +GET /rest/getArtists?u=alice&t=HASH&s=SALT&v=1.16.1&c=my-client&f=json ``` -### Legacy Password (Not Recommended) +Melodee also accepts the legacy `p` parameter as either a clear-text password +or `enc:` followed by the password's hexadecimal bytes. This mode can expose +reusable credentials and should be avoided except for clients that provide no +token-authentication option. Always use HTTPS. -``` -GET /rest/ping?u=username&p=password&v=1.16.1&c=myapp -``` +`ping` and `getOpenSubsonicExtensions` can be called without authentication. +Other routes normally authenticate the user. Local browser requests can also +use Melodee's internal cookie flow; third-party clients should not depend on +that bypass. ## Response Formats -Responses are available in JSON (default) or XML: - -``` -GET /rest/ping?f=json # JSON response -GET /rest/ping?f=xml # XML response -``` - -### JSON Response Structure - -```json -{ - "subsonic-response": { - "status": "ok", - "version": "1.16.1", - "type": "Melodee", - "serverVersion": "1.0.0.0", - "openSubsonic": true, - ... - } -} -``` - -### Error Response - -```json -{ - "subsonic-response": { - "status": "failed", - "version": "1.16.1", - "error": { - "code": 40, - "message": "Wrong username or password" - } - } -} -``` - -## Core Endpoints - -### System - -| Endpoint | Description | -|----------|-------------| -| ping | Test connectivity and authentication | -| getLicense | Get server license information | -| getOpenSubsonicExtensions | List supported OpenSubsonic extensions | - -### Browsing - -| Endpoint | Description | -|----------|-------------| -| getMusicFolders | Get all music libraries | -| getIndexes | Get artist index (A-Z listing) | -| getArtists | Get all artists | -| getArtist | Get artist details and albums | -| getAlbum | Get album details and songs | -| getSong | Get song details | -| getGenres | Get all genres | -| getMusicDirectory | Get directory contents | - -### Album/Song Lists - -| Endpoint | Description | -|----------|-------------| -| getAlbumList | Get albums by various criteria | -| getAlbumList2 | Get albums (ID3 tag based) | -| getRandomSongs | Get random songs | -| getSongsByGenre | Get songs by genre | -| getNowPlaying | Get currently playing songs | -| getStarred | Get starred items | -| getStarred2 | Get starred items (ID3 based) | - -### Searching - -| Endpoint | Description | -|----------|-------------| -| search2 | Search artists, albums, songs | -| search3 | Search (ID3 tag based) | - -### Playlists - -| Endpoint | Description | -|----------|-------------| -| getPlaylists | Get all playlists | -| getPlaylist | Get playlist details | -| createPlaylist | Create new playlist | -| updatePlaylist | Update playlist | -| deletePlaylist | Delete playlist | - -### Media Retrieval - -| Endpoint | Description | -|----------|-------------| -| stream | Stream audio file | -| download | Download audio file | -| getCoverArt | Get album/artist artwork | -| getLyrics | Get song lyrics | -| getAvatar | Get user avatar | - -### User Data - -| Endpoint | Description | -|----------|-------------| -| star | Star an item | -| unstar | Unstar an item | -| setRating | Set item rating | -| scrobble | Submit scrobble | -| getUser | Get user information | - -### Media Annotation - -| Endpoint | Description | -|----------|-------------| -| getSimilarSongs | Get similar songs | -| getSimilarSongs2 | Get similar songs (ID3 based) | -| getTopSongs | Get top songs for artist | - -## OpenSubsonic Extensions - -Melodee supports these OpenSubsonic extensions: - -```json -{ - "openSubsonicExtensions": [ - {"name": "melodeeExtensions", "versions": [1]}, - {"name": "apiKeyAuthentication", "versions": [1]}, - {"name": "formPost", "versions": [1]}, - {"name": "songLyrics", "versions": [1]}, - {"name": "transcodeOffset", "versions": [1]} - ] -} -``` - -### Extension Details - -- **apiKeyAuthentication**: Authenticate using API keys instead of password tokens -- **formPost**: Support for form-encoded POST requests -- **songLyrics**: Enhanced lyrics support with timing information -- **transcodeOffset**: Start transcoding from a specific position - -## Client Setup Guides - -### Supersonic - -[Supersonic](https://github.com/dweymouth/supersonic) is a modern, cross-platform desktop client. - -1. **Download** from GitHub releases for your platform -2. **Add Server**: - - Open Settings → Servers → Add Server - - Name: `My Melodee Server` - - URL: `http://your-server:port` - - Username: Your Melodee username - - Password: Your Melodee password -3. **Test Connection** and save -4. **Enjoy Features**: - - Gapless playback - - ReplayGain support - - Offline caching - - Keyboard shortcuts - - Queue management - -### Feishin (Subsonic Mode) - -[Feishin](https://github.com/jeffvli/feishin) supports both Jellyfin and Subsonic servers. - -1. **Download** from GitHub releases -2. **Add Server**: - - Click "Add Server" - - Select "Navidrome" or "Subsonic" as server type - - Enter URL: `http://your-server:port` -3. **Login** with your credentials -4. **Features**: - - Modern UI with themes - - Smart playlists - - Lyrics display - - Discord rich presence - -### Sublime Music - -[Sublime Music](https://github.com/sublime-music/sublime-music) is a GTK-based Linux client. - -1. **Install** via your package manager or pip: - ```bash - pip install sublime-music - ``` -2. **Configure**: - - Launch Sublime Music - - Add server in Settings - - Enter your Melodee server URL and credentials -3. **Features**: - - Native Linux integration - - Offline support - - Chromecast support - - MPRIS integration - -### Symphonium - -[Symphonium](https://symfonium.app/) is a premium Android client with extensive features. +The default response is XML. Select a format with `f`: -1. **Install** from Google Play Store -2. **Add Server**: - - Menu → Settings → Servers → Add - - Select "Subsonic" provider - - Enter server details -3. **Features**: - - Android Auto support - - Chromecast/DLNA - - Offline sync - - Material You theming +| Value | Result | +|-------|--------| +| `xml` or omitted | XML | +| `json` | JSON | +| `jsonp` | JSONP; also requires `callback` | -### DSub - -[DSub](https://github.com/daneren2005/Subsonic) is a mature Android client. - -1. **Install** from Google Play or F-Droid -2. **Configure Server**: - - Settings → Servers → Add Server - - Server URL: `http://your-server:port` - - Username and password -3. **Test Connection** -4. **Features**: - - Offline caching - - Playlist sync - - Shuffle by album - - Gapless playback - -### Ultrasonic - -[Ultrasonic](https://gitlab.com/ultrasonic/ultrasonic) is an open-source Android client. - -1. **Install** from F-Droid or Google Play -2. **Add Server**: - - Settings → Servers → Add Server - - Enter URL: `http://your-server:port` - - Enter credentials -3. **Features**: - - Open source - - Offline playback - - Background playback - - Playlist management - -### MeloAmp (Desktop) - -[MeloAmp](https://github.com/melodee-project/meloamp) is the official Melodee desktop client. - -1. **Download** from GitHub releases -2. **Connect**: - - Enter your Melodee server URL - - Login with your credentials -3. **Features**: - - Native Melodee API integration - - Fallback to OpenSubsonic - - Desktop integration - - Equalizer and themes - -### Melodee Player (Android) - -[Melodee Player](https://github.com/melodee-project/melodee-player) is the official Android client. - -1. **Install** from releases -2. **Configure** server URL and login -3. **Features**: - - Android Auto support - - Background playback - - Offline caching - - Material Design UI - -## Streaming - -### Basic Streaming - -``` -GET /rest/stream?id=song-id&u=user&t=token&s=salt&c=client&v=1.16.1 +```text +GET /rest/ping?f=json&v=1.16.1&c=my-client ``` -### Transcoding Parameters +Structured responses use the standard `subsonic-response` envelope with +`status`, protocol version, server type/version, data, or an OpenSubsonic error. +Streaming and image endpoints instead return binary content and the relevant +HTTP headers. -| Parameter | Description | Example | -|-----------|-------------|---------| -| maxBitRate | Max bitrate in kbps | `320` | -| format | Output format | `mp3`, `opus`, `flac` | -| timeOffset | Start position in seconds | `30` | -| estimateContentLength | Include Content-Length | `true` | +## Implemented Areas -### Example: Stream with Transcoding +Melodee provides routes for: -``` -GET /rest/stream?id=abc123&maxBitRate=128&format=mp3&u=user&t=token&s=salt&c=myapp&v=1.16.1 -``` - -## Scrobbling - -Submit plays to Last.fm and internal tracking: +- artists, albums, songs, genres, folders, indexes, lists, and search; +- stream, download, cover art, avatar, and lyrics retrieval; +- starring, ratings, scrobbling, playlists, bookmarks, and play queues; +- shares, internet radio, library scans, and conditional jukebox control; +- podcast subscriptions and episodes when podcasts and the user's podcast role + are enabled; +- selected user operations (`getUser` and `createUser`). -``` -GET /rest/scrobble?id=song-id&submission=true&u=user&t=token&s=salt&c=client&v=1.16.1 -``` +Chat, video, HLS, and caption routes return HTTP 410. `updateUser`, +`deleteUser`, `changePassword`, and `getUsers` return HTTP 501. These explicit +responses let clients detect unsupported functionality without mistaking it +for a missing route. -Parameters: -- `id`: Song ID -- `submission`: `true` for play complete, `false` for now playing -- `time`: Unix timestamp (optional) +## Server Extensions -## Playlist Operations +`getOpenSubsonicExtensions` currently advertises: -### Create Playlist +- `melodeeExtensions` +- `apiKeyAuthentication` +- `formPost` +- `songLyrics` +- `transcodeOffset` -``` -GET /rest/createPlaylist?name=My%20Playlist&songId=id1&songId=id2&u=user&t=token&s=salt&c=client&v=1.16.1 -``` +Clients should still negotiate individual features and handle an unsupported +response. Token-plus-salt authentication is the documented interoperable login +method for Melodee 2.2.0. -### Update Playlist - -``` -GET /rest/updatePlaylist?playlistId=123&name=New%20Name&songIdToAdd=id1&songIndexToRemove=0&u=user&t=token&s=salt&c=client&v=1.16.1 -``` - -### Delete Playlist - -``` -GET /rest/deletePlaylist?id=123&u=user&t=token&s=salt&c=client&v=1.16.1 -``` +## Client Setup -## Error Codes +In a compatible client, enter the origin of the Melodee installation, such as +`https://music.example.com`, not a native `/api/v1` URL. Select token +authentication if the client offers it and choose an OpenSubsonic/Subsonic API +version no newer than the client and server both support. -| Code | Description | -|------|-------------| -| 0 | Generic error | -| 10 | Required parameter missing | -| 20 | Incompatible client version | -| 30 | Incompatible server version | -| 40 | Wrong username or password | -| 41 | Token authentication not supported | -| 50 | User is not authorized | -| 60 | Trial period expired | -| 70 | Data not found | - -## API Testing - -Test connectivity and authentication: - -```bash -# Generate token -SALT=$(openssl rand -hex 16) -TOKEN=$(echo -n "your-password$SALT" | md5sum | cut -d' ' -f1) - -# Test ping -curl "http://your-server:port/rest/ping?u=username&t=$TOKEN&s=$SALT&v=1.16.1&c=curl&f=json" - -# Get artists -curl "http://your-server:port/rest/getArtists?u=username&t=$TOKEN&s=$SALT&v=1.16.1&c=curl&f=json" - -# Search -curl "http://your-server:port/rest/search3?query=love&u=username&t=$TOKEN&s=$SALT&v=1.16.1&c=curl&f=json" -``` - -## Compatibility Notes - -### Fully Supported - -- ✅ All browsing endpoints -- ✅ Streaming with transcoding -- ✅ Playlist management -- ✅ User ratings and stars -- ✅ Scrobbling (Last.fm) -- ✅ Cover art retrieval -- ✅ Lyrics support -- ✅ Similar songs/artists -- ✅ Podcasts (subscribe, download, stream, bookmarks) - -### Partially Supported - -- ⚠️ Bookmarks (basic support) -- ⚠️ Shares (via Melodee native shares) - -### Not Applicable - -- ❌ Video streaming (music server only) -- ❌ Chat (deprecated in spec) -- ❌ Jukebox (planned for future) - -## Troubleshooting - -### Authentication Failed - -1. Verify token calculation: `MD5(password + salt)` -2. Ensure salt is being sent correctly -3. Check username/password -4. Try API key authentication if supported by client - -### Streaming Issues - -1. Check audio file format is supported -2. Verify transcoding settings -3. Check server logs for errors -4. Try different maxBitRate values - -### Missing Content - -1. Ensure music is indexed in Melodee -2. Check user has library access -3. Run library scan from admin -4. Verify metadata is valid - -### Client Not Connecting - -1. Test `/rest/ping` endpoint directly -2. Check firewall settings -3. Verify server URL format -4. Check for HTTPS requirements - ---- - -## Compatibility Matrix - -For a detailed compatibility matrix including endpoint support status, client compatibility notes, and known limitations, see the [OpenSubsonic Compatibility Matrix](/opensubsonic-matrix/). - ---- +Client behavior varies by version and by which optional endpoints it assumes. +Passing login and basic browsing does not guarantee that every client feature +is implemented. Use the compatibility matrix and test streaming, seeking, +playlists, downloads, and scrobbling before relying on a client operationally. -For the complete OpenSubsonic specification, visit [opensubsonic.netlify.app](https://opensubsonic.netlify.app/). +See the upstream [OpenSubsonic API specification](https://opensubsonic.netlify.app/docs/endpoints/) +for parameter semantics; where it differs, Melodee's matrix and running +behavior take precedence. diff --git a/docs/pages/api.md b/docs/pages/api.md index f1012df70..13869beb0 100644 --- a/docs/pages/api.md +++ b/docs/pages/api.md @@ -1,354 +1,138 @@ --- -title: Melodee API +title: Melodee Native API +description: Authenticate with Melodee and integrate with its versioned native REST API. permalink: /api/ +tags: + - api + - jwt + - integration --- -# Melodee API +# Melodee Native API -This page documents the native Melodee REST API. For an overview of all available APIs (Melodee, OpenSubsonic, and Jellyfin), see the [API Overview](/apis/) page. +The native REST API is versioned beneath `/api/v1`. The running application +provides the authoritative request and response schemas: -For interactive API documentation and testing, visit the Scalar UI at `/scalar/v1` when Melodee is running. You can also download the OpenAPI specification at `/openapi/v1.json`. +- Scalar UI: `/scalar/v1` +- OpenAPI JSON: `/openapi/v1.json` -## Authentication - -All native endpoints (unless explicitly noted) require an API key associated with the user making the request. Provide it via header: - -``` -Authorization: Bearer -``` - -If an endpoint returns 401 with `{ error: "Authorization token is invalid" }`, verify the key or that the user is not locked. A 403 indicates the user is locked or blacklisted. +Use those resources for generated clients and exact parameters. This page +covers authentication, conventions, and route families. -## Versioning - -Prefix: `/api/v1/` (future versions will increment the path number; multiple versions may run side‑by‑side). +## Authentication -## Common Response Shape +Most native endpoints require a short-lived JWT, not a user's persistent GUID +API key. Authenticate with either `userName` or `email` plus a password: -Paginated list endpoints respond with: +```http +POST /api/v1/auth/authenticate +Content-Type: application/json -``` { - "meta": { - "totalCount": , - "pageSize": , - "page": , - "totalPages": - }, - "data": [ ... ] + "userName": "alice", + "email": null, + "password": "replace-me" } ``` -Errors: +The response includes `user`, `serverVersion`, `token`, `expiresAt`, +`refreshToken`, and `refreshTokenExpiresAt`. Send the `token` on protected +requests: +```http +Authorization: Bearer eyJ... ``` -{ "error": "Message" } -``` - -## Endpoints -### System +The default access-token lifetime is 15 minutes and the default refresh-token +lifetime is 30 days. Deployments can override both. Refresh tokens rotate: -| Method | Path | Description | Auth | -|--------|------|-------------|------| -| GET | /api/v1/System/info | Server type & semantic version. | No | -| GET | /api/v1/System/stats | Selected system statistics (songs, albums, etc.). | Yes | +```http +POST /api/v1/auth/refresh-token +Content-Type: application/json -#### GET /api/v1/System/info -Returns: -``` { - "serverType": "Melodee", - "name": "Melodee API", - "majorVersion": 1, - "minorVersion": 0, - "patchVersion": 0 + "refreshToken": "token-from-the-previous-response", + "deviceId": "optional-stable-device-id" } ``` -### Artists - -Base: `/api/v1/artists` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/artists/{id} | — | Get artist by ID (GUID). | -| GET | /api/v1/artists | q, page, pageSize, orderBy, orderDirection | List artists with pagination and filtering. | -| GET | /api/v1/artists/recent | limit | Get recently added artists. | -| GET | /api/v1/artists/{id}/albums | page, pageSize, orderBy, orderDirection | Get albums for an artist. | -| GET | /api/v1/artists/{id}/songs | q, page, pageSize, orderBy, orderDirection | Get songs for an artist. | -| POST | /api/v1/artists/starred/{apiKey}/{isStarred} | — | Toggle starred status for an artist. | -| POST | /api/v1/artists/setrating/{apiKey}/{rating} | — | Set rating for an artist (0-5). | -| POST | /api/v1/artists/hated/{apiKey}/{isHated} | — | Toggle hated status for an artist. | - -### Playlists - -Base: `/api/v1/playlists` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/playlists/{id} | — | Get playlist by ID. | -| GET | /api/v1/playlists | page, pageSize | List all playlists. | -| GET | /api/v1/playlists/{apiKey}/songs | page, pageSize | Get songs in a playlist. | -| POST | /api/v1/playlists | `{ name, comment, isPublic, songIds }` | Create a new playlist. | -| PUT | /api/v1/playlists/{apiKey} | `{ name, comment, isPublic }` | Update playlist metadata. | -| DELETE | /api/v1/playlists/{apiKey} | — | Delete a playlist. | -| POST | /api/v1/playlists/{apiKey}/songs | `[ songId1, songId2, ... ]` | Add songs to a playlist. | -| DELETE | /api/v1/playlists/{apiKey}/songs | `[ songId1, songId2, ... ]` | Remove songs from a playlist. | -| PUT | /api/v1/playlists/{apiKey}/songs/reorder | `{ songIds: [...] }` | Reorder songs (send full list of IDs in new order). | -| POST | /api/v1/playlists/{apiKey}/image | Form File | Upload playlist cover image. | -| DELETE | /api/v1/playlists/{apiKey}/image | — | Delete playlist cover image. | - -### User - -Base: `/api/v1/user` - -Endpoints for the currently authenticated user. - -| Method | Path | Description | -|--------|------|-------------| -| GET | /api/v1/user/me | Get current user profile. | -| GET | /api/v1/user/last-played | Get last played songs. | -| GET | /api/v1/user/playlists | Get user's playlists. | -| GET | /api/v1/user/songs/liked | Get liked (starred) songs. | -| GET | /api/v1/user/songs/disliked | Get disliked (hated) songs. | -| GET | /api/v1/user/songs/rated | Get rated songs. | -| GET | /api/v1/user/songs/top-rated | Get top rated songs (4+ stars). | -| GET | /api/v1/user/songs/recently-played | Get recently played songs. | -| GET | /api/v1/user/albums/* | Similar endpoints for albums (liked, disliked, etc.). | -| GET | /api/v1/user/artists/* | Similar endpoints for artists (liked, disliked, etc.). | -| GET | /api/v1/user/me/linked-providers | Get linked social accounts. | -| POST | /api/v1/user/me/link/google | Link Google account. | -| DELETE | /api/v1/user/me/link/google | Unlink Google account. | - -### Shares - -Base: `/api/v1/shares` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/shares/{apiKey} | — | Get share details. | -| GET | /api/v1/shares | page, pageSize | List user's shares. | -| POST | /api/v1/shares | `{ shareType, resourceId, description, ... }` | Create a share. | -| PUT | /api/v1/shares/{apiKey} | `{ description, isDownloadable, expiresAt }` | Update a share. | -| DELETE | /api/v1/shares/{apiKey} | — | Delete a share. | -| GET | /api/v1/shares/public/{shareUniqueId} | — | Public access to shared content (No Auth). | - -### Requests - -Base: `/api/v1/requests` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/requests | page, pageSize, query, mine, status | List requests. | -| GET | /api/v1/requests/{apiKey} | — | Get request details. | -| POST | /api/v1/requests | `{ category, description, ... }` | Create a new request. | -| PUT | /api/v1/requests/{apiKey} | `{ description, ... }` | Update a request. | -| POST | /api/v1/requests/{apiKey}/complete | — | Mark request as complete. | -| DELETE | /api/v1/requests/{apiKey} | — | Delete a request. | -| GET | /api/v1/requests/{apiKey}/comments | page, pageSize | List comments. | -| POST | /api/v1/requests/{apiKey}/comments | `{ body, parentCommentApiKey }` | Add a comment. | -| GET | /api/v1/requests/activity | — | Check for unread activity. | - -### Queue - -Base: `/api/v1/queue` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/queue | — | Get current user's play queue. | -| PUT | /api/v1/queue | `{ songIds, currentSongId, position }` | Save/Update play queue. | -| DELETE | /api/v1/queue | — | Clear play queue. | - -### Charts - -Base: `/api/v1/charts` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/charts | page, pageSize, tags, year | List charts. | -| GET | /api/v1/charts/{idOrSlug} | — | Get chart details. | -| GET | /api/v1/charts/{idOrSlug}/playlist | — | Get generated playlist tracks for a chart. | - -### Genres - -Base: `/api/v1/genres` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/genres | page, limit, orderBy, orderDirection | List genres. | -| GET | /api/v1/genres/{id}/songs | page, limit | Get songs in a genre. | -| POST | /api/v1/genres/starred/{id}/{isStarred} | — | Toggle starred status. | -| POST | /api/v1/genres/hated/{id}/{isHated} | — | Toggle hated status. | -| GET | /api/v1/genres/starred | — | Get starred genres. | -| GET | /api/v1/genres/hated | — | Get hated genres. | - -### Albums - -Base: `/api/v1/Albums` - -| Method | Path | Query | Description | -|--------|------|-------|-------------| -| GET | /api/v1/Albums/{id} | — | Album by id (GUID). | -| GET | /api/v1/Albums | page, pageSize, orderBy, orderDirection | Paginated list. | -| GET | /api/v1/Albums/recent | limit | Most recently added albums (CreatedAt desc). | -| GET | /api/v1/Albums/{id}/songs | — | All songs for an album (includes user rating if available). | - -Notes: - -- orderDirection: `asc` or `desc` (default desc). -- orderBy defaults to CreatedAt. - -### Songs - -Base: `/api/v1/Songs` - -| Method | Path | Query/Body | Description | -|--------|------|------------|-------------| -| GET | /api/v1/Songs | page, pageSize, orderBy, orderDirection | Paginated songs (user aware). | -| GET | /api/v1/Songs/recent | limit | Recently added songs. | -| POST | /api/v1/Songs/starred/{songId}/{isStarred} | — | Set or clear starred flag for user. | -| POST | /api/v1/Songs/setrating/{songId}/{rating} | — | Set rating (integer). | -| GET | /song/stream/{songId}/{userId}/{authToken} | Range headers | Stream (supports partial content). | - -Streaming: - -- This path omits the `/api/v1` version prefix intentionally for compatibility. -- authToken is a time‑limited HMAC token (client generates using user public key) encoded Base64. -- Supports `Range: bytes=...` for efficient seeking & partial delivery. - -### Search - -Base: `/api/v1/Search` - -| Method | Path | Body / Params | Description | -|--------|------|---------------|-------------| -| POST | /api/v1/Search | JSON `SearchRequest` | Multi‑type search (songs, albums, artists). | -| GET | /api/v1/Search/songs | q, page, pageSize, filterByArtistApiKey | Focused song search. | - -SearchRequest Fields (selected): - -| Field | Type | Notes | -|-------|------|-------| -| query | string | Search text | -| albumPageValue / artistPageValue / songPageValue | short | Per‑type paging | -| pageSize | short? | Default 50 | -| type | string? | CSV of enum flags (Songs, Albums, Artists, Data) | -| filterByArtistId | GUID? | Limit songs by artist | - -Response returns aggregated counts + typed collections. User ratings / starred info are included where applicable. - -### Scrobble - -Base: `/api/v1/Scrobble` - -| Method | Path | Body | Description | -|--------|------|------|-------------| -| POST | /api/v1/Scrobble | `{ songId, scrobbleType, playedDuration, playerName }` | Submit now playing or played scrobble event. | - -`scrobbleType` values: `NowPlaying`, `Played`. On success returns 200 with empty body. - -### Authorization & Blacklists - -Several endpoints enforce additional checks: - -- User locked -> 403 -- Blacklisted IP / Email (Search & Songs modifications, Streaming) -> 403 with `{ error: "User is blacklisted" }` - -### Rate / Concurrency Limits - -Streaming uses an internal limiter keyed per user to prevent excessive simultaneous streams (configurable in future releases). Exceeding yields HTTP 429. - -### Status Codes Summary - -| Code | Meaning | -|------|---------| -| 200 | Success | -| 206 | Partial Content (streaming range response) | -| 400 | Bad request / validation failure | -| 401 | Missing/invalid API key or auth token | -| 403 | User locked / blacklisted | -| 404 | Resource not found (album, song) | -| 429 | Too many concurrent streams | - -### Example: Paginated Albums - -Request: -``` -GET /api/v1/Albums?page=1&pageSize=25&orderBy=CreatedAt&orderDirection=desc -Authorization: Bearer -``` - -Response (truncated): -``` +Store access and refresh tokens as secrets. Do not put them in URLs, logs, +source control, or browser storage accessible to untrusted scripts. Use HTTPS +outside an isolated loopback connection. + +## Public and Special Authentication Routes + +`GET /api/v1/system/info` is public. Public shares use +`/api/v1/shares/public/{shareUniqueId}`. Browser sign-in and Google sign-in have +dedicated cookie endpoints under `/api/v1/auth`; consult OpenAPI for their +current request models. + +Song URLs returned by the API may use the non-versioned route +`/song/stream/{songApiKey}/{userApiKey}/{authToken}`. That route accepts a +short-lived, base64-encoded HMAC token bound to the user, song, and client +address. Consumers should use a stream URL supplied by Melodee rather than try +to manufacture one from a JWT or user API key. + +## Route Families + +| Route | Scope | +|-------|-------| +| `/api/v1/albums`, `/artists`, `/songs`, `/genres` | Library browsing and user annotations | +| `/api/v1/search`, `/recommendations`, `/charts` | Search and discovery | +| `/api/v1/artist-lookup`, `/audio` | External artist lookup and audio features | +| `/api/v1/playlists`, `/playlists/smart`, `/queue` | Playlists and the user's play queue | +| `/api/v1/user`, `/user/stats` | Current-user profile, activity, and statistics | +| `/api/v1/admin` | Administrator user operations | +| `/api/v1/requests`, `/shares` | Requests, comments, and shares | +| `/api/v1/podcasts` | Channels, episodes, discovery, bookmarks, and OPML | +| `/api/v1/scrobble`, `/analytics` | Playback reporting and listening analytics | +| `/api/v1/equalizer/presets`, `/playback/settings` | Playback preferences | +| `/api/v1/playback-backend` | Playback backend registration and status | +| `/api/v1/party-sessions`, `/party-endpoints`, `/endpoints` | Party sessions, queues, endpoints, playback, and moderation | +| `/api/v1/themes` | Theme listing, selection, import, export, and administration | +| `/api/v1/auth`, `/system` | Authentication and server information | + +Capabilities and administrator policies apply in addition to authentication. +A valid user token can therefore receive HTTP 403 for an operation the account +is not allowed to perform. + +## Pagination and Caching + +List endpoints generally return a `meta` object and a `data` collection: + +```json { - "meta": { "totalCount": 10234, "pageSize": 25, "page": 1, "totalPages": 410 }, - "data": [ { "id": "...", "name": "...", "songCount": 12, ... } ] + "meta": { + "totalCount": 100, + "pageSize": 25, + "page": 1, + "totalPages": 4 + }, + "data": [] } ``` -### Example: Stream a Song - -1. Obtain user public key & generate an HMAC based timed token (client library forthcoming). -2. Issue request: -``` -GET /song/stream/// -Range: bytes=0-524287 -``` -3. Handle 206 partial response & continue ranged requests as needed. - -### Recent Refactors - -The API layer was re‑architected to delegate operations to consolidated domain services (authentication, ratings, playlists, search, caching) reducing duplication and improving reliability without changing external contracts. - -### Official Clients & Ecosystem +Parameter names and response models vary by resource; do not assume every list +accepts the same sort fields. Some reads expose ETags and honor conditional +request headers. Use the generated OpenAPI document for the selected endpoint. -| Client | Platform | Highlights | Primary API Surface | -|--------|----------|-----------|---------------------| -| MeloAmp | Desktop (Linux / Win / macOS) | Artist/Album/Song browse, drag‑drop queue, theming, equalizer, starring, scrobbling, cross‑platform packaging | Native JSON (fallback OpenSubsonic possible) | -| Melodee Player | Android + Android Auto | Jetpack Compose UI, Media3 playback, voice commands, playlists, search, now‑playing bar, scrobbling | Native JSON (some legacy compatibility) | +## Errors and Limits -#### MeloAmp Details +API errors include a machine-readable code or message and can include a +correlation ID for log lookup. Common status codes are: -- Tech: Electron + React + Material‑UI + TypeScript. -- Auth: JWT (obtained via native login against Melodee API) stored locally. -- Features mapping to endpoints: - - Albums/Songs browsing -> `Albums`, `Songs`, `Search` endpoints. - - Star / rating actions -> `Songs/starred` and `Songs/setrating`. - - Scrobbling (play / complete) -> `Scrobble` endpoint. - - Queue operations are client‑side; future server playlist sync may use a native playlist endpoint (planned). -- Theming & equalizer are client-resident (no server configuration needed). - -#### Melodee Player (Android) Details - -- Tech: Kotlin, Jetpack Compose, Clean Architecture (data/domain/presentation), Media3 ExoPlayer, Retrofit. -- Android Auto: MediaBrowserService + MediaSession integration; voice command intents map to search + playback calls using `Search` and `Songs` endpoints. -- Caching: May locally cache minimal metadata for smooth navigation; server remains source of truth. -- Scrobbling: Dispatches NowPlaying then Played events to represent progress and completion. -- Supports pull‑to‑refresh to re-query / invalidate local caches via `If-None-Match` (future optimization with etag headers). - -## Homelab API Integration - -For homelab environments, the API enables several useful integrations: - -**Automation Scripts:** -- Automated library management -- Scheduled metadata updates -- Custom reporting and statistics - -**Home Automation:** -- Integration with home automation systems (Home Assistant, OpenHAB) -- Voice control via custom integrations -- Presence detection for automatic playback - -**Monitoring:** -- System health checks -- Library statistics tracking -- User activity monitoring - -If you build another client (CLI player, iOS app, web SPA, etc.) let us know and we’ll list it here. - ---- +| Status | Meaning | +|--------|---------| +| `400` | Request validation failed | +| `401` | JWT is missing, invalid, or expired | +| `403` | Account or capability policy denied the operation | +| `404` | Resource was not found | +| `409` | Resource state conflicts with the request | +| `412` | An ETag precondition failed | +| `429` | API, authentication, or streaming limit was exceeded | -Missing an endpoint? Open an issue with your use‑case so we can prioritize documenting or adding it. +Clients should honor `Retry-After` when present, refresh an expired access token +only once, and preserve the server's correlation ID in diagnostics. +See [API Overview](/apis/) for the compatibility APIs and [CLI Remote Server +Mode](/cli-remote-mode/) for the native operations available through `mcli`. diff --git a/docs/pages/apis.md b/docs/pages/apis.md index b747c8215..03cc96b7a 100644 --- a/docs/pages/apis.md +++ b/docs/pages/apis.md @@ -1,195 +1,67 @@ --- title: API Overview +description: Choose between Melodee's native, OpenSubsonic-compatible, and Jellyfin-compatible HTTP APIs. permalink: /apis/ +tags: + - api + - integration + - clients --- # API Overview -Melodee provides three distinct API surfaces to serve different use cases and client ecosystems. Understanding all options helps you choose the right API for your needs. - -## Available APIs - -| API | Best For | Client Examples | -|-----|----------|-----------------| -| [Melodee Native API](/api/) | New integrations, custom apps | MeloAmp, Melodee Player | -| [OpenSubsonic API](/api-opensubsonic/) | Subsonic-compatible clients | Supersonic, DSub, Ultrasonic | -| [Jellyfin API](/api-jellyfin/) | Jellyfin music clients | Finamp, Feishin, Streamyfin | - -## Melodee Native API - -The Melodee Native API provides a modern, strongly-typed REST interface with JSON responses. It's designed for new integrations and applications that want to take advantage of Melodee's full feature set. - -### Key Features -- **Modern REST design**: Clean, predictable endpoints with consistent JSON responses -- **Strong typing**: Well-defined data models with comprehensive documentation -- **Versioned**: API versioning ensures backward compatibility -- **Full feature access**: Access to all of Melodee's capabilities -- **Better performance**: Optimized for Melodee's architecture - -### Endpoint Structure -- Base URL: `/api/v1/` -- Example: `/api/v1/albums`, `/api/v1/songs`, `/api/v1/system/stats` - -### Authentication -All endpoints (except public ones) require a JWT Bearer token: -``` -Authorization: Bearer -``` - -### Response Format -Consistent response format for all endpoints: -```json -{ - "meta": { - "totalCount": 100, - "pageSize": 25, - "page": 1, - "totalPages": 4 - }, - "data": [...] -} -``` - -**[Full Melodee API Documentation →](/api/)** +Melodee 2.2.0 exposes three HTTP API surfaces. They share the same users and +media library, but their routes, credentials, and compatibility goals differ. + +| API | Public route | Authentication | Use it for | +|-----|--------------|----------------|------------| +| [Native API](/api/) | `/api/v1/*` | Melodee JWT bearer token | New Melodee integrations and administration | +| [OpenSubsonic](/api-opensubsonic/) | `/rest/*` | Subsonic username/token/salt or legacy password | Subsonic and OpenSubsonic clients | +| [Jellyfin compatibility](/api-jellyfin/) | Jellyfin root routes, internally `/api/jf/*` | Jellyfin access token | Music clients that speak the implemented Jellyfin subset | + +Compatibility layers do not expose every endpoint implemented by their +upstream projects. Review the [OpenSubsonic compatibility matrix](/opensubsonic-matrix/) +and the Jellyfin route list before selecting a client. + +## Native API + +The native API is versioned under `/api/v1`. Obtain a JWT from +`POST /api/v1/auth/authenticate`, then send it as a bearer token. It covers +library browsing, search, playlists, queues, user data, requests, shares, +podcasts, charts, playback, analytics, and administrative operations. + +The running server publishes native API discovery documents at: + +- Scalar UI: `/scalar/v1` +- OpenAPI JSON: `/openapi/v1.json` + +The OpenAPI document describes the native API; it is not a specification for +the OpenSubsonic or Jellyfin compatibility surfaces. ## OpenSubsonic API -The OpenSubsonic API provides compatibility with the Subsonic API specification, allowing Melodee to work with existing Subsonic-compatible clients and applications. - -### Key Features -- **Client compatibility**: Works with existing Subsonic clients (DSub, Supersonic, etc.) -- **Broad ecosystem**: Access to the wide range of existing Subsonic-compatible apps -- **Familiar endpoints**: Uses the well-established Subsonic API structure -- **Streaming compatibility**: Supports the same streaming protocols as Subsonic -- **OpenSubsonic extensions**: Enhanced features for modern clients - -### Endpoint Structure -- Base URL: `/rest/` -- Follows Subsonic API specification (e.g., `/rest/getAlbum`, `/rest/stream`) - -### Authentication -Uses Subsonic's token-based authentication: -- Token: `MD5(password + salt)` -- Parameters: `u`, `t`, `s`, `v`, `c` - -### Compatible Clients -- Supersonic, DSub, Ultrasonic, Sublime Music, Symphonium, Feishin (Subsonic mode) - -**[Full OpenSubsonic API Documentation →](/api-opensubsonic/)** - -## Jellyfin API - -The Jellyfin API enables popular Jellyfin music clients to connect to Melodee. This provides access to the growing ecosystem of Jellyfin-compatible music applications. - -### Key Features -- **Jellyfin client support**: Works with Finamp, Feishin, Streamyfin, Gelli -- **Full music browsing**: Artists, albums, songs, playlists, genres -- **Streaming**: Direct play and transcoding support -- **Playback tracking**: Session reporting and scrobbling integration -- **Playlist management**: Create, edit, and sync playlists - -### Endpoint Structure -- Clients connect to: `http://server:port/` -- Internal routing: `/api/jf/*` -- Automatic detection via `Authorization: MediaBrowser` header - -### Authentication -Uses Jellyfin's MediaBrowser authentication: -``` -Authorization: MediaBrowser Token="token", Client="app", Device="device", DeviceId="id", Version="1.0" -``` - -### Compatible Clients -- Finamp (iOS, Android, Desktop) -- Feishin (Desktop, Jellyfin mode) -- Streamyfin (iOS, Android) -- Gelli (Android) - -**[Full Jellyfin API Documentation →](/api-jellyfin/)** - -## Choosing the Right API - -### Use the Melodee Native API when: -- Building new applications or integrations -- You need access to Melodee-specific features (requests, charts, smart playlists) -- Performance is critical -- You prefer modern REST conventions -- You want strongly-typed responses - -### Use the OpenSubsonic API when: -- Integrating with existing Subsonic-compatible clients -- You have existing Subsonic-based integrations -- You need compatibility with third-party mobile apps -- Working with clients designed for Subsonic/Navidrome - -### Use the Jellyfin API when: -- You prefer Jellyfin ecosystem clients -- Using Finamp, Feishin, or Streamyfin -- You want feature parity with Jellyfin music features -- Migrating from a Jellyfin server - -## API Documentation and Exploration - -### Interactive API Documentation -Melodee provides interactive API documentation via Scalar where you can explore and test the native Melodee API: - -- **Melodee API (Scalar UI)**: Available at `/scalar/v1` when Melodee is running -- **OpenAPI Specification**: Available at `/openapi/v1.json` for download - -The Scalar UI allows you to: -- View detailed endpoint documentation -- Test API calls directly from the browser -- See example requests and responses -- Understand required parameters and authentication -- Download the OpenAPI specification for use with code generators - -### External Documentation -- **OpenSubsonic API**: [opensubsonic.netlify.app](https://opensubsonic.netlify.app/) -- **Jellyfin API**: [api.jellyfin.org](https://api.jellyfin.org/) - -## Client Compatibility Matrix - -| Client | Platform | Melodee API | OpenSubsonic | Jellyfin | -|--------|----------|-------------|--------------|----------| -| MeloAmp | Desktop | ✅ Primary | ✅ Fallback | ❌ | -| Melodee Player | Android | ✅ Primary | ✅ Fallback | ❌ | -| Supersonic | Desktop | ❌ | ✅ | ❌ | -| Feishin | Desktop | ❌ | ✅ | ✅ | -| Finamp | Mobile/Desktop | ❌ | ❌ | ✅ | -| Streamyfin | Mobile | ❌ | ❌ | ✅ | -| DSub | Android | ❌ | ✅ | ❌ | -| Ultrasonic | Android | ❌ | ✅ | ❌ | -| Symphonium | Android | ❌ | ✅ | ❌ | -| Sublime Music | Linux | ❌ | ✅ | ❌ | -| Gelli | Android | ❌ | ❌ | ✅ | - -## API Performance Considerations - -### Melodee Native API -- Optimized for Melodee's data structures -- Generally fastest response times -- Most efficient data serialization -- Best error handling and validation - -### OpenSubsonic API -- Transformation layer for compatibility -- Well-tested across many clients -- Extensive caching for performance -- Established streaming protocols - -### Jellyfin API -- URL rewriting middleware -- Efficient media browsing -- Session-based tracking -- Transcoding integration - -## Getting Started - -1. **Explore the API**: Visit `/scalar/v1` on your Melodee instance to interactively explore the Melodee API -2. **Create an account**: Register on your Melodee instance -3. **Get authentication**: Obtain JWT token, Subsonic token, or Jellyfin token depending on API -4. **Choose your client**: Select based on your platform and preferred API -5. **Test endpoints**: Use the Scalar UI or curl to test before implementing -6. **Download specs**: Get OpenAPI spec at `/openapi/v1.json` for code generation - -All three APIs provide access to Melodee's powerful music management features, so choose based on your specific needs and existing client ecosystem. \ No newline at end of file +OpenSubsonic endpoints accept GET and form-encoded POST requests at both +`/rest/endpoint` and `/rest/endpoint.view`. JSON, XML, and JSONP response modes +are supported. Melodee implements music browsing, search, streaming, +annotations, playlists, bookmarks, queues, shares, radio, scanning, jukebox, +and optional podcast endpoints. Deprecated chat, video, captions, and HLS +routes intentionally return HTTP 410. + +## Jellyfin Compatibility API + +Jellyfin clients connect to the Melodee server root. Middleware recognizes the +Jellyfin discovery and login paths and authenticated MediaBrowser requests, +then routes them internally beneath `/api/jf`. Support is limited to the music +and playlist endpoints documented on the [Jellyfin API page](/api-jellyfin/). +The database setting `jellyfin.enabled` controls this surface and defaults to +`true`. + +## Base URL and TLS + +Use the externally reachable origin only, for example +`https://music.example.com`; append the route for the selected API. Configure +`system.baseUrl` to the same public origin and terminate TLS either in Melodee +or a trusted reverse proxy. Do not expose plain HTTP credentials or access +tokens across an untrusted network. + +See [Configuration](/configuration/) for proxy and base-URL settings. diff --git a/docs/pages/archive.md b/docs/pages/archive.md index ceebe7575..644da4d90 100755 --- a/docs/pages/archive.md +++ b/docs/pages/archive.md @@ -1,7 +1,11 @@ --- layout: page title: Articles +description: Browse the archive of Melodee release announcements and project news. permalink: /archive/ +tags: + - news + - archive --- # News Archive diff --git a/docs/pages/backup.md b/docs/pages/backup.md index 2ae21aff7..946b2bfa0 100644 --- a/docs/pages/backup.md +++ b/docs/pages/backup.md @@ -1,538 +1,227 @@ --- title: Backup & Recovery +description: Back up and restore Melodee PostgreSQL data, persistent files, secrets, and generated search databases. permalink: /backup/ +tags: + - backup + - restore + - operations + - postgres --- -# Backup & Recovery Guide +# Backup & Recovery -This guide covers comprehensive backup strategies and disaster recovery procedures for Melodee in homelab environments, ensuring your music library and metadata remain safe. +A usable Melodee backup combines a logical PostgreSQL dump with the persistent +application files and the configuration needed to mount them. Copying only the +container image is not a backup. -## Understanding Melodee Data +## What to Protect -### Critical Data Components +| Data | Default location | Required? | +|------|------------------|-----------| +| PostgreSQL database | `melodee_db_data` | Yes | +| Published media and DecentDB files | `melodee_storage` | Yes | +| Inbound and Staging work | `melodee_inbound`, `melodee_staging` | If work is pending | +| Playlists and user images | `melodee_playlists`, `melodee_user_images` | Yes | +| Podcasts, themes, and templates | Corresponding named volumes | If used | +| ASP.NET data-protection keys | `melodee_data_protection_keys` | Recommended | +| Deployment configuration | `.env`, Compose files, proxy configuration | Yes | +| Logs | `melodee_logs` | Optional for recovery; useful for diagnosis | -Melodee stores data across multiple volumes and components: +The MusicBrainz and Artist Search DecentDB files live under Storage in the +default container layout. They can be rebuilt, but preserving them can save +substantial download and import time. -1. **Database (PostgreSQL)**: Core metadata, user accounts, playlists, ratings, play history -2. **Storage Volume**: Processed and organized music files -3. **User Images**: User avatars and profile images -4. **Playlists**: Custom playlist definitions and configurations -5. **Logs**: System logs for troubleshooting (optional to backup) +## Create a Backup -### Data Importance Ranking - -| Component | Criticality | Backup Frequency | Recovery Priority | -|-----------|-------------|------------------|-------------------| -| Database | Critical | Daily | 1st | -| Storage | Critical | Daily/Incremental | 2nd | -| User Images | Important | Weekly | 3rd | -| Playlists | Important | Weekly | 4th | -| Logs | Optional | As needed | Last | - -## Backup Strategies - -### Full Backup Approach - -A complete backup includes all critical components: +Choose a destination that is not inside a Melodee volume: ```bash -#!/bin/bash -# full-backup-melodee.sh - -BACKUP_ROOT="/backup/melodee" -DATE=$(date +%Y-%m-%d) -BACKUP_DIR="$BACKUP_ROOT/$DATE" -mkdir -p "$BACKUP_DIR" - -echo "Starting full backup: $DATE" - -# Stop services for consistent backup -echo "Stopping Melodee services..." -docker-compose down - -# Export database as SQL dump (more portable than volume export) -echo "Backing up database..." -docker exec melodee-db pg_dump -U melodeeuser -d melodeedb > "$BACKUP_DIR/db_dump.sql" - -# Export volumes -echo "Exporting volumes..." -docker run --rm -v melodee_storage:/volume -v "$BACKUP_DIR:/backup" alpine tar czf /backup/storage.tar.gz -C /volume . -docker run --rm -v melodee_user_images:/volume -v "$BACKUP_DIR:/backup" alpine tar czf /backup/user_images.tar.gz -C /volume . -docker run --rm -v melodee_playlists:/volume -v "$BACKUP_DIR:/backup" alpine tar czf /backup/playlists.tar.gz -C /volume . - -# Create backup manifest -cat > "$BACKUP_DIR/manifest.txt" << EOF -Backup Date: $DATE -Components: Database, Storage, User Images, Playlists -Database Size: $(du -h "$BACKUP_DIR/db_dump.sql" | cut -f1) -Storage Size: $(du -h "$BACKUP_DIR/storage.tar.gz" | cut -f1) -EOF - -# Start services -echo "Starting Melodee services..." -docker-compose up -d - -echo "Full backup completed: $BACKUP_DIR" +backup_dir="/srv/backups/melodee/$(date +%Y%m%d-%H%M%S)" +mkdir -p "$backup_dir" +chmod 700 "$backup_dir" ``` -### Incremental Backup Approach +### 1. Dump PostgreSQL -For large libraries, incremental backups are more practical: +`pg_dump` produces a transactionally consistent logical backup while +PostgreSQL is running: ```bash -#!/bin/bash -# incremental-backup-melodee.sh - -BACKUP_ROOT="/backup/melodee" -DATE=$(date +%Y-%m-%d) -INCREMENTAL_DIR="$BACKUP_ROOT/incremental/$DATE" -mkdir -p "$INCREMENTAL_DIR" - -echo "Starting incremental backup: $DATE" - -# Backup database (smaller, frequent backups) -docker exec melodee-db pg_dump -U melodeeuser -d melodeedb --format=custom > "$INCREMENTAL_DIR/db_backup.custom" - -# Only backup new/modified files in storage volume -# First, create a reference point if this is the first incremental backup -if [ ! -f "$BACKUP_ROOT/latest_timestamp" ]; then - # Full backup for first run - docker run --rm -v melodee_storage:/volume -v "$INCREMENTAL_DIR:/backup" alpine tar czf /backup/storage.tar.gz -C /volume . - touch "$BACKUP_ROOT/latest_timestamp" -else - # Find files modified since last backup - LAST_BACKUP_TIME=$(cat "$BACKUP_ROOT/latest_timestamp") - docker run --rm -v melodee_storage:/volume -v "$INCREMENTAL_DIR:/backup" alpine sh -c " - find /volume -newer /backup/last_backup_marker 2>/dev/null | - tar -czf /backup/storage_incremental.tar.gz -C /volume -T - - " || - # Fallback: backup everything if find fails - docker run --rm -v melodee_storage:/volume -v "$INCREMENTAL_DIR:/backup" alpine tar czf /backup/storage_incremental.tar.gz -C /volume . -fi - -# Update timestamp -date > "$BACKUP_ROOT/latest_timestamp" - -echo "Incremental backup completed: $INCREMENTAL_DIR" +docker compose exec -T melodee-db \ + pg_dump --username=melodeeuser --dbname=melodeedb --format=custom \ + > "$backup_dir/postgres.dump" ``` -### Database-Only Backup (Frequent) - -For critical metadata protection: +Confirm the dump can be read: ```bash -#!/bin/bash -# db-backup.sh - -BACKUP_DIR="/backup/melodee/database" -DATE=$(date +%Y-%m-%d-%H%M%S) -mkdir -p "$BACKUP_DIR" - -# Create compressed database dump -docker exec melodee-db pg_dump -U melodeeuser -d melodeedb | gzip > "$BACKUP_DIR/db_backup_$DATE.sql.gz" - -# Keep only last 7 days of database backups -find "$BACKUP_DIR" -name "db_backup_*.sql.gz" -mtime +7 -delete - -echo "Database backup completed: $BACKUP_DIR/db_backup_$DATE.sql.gz" +docker compose exec -T melodee-db pg_restore --list \ + < "$backup_dir/postgres.dump" > /dev/null ``` -## Backup Automation +Do not archive `/var/lib/postgresql/data` while PostgreSQL is running. A raw +database-volume copy is not a replacement for `pg_dump` unless it is made by a +database-aware snapshot procedure. -### Cron Scheduling +### 2. Stop Application Writes -Add to crontab for automated backups: +Stop the application while archiving its file volumes. PostgreSQL can remain +running: ```bash -# Edit crontab -crontab -e - -# Add backup schedules -# Daily database backup at 2 AM -0 2 * * * /path/to/db-backup.sh - -# Weekly full backup on Sundays at 3 AM -0 3 * * 0 /path/to/full-backup-melodee.sh - -# Daily incremental backup at 11 PM -0 23 * * * /path/to/incremental-backup-melodee.sh +docker compose stop melodee.blazor ``` -### Docker-based Backup - -Create a dedicated backup container: - -```yaml -# Add to compose.yml -services: - melodee-backup: - image: alpine:latest - volumes: - - db_data:/db_data:ro - - storage:/storage:ro - - user_images:/user_images:ro - - playlists:/playlists:ro - - /backup/melodee:/backup - - /var/run/docker.sock:/var/run/docker.sock - environment: - - BACKUP_SCHEDULE=daily - - RETENTION_DAYS=30 - command: > - sh -c " - apk add --no-cache postgresql-client tar gzip && - while true; do - DATE=$$(date +%Y-%m-%d) - BACKUP_DIR=/backup/$$DATE - mkdir -p $$BACKUP_DIR - - # Database backup - pg_dump -h melodee-db -U melodeeuser -d melodeedb | gzip > $$BACKUP_DIR/db_dump.sql.gz - - # Volume backups - tar czf $$BACKUP_DIR/storage.tar.gz -C /storage . - tar czf $$BACKUP_DIR/user_images.tar.gz -C /user_images . - tar czf $$BACKUP_DIR/playlists.tar.gz -C /playlists . - - # Cleanup old backups - find /backup -mindepth 1 -maxdepth 1 -type d -mtime +$$RETENTION_DAYS -exec rm -rf {} + - - # Wait for next backup - sleep 86400 - done - " - restart: unless-stopped -``` - -## Off-Site Backup Solutions +### 3. Archive Persistent Volumes -### Cloud Storage Integration +The helper below archives one Docker named volume at a time: -**AWS S3:** ```bash -#!/bin/bash -# s3-backup.sh - -# Install AWS CLI if not present -# apt-get install awscli - -# Sync local backups to S3 -aws s3 sync /backup/melodee s3://your-melodee-backup-bucket --delete - -# Verify backup integrity -aws s3 ls s3://your-melodee-backup-bucket --recursive +backup_volume() { + volume_name="$1" + archive_name="$2" + docker run --rm \ + --volume "${volume_name}:/source:ro" \ + --volume "${backup_dir}:/backup" \ + alpine:3.20 \ + tar -C /source -czf "/backup/${archive_name}.tar.gz" . +} + +backup_volume melodee_storage storage +backup_volume melodee_inbound inbound +backup_volume melodee_staging staging +backup_volume melodee_user_images user-images +backup_volume melodee_playlists playlists +backup_volume melodee_podcasts podcasts +backup_volume melodee_themes themes +backup_volume melodee_templates templates +backup_volume melodee_data_protection_keys data-protection-keys ``` -**rsync to Remote Server:** -```bash -#!/bin/bash -# remote-backup.sh +If a path is a bind mount, archive or snapshot the host directory with your +normal backup system instead of using `docker volume`. -# Ensure SSH key authentication is set up -rsync -avz --delete /backup/melodee/ user@remote-server:/path/to/remote/backup/melodee/ -``` +For Podman, replace `docker` with `podman` and confirm the volume names with +`podman volume ls`. -### Versioned Backup Strategy +### 4. Save Configuration and Restart ```bash -#!/bin/bash -# versioned-backup.sh - -BACKUP_ROOT="/backup/melodee" -DATE=$(date +%Y-%m-%d) -WEEK=$(date +%U) - -# Daily backups (keep 7 days) -DAILY_DIR="$BACKUP_ROOT/daily" -mkdir -p "$DAILY_DIR" -# Perform daily backup to this location - -# Weekly backups (keep 4 weeks) -WEEKLY_DIR="$BACKUP_ROOT/weekly/week_$WEEK" -if [ ! -d "$WEEKLY_DIR" ]; then - # Only backup if different from last week - mkdir -p "$WEEKLY_DIR" - # Perform weekly backup -fi - -# Monthly backups (keep 6 months) -MONTH=$(date +%Y-%m) -MONTHLY_DIR="$BACKUP_ROOT/monthly/$MONTH" -if [ ! -d "$MONTHLY_DIR" ]; then - mkdir -p "$MONTHLY_DIR" - # Perform monthly backup +cp --preserve=mode .env compose.yml "$backup_dir/" +if [ -f compose.override.yml ]; then + cp --preserve=mode compose.override.yml "$backup_dir/" fi +docker compose start melodee.blazor +curl --fail http://localhost:8080/health ``` -## Recovery Procedures +Treat `.env` and database dumps as secrets. Encrypt backup media, restrict file +permissions, and do not commit them to Git. -### Complete System Recovery +### 5. Create Checksums ```bash -#!/bin/bash -# recovery-full.sh - -RESTORE_DATE="$1" # Pass date as argument -if [ -z "$RESTORE_DATE" ]; then - echo "Usage: $0 YYYY-MM-DD" - exit 1 -fi - -BACKUP_DIR="/backup/melodee/$RESTORE_DATE" -if [ ! -d "$BACKUP_DIR" ]; then - echo "Backup directory not found: $BACKUP_DIR" - exit 1 -fi - -echo "Starting full recovery from: $RESTORE_DATE" - -# Stop services -echo "Stopping Melodee services..." -docker-compose down - -# Restore database first -echo "Restoring database..." -docker exec -i melodee-db psql -U melodeeuser -d melodeedb < "$BACKUP_DIR/db_dump.sql" - -# Restore volumes -echo "Restoring volumes..." -docker run --rm -v melodee_storage:/volume -v "$BACKUP_DIR:/backup" alpine sh -c "rm -rf /volume/* && tar xzf /backup/storage.tar.gz -C /volume" -docker run --rm -v melodee_user_images:/volume -v "$BACKUP_DIR:/backup" alpine sh -c "rm -rf /volume/* && tar xzf /backup/user_images.tar.gz -C /volume" -docker run --rm -v melodee_playlists:/volume -v "$BACKUP_DIR:/backup" alpine sh -c "rm -rf /volume/* && tar xzf /backup/playlists.tar.gz -C /volume" - -# Start services -echo "Starting Melodee services..." -docker-compose up -d - -echo "Full recovery completed. Verify system status with: docker-compose ps" +cd "$backup_dir" +sha256sum postgres.dump *.tar.gz > SHA256SUMS +sha256sum --check SHA256SUMS ``` -### Database-Only Recovery - -```bash -#!/bin/bash -# recovery-db.sh - -BACKUP_FILE="$1" -if [ ! -f "$BACKUP_FILE" ]; then - echo "Backup file not found: $BACKUP_FILE" - exit 1 -fi - -echo "Restoring database from: $BACKUP_FILE" - -# Stop Melodee to prevent database access -docker-compose stop melodee.blazor - -# Drop and recreate database -docker exec melodee-db dropdb -U melodeeuser melodeedb -docker exec melodee-db createdb -U melodeeuser melodeedb - -# Restore from backup -if [[ "$BACKUP_FILE" == *.gz ]]; then - gunzip -c "$BACKUP_FILE" | docker exec -i melodee-db psql -U melodeeuser -d melodeedb -elif [[ "$BACKUP_FILE" == *.custom ]]; then - echo "Custom format backup detected, using pg_restore" - gunzip -c "$BACKUP_FILE" | docker exec -i melodee-db pg_restore -U melodeeuser -d melodeedb --clean --if-exists -else - docker exec -i melodee-db psql -U melodeeuser -d melodeedb < "$BACKUP_FILE" -fi +Copy the completed backup to another failure domain. A backup on the same disk +as the live data does not protect against disk loss. -# Restart services -docker-compose start melodee.blazor +## Restore a Backup -echo "Database recovery completed" -``` +Test restores on an isolated host or Compose project before relying on the +procedure in an emergency. -### Partial Recovery (Individual Volume) +### 1. Stop Melodee ```bash -#!/bin/bash -# recovery-volume.sh - -VOLUME_NAME="$1" # e.g., "storage", "user_images", "playlists" -BACKUP_FILE="$2" # Path to specific volume backup - -if [ -z "$VOLUME_NAME" ] || [ -z "$BACKUP_FILE" ]; then - echo "Usage: $0 " - echo "Volume names: storage, user_images, playlists" - exit 1 -fi - -VOLUME_MAP="melodee_$VOLUME_NAME" - -echo "Restoring volume $VOLUME_NAME from: $BACKUP_FILE" - -# Stop services that might be using the volume -docker-compose stop melodee.blazor - -# Clear and restore the specific volume -docker run --rm -v "$VOLUME_MAP:/volume" -v "$(dirname "$BACKUP_FILE"):/backup" alpine sh -c "rm -rf /volume/* && tar xzf /backup/$(basename "$BACKUP_FILE") -C /volume" - -# Start services -docker-compose start melodee.blazor - -echo "Volume $VOLUME_NAME recovery completed" +docker compose down ``` -## Backup Verification - -### Automated Verification Script - -```bash -#!/bin/bash -# verify-backup.sh - -BACKUP_DIR="$1" -if [ -z "$BACKUP_DIR" ]; then - echo "Usage: $0 " - exit 1 -fi - -echo "Verifying backup in: $BACKUP_DIR" - -# Check if required backup files exist -REQUIRED_FILES=("db_dump.sql" "storage.tar.gz" "user_images.tar.gz" "playlists.tar.gz") -for file in "${REQUIRED_FILES[@]}"; do - if [ ! -f "$BACKUP_DIR/$file" ]; then - echo "ERROR: Missing backup file: $file" - exit 1 - else - echo "OK: Found $file ($(du -h "$BACKUP_DIR/$file" | cut -f1))" - fi -done - -# Verify database dump integrity -echo "Checking database dump integrity..." -if head -n 5 "$BACKUP_DIR/db_dump.sql" | grep -q "PostgreSQL"; then - echo "OK: Database dump appears valid" -else - echo "WARNING: Database dump may be corrupted" -fi - -# Check volume archives -for vol in storage user_images playlists; do - if tar -tzf "$BACKUP_DIR/$vol.tar.gz" >/dev/null 2>&1; then - echo "OK: $vol archive is valid" - else - echo "ERROR: $vol archive is corrupted" - fi -done - -echo "Backup verification completed" -``` +Do not use `-v`; the restore procedure reuses or deliberately replaces the +existing named volumes. -### Integrity Testing +### 2. Restore File Volumes -Regularly test recovery procedures: +Restore only into empty volumes, or explicitly remove their existing contents +after confirming the volume name and backup set. ```bash -#!/bin/bash -# test-recovery.sh - -# Create a test environment -TEST_DIR="/tmp/melodee-test-restore" -mkdir -p "$TEST_DIR" - -# Use a recent backup for testing -RECENT_BACKUP=$(ls -td /backup/melodee/*/ | head -n1) -echo "Testing recovery from: $RECENT_BACKUP" - -# Test database restore in isolation -echo "Testing database restore..." -# This would create a temporary database container and restore to it - -# Test volume extraction -echo "Testing volume extraction..." -for vol in storage user_images playlists; do - mkdir -p "$TEST_DIR/$vol" - tar xzf "$RECENT_BACKUP/$vol.tar.gz" -C "$TEST_DIR/$vol" --exclude="*.tmp" --exclude="*.tmp/*" - if [ $? -eq 0 ]; then - echo "OK: $vol extraction successful" - else - echo "ERROR: $vol extraction failed" - fi -done - -# Cleanup -rm -rf "$TEST_DIR" - -echo "Recovery test completed" +restore_volume() { + volume_name="$1" + archive_path="$2" + archive_name="$(basename "$archive_path")" + docker volume create "$volume_name" > /dev/null + docker run --rm \ + --volume "${volume_name}:/target" \ + --volume "$(dirname "$archive_path"):/backup:ro" \ + alpine:3.20 \ + sh -c 'rm -rf /target/* /target/.[!.]* /target/..?* 2>/dev/null || true; tar -C /target -xzf "/backup/$1"' \ + sh "$archive_name" +} + +restore_volume melodee_storage "$backup_dir/storage.tar.gz" +restore_volume melodee_inbound "$backup_dir/inbound.tar.gz" +restore_volume melodee_staging "$backup_dir/staging.tar.gz" +restore_volume melodee_user_images "$backup_dir/user-images.tar.gz" +restore_volume melodee_playlists "$backup_dir/playlists.tar.gz" +restore_volume melodee_podcasts "$backup_dir/podcasts.tar.gz" +restore_volume melodee_themes "$backup_dir/themes.tar.gz" +restore_volume melodee_templates "$backup_dir/templates.tar.gz" +restore_volume melodee_data_protection_keys "$backup_dir/data-protection-keys.tar.gz" ``` -## Disaster Recovery Plan - -### Immediate Response Steps - -1. **Assess the situation**: Determine what data is lost or corrupted -2. **Stop services**: Prevent further writes to corrupted data -3. **Identify recovery point**: Choose the most recent good backup -4. **Prepare recovery environment**: Ensure backup files are accessible -5. **Execute recovery**: Follow appropriate recovery procedure -6. **Verify operation**: Test that Melodee functions correctly - -### Recovery Scenarios - -**Database Corruption:** -- Restore database from backup -- Verify music files remain intact -- Re-index if necessary +The cleanup inside this example is destructive to the named target volume. Run +it only after verifying the target name and backup path. For bind mounts, +restore the host directories with the tool that created their backups. -**Volume Loss:** -- Restore specific volume from backup -- Restart affected services -- Verify file integrity +### 3. Restore PostgreSQL -**Complete System Failure:** -- Set up new system with same configuration -- Restore all components from backup -- Verify all functionality +Start only the database service, replace the application database, and feed the +custom-format dump to `pg_restore`: -## Best Practices - -### Backup Best Practices -- Test recovery procedures regularly -- Keep multiple backup copies in different locations -- Monitor backup success/failure -- Document your specific backup procedures -- Encrypt sensitive backup data -- Use compression to save space - -### Recovery Best Practices -- Maintain a recovery checklist -- Document your specific system configuration -- Keep recovery scripts tested and current -- Have a backup of your backup scripts -- Document any custom configurations +```bash +docker compose up -d melodee-db +docker compose exec -T melodee-db \ + dropdb --username=melodeeuser --if-exists --force melodeedb +docker compose exec -T melodee-db \ + createdb --username=melodeeuser melodeedb +docker compose exec -T melodee-db \ + pg_restore --username=melodeeuser --dbname=melodeedb \ + --no-owner --no-privileges \ + < "$backup_dir/postgres.dump" +``` -## Monitoring Backup Health +Restore `.env` and Compose overrides before starting the application so the +database password, JWT signing key, paths, and image version match the backup. -### Backup Monitoring Script +### 4. Start and Verify ```bash -#!/bin/bash -# monitor-backup.sh - -BACKUP_ROOT="/backup/melodee" -ALERT_EMAIL="admin@yourdomain.com" -MAX_AGE_DAYS=2 +docker compose up -d +docker compose ps +docker compose logs --tail=250 melodee.blazor +curl --fail http://localhost:8080/health +``` -# Check if recent backup exists -RECENT_BACKUP=$(find "$BACKUP_ROOT" -mindepth 1 -maxdepth 1 -type d -mtime -$MAX_AGE_DAYS | sort | tail -n1) +Sign in, run **Admin > Doctor**, browse a known album, stream a song, and inspect +library counts. Keep the backup untouched until verification is complete. -if [ -z "$RECENT_BACKUP" ]; then - echo "ALERT: No recent backup found (older than $MAX_AGE_DAYS days)" - # Send notification (email, push, etc.) - exit 1 -fi +## Backup Policy -# Check backup size (should be reasonable) -DB_SIZE=$(du -sm "$RECENT_BACKUP/db_dump.sql" 2>/dev/null | cut -f1) -if [ -z "$DB_SIZE" ] || [ "$DB_SIZE" -lt 1 ]; then - echo "ALERT: Database backup is too small ($DB_SIZE MB)" - exit 1 -fi +A practical policy includes: -echo "Backup monitoring: OK - Recent backup found from $(basename "$RECENT_BACKUP")" -``` +- Frequent PostgreSQL dumps +- File snapshots after imports or metadata curation +- A backup before every Melodee or PostgreSQL upgrade +- Multiple retention periods, such as daily, weekly, and monthly copies +- At least one encrypted off-host or offline copy +- Scheduled restore tests with recorded results -This comprehensive backup and recovery guide ensures your Melodee installation remains protected against data loss. Regular testing of recovery procedures is crucial to ensure they work when needed. \ No newline at end of file +Volume archives can be large. If the canonical media library is already +protected by storage snapshots, do not duplicate it blindly; document exactly +which system is authoritative and ensure its restore is tested with the matching +PostgreSQL backup. diff --git a/docs/pages/changelog.md b/docs/pages/changelog.md index 370938a1a..6e44c9a64 100644 --- a/docs/pages/changelog.md +++ b/docs/pages/changelog.md @@ -1,6 +1,10 @@ --- title: Changelog +description: Release history and notable changes for Melodee. permalink: /changelog/ +tags: + - release + - changelog --- # Changelog @@ -57,6 +61,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Enabled solution-wide XML documentation output so build-time unused-using analysis runs consistently, and corrected malformed API documentation and formatter configuration. +- Updated the documentation site's default Release, version menu, and search + scope to `2.2.0`; future version bumps now synchronize these values + automatically. +- Audited the complete public documentation set against the current source, + deployment files, CLI help, and API routes; corrected installation, + configuration, operations, feature-status, and compatibility guidance, and + refreshed site navigation, metadata, links, and release pages. ### Fixed @@ -80,6 +91,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 obsolete API, or XML documentation warnings. - Version bump automation now promotes release notes without moving the fresh Unreleased heading above the changelog's Jekyll front matter. +- Remote `mcli` commands now pass the normalized server origin to the HTTP + client, preventing `/api/v1` from being appended twice to request URLs. - ArtistSearch initialization now uses migrations for relational databases and schema creation for non-relational providers, preventing OpenSubsonic `getTopSongs` failures in in-memory hosts. @@ -89,6 +102,64 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Pinned `Microsoft.OpenApi` to patched version `2.7.5`, preventing crafted circular schema references from terminating the process during OpenAPI document parsing. +- Password-reset, SMTP, login, migration, profile-lookup, and blacklist logs no + longer persist email/user identifiers, reset URLs, template subjects, + configured SMTP hosts, tokens, or raw exception payloads. Reset links also + require a credential-free absolute HTTP(S) base URL without a query or + fragment. +- Startup configuration diagnostics now redact passwords, tokens, API keys, + connection strings, credential-bearing URLs, and unknown environment values + by default while preserving sanitized operational metadata. +- Both setup utilities now generate independent database and authentication + secrets without printing them and use one atomic writer with overwrite, + symlink, non-regular-file, and failure-cleanup protection. Secret files use + owner-only `0600` permissions on POSIX and the containing directory's ACL on + Windows. +- Python maintenance tooling now parses GitHub Link headers with bounded linear + work and no longer prints demo passwords, generated API/public keys, + encrypted passwords, or secret-bearing exception payloads. The code-scanning + exporter also confines requests, pagination links, redirects, and downloads + to HTTPS on the exact configured GitHub API origin. +- The destructive incoming cleanup tool now confines all reads and mutations to + a canonical cleanup root beneath an explicit trusted boundary, rejects + symlink/parent escapes, preflights ZIP members against Zip Slip, and prevents + SFV filenames from traversing outside the media root. Live cleanup uses pinned + no-follow descriptors and atomic no-replace quarantine moves on Linux and + fails closed when those primitives are unavailable. Extracted directories + and files are created privately as `0700` and `0600`. The boundary defaults to + the current working directory; use `--trusted-boundary` for an explicitly + authorized absolute media tree. +- Consolidated GitHub Actions, C#, JavaScript/TypeScript, and Python under the + advanced CodeQL workflow, removed repository-wide rule exclusions, and + replaced the invalid sanitizer summary with an auto-discovered, + flow-specific barrier model. Source changes fix the six original C# alerts; + the Python setup alert is handled by a hardened necessary persistence boundary + and narrow documented suppression. Fresh local verification reduced C# from + eight findings to zero and reports zero Actions and JavaScript/TypeScript + findings. Fresh local-threat-model Python analysis reports zero findings and + no SARIF warning/error notifications across the complete 45-query default + suite, while the 52-query security-extended suite also reports zero findings. +- Pinned all external actions across the six CI workflows to verified commits + (and the Gitleaks container to an immutable digest), and validated release + image digests before constructing multi-architecture manifests. +- Updated the runtime image to the official .NET 10 Ubuntu 26.04 variant and + current FFmpeg 8 packages, removed unused vulnerable inherited tooling, and + made the application process PID 1 under the unprivileged `melodee` account. + Trivy 0.72.0 validation reduced raw image findings from 416 to 140, with no + critical, high, fixable, .NET-package, or application-package findings among + the remainder. The 140 remaining entries represent 41 CVEs: 124 medium and 16 + low, a reduction of 276 findings (66.3%). +- Corrected Trivy SARIF severity filtering so GitHub code scanning receives the + intended critical/high policy while CI retains a complete all-severity JSON + report for review. +- Completed security verification includes a zero-warning solution build, 5,885 + passing .NET tests (34 skipped), zero vulnerable NuGet dependencies, and + successful Jekyll, GitHub Actions, YAML, shell, and real PostgreSQL container + checks. The production integration runs non-root as PID 1, reaches healthy + status, and keeps configured secret values out of logs. +- All 109 Python script tests pass with resource warnings treated as errors, + including 57 focused cleanup filesystem, ZIP, SFV, shutdown, and race + regressions. ## [2.1.4] - 2026-06-16 @@ -302,7 +373,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Podcast discovery, subscription, and playback. - Event scripting engine for custom automation. - MQL (Melodee Query Language) for advanced search and filtering. -- Scrobbling support (Last.fm and compatible services). +- Last.fm scrobbling support. - User sharing and playlist management. - Custom theming with Radzen theme support and custom CSS overrides. - Multi-library support (Inbound, Staging, Storage). @@ -311,7 +382,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Onboarding wizard for first-time setup. - Request system for user-submitted metadata corrections. - Radio station management. -- Chart generation and display. +- Chart import, album linking, and display. - User device profiles. - Multi-language localization (en-US, de-DE, es-ES, fr-FR, it-IT, ja-JP, pt-BR, ru-RU, zh-CN, ar-SA). - Docker multi-arch images (linux/amd64, linux/arm64) via GitHub Container Registry. @@ -325,7 +396,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed - Migrated to .NET 10 runtime. -- Migrated from SQLite to [DecentDB](https://github.com/sphildreth/decentdb) +- Continued using PostgreSQL as the primary application database and introduced + [DecentDB](https://github.com/sphildreth/decentdb) for generated MusicBrainz + and artist-search data stores. - Centralized configuration via `IMelodeeConfigurationFactory` with environment variable overrides. ### Fixed diff --git a/docs/pages/charts.md b/docs/pages/charts.md index c42e49ccb..6e4861662 100644 --- a/docs/pages/charts.md +++ b/docs/pages/charts.md @@ -1,244 +1,130 @@ --- title: Charts +description: Import curated album rankings, link them to library albums, and expose optional chart playlists. permalink: /charts/ +tags: + - administration + - charts + - playlists --- # Charts -Charts in Melodee allow you to import, manage, and explore curated lists of albums from music publications, websites, or your own custom rankings. Think of charts as "best of" lists—like Rolling Stone's 500 Greatest Albums, Pitchfork's Album of the Year, or your personal top albums of 2024. +Charts are ranked album lists such as publication year-end lists or a personal top 100. Melodee stores the source entry even when the album is not in your library, then tries to link each entry to a local album. -## What Are Charts? +## Browse charts -Charts are ranked lists of albums that can be: +Open **Charts** from the main navigation. Non-administrators see only charts marked **Visible**. The list shows each chart's source, year, item count, link completion, and generated-playlist status. -- **Imported** from external sources (CSV files) -- **Linked** to albums in your library automatically or manually -- **Browsed** to discover new music you might be missing -- **Converted** to playlists for easy listening +Two reports are available: -Charts help you answer questions like: -- "How many albums from the Rolling Stone 500 do I have?" -- "Which critically acclaimed albums am I missing?" -- "What should I listen to next from this year's best-of lists?" +- **Ranked report** combines linked and missing entries across charts. +- **Missing report** shows entries that are not linked to a local album. It can also be opened for one chart. -## Chart Structure +## Create or import a chart -Each chart consists of: +Chart management is restricted to administrators. -| Field | Description | -|-------|-------------| -| **Title** | The name of the chart (e.g., "Rolling Stone 500 Greatest Albums") | -| **Slug** | URL-friendly identifier (auto-generated from title) | -| **Source Name** | Where the chart came from (e.g., "Rolling Stone", "Pitchfork") | -| **Source URL** | Link to the original chart online | -| **Year** | The year the chart was published | -| **Description** | Optional notes about the chart | -| **Tags** | Categorization tags (e.g., "rock", "2024", "best-of") | -| **Visibility** | Whether the chart appears in public listings | +### Create with CSV -### Chart Items +Select **Create Chart**, enter the chart metadata, and paste the ranked items. A new chart requires at least one valid CSV row. -Each item in a chart represents an album entry: - -| Field | Description | -|-------|-------------| -| **Rank** | Position in the chart (1 = highest) | -| **Artist Name** | The artist as listed in the source | -| **Album Title** | The album name as listed in the source | -| **Release Year** | Optional year the album was released | -| **Link Status** | Whether the item is linked to your library | - -## Link Status - -Chart items have one of four link statuses: - -| Status | Description | -|--------|-------------| -| **Unlinked** | No matching album found in your library | -| **Linked** | Successfully matched to an album in your library | -| **Ambiguous** | Multiple potential matches found—needs manual resolution | -| **Ignored** | Manually marked to skip (e.g., if you don't want the album) | - -## Creating Charts - -### From CSV Import - -The easiest way to create a chart is by importing a CSV file: - -1. Navigate to **Charts** in the Melodee UI -2. Click **Create New Chart** -3. Fill in chart metadata (title, source, year, etc.) -4. Upload or paste CSV content - -#### CSV Format - -The CSV should have columns in this order: +Use one row per item, with no header row: ```csv -Rank,Artist,Album,Year (optional) 1,The Beatles,Abbey Road,1969 2,Pink Floyd,The Dark Side of the Moon,1973 3,Nirvana,Nevermind,1991 ``` -- **Rank**: Positive integer (1 = top of chart) -- **Artist**: Artist name as it appears on the chart -- **Album**: Album title as it appears on the chart -- **Year**: Optional release year +The columns are: + +1. positive, unique rank; +2. artist name; +3. album title; +4. optional release year. -Quotes are supported for values containing commas: +Quote fields that contain commas: ```csv -1,"Crosby, Stills, Nash & Young",Déjà Vu,1970 +1,"Crosby, Stills, Nash & Young","Déjà Vu",1970 ``` -### Manual Creation - -You can also create charts manually through the UI by adding items one at a time. - -## Automatic Linking - -When you import a chart, Melodee automatically attempts to link each item to albums in your library: - -1. **Exact Match**: Artist and album names match exactly (normalized) -2. **Fuzzy Match**: Partial matches when exact matching fails -3. **Artist Match Only**: Artist found but no matching album - -The linking process assigns confidence scores: - -| Confidence | Meaning | -|------------|---------| -| 1.0 | Exact match found | -| 0.7 - 0.8 | Fuzzy match or multiple possibilities | -| 0.5 | Artist found but album not matched | - -## Manual Linking - -For ambiguous or unlinked items, you can manually resolve them: - -1. Click on an unlinked/ambiguous chart item -2. Search for the correct album in your library -3. Select the match to link them - -You can also: -- **Ignore** items you don't want to track -- **Re-link** items if the automatic match was wrong - -## Generated Playlists - -Charts can automatically generate playlists from linked albums: - -1. Enable **"Generated Playlist"** on the chart -2. All songs from linked albums are included -3. Songs are ordered by chart rank, then track number - -This lets you listen through a "best of" list in ranked order. - -## Use Cases - -### Tracking Your Collection - -Import charts to see how your library compares: - -- **Rolling Stone 500**: See which classics you own -- **Grammy Winners**: Track award-winning albums -- **Decade Lists**: Compare your 80s, 90s, 2000s collections - -### Music Discovery - -Use unlinked items as a shopping/wishlist: - -- Filter charts by "Unlinked" status -- See highly-ranked albums you're missing -- Prioritize acquisitions based on chart positions - -### Personal Rankings - -Create your own charts: - -- "My Top 100 Albums of All Time" -- "Best Albums of 2024" -- "Desert Island Discs" - -### Playlist Generation - -Turn any chart into a listenable playlist: - -1. Import a chart (e.g., "Pitchfork Best New Albums 2024") -2. Enable generated playlist -3. Listen through the chart in ranked order - -## Managing Charts +Importing CSV into an existing chart replaces all of that chart's items and runs automatic linking. It also replaces any manual item links, so review the preview before saving. + +### Import JSON + +Select **Import JSON** on the Charts page to upload a file or paste JSON. The importer accepts this shape: + +```json +{ + "title": "Best Albums of 2025", + "sourceName": "Example publication", + "sourceUrl": "https://example.com/list", + "year": 2025, + "description": "Annual editors' list.", + "tags": ["2025", "editors-picks"], + "imageUrl": "https://example.com/chart-cover.jpg", + "isVisible": true, + "isGeneratedPlaylistEnabled": true, + "items": [ + { + "rank": 1, + "artistName": "Example Artist", + "albumTitle": "Example Album", + "releaseYear": 2025 + } + ] +} +``` -### Editing Charts +`title` and at least one item are required. Every item needs a positive, unique `rank`, `artistName`, and `albumTitle`. The other fields are optional. -- Update chart metadata (title, source, description) -- Re-run automatic linking after adding new albums -- Manually adjust link status for individual items +Example chart files are maintained in the repository's `design/charts/` directory. -### Chart Images +## Linking behavior -Upload cover images for charts to make them visually identifiable in the UI. +Each entry has one of four states: -### Filtering and Search +| State | Meaning | +|---|---| +| **Linked** | One local album was selected. | +| **Ambiguous** | More than one local album matched. | +| **Unlinked** | No local album matched. | +| **Ignored** | An administrator chose not to track this entry. | -Filter charts by: -- **Year**: Find charts from a specific year -- **Source**: Show only charts from a particular publication -- **Tags**: Filter by genre or category tags +Automatic linking first normalizes the artist and album names. It then tries: -## Reports +1. one exact album-title match for the normalized artist; +2. one partial title match for that artist; +3. otherwise, Ambiguous or Unlinked. -Melodee provides reports to help manage charts: +Release year is retained for display but is not part of automatic matching. Administrators can search for an album and resolve an entry manually, mark an entry ignored, or choose **Relink All**. Relinking can preserve or overwrite existing manual links. -### Missing Albums Report +The **ChartUpdateJob** retries unlinked chart entries after new albums arrive. Its default schedule is daily at 02:00 and is controlled by `jobs.chartUpdate.cronExpression`. -Shows all unlinked chart items across all charts, helping you identify: -- Albums you might want to acquire -- Items that need manual linking -- Patterns in what's missing from your library +## Generated playlists -### Ranked Report +When **Generated Playlist** is enabled, Melodee exposes every song from each linked album. Albums follow chart rank; songs follow disc/track order as represented by the song number. This is generated at request time and is not a user-owned editable playlist. -View chart items sorted by their ranking across all charts, highlighting the most acclaimed albums. +## Chart images -## API Access +The editor accepts an optional image and stores a GIF derivative in the library whose type is **Chart**. A Chart library is not seeded by default in 2.2.0. To use chart images: -Charts are accessible via the Melodee API: +1. create a writable directory and mount it into the Melodee container; +2. add an administrator-managed library with type **Chart** and that path; +3. upload the chart image from the chart editor. -``` -# List all charts -GET /api/v1/Charts +Chart metadata and items work without an image. -# Get chart by ID -GET /api/v1/Charts/{id} +## Native API -# Get chart by slug -GET /api/v1/Charts/slug/{slug} +The chart API is read-only; use the administrator UI for changes. It requires a native API bearer token and returns visible charts only. -# Get generated playlist tracks -GET /api/v1/Charts/{id}/playlist +```text +GET /api/v1/charts?page=1&pageSize=20 +GET /api/v1/charts/{numericIdOrSlug} +GET /api/v1/charts/{numericIdOrSlug}/playlist ``` -## Best Practices - -- **Use consistent source names**: Makes filtering easier (e.g., always "Rolling Stone" not "RS") -- **Include source URLs**: Helps verify original rankings -- **Tag charts appropriately**: Use tags like "rock", "2024", "best-of" for organization -- **Re-link periodically**: After adding albums, re-run linking to catch new matches -- **Review ambiguous items**: Manual resolution improves accuracy - -## Examples of Charts to Import - -| Chart | Source | Items | -|-------|--------|-------| -| 500 Greatest Albums | Rolling Stone | 500 | -| Album of the Year | Pitchfork | ~50/year | -| Mercury Prize Winners | Mercury Prize | ~30 | -| Grammy Album of the Year | Recording Academy | ~65 | -| 1001 Albums You Must Hear | Book | 1001 | -| NME Greatest Albums | NME | Various | - ---- - -Have questions about charts? Open an issue on GitHub or check the [API documentation](/api/) for technical details. +The list supports optional `tags`, `year`, and `source` query parameters. The playlist route returns an error unless generated playlists are enabled for that chart. See [Native API](/api/) for authentication and response conventions. diff --git a/docs/pages/cli-remote-mode.md b/docs/pages/cli-remote-mode.md index a122a74f9..b2fcf0b22 100644 --- a/docs/pages/cli-remote-mode.md +++ b/docs/pages/cli-remote-mode.md @@ -1,252 +1,152 @@ --- title: CLI Remote Server Mode -description: Use mcli to manage remote Melodee servers via REST API +description: Use mcli's supported remote commands with a Melodee JWT bearer token. +permalink: /cli-remote-mode/ tags: - cli - remote - api + - security --- # CLI Remote Server Mode -## Overview +Remote mode calls the native REST API over HTTP or HTTPS. In Melodee 2.2.0 it +supports exactly four operations: -`mcli` supports **Remote Server Mode**, allowing you to manage Melodee servers remotely over HTTPS using the REST API. This enables administration from any workstation without needing access to the server filesystem. +| Command | Required capability | +|---------|---------------------| +| `mcli system ... info` | None for server info; a token is still required by `mcli` | +| `mcli user me ...` | Authenticated user | +| `mcli user list ...` | Administrator | +| `mcli search ...` | Authenticated user | -## Quick Start +Library processing, jobs, configuration, backups, and destructive data commands +are local-only. -### Basic Remote Command +## Obtain a JWT -```bash -mcli --server https://demo.melodee.org --token YOUR_API_TOKEN system info -``` - -### Using Environment Variables +Authenticate against the native API. Avoid placing the password in shell +history; the example is suitable for an interactive test after substituting +safe input handling: ```bash -export MELODEE_SERVER=https://demo.melodee.org -export MELODEE_TOKEN=YOUR_API_TOKEN - -mcli system info -mcli user me -mcli user list +curl --fail --silent --show-error \ + --request POST https://music.example.com/api/v1/auth/authenticate \ + --header 'Content-Type: application/json' \ + --data '{"userName":"alice","email":null,"password":"replace-me"}' ``` -### Using Configuration Profiles +The response's `token` is a JWT access token. Its default lifetime is 15 +minutes. A user's persistent GUID API key is not interchangeable with this JWT. -Create a config file at: -- Linux/macOS: `~/.config/melodee/mcli.json` -- Windows: `%APPDATA%\melodee\mcli.json` - -```json -{ - "profiles": { - "demo": { - "server": "https://demo.melodee.org", - "token": "your-api-token-here" - }, - "prod": { - "server": "https://melodee.example.com", - "token": "your-prod-token-here" - } - }, - "defaults": { - "profile": "demo" - } -} -``` - -Then use the profile: +Set the token in the environment rather than a command-line option: ```bash -mcli --profile prod system info -mcli --profile demo user me +export MELODEE_SERVER=https://music.example.com +export MELODEE_TOKEN='eyJ...' ``` -## Global Options - -| Option | Environment Variable | Description | -|--------|---------------------|-------------| -| `--server ` | `MELODEE_SERVER` | Remote server URL (e.g., `https://demo.melodee.org`) | -| `--token ` | `MELODEE_TOKEN` | API authentication token (Bearer token) | -| `--profile ` | `MELODEE_PROFILE` | Profile name from config file | -| `--json` | N/A | Output compact JSON (default: pretty-printed) | - -## Precedence Rules - -Options are resolved in this order (highest priority first): - -1. **Command line flags** (`--server`, `--token`, `--profile`) -2. **Environment variables** (`MELODEE_SERVER`, `MELODEE_TOKEN`, `MELODEE_PROFILE`) -3. **Config file profile** (from `defaults.profile` or explicit `--profile`) - -## Remote Mode Commands - -The following commands work in both local and remote mode: - -### System Information - -```bash -mcli system info -``` - -Returns server version, name, and description. - -### Current User +## Commands ```bash mcli user me +mcli user list --limit 25 +mcli search 'Miles Davis' --limit 10 +mcli system info ``` -Returns information about the authenticated user. - -### List Users (Admin Only) +When using flags instead of environment variables, option placement follows +the command settings: ```bash -mcli user list +mcli user me --server https://music.example.com --token 'eyJ...' +mcli user list --server https://music.example.com --token 'eyJ...' +mcli search 'Miles Davis' --server https://music.example.com --token 'eyJ...' +mcli system --server https://music.example.com --token 'eyJ...' info ``` -Lists all users (requires admin privileges). - -### Search - -```bash -mcli search "Pink Floyd" -mcli search "Dark Side" --limit 10 -``` +`--server` is not a top-level option, so placing it before `search`, `system`, +or `user` produces a usage error. -Search for artists, albums, songs, and playlists. +## Profiles -## Output Formats +The CLI loads profiles from: -### Pretty JSON (Default) +- Linux/macOS: `${XDG_CONFIG_HOME:-$HOME/.config}/melodee/mcli.json` +- Windows: `%APPDATA%\melodee\mcli.json` -```bash -mcli system info +```json +{ + "profiles": { + "home": { + "server": "https://music.example.com", + "token": "eyJ..." + } + }, + "defaults": { + "profile": "home" + } +} ``` -Outputs formatted JSON with indentation. - -### Compact JSON +Use a named profile with the supported command: ```bash -mcli --json system info +mcli user me --profile home +mcli system --profile home info ``` -Outputs compact JSON on a single line. - -## Security - -### Token Safety - -**⚠️ SECURITY WARNING**: Never pass tokens on the command line in production scripts or shared environments. Tokens passed via `--token` are visible in shell history. - -**Recommended approaches** (in order of preference): - -1. **Use config file profiles** - Most secure for repeated use -2. **Use environment variables** - Good for CI/CD and scripts -3. **Use command line flags** - Only for one-off manual commands - -### Token Storage - -The config file (`mcli.json`) stores tokens in plain text. Ensure proper file permissions: +Profiles store tokens as clear text. Restrict the file to the owning user and +remember that short-lived JWTs expire: ```bash -# Linux/macOS chmod 600 ~/.config/melodee/mcli.json ``` -## Error Codes +## Precedence -`mcli` uses deterministic exit codes for remote mode: +Remote settings resolve from highest to lowest priority: -| Exit Code | Meaning | -|-----------|---------| -| 0 | Success | -| 2 | Usage/config error (missing server/token/profile) | -| 10 | Network error (DNS, connection refused, TLS handshake) | -| 11 | Timeout | -| 12 | Unauthorized/Forbidden (HTTP 401/403) | -| 13 | Not found (HTTP 404) | -| 14 | Server error (HTTP 5xx) | -| 15 | Unexpected/serialization error | +1. `--server`, `--token`, and `--profile` on the applicable command +2. `MELODEE_SERVER`, `MELODEE_TOKEN`, and `MELODEE_PROFILE` +3. The selected profile, including `defaults.profile` -## Examples +Remote mode activates only when a server URL resolves. A server without a token +returns a configuration error. -### Get System Info from Demo Server +## JSON Output -```bash -mcli --server https://demo.melodee.org --token demo-token system info -``` - -### List Users with Profile +The four remote-capable command settings expose `--json`. It selects compact +JSON instead of indented JSON where supported: ```bash -# First, create profile in ~/.config/melodee/mcli.json -mcli --profile demo user list +mcli user me --json +mcli search 'Miles Davis' --limit 5 --json +mcli system --json info ``` -### Search from Remote Server +## Exit Codes -```bash -export MELODEE_SERVER=https://demo.melodee.org -export MELODEE_TOKEN=your-token +| Code | Meaning | +|------|---------| +| `0` | Success | +| `2` | Missing remote configuration or token | +| `10` | DNS, connection, or TLS failure | +| `11` | Timeout | +| `12` | HTTP 401 or 403 | +| `13` | HTTP 404 | +| `14` | HTTP 5xx | +| `15` | Unexpected response or serialization failure | -mcli search "Miles Davis" --limit 5 --json -``` - -### Admin Operations - -```bash -# Requires admin token -mcli --server https://melodee.example.com --token admin-token user list -``` - -## Troubleshooting - -### "ERROR: Missing API token" - -Ensure you provide a token via `--token`, `MELODEE_TOKEN`, or a config profile. - -### "ERROR (401 Unauthorized): API token invalid or expired" - -Your token is invalid or has expired. Obtain a new token from the Melodee web interface. - -### "ERROR (403 Forbidden): Token does not have permission" - -Your token doesn't have the required permissions (e.g., admin access for `user list`). - -### "ERROR (404 Not Found): Endpoint not available" - -The server version doesn't support the requested endpoint. Ensure server and client versions are compatible. - -## Local Mode vs Remote Mode - -| Feature | Local Mode | Remote Mode | -|---------|-----------|-------------| -| **Trigger** | No `--server` flag | `--server` flag provided | -| **Access** | Direct database/filesystem access | REST API over HTTPS | -| **Authentication** | Not required | API token required | -| **Use Case** | Local administration | Remote administration | -| **Speed** | Faster (direct access) | Slower (network latency) | - -## Getting an API Token - -1. Log in to the Melodee web interface -2. Navigate to **User Settings → API Tokens** -3. Click **Generate New Token** -4. Copy the token and store it securely - -## Best Practices - -1. **Use profiles for frequent remote access** - Avoid typing server/token repeatedly -2. **Set default profile** - Configure `defaults.profile` in config file -3. **Rotate tokens regularly** - Generate new tokens periodically for security -4. **Use descriptive profile names** - e.g., `prod`, `staging`, `demo` -5. **Secure config file** - Ensure proper file permissions (chmod 600) +## Security -## See Also +- Prefer HTTPS except on an isolated loopback connection. +- Do not pass tokens through shared process listings or shell history. +- Unset `MELODEE_TOKEN` after use on shared systems. +- Do not commit `mcli.json`. +- Use an administrator token only for `user list`; use a least-privileged user + for search and self-service commands. -- [CLI Command Reference](/cli-commands) -- [API Documentation](/api/v1) -- [Configuration Guide](/configuration) +See [Native API authentication](/api/#authentication) and the +[CLI command reference](/cli/). diff --git a/docs/pages/cli.md b/docs/pages/cli.md index ad1ccaf47..14fe804e7 100644 --- a/docs/pages/cli.md +++ b/docs/pages/cli.md @@ -1,192 +1,126 @@ --- title: Command Line Interface (CLI) +description: Run Melodee administration, library, diagnostics, search, and maintenance commands with mcli. permalink: /cli/ +tags: + - cli + - administration + - automation --- # Command Line Interface (CLI) -The Melodee CLI (`mcli`) is a powerful command-line utility for managing music libraries, processing media files, and performing maintenance tasks. It provides direct access to Melodee's core functionality without requiring the web interface. - -## Getting Started - -### Installation - -The CLI is included with Melodee and is built as a standalone executable: +`mcli` provides local database/filesystem administration and a small remote API +mode. The command's built-in help is the source of truth for the installed +version: ```bash -# Build from source -dotnet build src/Melodee.Cli/Melodee.Cli.csproj - -# Run from debug folder -cd src/Melodee.Cli/bin/Debug/net10.0 -./mcli --help +mcli --help +mcli help library +mcli help library scan ``` -### Configuration +## Run the CLI -The CLI uses the same configuration as the web application. You can specify a custom configuration file using an environment variable: +The published application image includes the CLI at `/app/cli/mcli`: ```bash -# Point to a specific appsettings.json file -export MELODEE_APPSETTINGS_PATH="/path/to/appsettings.json" - -# Or inline -MELODEE_APPSETTINGS_PATH="/path/to/appsettings.json" ./mcli library list +docker compose exec melodee.blazor /app/cli/mcli --help +docker compose exec melodee.blazor /app/cli/mcli doctor ``` -**Environment Variables:** - -| Variable | Description | Example | -|----------|-------------|---------| -| `MELODEE_APPSETTINGS_PATH` | Path to custom appsettings.json | `/etc/melodee/appsettings.json` | -| `ASPNETCORE_ENVIRONMENT` | Environment name | `Development`, `Production` | - -## Command Structure - -The CLI uses a hierarchical command structure: +This is the simplest local mode because the container already has the same +connection strings and volume mounts as the server. -``` -mcli [BRANCH] [COMMAND] [OPTIONS] -``` - -## Available Commands - -| Command | Description | Documentation | -|---------|-------------|---------------| -| [album](/cli/album/) | Album data management and statistics | [View Details](/cli/album/) | -| [artist](/cli/artist/) | Artist data management and statistics | [View Details](/cli/artist/) | -| [configuration](/cli/configuration/) | Manage Melodee configuration settings | [View Details](/cli/configuration/) | -| [doctor](/cli/doctor/) | Run environment and configuration diagnostics | [View Details](/cli/doctor/) | -| [file](/cli/file/) | File analysis and inspection tools | [View Details](/cli/file/) | -| [import](/cli/import/) | Import data from external sources | [View Details](/cli/import/) | -| [job](/cli/job/) | Run background jobs and maintenance tasks | [View Details](/cli/job/) | -| [library](/cli/library/) | Library management and operations | [View Details](/cli/library/) | -| [parser](/cli/parser/) | Parse and analyze media metadata files | [View Details](/cli/parser/) | -| [tags](/cli/tags/) | Display and manage media file tags | [View Details](/cli/tags/) | -| [user](/cli/user/) | User account management | [View Details](/cli/user/) | -| [validate](/cli/validate/) | Validate media files and metadata | [View Details](/cli/validate/) | - -## Quick Examples +To build it from source: ```bash -# List all libraries -./mcli library list - -# Full scan workflow - process inbound → staging → storage → database -./mcli library scan - -# Silent scan for cron jobs (no output, exit code only) -./mcli library scan --silent - -# JSON scan output for scripting/automation -./mcli library scan --json - -# Show album statistics -./mcli album stats - -# Search for albums -./mcli album search "Beatles" - -# Find albums added in the last 7 days -./mcli album search --since 7 - -# Search and sort by year (newest first) -./mcli album search --sort Year --sort-dir desc - -# Bulk delete albums added in the last 5 days (with confirmation) -./mcli album search --since 5 --delete - -# Search for an artist -./mcli artist search "Beatles" - -# Find artists added in the last 30 days, sorted by album count -./mcli artist search --since 30 --sort Albums --sort-dir desc - -# Find duplicate artists with high confidence -./mcli artist find-duplicates -m 0.9 - -# Find and merge duplicate artists -./mcli artist find-duplicates -m 0.95 --merge - -# List all users -./mcli user list +dotnet build src/Melodee.Cli/Melodee.Cli.csproj +src/Melodee.Cli/bin/Debug/net10.0/mcli --help +``` -# Create a new user -./mcli user create -u "demo" -e "demo@melodee.org" -p "SecurePass123!" +## Local Configuration -# List background jobs with next run times -./mcli job list +Local mode opens PostgreSQL, DecentDB, and library paths directly. Run it on a +trusted machine with access to those resources. -# Run a specific job -./mcli job run -j ArtistHousekeepingJob +| Variable | Purpose | +|----------|---------| +| `MELODEE_APPSETTINGS_PATH` | Path to a specific `appsettings.json` file | +| `ASPNETCORE_ENVIRONMENT` | Selects `appsettings.{Environment}.json` | +| `MELODEE_ENVIRONMENT` | CLI environment fallback | +| `ConnectionStrings__DefaultConnection` | PostgreSQL override | +| `ConnectionStrings__MusicBrainzConnection` | MusicBrainz DecentDB override | +| `ConnectionStrings__ArtistSearchEngineConnection` | Artist Search DecentDB override | -# Get a configuration value -./mcli configuration get imaging.smallSize +```bash +MELODEE_APPSETTINGS_PATH=/etc/melodee/appsettings.json mcli library list ``` -## Common Options - -These options are available across most commands: +## Command Reference + +| Command | Purpose | Reference | +|---------|---------|-----------| +| `album` | List, search, inspect, and delete albums | [Album](/cli/album/) | +| `artist` | List, search, deduplicate, and delete artists | [Artist](/cli/artist/) | +| `backup export` | Export settings and libraries | [Backup](/cli/backup/) | +| `configuration` | Read and change database-backed settings | [Configuration](/cli/configuration/) | +| `doctor` | Diagnose the installation | [Doctor](/cli/doctor/) | +| `file mpeg` | Inspect an audio file | [File](/cli/file/) | +| `import` | Import user favorite songs from CSV | [Import](/cli/import/) | +| `job` | List and run background jobs | [Job](/cli/job/) | +| `library` | Process, move, scan, validate, and report on libraries | [Library](/cli/library/) | +| `parser parse` | Parse CUE, M3U, NFO, or SFV metadata | [Parser](/cli/parser/) | +| `search` | Search artists, albums, songs, and playlists | [Search](/cli/search/) | +| `system info` | Print server information | [System](/cli/system/) | +| `tags show` | Inspect tags in a media file | [Tags](/cli/tags/) | +| `user` | Create, list, inspect, and delete users | [User](/cli/user/) | +| `validate` | Validate `melodee.json` album metadata | [Validate](/cli/validate/) | + +## Common Examples -| Option | Description | -|--------|-------------| -| `-h`, `--help` | Show help information for the command | -| `--json` | Output results as JSON (structured output for automation) | -| `--raw` | Output in JSON format for scripting | -| `--silent` | Suppress all output (for cron jobs, returns exit code only) | -| `--verbose` | Output verbose debug and timing information (default: false) | +```bash +# Deep installation checks +mcli doctor --write-test -## Exit Codes +# Full Inbound -> Staging -> Storage -> PostgreSQL workflow +mcli library scan -The CLI uses standard exit codes: +# JSON scan summary without progress rendering +mcli library scan --json -| Code | Meaning | -|------|---------| -| `0` | Success | -| `1` | Error or failure | +# Search recent albums +mcli album search --since 7 --sort Added --sort-dir desc -These can be used in scripts for error handling: +# Create a user +mcli user create --username alice --email alice@example.com --password 'replace-me' -```bash -#!/bin/bash -./mcli library stats --library "Storage" -if [ $? -eq 0 ]; then - echo "Success!" -else - echo "Failed!" -fi +# Export configuration without clear-text secrets +mcli backup export --output melodee-settings.json --redact-secrets ``` -## Troubleshooting +## Output and Exit Codes -### "Invalid library Name" Error +Output flags are command-specific; `--json`, `--raw`, `--silent`, and +`--verbose` are not universal global options. Check the target command's help +before using it in automation. -**Problem:** The CLI can't find the library in the database. +Most commands use `0` for success and a nonzero value for failure, but some +legacy commands have command-specific behavior. Remote mode reserves exit codes +10-15; see [CLI Remote Server Mode](/cli-remote-mode/#exit-codes). Test the exact +command and version before making an automation decision solely from its exit +status. -**Solution:** -1. Check that you're using the correct configuration file: - ```bash - MELODEE_APPSETTINGS_PATH="/path/to/appsettings.json" ./mcli library list - ``` -2. Verify the library exists: - ```bash - ./mcli library list - ``` -3. Check connection string in `appsettings.json` +## Destructive Commands -### Configuration Not Found - -**Problem:** The CLI can't find `appsettings.json`. - -**Solution:** -Use the `MELODEE_APPSETTINGS_PATH` environment variable: -```bash -export MELODEE_APPSETTINGS_PATH="/full/path/to/appsettings.json" -``` +Commands such as `album delete`, `artist delete`, `library clean`, +`library purge`, duplicate merging, and `configuration set --remove` can remove +database records or files. Create a [backup](/backup/), omit confirmation-bypass +flags on the first run, and use preview or JSON modes where provided. -## See Also +## Remote Mode -- [Background Jobs](/jobs/) - Automated job scheduling -- [Libraries](/libraries/) - Library concepts and management -- [Configuration Reference](/configuration-reference/) - All configuration settings -- [API Reference](/api/) - REST API documentation +Only `search`, `system info`, `user me`, and `user list` currently use the +remote REST client. Other commands remain local even if similarly named. See +[CLI Remote Server Mode](/cli-remote-mode/) for token acquisition, exact option +placement, and security guidance. diff --git a/docs/pages/cli/album.md b/docs/pages/cli/album.md index 195e7d588..212fff936 100644 --- a/docs/pages/cli/album.md +++ b/docs/pages/cli/album.md @@ -1,533 +1,38 @@ --- title: CLI - Album Commands +description: List, search, inspect image issues, summarize, and delete albums with mcli. permalink: /cli/album/ -layout: page +tags: + - cli + - albums + - administration --- # Album Commands -The `album` branch provides commands for managing album data, searching albums, viewing statistics, and detecting image issues. - -## Overview - -```bash -mcli album [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Alias | Description | -|---------|-------|-------------| -| `delete` | `rm` | Delete an album from the database | -| `image-issues` | `img` | Find albums with missing, invalid, or misnumbered images | -| `list` | `ls` | List albums in the database | -| `search` | `s` | Search for albums by name | -| `stats` | | Show album statistics grouped by status | - ---- - -## album list - -Lists albums in the database with their details. - -### Usage - -```bash -mcli album list [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-n`, `--limit` | | `50` | Maximum number of results to return | -| `--raw` | | `false` | Output results in JSON format | -| `-s`, `--status` | | | Filter by album status (Ok, New, Invalid) | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples - -```bash -# List first 50 albums -./mcli album list - -# List 100 albums -./mcli album list -n 100 - -# List only albums with Ok status -./mcli album list --status Ok - -# List albums with Invalid status -./mcli album list -s Invalid - -# Output as JSON for scripting -./mcli album list --raw -``` - -### Output - -``` -╭────────────────────────────────────────────────────────────────────────────────────╮ -│ Artist │ Album │ Year │ Songs │ Status │ -├────────────────────────────────────────────────────────────────────────────────────┤ -│ The Beatles │ Abbey Road │ 1969 │ 17 │ Ok │ -│ Pink Floyd │ The Dark Side of the Moon │ 1973 │ 10 │ Ok │ -│ Led Zeppelin │ IV │ 1971 │ 8 │ New │ -╰────────────────────────────────────────────────────────────────────────────────────╯ - -Showing 3 of 1,234 albums -``` - ---- - -## album search - -Search for albums by name with optional date filtering, sorting, and bulk delete. - -### Usage - -```bash -mcli album search [QUERY] [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `QUERY` | No | Search query for album name. Use `*` or omit to match all albums. | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--delete` | | `false` | ⚠️ Delete all albums matching the search criteria | -| `--keep-files` | | `false` | Keep album files on disk when deleting (database only) | -| `-n`, `--limit` | | `25` | Maximum number of results to return | -| `--raw` | | `false` | Output results in JSON format | -| `--since` | | | Only show albums created within the last N days | -| `--sort` | | | Sort by column: Artist, Album, Year, Songs, Added, Status | -| `--sort-dir` | | `asc` | Sort direction: `asc` or `desc` | -| `--verbose` | | `false` | Output verbose debug and timing results | -| `-y`, `--yes` | | `false` | Skip confirmation prompt when deleting | - -### Examples - -```bash -# Search for albums containing "dark" -./mcli album search "dark" - -# Search with more results -./mcli album search "best of" -n 50 - -# Find all albums added in the last 7 days -./mcli album search --since 7 - -# Find all albums added today -./mcli album search --since 1 - -# Find albums added in last 30 days matching "beatles" -./mcli album search "beatles" --since 30 - -# JSON output for scripting -./mcli album search "greatest hits" --raw - -# Get recently added albums as JSON -./mcli album search --since 7 --raw -``` - -### Sorting Examples - -```bash -# Sort by artist name (A-Z) -./mcli album search --sort Artist - -# Sort by artist name (Z-A) -./mcli album search --sort Artist --sort-dir desc - -# Sort by year (oldest first) -./mcli album search --sort Year - -# Sort by year (newest first) -./mcli album search --sort Year --sort-dir desc - -# Albums added in last 10 days, sorted by added date (oldest first) -./mcli album search --since 10 --sort Added - -# Albums added in last 10 days, sorted by added date (newest first) -./mcli album search --since 10 --sort Added --sort-dir desc - -# Sort by song count (most songs first) -./mcli album search --sort Songs --sort-dir desc - -# Sort all albums by status -./mcli album search --sort Status -``` - -### Bulk Delete Examples - -⚠️ **WARNING: Delete operations are permanent and cannot be undone!** - -```bash -# Delete all albums added in the last 5 days (with confirmation) -./mcli album search --since 5 --delete - -# Delete albums matching "test" (with confirmation) -./mcli album search "test" --delete - -# Delete without confirmation (USE WITH CAUTION) -./mcli album search --since 1 --delete -y - -# Delete from database but keep files on disk -./mcli album search "duplicate" --delete --keep-files - -# Delete all albums matching query, keeping files, no confirmation -./mcli album search "bad import" --delete --keep-files -y -``` - -### Output - -``` -Search results for: dark - -╭────────────────────────────────────────────────────────────────────────────────────╮ -│ Artist │ Album │ Year │ Songs │ Status │ -├────────────────────────────────────────────────────────────────────────────────────┤ -│ Pink Floyd │ The Dark Side of the Moon │ 1973 │ 10 │ Ok │ -│ Metallica │ The Dark Side of Metallica │ 1991 │ 12 │ Ok │ -╰────────────────────────────────────────────────────────────────────────────────────╯ - -Found 2 matching albums (showing 2) -``` - -### Output with --since - -When using `--since`, results are sorted by creation date (newest first) and include an "Added" column using ISO8601 format (YYYYMMDDTHHMMSS): - -``` -Albums created in the last 7 days - -╭──────────────────────────┬────────────────────────────┬──────┬───────┬─────────────────┬────────╮ -│ Artist │ Album │ Year │ Songs │ Added │ Status │ -├──────────────────────────┼────────────────────────────┼──────┼───────┼─────────────────┼────────┤ -│ Taylor Swift │ The Tortured Poets Dept. │ 2024 │ 16 │ 20241230T142300 │ Ok │ -│ Billie Eilish │ Hit Me Hard and Soft │ 2024 │ 10 │ 20241229T091500 │ Ok │ -│ Sabrina Carpenter │ Short n' Sweet │ 2024 │ 12 │ 20241228T184200 │ Ok │ -╰──────────────────────────┴────────────────────────────┴──────┴───────┴─────────────────┴────────╯ - -Found 3 matching albums (showing 3) -``` - -### Delete Confirmation - -When using `--delete`, a confirmation prompt is shown with details about what will be deleted: - -``` -Albums created in the last 5 days - -╭──────────────────────────┬────────────────────────────┬──────┬───────┬─────────────┬────────╮ -│ Artist │ Album │ Year │ Songs │ Added │ Status │ -├──────────────────────────┼────────────────────────────┼──────┼───────┼─────────────┼────────┤ -│ Test Artist │ Test Album │ 2024 │ 5 │ 12-30 10:00 │ Ok │ -│ Another Test │ Bad Import │ 2024 │ 12 │ 12-29 15:30 │ Ok │ -╰──────────────────────────┴────────────────────────────┴──────┴───────┴─────────────┴────────╯ - -Found 2 matching albums (showing 2) - -─────────────────────── ⚠️ DESTRUCTIVE OPERATION ⚠️ ─────────────────────── - -This will permanently delete: - • 2 album(s) - • 17 song(s) - • All associated files on disk - -This action cannot be undone! - -Are you sure you want to delete these albums? [y/n] (n): -``` - -### Safety Features - -- **Confirmation required**: By default, you must confirm before deletion -- **Locked albums skipped**: Albums marked as locked will not be deleted -- **Clear summary**: Shows exactly how many albums, songs, and files will be affected -- **Keep files option**: Use `--keep-files` to remove from database only -- **Progress indicator**: Shows deletion progress for large batch operations - ---- - -## album stats - -Show album statistics grouped by status with detailed breakdowns. - -### Usage - ```bash -mcli album stats [OPTIONS] +mcli album [--verbose] COMMAND ``` -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | +| Command | Key arguments and options | +|---------|---------------------------| +| `list` | `--limit` (50), `--status`, `--raw` | +| `search [QUERY]` | `--since`, `--limit` (25), `--sort`, `--sort-dir`, `--raw`, `--delete`, `--keep-files`, `--yes` | +| `stats` | `--raw` | +| `image-issues` | `--missing`, `--invalid`, `--misnumbered`, `--limit` (100), `--raw` | +| `delete ID` | `--keep-files`, `--yes` | -### Examples +Valid `list --status` values are `Ok`, `New`, `NeedsAttention`, `Duplicate`, and +`Invalid`. Use `*` as the search query to match all albums; the query may be +omitted when `--since` is supplied. ```bash -# Show album statistics -./mcli album stats - -# JSON output for monitoring -./mcli album stats --raw +mcli album list --status NeedsAttention --limit 100 +mcli album search 'Abbey Road' --sort Year --sort-dir asc +mcli album search --since 7 --raw +mcli album image-issues --limit 50 ``` -### Output - -``` - Album Statistics -╭─────────────────────┬───────────┬───────╮ -│ Metric │ Count │ % │ -├─────────────────────┼───────────┼───────┤ -│ Total Albums │ 1,234 │ 100% │ -│ Total Songs │ 15,678 │ --- │ -│ ─────────────────── │ ───────── │ ───── │ -│ Missing Images │ 15 │ 1.2% │ -│ Locked │ 3 │ 0.2% │ -╰─────────────────────┴───────────┴───────╯ - - Albums by Status -╭─────────┬───────────┬───────╮ -│ Status │ Count │ % │ -├─────────┼───────────┼───────┤ -│ Ok │ 1,200 │ 97.2% │ -│ New │ 30 │ 2.4% │ -│ Invalid │ 4 │ 0.3% │ -╰─────────┴───────────┴───────╯ -``` - -### JSON Output - -```json -{ - "TotalAlbums": 1234, - "LockedAlbums": 3, - "AlbumsWithNoImages": 15, - "TotalSongs": 15678, - "StatusCounts": [ - { "Status": "Ok", "Count": 1200 }, - { "Status": "New", "Count": 30 }, - { "Status": "Invalid", "Count": 4 } - ] -} -``` - ---- - -## album delete - -Delete an album from the database with optional file deletion. - -### Usage - -```bash -mcli album delete [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `ID` | Yes | Album ID to delete | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--keep-files` | | `false` | Keep the album directory on disk (do not delete files) | -| `--verbose` | | `false` | Output verbose debug and timing results | -| `-y`, `--yes` | | `false` | Skip confirmation prompt | - -### Examples - -```bash -# Delete album (with confirmation, deletes files) -./mcli album delete 123 - -# Delete album but keep files on disk -./mcli album delete 123 --keep-files - -# Delete without confirmation (scripting) -./mcli album delete 123 -y - -# Delete from database only, no confirmation -./mcli album delete 123 --keep-files -y -``` - -### Output - -``` -Album: Abbey Road -Artist: The Beatles -Songs: 17 -Directory: /mnt/music/library/The Beatles/Abbey Road - -Delete album 'Abbey Road' and ALL files on disk? [y/n] (n): y - -✓ Album 'Abbey Road' deleted successfully. -``` - -### Safety Notes - -- ⚠️ **Locked albums cannot be deleted.** Unlock the album first. -- ⚠️ **Default behavior deletes files from disk.** Use `--keep-files` to preserve. -- The command shows album details and requires confirmation by default. - ---- - -## album image-issues - -Find albums with missing, invalid, or incorrectly numbered images. - -### Usage - -```bash -mcli album image-issues [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--invalid` | | `true` | Include albums with invalid images (wrong size, not square, etc.) | -| `-n`, `--limit` | | `100` | Maximum number of results to return | -| `--misnumbered` | | `true` | Include albums with incorrectly numbered images | -| `--missing` | | `true` | Include albums with missing images | -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Image Naming Convention - -Melodee expects album images to follow this naming pattern: - -``` -i-XX-Type.jpg -``` - -Where: -- `i-` is a required prefix -- `XX` is a two-digit sequential number starting from `01` -- `Type` is the image type (e.g., `Front`, `Back`, `Inside`) - -**Examples of valid image names:** -- `i-01-Front.jpg` (primary front cover) -- `i-02-Back.jpg` (back cover) -- `i-03-Inside.jpg` (inside artwork) - -### Issue Types - -**Missing:** -Albums with no images at all. - -**Invalid:** -Images that fail validation: -- Not square (width ≠ height) -- Below minimum size -- Corrupted or unreadable - -**Misnumbered:** -Images that don't follow sequential numbering: -- Gaps in numbering (e.g., `i-01`, `i-03` missing `i-02`) -- Wrong format (e.g., `cover.jpg` instead of `i-01-Front.jpg`) -- Starting number not `01` - -### Examples - -```bash -# Find all image issues -./mcli album image-issues - -# Find only albums with missing images -./mcli album image-issues --invalid=false --misnumbered=false - -# Find only albums with invalid images -./mcli album image-issues --missing=false --misnumbered=false - -# Find only misnumbered images -./mcli album image-issues --missing=false --invalid=false - -# Limit results -./mcli album image-issues -n 50 - -# JSON output for automation -./mcli album image-issues --raw -``` - -### Output - -``` -Scanning albums for image issues... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:05 - - Missing Images (5) -╭──────┬─────────────────────────┬────────────────────────────────┬─────────────────╮ -│ ID │ Artist │ Album │ Details │ -├──────┼─────────────────────────┼────────────────────────────────┼─────────────────┤ -│ 123 │ The Beatles │ Please Please Me │ No images found │ -│ 456 │ Rolling Stones │ Exile on Main St. │ No images found │ -╰──────┴─────────────────────────┴────────────────────────────────┴─────────────────╯ - - Invalid Images (2) -╭──────┬─────────────────────────┬────────────────────────────────┬──────────────────────────────────╮ -│ ID │ Artist │ Album │ Details │ -├──────┼─────────────────────────┼────────────────────────────────┼──────────────────────────────────┤ -│ 789 │ Pink Floyd │ Animals │ i-01-Front.jpg: Image not square │ -╰──────┴─────────────────────────┴────────────────────────────────┴──────────────────────────────────╯ - - Misnumbered Images (3) -╭──────┬─────────────────────────┬────────────────────────────────┬────────────────────────────────────╮ -│ ID │ Artist │ Album │ Details │ -├──────┼─────────────────────────┼────────────────────────────────┼────────────────────────────────────┤ -│ 101 │ Led Zeppelin │ Houses of the Holy │ i-03-Front.jpg (expected 01) │ -│ 102 │ Queen │ A Night at the Opera │ cover.jpg (invalid format) │ -╰──────┴─────────────────────────┴────────────────────────────────┴────────────────────────────────────╯ - -Found 10 albums with image issues -``` - -### JSON Output - -```json -[ - { - "AlbumId": 123, - "AlbumName": "Please Please Me", - "ArtistName": "The Beatles", - "Directory": "/mnt/music/library/The Beatles/Please Please Me", - "IssueType": "Missing", - "Details": "No images found" - }, - { - "AlbumId": 789, - "AlbumName": "Animals", - "ArtistName": "Pink Floyd", - "Directory": "/mnt/music/library/Pink Floyd/Animals", - "IssueType": "Invalid", - "Details": "i-01-Front.jpg: Image not square [800x600]" - } -] -``` - -### Use Cases - -- **Pre-release audit:** Check albums before moving to storage -- **Quality control:** Identify albums needing artwork attention -- **Automated monitoring:** Integrate with scripts for regular checks -- **Batch fixing:** Export JSON and process with external tools - ---- - -## See Also - -- [Artist Commands](/cli/artist/) - Artist data management -- [Library Commands](/cli/library/) - Library operations -- [CLI Overview](/cli/) - Main CLI documentation +`search --delete` and `delete` are destructive. Without `--keep-files`, deletion +can remove associated files as well as database records. Back up first and let +the confirmation prompt run before using `--yes`. diff --git a/docs/pages/cli/artist.md b/docs/pages/cli/artist.md index 13a865e2a..2b812d96c 100644 --- a/docs/pages/cli/artist.md +++ b/docs/pages/cli/artist.md @@ -1,741 +1,34 @@ --- title: CLI - Artist Commands +description: List, search, find duplicates, summarize, and delete artists with mcli. permalink: /cli/artist/ -layout: page +tags: + - cli + - artists + - administration --- # Artist Commands -The `artist` branch provides commands for managing artist data, searching artists, viewing statistics, and identifying potential issues like duplicates or missing images. - -## Overview - -```bash -mcli artist [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Alias | Description | -|---------|-------|-------------| -| `delete` | `rm` | Delete an artist from the database | -| `find-duplicates` | `fd` | Find and optionally merge duplicate artists using advanced similarity detection | -| `list` | `ls` | List artists in the database | -| `search` | `s` | Search for artists by name | -| `stats` | | Show artist statistics including missing images and potential duplicates | - ---- - -## artist list - -Lists artists in the database with their album and song counts. - -### Usage - -```bash -mcli artist list [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-n`, `--limit` | | `50` | Maximum number of results to return | -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples - -```bash -# List first 50 artists -./mcli artist list - -# List 100 artists -./mcli artist list -n 100 - -# Output as JSON for scripting -./mcli artist list --raw -``` - -### Output - -``` -╭─────────────────────────────┬────────┬───────┬───────┬────────┬──────────╮ -│ Name │ Albums │ Songs │ Plays │ Rating │ Status │ -├─────────────────────────────┼────────┼───────┼───────┼────────┼──────────┤ -│ The Beatles │ 13 │ 227 │ 1542 │ 4.8★ │ ✓ │ -│ Pink Floyd │ 15 │ 164 │ 892 │ 4.7★ │ ✓ │ -│ Led Zeppelin │ 9 │ 92 │ 456 │ 4.5★ │ 🔒 Locked │ -╰─────────────────────────────┴────────┴───────┴───────┴────────┴──────────╯ - -Showing 3 of 1,234 artists -``` - -### JSON Output - -```json -[ - { - "Id": 1, - "ApiKey": "a1b2c3d4-...", - "Name": "The Beatles", - "NameNormalized": "beatles", - "AlbumCount": 13, - "SongCount": 227, - "IsLocked": false, - "LibraryId": 3, - "CreatedAt": "2024-01-15T10:30:00Z", - "LastPlayedAt": "2024-12-30T08:15:00Z", - "PlayedCount": 1542, - "CalculatedRating": 4.8 - } -] -``` - ---- - -## artist search - -Search for artists by name with optional date filtering, sorting, and bulk delete. - -### Usage - -```bash -mcli artist search [QUERY] [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `QUERY` | No | Search query for artist name. Use `*` or omit to match all artists. | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--delete` | | `false` | ⚠️ Delete all artists matching the search criteria | -| `--keep-files` | | `false` | Keep artist files on disk when deleting (database only) | -| `-n`, `--limit` | | `25` | Maximum number of results to return | -| `--raw` | | `false` | Output results in JSON format | -| `--since` | | | Only show artists created within the last N days | -| `--sort` | | | Sort by column: Name, Albums, Songs, Added, Rating | -| `--sort-dir` | | `asc` | Sort direction: `asc` or `desc` | -| `--verbose` | | `false` | Output verbose debug and timing results | -| `-y`, `--yes` | | `false` | Skip confirmation prompt when deleting | - -### Examples - -```bash -# Search for artists containing "beatles" -./mcli artist search "beatles" - -# Search with more results -./mcli artist search "john" -n 50 - -# Find all artists added in the last 7 days -./mcli artist search --since 7 - -# Find all artists added today -./mcli artist search --since 1 - -# JSON output for scripting -./mcli artist search "pink" --raw - -# Get recently added artists as JSON -./mcli artist search --since 7 --raw -``` - -### Sorting Examples - -```bash -# Sort by name (A-Z) -./mcli artist search --sort Name - -# Sort by name (Z-A) -./mcli artist search --sort Name --sort-dir desc - -# Sort by album count (most albums first) -./mcli artist search --sort Albums --sort-dir desc - -# Sort by song count (most songs first) -./mcli artist search --sort Songs --sort-dir desc - -# Artists added in last 10 days, sorted by added date (oldest first) -./mcli artist search --since 10 --sort Added - -# Artists added in last 10 days, sorted by added date (newest first) -./mcli artist search --since 10 --sort Added --sort-dir desc - -# Sort by rating (highest first) -./mcli artist search --sort Rating --sort-dir desc -``` - -### Bulk Delete Examples - -⚠️ **WARNING: Delete operations are permanent and cannot be undone!** - -```bash -# Delete all artists added in the last 5 days (with confirmation) -./mcli artist search --since 5 --delete - -# Delete artists matching "test" (with confirmation) -./mcli artist search "test" --delete - -# Delete without confirmation (USE WITH CAUTION) -./mcli artist search --since 1 --delete -y - -# Delete from database but keep files on disk -./mcli artist search "duplicate" --delete --keep-files - -# Delete all artists matching query, keeping files, no confirmation -./mcli artist search "bad import" --delete --keep-files -y -``` - -### Output - -``` -Search results for: beatles - -╭─────────────────────────────┬────────┬───────┬────────╮ -│ Name │ Albums │ Songs │ Rating │ -├─────────────────────────────┼────────┼───────┼────────┤ -│ The Beatles │ 13 │ 227 │ 4.8★ │ -│ The Bootleg Beatles │ 2 │ 24 │ 3.5★ │ -╰─────────────────────────────┴────────┴───────┴────────╯ - -Found 2 matching artists (showing 2) -``` - -### Output with --since - -When using `--since`, results are sorted by creation date (newest first) and include an "Added" column using ISO8601 format (YYYYMMDDTHHMMSS): - -``` -Artists created in the last 7 days - -╭─────────────────────────────┬────────┬───────┬─────────────────┬────────╮ -│ Name │ Albums │ Songs │ Added │ Rating │ -├─────────────────────────────┼────────┼───────┼─────────────────┼────────┤ -│ New Artist │ 2 │ 24 │ 20241230T142300 │ --- │ -│ Another New Artist │ 1 │ 10 │ 20241229T091500 │ --- │ -╰─────────────────────────────┴────────┴───────┴─────────────────┴────────╯ - -Found 2 matching artists (showing 2) -``` - -### Delete Confirmation - -When using `--delete`, a confirmation prompt is shown with details about what will be deleted: - -``` -Artists created in the last 5 days - -╭─────────────────────────────┬────────┬───────┬─────────────────┬────────╮ -│ Name │ Albums │ Songs │ Added │ Rating │ -├─────────────────────────────┼────────┼───────┼─────────────────┼────────┤ -│ Test Artist │ 2 │ 15 │ 20241230T100000 │ --- │ -│ Bad Import Artist │ 3 │ 32 │ 20241229T153000 │ --- │ -╰─────────────────────────────┴────────┴───────┴─────────────────┴────────╯ - -Found 2 matching artists (showing 2) - -─────────────────────── ⚠️ DESTRUCTIVE OPERATION ⚠️ ─────────────────────── - -This will permanently delete: - • 2 artist(s) - • 5 album(s) - • 47 song(s) - • All associated files on disk - -This action cannot be undone! - -Are you sure you want to delete these artists? [y/n] (n): -``` - -### Safety Features - -- **Confirmation required**: By default, you must confirm before deletion -- **Locked artists skipped**: Artists marked as locked will not be deleted -- **Clear summary**: Shows exactly how many artists, albums, songs, and files will be affected -- **Keep files option**: Use `--keep-files` to remove from database only -- **Progress indicator**: Shows deletion progress for large batch operations - ---- - -## artist stats - -Show artist statistics including counts, missing images, and potential duplicate detection. - -### Usage - ```bash -mcli artist stats [OPTIONS] +mcli artist [--verbose] COMMAND ``` -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples +| Command | Key arguments and options | +|---------|---------------------------| +| `list` | `--library`, `--limit` (50), `--raw` | +| `search [QUERY]` | `--since`, `--limit` (25), `--sort`, `--sort-dir`, `--raw`, `--delete`, `--keep-files`, `--yes` | +| `stats` | `--raw` | +| `find-duplicates` | `--artist-id`, `--source`, `--min-score` (0.7), `--include-low-confidence`, `--limit`, `--json`, `--merge` | +| `delete ID` | `--keep-files`, `--yes` | ```bash -# Show artist statistics -./mcli artist stats - -# JSON output for monitoring -./mcli artist stats --raw +mcli artist list --library Storage --limit 100 +mcli artist search 'Miles Davis' --sort Albums --sort-dir desc +mcli artist find-duplicates --min-score 0.9 --json +mcli artist find-duplicates --source musicbrainz --limit 20 ``` -### Output - -``` - Artist Statistics -╭─────────────────────┬───────────┬───────╮ -│ Metric │ Count │ % │ -├─────────────────────┼───────────┼───────┤ -│ Total Artists │ 1,234 │ 100% │ -│ Total Albums │ 15,678 │ --- │ -│ Total Songs │ 187,234 │ --- │ -│ ─────────────────── │ ───────── │ ───── │ -│ Missing Images │ 23 │ 1.9% │ -│ No Albums │ 5 │ 0.4% │ -│ Locked │ 12 │ 1.0% │ -│ Ready to Process │ 8 │ 0.6% │ -╰─────────────────────┴───────────┴───────╯ - - Potential Duplicates -╭─────────────────────────────┬───────╮ -│ Normalized Name │ Count │ -├─────────────────────────────┼───────┤ -│ beatles │ 3 │ -│ prince │ 2 │ -│ queen │ 2 │ -╰─────────────────────────────┴───────╯ - -⚠ Found 3 artist names with potential duplicates -``` - -### Duplicate Detection - -The stats command detects potential duplicate artists by grouping them by their normalized name. Normalized names: -- Are lowercase -- Remove articles ("The", "A", "An") -- Remove special characters -- Collapse whitespace - -**Example duplicates:** -- "The Beatles" and "Beatles" both normalize to "beatles" -- "Prince" and "PRINCE" both normalize to "prince" - -### JSON Output - -```json -{ - "TotalArtists": 1234, - "LockedArtists": 12, - "ArtistsWithNoImages": 23, - "ArtistsWithNoAlbums": 5, - "ArtistsReadyToProcess": 8, - "TotalAlbums": 15678, - "TotalSongs": 187234, - "PotentialDuplicates": [ - { "Name": "beatles", "Count": 3 }, - { "Name": "prince", "Count": 2 }, - { "Name": "queen", "Count": 2 } - ] -} -``` - -### Use Cases - -- **Quality control:** Identify artists needing images or attention -- **Duplicate cleanup:** Find and merge duplicate artist entries -- **Monitoring:** Track library health over time -- **Reporting:** Generate statistics for dashboards - ---- - -## artist find-duplicates - -Find potential duplicate artists using advanced similarity detection algorithms. This command identifies artists that may be duplicates based on name similarity, name flipping (e.g., "John Smith" vs "Smith, John"), shared external IDs, overlapping albums, and more. - -### Usage - -```bash -mcli artist find-duplicates [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--merge` | | `false` | Automatically merge detected duplicates into the "Keep" artist | -| `-m`, `--min-score` | | `0.85` | Minimum similarity score (0.0-1.0). Higher values = stricter matching. | -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | -| `-y`, `--yes` | | `false` | Skip confirmation prompt when merging | - -### Similarity Score Guidelines - -| Score | Description | Typical Matches | -|-------|-------------|-----------------| -| 0.97+ | Very high confidence | Name flips ("Miles Davis" ↔ "Davis, Miles") | -| 0.92-0.96 | High confidence | Minor spelling variations, missing articles | -| 0.90-0.91 | Medium confidence | Similar names, abbreviations | -| 0.85-0.89 | Lower confidence | May include false positives | - -### Examples - -```bash -# Find duplicates with default threshold (0.85) -./mcli artist find-duplicates - -# Find only high-confidence duplicates (0.9+) -./mcli artist find-duplicates -m 0.9 - -# Find very high confidence duplicates only -./mcli artist find-duplicates -m 0.97 - -# Find duplicates and merge them (with confirmation) -./mcli artist find-duplicates -m 0.9 --merge - -# Find and merge without confirmation (USE WITH CAUTION) -./mcli artist find-duplicates -m 0.95 --merge -y - -# JSON output for scripting -./mcli artist find-duplicates -m 0.9 --raw -``` - -### Output - -``` -╭─────────────────┬───────┬────────────────────────┬───────┬────────────────────────────────────┬───────┬────────────┬─────────────────────────────────╮ -│ Group │ Score │ Keep (←) │ ID │ Merge (→) │ ID │ Shared IDs │ Reasons │ -├─────────────────┼───────┼────────────────────────┼───────┼────────────────────────────────────┼───────┼────────────┼─────────────────────────────────┤ -│ artist-dup-0029 │ 0.97 │ Miles Davis │ 3224 │ Davis, Miles │ 5981 │ - │ NameFlip │ -│ │ │ │ │ │ │ │ │ -│ artist-dup-0035 │ 0.97 │ George Benson │ 1686 │ Benson, George │ 6646 │ - │ NameFlip │ -│ │ │ │ │ │ │ │ │ -│ artist-dup-0003 │ 0.90 │ The Beatles │ 1220 │ Beatles │ 947 │ - │ ExactName │ -│ │ │ │ │ │ │ │ │ -│ artist-dup-0075 │ 0.90 │ Pixies │ 14228 │ The Pixies │ 14724 │ - │ ExactName, Albums, AlbumOverlap │ -╰─────────────────┴───────┴────────────────────────┴───────┴────────────────────────────────────┴───────┴────────────┴─────────────────────────────────╯ - -Found 125 duplicate group(s) with 126 pair(s) involving 251 artist(s) - High confidence (≥0.9): 125 -``` - -### Detection Reasons - -The command identifies duplicates using multiple detection methods: - -| Reason | Description | -|--------|-------------| -| `ExactName` | Names match exactly after normalization (removing "The", punctuation, etc.) | -| `NameFlip` | Names match when first/last are swapped ("John Smith" = "Smith, John") | -| `NameSim` | Names are similar based on Levenshtein distance | -| `SharedMusicBrainzId` | Artists share the same MusicBrainz ID | -| `SharedSpotifyId` | Artists share the same Spotify ID | -| `SharedDiscogsId` | Artists share the same Discogs ID | -| `Albums` | Artists have albums with identical names | -| `AlbumOverlap` | Artists have multiple overlapping album titles | - -### Merge Behavior - -When using `--merge`, the command will: - -1. **Keep the "Keep" artist** - The artist in the "Keep (←)" column is preserved -2. **Transfer all data** from merged artists: - - Albums are reassigned to the kept artist - - Songs are reassigned to the kept artist - - Contributor relationships are updated - - User ratings/stars are transferred - - Play counts are aggregated - - External IDs (MusicBrainz, Spotify, Discogs) are merged if not already present - - Alternate names are preserved for searchability -3. **Delete the merged artists** - Artists in the "Merge (→)" column are deleted - -### Merge Output - -``` -Are you sure you want to merge 125 duplicate group(s)? [y/n] (n): y - ✓ Merged 1 artist(s) into Miles Davis - ✓ Merged 1 artist(s) into George Benson - ✓ Merged 1 artist(s) into The Beatles - ✗ Error merging into Pixies: Duplicate album detected - -Merging artists... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% - -Merge complete: 122 succeeded, 3 failed - Log file can be found here /mnt/melodee/logs/mcli -``` - -### What Gets Merged - -When artist B is merged into artist A: - -| Data Type | Merge Behavior | -|-----------|----------------| -| **Albums** | All albums from B are reassigned to A | -| **Songs** | All songs from B are reassigned to A | -| **Contributors** | Artist contributor relationships are updated to reference A | -| **User Ratings** | Ratings for B's content are preserved under A | -| **User Stars** | Stars are transferred to A (avoids duplicates) | -| **Play History** | Play counts are aggregated | -| **External IDs** | MusicBrainz, Spotify, Discogs IDs from B are added to A if A doesn't have them | -| **Alternate Names** | B's name variants are added to A's alternate names for searchability | -| **Images** | A's images are kept; B's images are not transferred | -| **Metadata** | A's metadata (bio, etc.) takes precedence | - -### Safety Features - -- **Confirmation required**: By default, merge requires user confirmation -- **Locked artists skipped**: Artists marked as locked cannot be merged or deleted -- **Duplicate album handling**: If both artists have albums with identical titles, the merge handles this gracefully -- **Transaction safety**: Each merge is performed in a database transaction -- **Detailed logging**: All merge operations are logged to the log file for audit purposes -- **Error recovery**: Failed merges don't affect other merge operations - -### Troubleshooting Merge Failures - -Common reasons for merge failures: - -1. **Duplicate unique constraints**: Both artists may have albums that, when combined, violate uniqueness constraints -2. **Database locks**: Another process may be accessing the artist records -3. **Locked artists**: The target artist may be locked - -Check the log file for detailed error information: -```bash -tail -100 /mnt/melodee/logs/mcli/melodee-mcli-*.log -``` - -### Best Practices - -1. **Start with high threshold**: Begin with `-m 0.95` or higher to process obvious duplicates first -2. **Review before merging**: Run without `--merge` first to review the detected duplicates -3. **Backup database**: Before large merge operations, backup your database -4. **Process incrementally**: Merge in batches rather than all at once -5. **Check logs**: Review the log file after merge operations for any issues - -### JSON Output - -```json -{ - "Groups": [ - { - "GroupId": "artist-dup-0029", - "Score": 0.97, - "KeepArtist": { - "Id": 3224, - "Name": "Miles Davis", - "AlbumCount": 45, - "SongCount": 523 - }, - "MergeArtists": [ - { - "Id": 5981, - "Name": "Davis, Miles", - "AlbumCount": 2, - "SongCount": 24 - } - ], - "SharedIds": [], - "Reasons": ["NameFlip"] - } - ], - "Summary": { - "TotalGroups": 125, - "TotalPairs": 126, - "TotalArtists": 251, - "HighConfidence": 125 - } -} -``` - -### Workflow Example - -```bash -# Step 1: Review high-confidence duplicates -./mcli artist find-duplicates -m 0.95 - -# Step 2: If results look good, merge them -./mcli artist find-duplicates -m 0.95 --merge - -# Step 3: Review medium-confidence duplicates more carefully -./mcli artist find-duplicates -m 0.90 - -# Step 4: Manually review the output, then merge if appropriate -./mcli artist find-duplicates -m 0.90 --merge - -# Step 5: Check for any remaining duplicates at lower threshold -./mcli artist find-duplicates -m 0.85 -``` - ---- - -## artist delete - -Delete an artist and all associated albums from the database. - -### Usage - -```bash -mcli artist delete [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `ID` | Yes | Artist ID to delete | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--keep-files` | | `false` | Keep the artist directory on disk (do not delete files) | -| `--verbose` | | `false` | Output verbose debug and timing results | -| `-y`, `--yes` | | `false` | Skip confirmation prompt | - -### Examples - -```bash -# Delete artist (with confirmation, deletes files) -./mcli artist delete 42 - -# Delete artist but keep files on disk -./mcli artist delete 42 --keep-files - -# Delete without confirmation (scripting) -./mcli artist delete 42 -y - -# Delete from database only, no confirmation -./mcli artist delete 42 --keep-files -y -``` - -### Output - -``` -Artist: The Beatles -Albums: 13 -Songs: 227 -Directory: /mnt/music/library/The Beatles - -Delete artist 'The Beatles' and ALL files on disk? [y/n] (n): y - -✓ Artist 'The Beatles' deleted successfully. -``` - -### What Gets Deleted - -When deleting an artist: -1. **Database records:** - - Artist record - - All associated albums - - All associated songs - - All contributor associations - - User ratings and stars for this artist - -2. **Filesystem (unless `--keep-files`):** - - Artist directory - - All album subdirectories - - All music files and images - -### Safety Notes - -- ⚠️ **Locked artists cannot be deleted.** Unlock the artist first through the web interface. -- ⚠️ **Default behavior deletes files from disk.** Use `--keep-files` to preserve files. -- ⚠️ **This is a destructive operation.** The command shows details and requires confirmation by default. -- ✅ **Library statistics are updated** after deletion to maintain accurate counts. - -### Preserving Files - -When using `--keep-files`: -1. The command temporarily moves the artist directory to a backup location -2. Deletes the database records -3. Restores the directory to its original location -4. Files remain intact for potential re-import or manual handling - ---- - -## Workflow Examples - -### Finding and Cleaning Up Duplicates (Recommended) - -```bash -# 1. Find high-confidence duplicates first -./mcli artist find-duplicates -m 0.95 - -# 2. Review the output - check that "Keep" and "Merge" artists are correct - -# 3. Merge the high-confidence duplicates -./mcli artist find-duplicates -m 0.95 --merge - -# 4. Lower threshold and review medium-confidence matches -./mcli artist find-duplicates -m 0.90 - -# 5. If results look good, merge them -./mcli artist find-duplicates -m 0.90 --merge - -# 6. Check logs for any issues -tail -50 /mnt/melodee/logs/mcli/melodee-mcli-*.log -``` - -### Legacy: Finding Duplicates Manually - -```bash -# 1. Find potential duplicates using stats -./mcli artist stats | grep -A 20 "Potential Duplicates" - -# 2. Search for specific duplicate -./mcli artist search "beatles" - -# 3. Review each artist in web interface or use API to compare - -# 4. Delete the duplicate (keep files if unsure) -./mcli artist delete 789 --keep-files -``` - -### Batch Processing with Scripts - -```bash -#!/bin/bash -# Find artists without images and export for review - -./mcli artist stats --raw | \ - jq -r '.PotentialDuplicates[] | "\(.Name): \(.Count) entries"' > duplicates.txt - -echo "Potential duplicates saved to duplicates.txt" -``` - -### Integration with Monitoring - -```bash -#!/bin/bash -# Daily artist health check - -STATS=$(./mcli artist stats --raw) - -NO_IMAGES=$(echo "$STATS" | jq '.ArtistsWithNoImages') -DUPLICATES=$(echo "$STATS" | jq '.PotentialDuplicates | length') - -if [ "$NO_IMAGES" -gt 10 ]; then - echo "WARNING: $NO_IMAGES artists missing images" -fi - -if [ "$DUPLICATES" -gt 0 ]; then - echo "WARNING: $DUPLICATES potential duplicate artist names found" -fi -``` - ---- - -## See Also - -- [Album Commands](/cli/album/) - Album data management -- [Library Commands](/cli/library/) - Library operations -- [CLI Overview](/cli/) - Main CLI documentation +Run duplicate discovery without `--merge` first and inspect the suggested +primary artist. Merging, search deletion, and direct deletion can change or +remove records and files; create a backup before using them. diff --git a/docs/pages/cli/backup.md b/docs/pages/cli/backup.md new file mode 100644 index 000000000..c65ba3562 --- /dev/null +++ b/docs/pages/cli/backup.md @@ -0,0 +1,33 @@ +--- +title: CLI - Backup Export +description: Export Melodee settings and library definitions with mcli. +permalink: /cli/backup/ +tags: + - cli + - backup + - configuration +--- + +# Backup Export + +`backup export` writes database-backed settings and library definitions to +JSON. It does not back up PostgreSQL data or media files. + +```bash +mcli backup export [--output PATH] [--stdout] [--redact-secrets] [--raw] +``` + +Use `--redact-secrets` when the export will be attached to a support request or +stored somewhere less protected than a production backup: + +```bash +mcli backup export --output melodee-settings.json --redact-secrets +mcli backup export --stdout --redact-secrets > melodee-settings.json +``` + +Without `--redact-secrets`, settings whose keys contain `secret`, `token`, or +`password` can be written in clear text. Protect the resulting file accordingly. + +This command is useful for configuration auditing and recovery planning, but a +complete recovery set also needs a PostgreSQL dump and copies of each Docker +volume or host-mounted library. See [Backup and Restore](/backup/). diff --git a/docs/pages/cli/configuration.md b/docs/pages/cli/configuration.md index 21cf5a489..b67508149 100644 --- a/docs/pages/cli/configuration.md +++ b/docs/pages/cli/configuration.md @@ -1,326 +1,39 @@ --- title: CLI - Configuration Commands +description: List, read, set, and remove Melodee database-backed settings with mcli. permalink: /cli/configuration/ -layout: page +tags: + - cli + - configuration + - administration --- # Configuration Commands -The `configuration` branch provides commands for viewing and modifying Melodee configuration settings stored in the database. - -## Overview - -```bash -mcli configuration [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `get` | Get a specific configuration setting value | -| `list` | List all configuration settings | -| `set` | Modify a configuration setting | - ---- - -## configuration list - -Lists all configuration settings stored in the database. - -### Usage - -```bash -mcli configuration list [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-c`, `--category` | | | Filter settings by category prefix | -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Examples - -```bash -# List all settings -./mcli configuration list - -# Filter by category -./mcli configuration list --category imaging - -# JSON output -./mcli configuration list --raw -``` - -### Output - -``` -╭────────────────────────────────────────────┬─────────────────────────┬─────────────────────────────────────────╮ -│ Key │ Value │ Comment │ -├────────────────────────────────────────────┼─────────────────────────┼─────────────────────────────────────────┤ -│ imaging.smallSize │ 160 │ Small image dimension in pixels │ -│ imaging.mediumSize │ 320 │ Medium image dimension in pixels │ -│ imaging.largeSize │ 640 │ Large image dimension in pixels │ -│ imaging.minimumImageSize │ 300 │ Minimum accepted image size │ -│ jobs.libraryProcess.cronExpression │ 0 */5 * * * ? │ Cron expression for library processing │ -│ processing.doDeleteOriginalAfterConversion │ true │ Delete original files after conversion │ -╰────────────────────────────────────────────┴─────────────────────────┴─────────────────────────────────────────╯ - -Total: 156 settings -``` - -### Configuration Categories - -Settings are organized by category prefix: - -| Category | Description | -|----------|-------------| -| `imaging.*` | Image processing and sizing | -| `jobs.*` | Background job scheduling | -| `processing.*` | Media processing options | -| `conversion.*` | Audio conversion settings | -| `scripting.*` | Pre/post processing scripts | -| `validation.*` | Validation rules | - ---- - -## configuration get - -Get the value of a specific configuration setting. - -### Usage - -```bash -mcli configuration get [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `KEY` | Yes | The configuration key to retrieve | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--raw` | | `false` | Output only the value (no formatting) | -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Examples - -```bash -# Get a specific setting -./mcli configuration get imaging.smallSize - -# Get raw value for scripting -./mcli configuration get imaging.smallSize --raw - -# Use in shell scripts -SMALL_SIZE=$(./mcli configuration get imaging.smallSize --raw) -echo "Small image size: $SMALL_SIZE" -``` - -### Output - -``` -Key: imaging.smallSize -Value: 160 -Comment: Small image dimension in pixels -``` - -### Raw Output - -``` -160 -``` - ---- - -## configuration set - -Modify a configuration setting value. - -### Usage - -```bash -mcli configuration set [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `KEY` | Yes | The configuration key to modify | -| `VALUE` | Yes | The new value to set | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Examples - -```bash -# Set image size -./mcli configuration set imaging.smallSize 200 - -# Set cron expression (quote special characters) -./mcli configuration set jobs.libraryProcess.cronExpression "0 */10 * * * ?" - -# Set boolean value -./mcli configuration set processing.doDeleteOriginalAfterConversion false -``` - -### Output - -``` -✓ Setting updated successfully - -Key: imaging.smallSize -Old Value: 160 -New Value: 200 -``` - -### Value Types - -Configuration values are stored as strings but interpreted based on context: - -| Type | Examples | -|------|----------| -| Integer | `160`, `1024`, `0` | -| Boolean | `true`, `false` | -| String | `"value"`, `path/to/file` | -| Cron Expression | `0 */5 * * * ?` | - -### Common Settings - -**Image Processing:** - -```bash -# Set thumbnail sizes -./mcli configuration set imaging.smallSize 160 -./mcli configuration set imaging.mediumSize 320 -./mcli configuration set imaging.largeSize 640 - -# Set minimum image size requirement -./mcli configuration set imaging.minimumImageSize 300 -``` - -**Job Scheduling:** - -```bash -# Process library every 5 minutes -./mcli configuration set jobs.libraryProcess.cronExpression "0 */5 * * * ?" - -# Run inbound processing hourly -./mcli configuration set jobs.inboundProcess.cronExpression "0 0 * * * ?" - -# Disable a job (empty cron) -./mcli configuration set jobs.someJob.cronExpression "" -``` - -**Processing Options:** - -```bash -# Keep original files after conversion -./mcli configuration set processing.doDeleteOriginalAfterConversion false - -# Set maximum parallel processing -./mcli configuration set processing.maxParallelProcesses 4 -``` - ---- - -## Configuration Management Best Practices - -### Backup Before Changes - -```bash -# Export current configuration -./mcli configuration list --raw > config-backup-$(date +%Y%m%d).json -``` - -### Environment-Specific Settings - -Use different configuration for development and production: - -```bash -# Development - faster processing, smaller images -MELODEE_APPSETTINGS_PATH="/etc/melodee/appsettings.Development.json" \ - ./mcli configuration set imaging.smallSize 100 - -# Production - full quality -MELODEE_APPSETTINGS_PATH="/etc/melodee/appsettings.json" \ - ./mcli configuration set imaging.smallSize 160 -``` - -### Scripted Configuration - -```bash -#!/bin/bash -# Apply standard configuration settings - -settings=( - "imaging.smallSize:160" - "imaging.mediumSize:320" - "imaging.largeSize:640" - "processing.doDeleteOriginalAfterConversion:true" -) - -for setting in "${settings[@]}"; do - key="${setting%%:*}" - value="${setting#*:}" - ./mcli configuration set "$key" "$value" -done - -echo "Configuration applied successfully" -``` - ---- - -## Troubleshooting - -### Setting Not Found - -``` -Error: Setting 'invalid.key' not found -``` - -**Solution:** Use `configuration list` to see available settings: - ```bash -./mcli configuration list | grep -i "keyword" +mcli configuration [--verbose] COMMAND ``` -### Value Type Mismatch +| Command | Usage | +|---------|-------| +| `list` | `mcli configuration list [--filter PATTERN] [--raw]` | +| `get` | `mcli configuration get KEY [--raw]` | +| `set` | `mcli configuration set [KEY] [VALUE] [--remove]` | -``` -Error: Cannot convert 'abc' to integer for setting 'imaging.smallSize' -``` - -**Solution:** Ensure the value matches the expected type. Check the current value for guidance: +The `--filter` option supports wildcard patterns such as `imaging.*`. It is +named `--filter` (`-f`), not `--category`. ```bash -./mcli configuration get imaging.smallSize +mcli configuration list --filter 'jobs.*' +mcli configuration get streaming.maxConcurrentStreams.perUser --raw +mcli configuration set system.baseUrl https://music.example.com +mcli configuration set obsolete.custom.setting --remove ``` -### Changes Not Taking Effect - -Configuration changes require the Melodee service to reload. Either: - -1. Restart the Melodee service -2. Wait for automatic configuration refresh (if enabled) -3. Trigger a configuration reload through the web interface - ---- - -## See Also +Values are persisted as strings and interpreted by their consumers. Preserve +quotes around cron expressions, arrays, or values containing shell metacharacters. +An environment override takes precedence over the stored value. -- [Configuration Reference](/configuration-reference/) - Complete list of all settings -- [CLI Overview](/cli/) - Main CLI documentation -- [Jobs](/jobs/) - Background job configuration +Use `--remove` only for a custom or explicitly obsolete setting. Removing a +seeded setting may disable a feature or cause it to fall back unexpectedly. +Create a redacted [configuration export](/cli/backup/) before bulk changes. diff --git a/docs/pages/cli/doctor.md b/docs/pages/cli/doctor.md index 6e2132e50..343c5ad82 100644 --- a/docs/pages/cli/doctor.md +++ b/docs/pages/cli/doctor.md @@ -1,75 +1,36 @@ --- title: CLI - Doctor Command +description: Diagnose Melodee connections, paths, tools, secrets, and DecentDB compatibility with mcli doctor. permalink: /cli/doctor/ -layout: page +tags: + - cli + - diagnostics + - troubleshooting --- # Doctor Command -`mcli doctor` runs a set of diagnostics to confirm the CLI can load configuration, connect to databases, and access library paths. - -## Usage - -```bash -mcli doctor [OPTIONS] -``` - -## Options - -| Option | Default | Description | -|--------|---------|-------------| -| `--raw` | `false` | Output structured JSON instead of formatted tables | -| `--verbose` | `false` | Include extra diagnostic details | -| `--write-test` | `false` | Create+delete a temp file in each library directory to validate write access | - -## What It Checks - -1. **Configuration** - - Confirms configuration is loadable and shows where it came from (`MELODEE_APPSETTINGS_PATH` or local `appsettings*.json`) - - Validates required connection strings exist -2. **Database connectivity** - - Postgres (`DefaultConnection`) - - MusicBrainz DecentDB (`MusicBrainzConnection`) - - ArtistSearchEngine DecentDB (`ArtistSearchEngineConnection`) -3. **Library paths** - - Ensures each configured library path exists - - Optionally validates write access with `--write-test` - -## Examples - -### Basic health check - ```bash -./mcli doctor +mcli doctor [--raw] [--verbose] [--write-test] ``` -### Validate write permissions (recommended for deployments) +| Option | Purpose | +|--------|---------| +| `--raw` | Emit JSON suitable for inspection or automation | +| `--verbose` | Include detailed diagnostic and timing output | +| `--write-test` | Create and delete a temporary file in library directories | -```bash -./mcli doctor --write-test -``` - -### JSON output for automation +Doctor checks startup configuration, PostgreSQL and DecentDB connections, +expected search schemas, configured library paths, required executables, +security keys, and other installation prerequisites. The write test is +non-destructive but requires the same mounts and effective permissions as the +application. ```bash -./mcli doctor --raw | jq '.checks[] | select(.success==false)' -``` - -## Example Output - +mcli doctor +mcli doctor --write-test +mcli doctor --raw > doctor-result.json ``` -✓ Configuration -✓ Database: Postgres -✓ Database: MusicBrainz (DecentDB) -✓ Database: ArtistSearchEngine (DecentDB) -✓ Libraries - -All checks passed. -``` - -## Exit Codes -| Code | Meaning | -|------|---------| -| `0` | All checks passed | -| `1` | One or more checks failed | +DecentDB error 8 means a generated database uses an unsupported file format. +Follow [DecentDB Usage & Migration](/decentdb/) rather than deleting PostgreSQL. diff --git a/docs/pages/cli/file.md b/docs/pages/cli/file.md index f2ec2a590..6ba1e2069 100644 --- a/docs/pages/cli/file.md +++ b/docs/pages/cli/file.md @@ -1,143 +1,25 @@ --- title: CLI - File Commands +description: Inspect MPEG audio files with mcli. permalink: /cli/file/ -layout: page +tags: + - cli + - media + - diagnostics --- # File Commands -The `file` branch provides commands for analyzing and inspecting individual media files. - -## Overview - -```bash -mcli file [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `mpeg` | Analyze MPEG audio files and show detailed information | - ---- - -## file mpeg - -Analyzes an MPEG audio file and displays detailed technical information. This is primarily a **diagnostic tool** for troubleshooting media file issues. - -### Usage +`file mpeg` loads one file, displays its MPEG information, and reports whether +Melodee considers it a valid MPEG audio file. ```bash -mcli file mpeg [OPTIONS] +mcli file /music/example.mp3 mpeg ``` -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `FILENAME` | Yes | Path to the MPEG audio file to analyze | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Examples - -```bash -# Analyze an MP3 file -./mcli file mpeg "/path/to/song.mp3" - -# Check if a file is valid MPEG -./mcli file mpeg "/path/to/suspicious-file.mp3" -``` - -### Output - -``` -╭─────────────────────────────────────────────────────────────╮ -│ MPEG File Analysis │ -├─────────────────────────────────────────────────────────────┤ -│ File: /path/to/song.mp3 │ -│ Size: 8.4 MB │ -│ Valid MPEG: ✓ Yes │ -├─────────────────────────────────────────────────────────────┤ -│ Format Information │ -├─────────────────────────────────────────────────────────────┤ -│ MPEG Version: MPEG-1 Layer 3 │ -│ Bitrate: 320 kbps (CBR) │ -│ Sample Rate: 44100 Hz │ -│ Channels: Stereo │ -│ Duration: 3:42 │ -├─────────────────────────────────────────────────────────────┤ -│ Frame Analysis │ -├─────────────────────────────────────────────────────────────┤ -│ Total Frames: 8,532 │ -│ First Frame Offset: 2,048 bytes │ -│ Frame Consistency: 100% │ -╰─────────────────────────────────────────────────────────────╯ -``` - -### Information Displayed - -**File Properties:** -- File path and size -- MPEG validity check - -**Format Details:** -- MPEG version (MPEG-1, MPEG-2, MPEG-2.5) -- Layer (Layer I, II, or III) -- Bitrate (CBR or VBR average) -- Sample rate -- Channel mode (Mono, Stereo, Joint Stereo, Dual Channel) -- Duration - -**Frame Analysis:** -- Total frame count -- First frame offset (detects ID3v2 headers) -- Frame consistency percentage - -### Use Cases - -- **Troubleshooting:** Identify why a file won't play or import -- **Quality check:** Verify bitrate and encoding quality -- **Corruption detection:** Check for frame consistency issues -- **Format verification:** Confirm file is actually valid MPEG - -### Invalid File Output - -``` -╭─────────────────────────────────────────────────────────────╮ -│ MPEG File Analysis │ -├─────────────────────────────────────────────────────────────┤ -│ File: /path/to/file.mp3 │ -│ Size: 4.2 MB │ -│ Valid MPEG: ✗ No │ -├─────────────────────────────────────────────────────────────┤ -│ Issues Found: │ -│ - No valid MPEG frame header found │ -│ - File may be corrupted or not an MPEG audio file │ -╰─────────────────────────────────────────────────────────────╯ -``` - -### Scripting Example - -```bash -#!/bin/bash -# Check all MP3 files in a directory - -for file in *.mp3; do - echo "Checking: $file" - ./mcli file mpeg "$file" | grep "Valid MPEG" -done -``` - ---- - -## See Also +The CLI process must be able to read the supplied path. When running inside the +application container, use the container path, such as `/storage/...`, rather +than a host-only path. -- [Tags Commands](/cli/tags/) - View ID3 tags from media files -- [Parser Commands](/cli/parser/) - Parse metadata files -- [CLI Overview](/cli/) - Main CLI documentation +For metadata tags rather than MPEG frame information, use +[`mcli tags show`](/cli/tags/). diff --git a/docs/pages/cli/import.md b/docs/pages/cli/import.md index dba373074..69e3e4c7c 100644 --- a/docs/pages/cli/import.md +++ b/docs/pages/cli/import.md @@ -1,165 +1,31 @@ --- title: CLI - Import Commands +description: Import a user's favorite songs from a CSV file with mcli. permalink: /cli/import/ -layout: page +tags: + - cli + - import + - users --- # Import Commands -The `import` branch provides commands for importing data from external sources into Melodee. +`import user-favorite-songs` maps CSV columns to artists, albums, and songs, +then adds matching songs to a user's favorites. -## Overview - -```bash -mcli import [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `user-favorite-songs` | Import user favorite songs from a CSV file | - ---- - -## import user-favorite-songs - -Imports user favorite songs from a CSV file, marking them as starred in the database. - -### Usage - -```bash -mcli import user-favorite-songs [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `CSV_FILE` | Yes | Path to the CSV file containing favorite songs | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### CSV Format - -The CSV file should contain song information for matching. Supported columns: - -| Column | Required | Description | -|--------|----------|-------------| -| `artist` | Yes | Artist name | -| `album` | No | Album name (improves matching accuracy) | -| `title` | Yes | Song title | -| `track_number` | No | Track number on album | - -### Example CSV - -```csv -artist,album,title,track_number -The Beatles,Abbey Road,Come Together,1 -Pink Floyd,The Dark Side of the Moon,Time,4 -Led Zeppelin,IV,Stairway to Heaven,4 -Queen,A Night at the Opera,Bohemian Rhapsody,11 -``` - -### Examples - -```bash -# Import favorites from CSV -./mcli import user-favorite-songs "/path/to/favorites.csv" - -# Import with verbose output -./mcli import user-favorite-songs "/path/to/favorites.csv" --verbose -``` - -### Output - -``` -Importing user favorites from: /path/to/favorites.csv - -Processing 100 entries... -╭────────────────────┬───────╮ -│ Status │ Count │ -├────────────────────┼───────┤ -│ ✓ Matched & Starred│ 87 │ -│ ⚠ Not Found │ 10 │ -│ ⚠ Multiple Matches │ 3 │ -╰────────────────────┴───────╯ - -Import complete: 87 songs starred -``` - -### Matching Logic - -The import process attempts to match songs using: - -1. **Exact match:** Artist name + Song title + Album name (if provided) -2. **Normalized match:** Case-insensitive, ignoring special characters -3. **Fuzzy match:** Allows for slight variations in naming - -### Not Found Report - -When songs can't be matched, they're reported: - -``` -Songs not found: -- Artist: "The Beetles", Title: "Help!" (possible typo: "The Beatles") -- Artist: "Unknown Artist", Title: "Unknown Song" -``` - -### Multiple Match Handling - -When multiple songs match: - -``` -Multiple matches found: -- Artist: "Queen", Title: "We Will Rock You" - → 3 versions found (different albums) - → First match used -``` - -### Use Cases - -- **Migration:** Import favorites from another music player -- **Backup restore:** Restore starred songs from export -- **Bulk starring:** Mark many songs as favorites at once - -### Creating Export Files - -From other services: - -**Spotify:** Use third-party tools to export liked songs to CSV - -**iTunes/Apple Music:** ```bash -# Export playlist to CSV using AppleScript or third-party tools +mcli import favorites.csv user-favorite-songs \ + USER_API_KEY ArtistColumn AlbumColumn SongColumn --pretend ``` -**Last.fm:** -```bash -# Use Last.fm API to export loved tracks -``` - -### Scripting Example - -```bash -#!/bin/bash -# Import favorites for multiple users - -for user_file in /backups/favorites/*.csv; do - username=$(basename "$user_file" .csv) - echo "Importing favorites for: $username" - ./mcli import user-favorite-songs "$user_file" -done -``` - ---- +The positional values are: -## See Also +1. CSV filename +2. User API key (GUID) +3. Artist-name column +4. Album-name column +5. Song-name column -- [CLI Overview](/cli/) - Main CLI documentation -- [Artist Commands](/cli/artist/) - Artist data management -- [Album Commands](/cli/album/) - Album data management +Run with `--pretend` first. It performs matching and reports what would happen +without changing favorites. Column names must match the CSV header, and the CLI +must have local access to PostgreSQL and the CSV file. diff --git a/docs/pages/cli/job.md b/docs/pages/cli/job.md index df31ed724..3a3d3d2a0 100644 --- a/docs/pages/cli/job.md +++ b/docs/pages/cli/job.md @@ -1,412 +1,36 @@ --- title: CLI - Job Commands +description: List and run Melodee background jobs synchronously with mcli. permalink: /cli/job/ -layout: page +tags: + - cli + - jobs + - administration --- # Job Commands -The `job` branch provides commands for viewing and running background maintenance jobs. +The `job` group runs background work synchronously in the current CLI process. +It is local-only and uses the same databases, paths, and settings as the server. -## Overview +| Command | Purpose | +|---------|---------| +| `job list [--raw]` | Show known jobs, execution history, and statistics | +| `job run --job NAME` | Run a registered job by class name | +| `job artistsearchengine-refresh` | Refresh the artist search database | +| `job musicbrainz-update` | Download MusicBrainz data and rebuild its local database | ```bash -mcli job [COMMAND] [OPTIONS] +mcli job list +mcli job run --job ChartUpdateJob +mcli job --batchsize 500 artistsearchengine-refresh +mcli job musicbrainz-update ``` -**Available Commands:** +`--batchsize` (`-b`) and `--verbose` are options on the `job` command and must +appear before its subcommand. The batch-size override applies only to commands +that consume it. -| Command | Description | -|---------|-------------| -| `artistsearchengine-refresh` | Run artist search engine refresh job | -| `list` | List all known background jobs with their execution history and statistics | -| `musicbrainz-update` | Run MusicBrainz database update job | -| `run` | Run a specific background job by name | - ---- - -## job list - -Lists all known background jobs with their execution history and statistics. - -### Usage - -```bash -mcli job list [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Examples - -```bash -# List all jobs with statistics -./mcli job list - -# JSON output for monitoring -./mcli job list --raw -``` - -### Output - -``` -╭──────┬────────────┬────────────────┬─────────────╮ -│ Jobs │ Total Runs │ Failures (24h) │ Active Jobs │ -├──────┼────────────┼────────────────┼─────────────┤ -│ 9 │ 156 │ 0 │ 4 │ -╰──────┴────────────┴────────────────┴─────────────╯ - -╭─────────────────────────────────────────┬──────┬─────────┬──────────┬─────────────────┬─────────────────┬─────────╮ -│ Job Name │ Runs │ Success │ Avg Time │ Last Run │ Next Run * │ Status │ -├─────────────────────────────────────────┼──────┼─────────┼──────────┼─────────────────┼─────────────────┼─────────┤ -│ ArtistHousekeepingJob │ 45 │ 100% │ 92ms │ 20241230T130000 │ 20241230T140000 │ ✓ OK │ -│ ArtistSearchEngineHousekeepingJob │ 12 │ 100% │ 1.2sec │ 20241230T120000 │ 20241231T000000 │ ✓ OK │ -│ ChartUpdateJob │ 8 │ 100% │ 450ms │ 20241230T060000 │ 20241231T060000 │ ✓ OK │ -│ LibraryInboundProcessJob │ 56 │ 98% │ 57ms │ 20241230T130500 │ 20241230T131000 │ ✓ OK │ -│ LibraryProcessJob │ 23 │ 100% │ 2.3sec │ 20241230T130000 │ 20241230T140000 │ ✓ OK │ -│ MusicBrainzUpdateDatabaseJob │ 1 │ 100% │ 45.2min │ 20241228T020000 │ 20250104T020000 │ ✓ OK │ -│ NowPlayingCleanupJob │ 11 │ 100% │ 12ms │ 20241230T130000 │ 20241230T130500 │ ✓ OK │ -│ StagingAlbumRevalidationJob │ 2 │ 100% │ 3.4s │ 20241230T020000 │ 20250106T020000 │ ✓ OK │ -│ StagingAutoMoveJob │ 15 │ 100% │ 890ms │ 20241230T130000 │ 20241230T130500 │ ✓ OK │ -╰─────────────────────────────────────────┴──────┴─────────┴──────────┴─────────────────┴─────────────────┴─────────╯ - -* Next Run times are approximate, calculated from cron expressions -``` - -### Summary Statistics - -| Metric | Description | -|--------|-------------| -| Jobs | Total number of configured jobs | -| Total Runs | Sum of all job executions | -| Failures (24h) | Jobs that failed in the last 24 hours | -| Active Jobs | Jobs that have run at least once | - -### Job Status Indicators - -| Status | Description | -|--------|-------------| -| ✓ OK | Job last ran successfully | -| ✗ Failed | Job last run failed | -| No runs | Job has never been executed | - -### JSON Output - -```json -[ - { - "JobName": "ArtistHousekeepingJob", - "RunCount": 45, - "SuccessRate": 100, - "AverageRunTimeMs": 92, - "LastRunAt": "2024-12-30T13:00:00Z", - "LastRunStatus": "Success" - } -] -``` - ---- - -## job run - -Run a specific background job by name. - -### Usage - -```bash -mcli job run -j [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-j`, `--job-name` | | **Required** | Name of the job to run | -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Available Jobs - -| Job Name | Description | -|----------|-------------| -| `ArtistHousekeepingJob` | Cleans up artist data and removes orphaned records | -| `ArtistSearchEngineHousekeepingJob` | Updates artist search engine database | -| `ChartUpdateJob` | Updates music charts | -| `LibraryInboundProcessJob` | Processes files in the inbound library | -| `LibraryProcessJob` | General library processing | -| `MusicBrainzUpdateDatabaseJob` | Downloads and updates MusicBrainz database | -| `NowPlayingCleanupJob` | Cleans up old now-playing records | -| `StagingAlbumRevalidationJob` | Re-validates albums with invalid artists | -| `StagingAutoMoveJob` | Automatically moves OK albums from staging | - -### Examples - -```bash -# Run artist housekeeping -./mcli job run -j ArtistHousekeepingJob - -# Run library processing -./mcli job run -j LibraryProcessJob - -# Run MusicBrainz update (long-running) -./mcli job run -j MusicBrainzUpdateDatabaseJob -``` - -### Output - -``` -Starting job: ArtistHousekeepingJob -✓ Job completed successfully: ArtistHousekeepingJob -``` - -### Error Output - -``` -Starting job: InvalidJobName -✗ Job 'InvalidJobName' not found. Available jobs: - - ArtistHousekeepingJob - - ArtistSearchEngineHousekeepingJob - - ChartUpdateJob - - LibraryInboundProcessJob - - LibraryProcessJob - - MusicBrainzUpdateDatabaseJob - - NowPlayingCleanupJob - - StagingAutoMoveJob -``` - ---- - -## job artistsearchengine-refresh - -Refreshes the artist search engine database by updating local data from external search engines. - -### Usage - -```bash -mcli job artistsearchengine-refresh [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### What It Does - -1. Queries configured search engines for artist information -2. Updates local artist database with new albums and metadata -3. Refreshes artist images and biographical data -4. Records job execution history - -### Example - -```bash -./mcli job artistsearchengine-refresh -``` - -### Output - -``` -Starting Artist Search Engine Refresh... -Processing: 1,234 artists -✓ Updated: 156 artists with new data -✓ Completed in 2m 34s -``` - ---- - -## job musicbrainz-update - -Downloads and updates the local MusicBrainz database used for metadata enrichment during scanning. - -### Usage - -```bash -mcli job musicbrainz-update [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### What It Does - -1. Downloads the latest MusicBrainz data dump -2. Extracts and processes the data -3. Creates/updates the local SQLite database -4. Enables offline metadata lookups - -### Requirements - -- Sufficient disk space (several GB for the dump) -- Network access to MusicBrainz servers -- May take 30+ minutes depending on connection speed - -### Example - -```bash -./mcli job musicbrainz-update -``` - -### Output - -``` -Starting MusicBrainz Database Update... -Downloading data dump... 2.3 GB / 2.3 GB (100%) -Extracting archive... Done -Processing data... - - Artists: 1,234,567 - - Albums: 2,345,678 - - Recordings: 12,345,678 -✓ Database updated successfully -✓ Completed in 45m 12s -``` - ---- - -## Scheduling Jobs - -Jobs are normally scheduled automatically using cron expressions. The CLI commands allow manual execution for: - -- Testing job functionality -- Running jobs outside their normal schedule -- Recovering from failed scheduled runs -- Initial setup and verification - -### View Job Schedule - -```bash -# Get cron expression for a job -./mcli configuration get jobs.libraryProcess.cronExpression -``` - -### Modify Job Schedule - -```bash -# Run every 5 minutes -./mcli configuration set jobs.libraryProcess.cronExpression "0 */5 * * * ?" - -# Run every hour -./mcli configuration set jobs.libraryProcess.cronExpression "0 0 * * * ?" - -# Run at 2 AM daily -./mcli configuration set jobs.libraryProcess.cronExpression "0 0 2 * * ?" - -# Disable job (empty expression) -./mcli configuration set jobs.someJob.cronExpression "" -``` - ---- - -## Scripting and Automation - -### Daily Maintenance Script - -```bash -#!/bin/bash -# Daily maintenance jobs - -export MELODEE_APPSETTINGS_PATH="/etc/melodee/appsettings.json" - -echo "Running daily maintenance..." - -# Housekeeping -./mcli job run -j ArtistHousekeepingJob -./mcli job run -j NowPlayingCleanupJob - -# Process any pending items -./mcli job run -j LibraryInboundProcessJob -./mcli job run -j StagingAutoMoveJob - -echo "Daily maintenance complete" -``` - -### Monitoring Job Status - -```bash -#!/bin/bash -# Check for job failures - -FAILURES=$(./mcli job list --raw | jq '[.[] | select(.LastRunStatus == "Failed")] | length') - -if [ "$FAILURES" -gt 0 ]; then - echo "WARNING: $FAILURES job(s) failed" - ./mcli job list --raw | jq '.[] | select(.LastRunStatus == "Failed") | .JobName' - exit 1 -fi - -echo "All jobs healthy" -exit 0 -``` - -### Weekly MusicBrainz Update - -```bash -#!/bin/bash -# Weekly MusicBrainz database refresh (run Sunday at 2 AM) - -export MELODEE_APPSETTINGS_PATH="/etc/melodee/appsettings.json" - -echo "Starting weekly MusicBrainz update..." -./mcli job musicbrainz-update - -if [ $? -eq 0 ]; then - echo "MusicBrainz update completed successfully" -else - echo "MusicBrainz update failed" >&2 - exit 1 -fi -``` - ---- - -## Troubleshooting - -### Job Not Found - -``` -Error: Job 'InvalidName' not found -``` - -**Solution:** Check exact job name with `job list`: - -```bash -./mcli job list | grep -i "keyword" -``` - -### Job Fails Immediately - -**Possible causes:** -1. Database connection issues -2. Missing configuration -3. Insufficient permissions - -**Solution:** Check logs and try with verbose output: - -```bash -./mcli job run -j JobName --verbose -``` - -### Job Runs But No Effect - -**Possible causes:** -1. No data to process -2. Job skipped due to recent run -3. Configuration preventing action - -**Solution:** Check job conditions and force run if needed. - ---- - -## See Also - -- [Background Jobs](/jobs/) - Detailed job documentation -- [Configuration Commands](/cli/configuration/) - Modify job schedules -- [CLI Overview](/cli/) - Main CLI documentation +An ad hoc run can overlap a Quartz-scheduled run. Check the Jobs administration +page before starting resource-intensive work, and see [Background Jobs](/jobs/) +for configured schedules and dependencies. diff --git a/docs/pages/cli/library.md b/docs/pages/cli/library.md index bcff5ff53..0c467f1bc 100644 --- a/docs/pages/cli/library.md +++ b/docs/pages/cli/library.md @@ -1,863 +1,65 @@ --- title: CLI - Library Commands +description: Process, scan, inspect, repair, and purge Melodee libraries with mcli. permalink: /cli/library/ -layout: page +tags: + - cli + - libraries + - administration --- # Library Commands -The `library` branch provides commands for managing music libraries, processing media, and maintaining metadata. +Library commands operate directly on local databases and mounted media paths. +Use the same effective user and mounts as the server. -## Overview +| Command | Important options | +|---------|-------------------| +| `album-report` | `--library`, `--full`, `--raw` | +| `clean` | `--library` | +| `find-duplicate-dirs` | `--library`, `--artist`, `--limit`, `--search`, `--merge`, `--json` | +| `list` | `--library`, `--raw` | +| `move-ok` | `--library`, `--to-library`, or path mode with `--from-path` and `--to-path` | +| `process` | `--library`, `--copy`, `--force`, `--limit`, `--pre-script`, or path mode with `--inbound` and `--staging` | +| `purge` | `--library` | +| `rebuild [only-path]` | `--library`, `--only-missing`, `--skip-images`, `--limit` | +| `scan` | `--force`, `--json`, `--silent`, `--verbose` | +| `stats` | `--library`, `--borked`, `--raw` | +| `validate` | `--library`, `--fix`, `--json` | -```bash -mcli library [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Alias | Description | -|---------|-------|-------------| -| `album-report` | `ar` | Show report of albums found in library | -| `clean` | `c` | Clean library and delete folders without media files | -| `find-duplicate-dirs` | `fdd` | Find duplicate album directories and resolve using metadata | -| `list` | `ls` | List all libraries with their details | -| `move-ok` | `m` | Move 'Ok' status albums to another library | -| `process` | `p` | Process media from inbound to staging (step 1 only) | -| `purge` | | Purge library data from database | -| `rebuild` | `r` | Rebuild melodee metadata albums in library | -| `scan` | `s` | **Full scan workflow** - process inbound → staging → storage → database | -| `stats` | `ss` | Show statistics for a specific library | -| `validate` | `v` | Validate library integrity (DB vs disk consistency) | - ---- - -## library list - -Lists all libraries configured in the database with their details. - -### Usage - -```bash -mcli library list [OPTIONS] -``` - -### Options - -| Option | Default | Description | -|--------|---------|-------------| -| `--raw` | `false` | Output in JSON format for scripting | -| `--verbose` | `false` | Include verbose debug output | - -### Examples - -```bash -# List all libraries -./mcli library list - -# JSON output for scripting -./mcli library list --raw -``` - -### Output - -``` -╭──────────┬─────────┬──────────────────────────────┬─────────┬────────┬───────┬─────────────────┬───────────╮ -│ Name │ Type │ Path │ Artists │ Albums │ Songs │ Last Scan │ Status │ -├──────────┼─────────┼──────────────────────────────┼─────────┼────────┼───────┼─────────────────┼───────────┤ -│ Inbound │ Inbound │ /mnt/music/inbound │ 0 │ 0 │ 0 │ N/A │ ✓ OK │ -│ Staging │ Staging │ /mnt/music/staging │ 42 │ 156 │ 1892 │ 20241230T130000 │ ⚠ Needs │ -│ Storage │ Storage │ /mnt/music/library │ 1,234 │ 15,678 │187234 │ 20241230T130500 │ ✓ OK │ -╰──────────┴─────────┴──────────────────────────────┴─────────┴────────┴───────┴─────────────────┴───────────╯ - -Total libraries: 3 -⚠ 1 library(ies) need scanning -``` - -### Library Types - -| Type | Description | Scanning | -|------|-------------|----------| -| Inbound | New files to be processed | N/A | -| Staging | Processed files awaiting review | Yes | -| Storage | Final destination for approved albums | Yes | -| UserImages | User-uploaded images | N/A | -| Playlist | Playlist files | N/A | - ---- - -## library stats - -Shows detailed statistics for a specific library. - -### Usage - -```bash -mcli library stats --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--borked` | | `false` | Show only issues, skip informational stats | -| `--library` | `-l` | **Required** | Name of the library to analyze | -| `--raw` | | `false` | Output in JSON format | -| `--verbose` | | `false` | Include verbose debug output | - -### Examples - -```bash -# Show all statistics -./mcli library stats --library "Storage" - -# Show only problems -./mcli library stats -l "Staging" --borked - -# JSON output -./mcli library stats -l "Storage" --raw -``` - -### Output - -``` -╭─────────────────────────────────────────────╮ -│ Library: Storage │ -│ Path: /mnt/music/library │ -│ Type: Storage │ -╰─────────────────────────────────────────────╯ - - Artist Statistics -╭─────────────────────┬───────────┬───────╮ -│ Metric │ Count │ % │ -├─────────────────────┼───────────┼───────┤ -│ Total Artists │ 1,234 │ 100% │ -│ With Images │ 1,200 │ 97.2% │ -│ Locked │ 12 │ 1.0% │ -╰─────────────────────┴───────────┴───────╯ - - Album Statistics -╭─────────────────────┬───────────┬───────╮ -│ Metric │ Count │ % │ -├─────────────────────┼───────────┼───────┤ -│ Total Albums │ 15,678 │ 100% │ -│ Ok Status │ 15,500 │ 98.9% │ -│ Needs Attention │ 120 │ 0.8% │ -│ Missing Images │ 58 │ 0.4% │ -╰─────────────────────┴───────────┴───────╯ -``` - ---- - -## library album-report - -Generates a report of albums grouped by status. - -### Usage - -```bash -mcli library album-report --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--full` | | `false` | Show detailed list instead of summary | -| `--library` | `-l` | **Required** | Name of the library | -| `--raw` | | `false` | Output in JSON format | -| `--verbose` | | `false` | Include verbose debug output | - -### Examples - -```bash -# Summary report -./mcli library album-report --library "Staging" - -# Full detailed report -./mcli library album-report -l "Staging" --full -``` - -### Summary Output - -``` -╭────────────────────┬───────╮ -│ Status │ Count │ -├────────────────────┼───────┤ -│ ✓ Ok │ 1,204 │ -│ ⚠ Ok (Invalid) │ 3 │ -│ ✗ HasNoImages │ 15 │ -│ ✗ HasNoTracks │ 2 │ -╰────────────────────┴───────╯ - -Use --full to see detailed list. -``` - ---- - -## library clean - -⚠️ **DESTRUCTIVE OPERATION** - -Removes directories without media files from a library. +## Common Workflows -### Usage +Run the complete Inbound to Staging to Storage to PostgreSQL workflow: ```bash -mcli library clean --library [OPTIONS] +mcli library scan +mcli library scan --json ``` -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--library` | `-l` | **Required** | Name of the library to clean | -| `--verbose` | | `false` | Include verbose debug output | - -### What It Does - -1. Scans library for all directories -2. Identifies directories without media files -3. Preserves image-only directories if parent has media -4. **Permanently deletes** empty/non-media directories - -### Safety - -- ❌ Won't run on locked libraries -- ✅ Safe for Staging and Storage libraries -- ⚠️ Use caution with Inbound libraries - -### Example +Inspect integrity before allowing removal of orphaned database records: ```bash -./mcli library clean --library "Staging" +mcli library validate --library Storage --json +mcli library validate --library Storage --fix ``` ---- - -## library rebuild - -Regenerates Melodee metadata files by reading music files. - -### Usage +Review duplicate directories before merging them: ```bash -mcli library rebuild --library [PATH] [OPTIONS] +mcli library find-duplicate-dirs --library Storage --search --json +mcli library find-duplicate-dirs --library Storage --search --merge ``` -### Arguments - -| Argument | Description | -|----------|-------------| -| `[PATH]` | Optional. Rebuild only this specific album path | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--library` | `-l` | **Required** | Name of the library | -| `--limit` | `-n` | | Maximum number of albums to process and then quit | -| `--only-missing` | | `true` | Only create missing metadata files | -| `--skip-images` | | `false` | Skip searching for, downloading and processing images. Existing images are kept unless invalid. | -| `--verbose` | | `false` | Include verbose debug output | - -### Examples - -```bash -# Create only missing metadata -./mcli library rebuild --library "Storage" - -# Full rebuild (recreate all) -./mcli library rebuild -l "Storage" --only-missing false - -# Rebuild specific album -./mcli library rebuild -l "Storage" "The Beatles/Abbey Road" - -# Rebuild without processing images (faster, keeps existing images) -./mcli library rebuild -l "Storage" --skip-images - -# Process only first 100 albums -./mcli library rebuild -l "Storage" --limit 100 -``` - -### Safety - -- ✅ Non-destructive: Only creates/updates metadata files -- ❌ Won't run on Inbound or locked libraries - ---- - -## library scan - -Performs a **full library scan workflow** - the complete media ingestion pipeline. - -### Usage - -```bash -mcli library scan [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--force` | | `false` | Force processing even if recently scanned | -| `--json` | | `false` | Output results as JSON (implies --silent for progress) | -| `--silent` | | `false` | Suppress all output (silent mode) | -| `--verbose` | | `false` | Include verbose debug output | - -### What It Does - -This command orchestrates the **entire media ingestion pipeline** in sequence: - -| Step | Job | Description | -|------|-----|-------------| -| 1 | LibraryInboundProcessJob | Processes raw files from inbound → staging | -| 2 | StagingAlbumRevalidationJob | Re-validates albums with invalid artists | -| 3 | StagingAutoMoveJob | Moves approved albums from staging → storage | -| 4 | LibraryInsertJob | Inserts albums from storage into database | - -This is the **recommended way to add new music** - a single command that handles everything. +`--merge` requires `--search`. The older `--delete` option is deprecated and is +now an alias for merge behavior. -### Examples - -```bash -# Standard scan - process everything end-to-end -./mcli library scan - -# Force full reprocessing -./mcli library scan --force - -# Silent mode for cron jobs (no output, exit code only) -./mcli library scan --silent - -# JSON output for scripting and automation -./mcli library scan --json - -# Check exit code in scripts -./mcli library scan --silent && echo "Scan successful" || echo "Scan failed" -``` - -### JSON Output Example - -When using `--json`, the command outputs structured JSON: - -```json -{ - "success": true, - "durationSeconds": 145.23, - "duration": "00:02:25", - "steps": [ - { "name": "Processing inbound files", "success": true, "durationSeconds": 83.5 }, - { "name": "Revalidating staging albums", "success": true, "durationSeconds": 5.1 }, - { "name": "Moving approved albums to storage", "success": true, "durationSeconds": 12.3 }, - { "name": "Inserting albums into database", "success": true, "durationSeconds": 44.3 } - ], - "summary": { - "inboundProcessing": { "newArtists": 5, "newAlbums": 12, "newSongs": 145 }, - "stagingRevalidation": { "albumsRevalidated": 3, "albumsNowValid": 2 }, - "storageTransfer": { "albumsMoved": 14 }, - "databaseInsert": { "artistsInserted": 5, "albumsInserted": 14, "songsInserted": 168 } - }, - "errors": [] -} -``` - -### Interactive Output - -``` -╭─────────────────────────────────────╮ -│ Library Scan Configuration │ -├─────────────────────────────────────┤ -│ Force Mode No │ -│ Verbose No │ -╰─────────────────────────────────────╯ - -✓ Processing inbound files (01:23) -✓ Revalidating staging albums (00:05) -✓ Moving approved albums to storage (00:12) -✓ Inserting albums into database (00:45) - -── Library scan completed in 00:02:25 ── - -╭─────────────────────────────────────╮ -│ Scan Summary │ -├─────────────────────────────────────┤ -│ Inbound Processing │ -│ New artists discovered 5 │ -│ New albums discovered 12 │ -│ New songs discovered 145 │ -├─────────────────────────────────────┤ -│ Database Insert │ -│ Artists inserted 5 │ -│ Albums inserted 14 │ -│ Songs inserted 168 │ -╰─────────────────────────────────────╯ -``` - ---- - -## library process - -Processes media from Inbound to Staging library. - -### Usage - -```bash -mcli library process --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--copy` | | `true` | Copy files instead of moving | -| `--force` | | `true` | Override existing metadata files | -| `--inbound` | | | Inbound path (path-based mode) | -| `--library` | `-l` | **Required** | Name of the inbound library | -| `--limit` | | | Maximum albums to process | -| `--pre-script` | | | Script to run before processing | -| `--staging` | | | Staging path (path-based mode) | -| `--verbose` | | `false` | Include verbose debug output | - -### What It Does - -1. Scans inbound library for media files -2. Parses ID3/Vorbis tags -3. Extracts album artwork -4. Creates `melodee.json` metadata files -5. Moves/copies to staging - -### Examples - -```bash -# Process all from Inbound -./mcli library process --library "Inbound" - -# Process 10 albums -./mcli library process -l "Inbound" --limit 10 - -# Path-based mode -./mcli library process --inbound "/mnt/incoming" --staging "/mnt/staging" -``` - ---- - -## library move-ok - -Moves albums with "Ok" status between libraries. - -### Usage - -```bash -mcli library move-ok --library --to-library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--from-path` | | | Source path (path-based mode) | -| `--library` | `-l` | **Required** | Source library name | -| `--to-library` | | **Required** | Destination library name | -| `--to-path` | | | Destination path (path-based mode) | -| `--verbose` | | `false` | Include verbose debug output | - -### Examples - -```bash -# Move from Staging to Storage -./mcli library move-ok --library "Staging" --to-library "Storage" - -# Path-based mode -./mcli library move-ok --from-path "/mnt/staging" --to-path "/mnt/library" -``` - ---- - -## library purge - -⚠️ **DESTRUCTIVE OPERATION** - -Purges all data for a library from the database. - -### Usage - -```bash -mcli library purge --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--force` | | `false` | Ignore last scan timestamp | -| `--library` | `-l` | **Required** | Name of the library to purge | -| `--verbose` | | `false` | Include verbose debug output | - -### What Gets Deleted - -- All artist records for this library -- All album records -- All song records -- Library statistics - -### What Is NOT Deleted - -- Filesystem files (only database records) -- The library configuration itself - -### Safety - -- ❌ Won't run on locked libraries -- Confirmation required - -### Example - -```bash -./mcli library purge --library "Staging" -``` - ---- - -## library validate - -Validates library integrity by checking bidirectional consistency between database records and files on disk. - -### Usage - -```bash -mcli library validate --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--fix` | | `false` | Remove orphaned database records | -| `--json` | | `false` | Output results as JSON | -| `--library` | `-l` | **Required** | Name of the storage library to validate | -| `--verbose` | | `false` | Include verbose debug output | - -### What It Validates - -1. **Database → Disk**: Checks that all artists, albums, and songs in the database have corresponding files/directories on disk -2. **Disk → Database**: Checks that all album directories with media files on disk are registered in the database - -### Issues Detected - -| Issue Type | Description | Fix Option | -|------------|-------------|------------| -| Orphaned Artists | Artist in DB but directory missing on disk | `--fix` removes from DB | -| Orphaned Albums | Album in DB but directory missing on disk | `--fix` removes from DB | -| Missing Songs | Song in DB but file missing on disk | `--fix` removes from DB | -| Unregistered Directories | Album directory on disk not in DB | Run `library scan` to add | - -### Examples - -```bash -# Basic validation - check for issues -./mcli library validate --library "Storage" - -# JSON output for scripting -./mcli library validate -l "Storage" --json - -# Auto-fix orphaned DB records -./mcli library validate -l "Storage" --fix - -# Pipe JSON to jq for processing -./mcli library validate -l "Storage" --json | jq '.issues.orphanedAlbums | length' -``` - -### Example Output - -``` -╭───────────────────────────────────╮ -│ Library Validation │ -├───────────────────────────────────┤ -│ Library Storage │ -│ Path /mnt/music/library │ -│ Fix Mode No │ -╰───────────────────────────────────╯ - -✓ Validating database records against disk -✓ Validating disk directories against database - -── Validation completed in 00:02.345 ── - -╭─────────────────┬─────────┬────────╮ -│ Category │ Checked │ Issues │ -├─────────────────┼─────────┼────────┤ -│ Artists │ 1,234 │ 0 │ -│ Albums │ 15,678 │ 3 │ -│ Songs │ 187,234 │ 12 │ -│ Disk Dirs │ 15,700 │ 5 │ -╰─────────────────┴─────────┴────────╯ - -✗ Library validation failed - 20 issue(s) found. -Tip: Use --fix to remove orphaned database records. -Tip: Run 'mcli library scan' to add unregistered directories. -``` - -### JSON Output Example - -```json -{ - "success": false, - "libraryName": "Storage", - "libraryPath": "/mnt/music/library", - "durationSeconds": 2.345, - "summary": { - "artistsChecked": 1234, - "albumsChecked": 15678, - "songsChecked": 187234, - "directoriesScanned": 15700 - }, - "issues": { - "orphanedArtists": [], - "orphanedAlbums": [ - { "id": 123, "name": "Missing Album", "directory": "M/Mi/Missing Artist/Missing Album/" } - ], - "missingSongs": [ - { "id": 456, "title": "Lost Track", "fileName": "01 - Lost Track.mp3", "albumName": "Some Album" } - ], - "unregisteredDirectories": [ - "/mnt/music/library/N/Ne/New Artist/New Album" - ] - }, - "fixed_": null -} -``` - -### Safety - -- ✅ Read-only by default (only reports issues) -- ⚠️ `--fix` modifies database (removes orphaned records) -- ❌ Only works with Storage libraries - ---- - -## library find-duplicate-dirs - -Finds duplicate album directories (same album name, different years) and optionally resolves them using metadata searches from MusicBrainz, Spotify, and other sources. - -### Usage - -```bash -mcli library find-duplicate-dirs --library [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--artist` | `-a` | | Filter to specific artist name (partial match) | -| `--delete` | | `false` | Delete incorrect duplicates (requires `--search`) | -| `--json` | | `false` | Output results as JSON | -| `--library` | `-l` | **Required** | Name of the storage library to scan | -| `--limit` | `-n` | | Maximum number of duplicate groups to process | -| `--search` | | `false` | Search metadata sources to determine correct year | -| `--verbose` | | `false` | Include verbose debug output | - -### What It Does - -1. **Scans** the library for album directories with the same normalized name but different years -2. **Identifies** potential duplicates (e.g., "Hot Pink [2019]" and "Hot Pink [2020]") -3. **Searches** metadata sources (when `--search` is used) to determine the correct release year -4. **Suggests** which directories to keep and which to delete -5. **Deletes** incorrect directories (when `--delete` is used) - -### Examples - -```bash -# Find all duplicate album directories -./mcli library find-duplicate-dirs --library "Storage" - -# Filter to a specific artist -./mcli library find-duplicate-dirs -l "Storage" --artist "Doja Cat" - -# Search metadata to determine correct year -./mcli library find-duplicate-dirs -l "Storage" --search - -# Search and auto-delete incorrect duplicates -./mcli library find-duplicate-dirs -l "Storage" --search --delete - -# JSON output for scripting -./mcli library find-duplicate-dirs -l "Storage" --search --json - -# Limit to first 10 groups -./mcli library find-duplicate-dirs -l "Storage" --limit 10 -``` - -### Example Output (Basic) - -``` -╭─────────────┬──────────────────┬──────────────────────────────┬──────┬───────╮ -│ Artist │ Album │ Directory │ Year │ Files │ -├─────────────┼──────────────────┼──────────────────────────────┼──────┼───────┤ -│ Doja Cat │ Hot Pink │ Hot Pink [2019] │ 2019 │ 12 │ -│ │ │ Hot Pink [2020] │ 2020 │ 12 │ -│ │ │ │ │ │ -│ Pink Floyd │ The Wall │ The Wall [1979] │ 1979 │ 26 │ -│ │ │ The Wall [1980] │ 1980 │ 26 │ -╰─────────────┴──────────────────┴──────────────────────────────┴──────┴───────╯ - -Found 2 duplicate album group(s) with 4 directories -``` - -### Example Output (With Metadata Search) - -``` -╭─────────────┬──────────────────┬──────────────────────┬──────┬───────┬───────────────┬─────────────────┬──────────╮ -│ Artist │ Album │ Directory │ Year │ Files │ Metadata Year │ Source │ Status │ -├─────────────┼──────────────────┼──────────────────────┼──────┼───────┼───────────────┼─────────────────┼──────────┤ -│ Doja Cat │ Hot Pink │ Hot Pink [2019] │ 2019 │ 12 │ 2019 │ SearchEngine │ ✓ Keep │ -│ │ │ Hot Pink [2020] │ 2020 │ 12 │ │ │ ✗ Delete │ -│ │ │ │ │ │ │ │ │ -│ Pink Floyd │ The Wall │ The Wall [1979] │ 1979 │ 26 │ 1979 │ SearchEngine │ ✓ Keep │ -│ │ │ The Wall [1980] │ 1980 │ 26 │ │ │ ✗ Delete │ -╰─────────────┴──────────────────┴──────────────────────┴──────┴───────┴───────────────┴─────────────────┴──────────╯ - -Found 2 duplicate album group(s) with 4 directories - Resolved via metadata: 2 - Directories suggested for deletion: 2 -``` - -### JSON Output Example - -```json -{ - "durationSeconds": 12.34, - "totalGroups": 2, - "totalDirectories": 4, - "resolvedGroups": 2, - "directoriesToDelete": 2, - "groups": [ - { - "artistName": "Doja Cat", - "albumName": "Hot Pink", - "metadataYear": 2019, - "metadataSource": "SearchEngine", - "musicBrainzId": "abc123-...", - "spotifyId": "xyz789", - "suggestedCorrectDirectory": "/mnt/music/library/D/Do/Doja Cat/Hot Pink [2019]", - "suggestedDeleteDirectories": [ - "/mnt/music/library/D/Do/Doja Cat/Hot Pink [2020]" - ], - "directories": [ - { - "path": "/mnt/music/library/D/Do/Doja Cat/Hot Pink [2019]", - "albumName": "Hot Pink", - "year": 2019, - "fileCount": 12, - "isCorrectYear": true - }, - { - "path": "/mnt/music/library/D/Do/Doja Cat/Hot Pink [2020]", - "albumName": "Hot Pink", - "year": 2020, - "fileCount": 12, - "isCorrectYear": false - } - ] - } - ] -} -``` - -### How Year Detection Works - -The command detects album years from: - -1. **Directory name patterns**: `[2019]`, `(2019)`, or `2019` prefix -2. **Melodee metadata files**: `melodee.json` if present -3. **Metadata searches**: MusicBrainz, Spotify, and other configured search engines - -### Safety - -- ✅ Read-only by default (only reports duplicates) -- ⚠️ `--search` queries external metadata APIs -- ⚠️ `--delete` permanently removes directories (requires confirmation) -- ❌ Only works with Storage libraries - -### Use Cases - -1. **Post-migration cleanup**: After importing a large music collection, find and resolve duplicate albums with inconsistent years -2. **Quality assurance**: Identify albums that may have been ripped or tagged with incorrect years -3. **Library maintenance**: Periodic checks to keep your library clean and consistent - ---- - -## Workflow Examples - -### Add New Music (Recommended) - -```bash -# Single command to process all new music end-to-end -./mcli library scan -``` - -This is the simplest and recommended approach. It handles: -- Processing files from inbound -- Validating and revalidating albums -- Moving approved albums to storage -- Adding them to the database - -### Force Full Reprocessing - -```bash -# Force reprocess everything, ignoring timestamps -./mcli library scan --force -``` - -### Full Library Rebuild - -```bash -#!/bin/bash -# Complete library rebuild (metadata regeneration) - -LIBRARY="Storage" - -echo "Starting full rebuild of $LIBRARY..." - -# Clean first -./mcli library clean -l "$LIBRARY" - -# Full metadata rebuild -./mcli library rebuild -l "$LIBRARY" --only-missing false - -# Rescan into database -./mcli library scan --force - -echo "Rebuild complete" -``` - -### Validate and Fix Library Integrity - -```bash -#!/bin/bash -# Check and fix library integrity issues - -LIBRARY="Storage" - -echo "Validating library integrity..." - -# First, check for issues -./mcli library validate -l "$LIBRARY" - -# If issues found, fix orphaned DB records -./mcli library validate -l "$LIBRARY" --fix - -# Add any unregistered directories to DB -./mcli library scan - -echo "Library validation complete" -``` - ---- +## Destructive Operations -## See Also +`clean` deletes directories without media files. `purge` deletes a library's +artists, albums, and songs and resets its statistics. `validate --fix`, moving, +processing with move semantics, and duplicate merging also change persistent +state. Back up PostgreSQL and media first, target a library explicitly, and run +the available report or JSON mode before making changes. -- [Libraries](/libraries/) - Library concepts -- [Album Commands](/cli/album/) - Album operations -- [CLI Overview](/cli/) - Main CLI documentation +See [Libraries](/libraries/) for the directory workflow and [Backup and +Restore](/backup/) for a complete recovery set. diff --git a/docs/pages/cli/parser.md b/docs/pages/cli/parser.md index 14f1e2316..1a0f1c68c 100644 --- a/docs/pages/cli/parser.md +++ b/docs/pages/cli/parser.md @@ -1,331 +1,24 @@ --- -title: CLI - Parser Commands +title: CLI - Parser Command +description: Parse supported album metadata files with mcli. permalink: /cli/parser/ -layout: page +tags: + - cli + - metadata + - diagnostics --- -# Parser Commands +# Parser Command -The `parser` branch provides commands for parsing and analyzing media metadata files. This is primarily a **diagnostic tool** for troubleshooting import and metadata issues. - -## Overview - -```bash -mcli parser [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `parse` | Parse a media file and show extracted metadata | - ---- - -## parser parse - -Parses various music metadata files and extracts information for inspection. - -### Usage - -```bash -mcli parser parse [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `FILENAME` | Yes | Path to the file to parse | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Show detailed parsing results in JSON format | - -### Supported File Types - -| Extension | Type | Description | -|-----------|------|-------------| -| `.cue` | CUE Sheet | Track listings with timestamps for splitting | -| `.nfo` | NFO File | Release information and metadata | -| `.sfv` | SFV File | Simple File Verification checksums | -| `.m3u` | M3U Playlist | Playlist files with track ordering | - ---- - -## CUE Sheet Parsing - -CUE sheets describe how to split a single audio file into multiple tracks. - -### Example - -```bash -./mcli parser parse "/path/to/album.cue" -``` - -### Sample CUE File - -``` -PERFORMER "The Beatles" -TITLE "Abbey Road" -FILE "abbey_road.flac" WAVE - TRACK 01 AUDIO - TITLE "Come Together" - PERFORMER "The Beatles" - INDEX 01 00:00:00 - TRACK 02 AUDIO - TITLE "Something" - PERFORMER "The Beatles" - INDEX 01 04:20:00 -``` - -### Output - -```json -{ - "Type": "CUE", - "IsValid": true, - "Artist": "The Beatles", - "Album": "Abbey Road", - "AudioFile": "abbey_road.flac", - "Tracks": [ - { - "Number": 1, - "Title": "Come Together", - "Artist": "The Beatles", - "StartTime": "00:00:00", - "Duration": "04:20" - }, - { - "Number": 2, - "Title": "Something", - "Artist": "The Beatles", - "StartTime": "04:20:00", - "Duration": null - } - ] -} -``` - ---- - -## NFO File Parsing - -NFO files contain release information, often in scene releases. - -### Example - -```bash -./mcli parser parse "/path/to/album.nfo" -``` - -### Sample NFO Content - -``` -Artist......: The Beatles -Album.......: Abbey Road -Year........: 1969 -Genre.......: Rock -Tracks......: 17 -Quality.....: FLAC / Lossless -``` - -### Output - -```json -{ - "Type": "NFO", - "IsValid": true, - "Artist": "The Beatles", - "Album": "Abbey Road", - "Year": 1969, - "Genre": "Rock", - "TrackCount": 17, - "Quality": "FLAC / Lossless" -} -``` - ---- - -## SFV File Parsing - -SFV files contain CRC32 checksums for file verification. - -### Example - -```bash -./mcli parser parse "/path/to/album.sfv" -``` - -### Sample SFV Content - -``` -; Generated by sfv tool -01 - Come Together.mp3 A1B2C3D4 -02 - Something.mp3 E5F6G7H8 -03 - Maxwell's Silver Hammer.mp3 I9J0K1L2 -``` - -### Output - -```json -{ - "Type": "SFV", - "IsValid": true, - "Files": [ - { - "Filename": "01 - Come Together.mp3", - "Checksum": "A1B2C3D4", - "Exists": true, - "Valid": true - }, - { - "Filename": "02 - Something.mp3", - "Checksum": "E5F6G7H8", - "Exists": true, - "Valid": true - } - ], - "Summary": { - "Total": 3, - "Valid": 3, - "Missing": 0, - "Corrupted": 0 - } -} -``` - ---- - -## M3U Playlist Parsing - -M3U files are playlists that list audio files in order. - -### Example +`parser parse` exercises Melodee's directory metadata parsers for CUE, SFV, +M3U, and NFO files. ```bash -./mcli parser parse "/path/to/playlist.m3u" +mcli parser /music/album/album.cue parse --verbose +mcli parser /music/album/album.nfo parse ``` -### Sample M3U Content - -``` -#EXTM3U -#EXTINF:259,The Beatles - Come Together -/music/The Beatles/Abbey Road/01 - Come Together.mp3 -#EXTINF:182,The Beatles - Something -/music/The Beatles/Abbey Road/02 - Something.mp3 -``` - -### Output - -```json -{ - "Type": "M3U", - "IsValid": true, - "Format": "Extended M3U", - "Entries": [ - { - "Path": "/music/The Beatles/Abbey Road/01 - Come Together.mp3", - "Duration": 259, - "Title": "The Beatles - Come Together", - "Exists": true - }, - { - "Path": "/music/The Beatles/Abbey Road/02 - Something.mp3", - "Duration": 182, - "Title": "The Beatles - Something", - "Exists": true - } - ], - "TotalDuration": "7:21", - "TrackCount": 2 -} -``` - ---- - -## Use Cases - -### Debugging Import Issues - -```bash -# Check why CUE sheet isn't being processed correctly -./mcli parser parse "/path/to/problem.cue" - -# Verify NFO metadata extraction -./mcli parser parse "/path/to/release.nfo" -``` - -### Validating File Integrity - -```bash -# Check SFV for corrupted files -./mcli parser parse "/path/to/album.sfv" -``` - -### Analyzing Release Structure - -```bash -# Understand a release's organization -./mcli parser parse "/path/to/album.cue" -./mcli parser parse "/path/to/album.nfo" -./mcli parser parse "/path/to/album.sfv" -``` - -### Scripting - -```bash -#!/bin/bash -# Parse all CUE files in a directory - -for cue in *.cue; do - echo "=== $cue ===" - ./mcli parser parse "$cue" | jq '.Tracks | length' -done -``` - ---- - -## Error Handling - -### Invalid File - -```json -{ - "Type": "CUE", - "IsValid": false, - "Error": "Invalid CUE syntax at line 15", - "Line": 15, - "Content": "TRACK 01 INVALID" -} -``` - -### Missing Referenced Files - -```json -{ - "Type": "CUE", - "IsValid": true, - "Warnings": [ - "Referenced audio file not found: abbey_road.flac" - ] -} -``` - -### Unsupported Format - -``` -Error: Unsupported file type: .xyz -Supported types: .cue, .nfo, .sfv, .m3u -``` - ---- - -## See Also - -- [Tags Commands](/cli/tags/) - View ID3 tags from media files -- [File Commands](/cli/file/) - Analyze MPEG files -- [CLI Overview](/cli/) - Main CLI documentation +The surrounding album directory and referenced media files may be needed to +produce a valid result. `--verbose` includes the parser result and timing data. +This is a diagnostic command; test it against a copy if the configured parser +is allowed to normalize supporting files. diff --git a/docs/pages/cli/search.md b/docs/pages/cli/search.md new file mode 100644 index 000000000..dd603ac5c --- /dev/null +++ b/docs/pages/cli/search.md @@ -0,0 +1,35 @@ +--- +title: CLI - Search Command +description: Search Melodee artists, albums, songs, and playlists locally or remotely with mcli. +permalink: /cli/search/ +tags: + - cli + - search + - api +--- + +# Search Command + +```bash +mcli search QUERY [--limit 25] [--verbose] +``` + +Local mode searches using the configured databases: + +```bash +mcli search 'Miles Davis' --limit 10 +``` + +When a remote server resolves from `--server`, environment variables, or a +profile, the command calls the native REST API: + +```bash +MELODEE_SERVER=https://music.example.com \ +MELODEE_TOKEN='eyJ...' \ +mcli search 'Miles Davis' --limit 10 --json +``` + +Results can include artists, albums, songs, and playlists. `--json` is a remote +mode option and selects compact JSON; local presentation is controlled by the +local command implementation. See [CLI Remote Server Mode](/cli-remote-mode/) +for authentication and precedence. diff --git a/docs/pages/cli/system.md b/docs/pages/cli/system.md new file mode 100644 index 000000000..5c5624859 --- /dev/null +++ b/docs/pages/cli/system.md @@ -0,0 +1,28 @@ +--- +title: CLI - System Command +description: Display Melodee server identity and version information locally or remotely with mcli. +permalink: /cli/system/ +tags: + - cli + - diagnostics + - api +--- + +# System Command + +`system info` reports the server name, description, and version. + +```bash +mcli system info +``` + +Remote connection options belong to the `system` group, before `info`: + +```bash +mcli system --server https://music.example.com --token 'eyJ...' info +mcli system --profile home --json info +``` + +The CLI currently requires a token whenever remote mode is active, even though +the underlying server-information endpoint is public. See [CLI Remote Server +Mode](/cli-remote-mode/) for token acquisition and secure storage guidance. diff --git a/docs/pages/cli/tags.md b/docs/pages/cli/tags.md index ea191e320..624576bde 100644 --- a/docs/pages/cli/tags.md +++ b/docs/pages/cli/tags.md @@ -1,220 +1,28 @@ --- -title: CLI - Tags Commands +title: CLI - Tag Commands +description: Display embedded metadata tags from a media file with mcli. permalink: /cli/tags/ -layout: page +tags: + - cli + - metadata + - diagnostics --- -# Tags Commands +# Tag Commands -The `tags` branch provides commands for displaying and analyzing media file tags (ID3, Vorbis Comments, etc.). - -## Overview +`tags show` reads a media file and displays the embedded tags recognized by +Melodee. ```bash -mcli tags [COMMAND] [OPTIONS] +mcli tags /music/example.flac show +mcli tags /music/example.mp3 show --verbose ``` -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `show` | Display all tags from a media file | - ---- - -## tags show - -Displays all known tags from a media file including ID3v1, ID3v2, Vorbis Comments, APE tags, and other metadata. - -### Usage +Use `--onlytags` (`-o`) to emit only the tag values as comma-separated output: ```bash -mcli tags show [OPTIONS] +mcli tags /music/example.flac show --onlytags ``` -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `FILENAME` | Yes | Path to the media file to analyze | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--verbose` | | `true` | Output verbose debug and timing results | - -### Supported Formats - -| Format | Tag Types | -|--------|-----------| -| MP3 | ID3v1, ID3v2.3, ID3v2.4, APE | -| FLAC | Vorbis Comments, ID3v2 | -| OGG | Vorbis Comments | -| M4A/AAC | iTunes/MP4 tags | -| WAV | INFO chunks, ID3v2 | -| WMA | ASF metadata | - -### Examples - -```bash -# Show tags from MP3 file -./mcli tags show "/path/to/song.mp3" - -# Show tags from FLAC file -./mcli tags show "/path/to/song.flac" -``` - -### Output - -``` -╭─────────────────────────────────────────────────────────────────────────────╮ -│ File: /path/to/song.mp3 │ -│ Format: MPEG Audio (MP3) │ -│ Tag Types: ID3v2.4, ID3v1 │ -╰─────────────────────────────────────────────────────────────────────────────╯ - - ID3v2.4 Tags -╭─────────────────────────┬───────────────────────────────────────────────────╮ -│ Tag │ Value │ -├─────────────────────────┼───────────────────────────────────────────────────┤ -│ TIT2 (Title) │ Come Together │ -│ TPE1 (Artist) │ The Beatles │ -│ TALB (Album) │ Abbey Road │ -│ TRCK (Track) │ 1/17 │ -│ TYER (Year) │ 1969 │ -│ TCON (Genre) │ Rock │ -│ TPOS (Disc) │ 1/1 │ -│ TPE2 (Album Artist) │ The Beatles │ -│ TCMP (Compilation) │ 0 │ -│ APIC (Picture) │ Front Cover (image/jpeg, 45.2 KB) │ -│ TXXX:MusicBrainz Album Id│ 12345678-1234-1234-1234-123456789012 │ -╰─────────────────────────┴───────────────────────────────────────────────────╯ - - ID3v1 Tags -╭─────────────────────────┬───────────────────────────────────────────────────╮ -│ Tag │ Value │ -├─────────────────────────┼───────────────────────────────────────────────────┤ -│ Title │ Come Together │ -│ Artist │ The Beatles │ -│ Album │ Abbey Road │ -│ Year │ 1969 │ -│ Genre │ Rock (17) │ -│ Track │ 1 │ -╰─────────────────────────┴───────────────────────────────────────────────────╯ -``` - -### Common Tags - -**Standard Tags:** - -| ID3v2 Frame | Vorbis Comment | Description | -|-------------|----------------|-------------| -| TIT2 | TITLE | Song title | -| TPE1 | ARTIST | Performing artist | -| TALB | ALBUM | Album name | -| TRCK | TRACKNUMBER | Track number | -| TYER/TDRC | DATE | Year/date | -| TCON | GENRE | Genre | -| TPE2 | ALBUMARTIST | Album artist | -| TPOS | DISCNUMBER | Disc number | -| APIC | (embedded) | Album artwork | - -**Extended Tags:** - -| ID3v2 Frame | Vorbis Comment | Description | -|-------------|----------------|-------------| -| TXXX:MusicBrainz Album Id | MUSICBRAINZ_ALBUMID | MusicBrainz release ID | -| TXXX:MusicBrainz Artist Id | MUSICBRAINZ_ARTISTID | MusicBrainz artist ID | -| TXXX:MusicBrainz Track Id | MUSICBRAINZ_TRACKID | MusicBrainz recording ID | -| TXXX:ISRC | ISRC | International Standard Recording Code | - -### FLAC Output Example - -``` -╭─────────────────────────────────────────────────────────────────────────────╮ -│ File: /path/to/song.flac │ -│ Format: FLAC Audio │ -│ Tag Types: Vorbis Comments │ -╰─────────────────────────────────────────────────────────────────────────────╯ - - Vorbis Comments -╭─────────────────────────┬───────────────────────────────────────────────────╮ -│ Tag │ Value │ -├─────────────────────────┼───────────────────────────────────────────────────┤ -│ TITLE │ Come Together │ -│ ARTIST │ The Beatles │ -│ ALBUM │ Abbey Road │ -│ TRACKNUMBER │ 1 │ -│ TRACKTOTAL │ 17 │ -│ DATE │ 1969 │ -│ GENRE │ Rock │ -│ ALBUMARTIST │ The Beatles │ -│ DISCNUMBER │ 1 │ -│ DISCTOTAL │ 1 │ -╰─────────────────────────┴───────────────────────────────────────────────────╯ - - Embedded Pictures -╭───────┬────────────┬────────────┬──────────╮ -│ Index │ Type │ Format │ Size │ -├───────┼────────────┼────────────┼──────────┤ -│ 0 │ Front │ image/jpeg │ 245.3 KB │ -│ 1 │ Back │ image/jpeg │ 198.7 KB │ -╰───────┴────────────┴────────────┴──────────╯ -``` - -### Use Cases - -**Debugging Tag Issues:** -```bash -# Why isn't the artist showing correctly? -./mcli tags show "/path/to/song.mp3" | grep -i artist -``` - -**Verifying Metadata:** -```bash -# Check if MusicBrainz IDs are present -./mcli tags show "/path/to/song.flac" | grep -i musicbrainz -``` - -**Comparing Tag Versions:** -```bash -# Compare ID3v1 vs ID3v2 tags -./mcli tags show "/path/to/song.mp3" -``` - -### Scripting Examples - -**Check for Missing Tags:** -```bash -#!/bin/bash -# Find files missing album artwork - -for file in *.mp3; do - if ! ./mcli tags show "$file" | grep -q "APIC\|Picture"; then - echo "Missing artwork: $file" - fi -done -``` - -**Export Tag Summary:** -```bash -#!/bin/bash -# Create a tag inventory - -echo "File,Title,Artist,Album" > tags.csv -for file in *.mp3; do - title=$(./mcli tags show "$file" | grep "TIT2" | cut -d'│' -f3 | xargs) - artist=$(./mcli tags show "$file" | grep "TPE1" | cut -d'│' -f3 | xargs) - album=$(./mcli tags show "$file" | grep "TALB" | cut -d'│' -f3 | xargs) - echo "\"$file\",\"$title\",\"$artist\",\"$album\"" >> tags.csv -done -``` - ---- - -## See Also - -- [File Commands](/cli/file/) - Analyze MPEG files -- [Parser Commands](/cli/parser/) - Parse metadata files -- [CLI Overview](/cli/) - Main CLI documentation +The CLI process must have read access to the path. For MPEG frame diagnostics +instead of metadata, use [`mcli file mpeg`](/cli/file/). diff --git a/docs/pages/cli/user.md b/docs/pages/cli/user.md index 24b8a4a99..97de0f3cf 100644 --- a/docs/pages/cli/user.md +++ b/docs/pages/cli/user.md @@ -1,255 +1,40 @@ --- title: CLI - User Commands +description: Create, list, inspect, and delete Melodee users with mcli. permalink: /cli/user/ -layout: page +tags: + - cli + - users + - administration --- # User Commands -The `user` branch provides commands for managing user accounts, including creating, listing, and deleting users from the command line. - -## Overview - -```bash -mcli user [COMMAND] [OPTIONS] -``` - -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `create` | Create a new user account | -| `delete` | Delete a user from the database | -| `list` | List all users in the system | - ---- - -## user create - -Creates a new user account in the system. - -### Usage - -```bash -mcli user create --username --email --password [OPTIONS] -``` - -### Options - -| Option | Alias | Required | Default | Description | -|--------|-------|----------|---------|-------------| -| `--username` | `-u` | Yes | | Username for the new account | -| `--email` | `-e` | Yes | | Email address for the new account | -| `--password` | `-p` | Yes | | Password (minimum 8 characters) | -| `--force` | `-f` | No | `false` | Delete existing user with same username or email before creating | -| `--verbose` | | No | `false` | Output verbose debug and timing results | - -### Examples - -```bash -# Create a new user -./mcli user create --username "john" --email "john@example.com" --password "SecurePass123!" - -# Create a user with short options -./mcli user create -u "john" -e "john@example.com" -p "SecurePass123!" - -# Force recreate an existing user (deletes existing user first) -./mcli user create -u "demo" -e "demo@melodee.org" -p "Mel0deeR0cks!" --force -``` - -### Output - -``` -✓ User 'john' created successfully. -``` - -### Error Output - -``` -✗ Failed to create user: User already exists. Use --force to replace. -``` - -### Docker/Podman Usage - -When running in a container, use `exec` to run the CLI: - -```bash -# Docker -docker compose exec melodee.blazor /app/cli/mcli user create \ - --username "demo" \ - --email "demo@melodee.org" \ - --password "Mel0deeR0cks!" - -# Podman -podman compose exec melodee.blazor /app/cli/mcli user create \ - --username "demo" \ - --email "demo@melodee.org" \ - --password "Mel0deeR0cks!" -``` - -### Notes - -- The `--force` option will delete any existing user with the same username or email address before creating the new user -- Passwords must be at least 8 characters long -- User accounts created via CLI have full administrator privileges - ---- - -## user list - -Lists all user accounts in the system. - -### Usage - -```bash -mcli user list [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-n`, `--limit` | | `50` | Maximum number of users to return | -| `--raw` | | `false` | Output results in JSON format | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples +| Command | Purpose | +|---------|---------| +| `create` | Create a local user with `--username`, `--email`, and `--password` | +| `delete ID` | Delete a local user; `--yes` skips confirmation | +| `list` | List users locally or remotely; `--limit` defaults to 50 | +| `me` | Show the current user locally or remotely | ```bash -# List first 50 users -./mcli user list - -# List up to 100 users -./mcli user list -n 100 - -# Output as JSON for scripting -./mcli user list --raw -``` - -### Output - -``` -╭──────┬──────────────┬───────────────────────┬─────────────────┬───────────╮ -│ ID │ Username │ Email │ Created │ Status │ -├──────┼──────────────┼───────────────────────┼─────────────────┼───────────┤ -│ 1 │ admin │ admin@example.com │ 20240115T103000 │ ✓ │ -│ 2 │ demo │ demo@melodee.org │ 20241230T142300 │ ✓ │ -│ 3 │ john │ john@example.com │ 20241230T150000 │ 🔒 Locked │ -╰──────┴──────────────┴───────────────────────┴─────────────────┴───────────╯ - -Showing 3 of 3 users +mcli user create --username alice --email alice@example.com --password 'replace-me' +mcli user list --limit 25 +mcli user me +mcli user delete 42 ``` -### JSON Output - -```json -[ - { - "Id": 1, - "ApiKey": "a1b2c3d4-...", - "UserName": "admin", - "Email": "admin@example.com", - "IsLocked": false, - "CreatedAt": "2024-01-15T10:30:00Z", - "LastLoginAt": "2024-12-30T08:15:00Z", - "LastActivityAt": "2024-12-30T14:23:00Z" - } -] -``` - ---- - -## user delete - -Deletes a user account from the system. +`create --force` deletes and recreates an existing matching account. This can +remove related account state, so prefer correcting the existing user in the +administration UI. -### Usage +Only `list` and `me` support remote mode. Remote `list` requires an administrator +JWT; remote `me` requires an authenticated user's JWT: ```bash -mcli user delete [OPTIONS] -``` - -### Arguments - -| Argument | Required | Description | -|----------|----------|-------------| -| `ID` | Yes | User ID to delete | - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `-y`, `--yes` | | `false` | Skip confirmation prompt | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples - -```bash -# Delete user (with confirmation) -./mcli user delete 42 - -# Delete without confirmation (scripting) -./mcli user delete 42 -y -``` - -### Output - +mcli user me --server https://music.example.com --token 'eyJ...' +mcli user list --server https://music.example.com --token 'eyJ...' --limit 25 ``` -User: john -Email: john@example.com -Created: 2024-12-30T15:00:00Z - -Delete user 'john'? [y/n] (n): y - -✓ User 'john' deleted successfully. -``` - -### Safety Notes - -- ⚠️ **Locked users cannot be deleted.** Unlock the user first through the web interface. -- ⚠️ **This is a destructive operation.** The command shows details and requires confirmation by default. -- ⚠️ **User data associations** (stars, ratings, playlists) may be affected by deletion. - ---- - -## Workflow Examples - -### Setting Up a Demo User - -```bash -# Create or replace a demo user -./mcli user create \ - --username "demo" \ - --email "demo@melodee.org" \ - --password "Mel0deeR0cks!" \ - --force -``` - -### Listing Users for Automation - -```bash -#!/bin/bash -# Get user count for monitoring -USER_COUNT=$(./mcli user list --raw | jq 'length') -echo "Total users: $USER_COUNT" -``` - -### Bulk User Management Script - -```bash -#!/bin/bash -# Create multiple users from a file -# users.txt format: username,email,password - -while IFS=',' read -r username email password; do - ./mcli user create -u "$username" -e "$email" -p "$password" -done < users.txt -``` - ---- - -## See Also -- [CLI Overview](/cli/) - Main CLI documentation -- [Configuration Commands](/cli/configuration/) - Manage settings -- [Library Commands](/cli/library/) - Library operations +See [CLI Remote Server Mode](/cli-remote-mode/) for profiles, environment +variables, and exit codes. diff --git a/docs/pages/cli/validate.md b/docs/pages/cli/validate.md index c9f51485c..e1e3c26a7 100644 --- a/docs/pages/cli/validate.md +++ b/docs/pages/cli/validate.md @@ -1,264 +1,32 @@ --- title: CLI - Validate Commands +description: Validate Melodee album metadata files and database records with mcli. permalink: /cli/validate/ -layout: page +tags: + - cli + - metadata + - validation --- # Validate Commands -The `validate` branch provides commands for validating media files and Melodee metadata. +`validate` and `validate album` validate `melodee.json` album metadata. Select a +target with one of the supported identifiers: -## Overview +| Option | Target | +|--------|--------| +| `--file PATH` | A specific `melodee.json` file | +| `--id ID` | A metadata record ID | +| `--apiKey GUID` | One album API key | +| `--artistApiKey GUID` | All albums for an artist API key | +| `--library NAME` | A named library | ```bash -mcli validate [COMMAND] [OPTIONS] +mcli validate album --file /storage/Artist/Album/melodee.json +mcli validate --library Staging --json +mcli validate --apiKey 00000000-0000-0000-0000-000000000000 --verbose ``` -**Available Commands:** - -| Command | Description | -|---------|-------------| -| `album` | Validate a Melodee metadata file (melodee.json) | - ---- - -## validate album - -Validates a Melodee metadata file (`melodee.json`) and reports any issues found. Can also validate all albums for a given artist. - -### Usage - -```bash -mcli validate album [OPTIONS] -``` - -### Options - -| Option | Alias | Default | Description | -|--------|-------|---------|-------------| -| `--file` | | | Path to the melodee.json file to validate | -| `--apiKey` | | | ApiKey of an album to validate | -| `--artistApiKey` | | | ApiKey of an artist to validate all albums for | -| `--library` | | | Name of Library (used with --id) | -| `--id` | | | Id of Melodee Data File to validate (used with --library) | -| `--verbose` | | `false` | Output verbose debug and timing results | - -### Examples - -```bash -# Validate a melodee.json file -./mcli validate album --file "/path/to/album/melodee.json" - -# Validate an album by ApiKey -./mcli validate album --apiKey "12345678-1234-1234-1234-123456789012" - -# Validate all albums for an artist by ApiKey -./mcli validate album --artistApiKey "87654321-4321-4321-4321-210987654321" - -# Validate with verbose output -./mcli validate album --file "/path/to/album/melodee.json" --verbose -``` - -### Output (Single Album - Valid) - -``` -╭─────────────────────────────────────────────────────────────────────────────╮ -│ Validation Results │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ File: /path/to/album/melodee.json │ -│ Status: ✓ Valid │ -├─────────────────────────────────────────────────────────────────────────────┤ -│ Album: Abbey Road │ -│ Artist: The Beatles │ -│ Year: 1969 │ -│ Tracks: 17 │ -│ Images: 2 │ -╰─────────────────────────────────────────────────────────────────────────────╯ -``` - -### Output (Artist Albums Validation) - -``` -╭─────────────────────────────────────────────────────────────────────────────╮ -│ Artist Albums Validation Summary │ -├───────────┬─────────────────────────────────────────────────────────────────┤ -│ Property │ Value │ -├───────────┼─────────────────────────────────────────────────────────────────┤ -│ Artist │ The Beatles │ -│ Total │ 13 │ -│ Valid │ 11 │ -│ Invalid │ 2 │ -│ Status │ ✗ Issues Found │ -╰───────────┴─────────────────────────────────────────────────────────────────╯ - -╭─────────────────────────────────────────────────────────────────────────────╮ -│ Album Details │ -├──────────────────────────────┬──────┬────────┬───────┬───────┬──────────────┤ -│ Album │ Year │ Status │ Dir │ Cover │ Issues │ -├──────────────────────────────┼──────┼────────┼───────┼───────┼──────────────┤ -│ Abbey Road │ 1969 │ ✓ │ ✓ │ ✓ │ 0 │ -│ Let It Be │ 1970 │ ✗ │ ✗ │ ✗ │ 2 │ -│ ... │ │ │ │ │ │ -╰──────────────────────────────┴──────┴────────┴───────┴───────┴──────────────╯ - -Issues Found: - -Let It Be (1970): - ✗ Album directory does not exist: /library/The Beatles/Let It Be - ✗ melodee.json file not found -``` - -### Validation Rules - -**Required Fields:** - -| Field | Description | -|-------|-------------| -| Artist | Album artist name | -| Album | Album title | -| Tracks | At least one track | - -**Track Validation:** - -| Rule | Description | -|------|-------------| -| Title | Each track must have a title | -| Track Number | Must be a positive integer | -| Duration | Must be greater than 0 | -| File Reference | Referenced audio file must exist | - -**Image Validation:** - -| Rule | Description | -|------|-------------| -| Front Cover | At least one front cover image | -| Format | Images must be JPEG, PNG, or WebP | -| Size | Images should meet minimum size requirements | - -**Directory Validation (Artist Mode):** - -| Rule | Description | -|------|-------------| -| Directory Exists | Album directory must exist on disk | -| melodee.json | Metadata file must be present | -| Cover Image | At least one image file present | - -**Metadata Quality:** - -| Rule | Severity | Description | -|------|----------|-------------| -| MusicBrainz IDs | Warning | Recommended for accurate matching | -| Genre | Warning | Helps with categorization | -| Year | Warning | Release year recommended | - -### melodee.json Structure - -```json -{ - "Artist": "The Beatles", - "Album": "Abbey Road", - "Year": 1969, - "Genre": "Rock", - "Tracks": [ - { - "Number": 1, - "Title": "Come Together", - "Duration": 259, - "File": "01 - Come Together.mp3", - "MusicBrainzId": "12345678-1234-1234-1234-123456789012" - } - ], - "Images": [ - { - "Type": "Front", - "File": "i-01-Front.jpg" - } - ], - "MusicBrainzAlbumId": "87654321-4321-4321-4321-210987654321" -} -``` - -### Use Cases - -**Pre-Import Validation:** -```bash -# Validate before moving to library -./mcli validate album "/staging/Artist/Album/melodee.json" -if [ $? -eq 0 ]; then - echo "Album is valid, ready to move" -else - echo "Fix issues before importing" -fi -``` - -**Batch Validation:** -```bash -#!/bin/bash -# Validate all melodee.json files in staging - -find /staging -name "melodee.json" | while read file; do - if ! ./mcli validate album "$file" > /dev/null 2>&1; then - echo "Invalid: $(dirname "$file")" - fi -done -``` - -**Quality Report:** -```bash -#!/bin/bash -# Generate validation report - -echo "Validation Report - $(date)" > report.txt -echo "=========================" >> report.txt - -find /library -name "melodee.json" | while read file; do - echo "" >> report.txt - echo "Album: $(dirname "$file")" >> report.txt - ./mcli validate album "$file" 2>&1 >> report.txt -done -``` - -### Exit Codes - -| Code | Meaning | -|------|---------| -| 0 | Validation passed (no errors) | -| 1 | Validation failed (errors found) | - -Use exit codes in scripts: - -```bash -./mcli validate album "$file" -case $? in - 0) echo "Valid" ;; - 1) echo "Invalid - needs attention" ;; -esac -``` - -### Common Issues - -**Missing melodee.json:** -``` -Error: File not found: /path/to/album/melodee.json -``` -**Solution:** Run `library rebuild` to generate metadata files. - -**Corrupted JSON:** -``` -Error: Invalid JSON syntax at line 15 -``` -**Solution:** Manually fix the JSON or regenerate with `library rebuild`. - -**Missing Audio Files:** -``` -Error: Track 3 references missing file: 03 - Song.mp3 -``` -**Solution:** Ensure all referenced audio files exist in the album directory. - ---- - -## See Also - -- [Library Commands](/cli/library/) - Rebuild metadata files -- [Album Commands](/cli/album/) - Album management -- [CLI Overview](/cli/) - Main CLI documentation +Use `--json` (`-j`) for structured results. This command validates album +metadata; [`library validate`](/cli/library/) instead compares PostgreSQL +records with files on disk and optionally removes orphaned records. diff --git a/docs/pages/configuration-reference.md b/docs/pages/configuration-reference.md index 745836e1b..4cef7ba47 100644 --- a/docs/pages/configuration-reference.md +++ b/docs/pages/configuration-reference.md @@ -1,361 +1,207 @@ --- title: Configuration Reference +description: Reference for Melodee 2.2.0 Compose variables, .NET host settings, and database-backed settings. permalink: /configuration-reference/ +tags: + - configuration + - environment-variables + - reference --- # Configuration Reference -This page provides a comprehensive reference of all configuration options available in Melodee, organized by category. - -## Environment Variables - -### Database Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `DB_PASSWORD` | - | Required: Password for PostgreSQL database | -| `DB_MIN_POOL_SIZE` | 10 | Minimum number of database connections in the pool | -| `DB_MAX_POOL_SIZE` | 50 | Maximum number of database connections in the pool | -| `ConnectionStrings__DefaultConnection` | - | Full connection string for PostgreSQL database | - -### Application Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `MELODEE_PORT` | 8080 | Port on which Melodee will run | -| `MELODEE_STORAGE_PATH` | /app/storage | Path for processed music files | -| `MELODEE_INBOUND_PATH` | /app/inbound | Path for new music files to be processed | -| `MELODEE_STAGING_PATH` | /app/staging | Path for music files awaiting review | -| `MELODEE_USER_IMAGES_PATH` | /app/user-images | Path for user avatars and images | -| `MELODEE_PLAYLISTS_PATH` | /app/playlists | Path for playlist definitions | -| `MELODEE_TEMPLATES_PATH` | /app/templates | Path for email templates organized by language | -| `SEARCHENGINE_MUSICBRAINZ_STORAGEPATH` | /app/storage/_search-engines/musicbrainz | Path for MusicBrainz database | - -### Authentication Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `Auth__Google__Enabled` | false | Enable Google OAuth authentication | -| `Auth__Google__ClientId` | - | Google OAuth Client ID | -| `Auth__Google__AllowedHostedDomains` | - | Restrict Google login to specific domains | -| `Auth__Google__AutoLinkEnabled` | false | Automatically link Google accounts | -| `Auth__Tokens__AccessTokenLifetimeMinutes` | 15 | Access token lifetime in minutes | -| `Auth__Tokens__RefreshTokenLifetimeDays` | 30 | Refresh token lifetime in days | -| `Auth__SelfRegistrationEnabled` | true | Allow user self-registration | - -### JWT Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `Jwt__Key` | - | Required: JWT signing key (256-bit secret) | -| `Jwt__Issuer` | melodee | JWT token issuer | -| `Jwt__Audience` | melodee-clients | JWT token audience | - -### Streaming Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `Streaming__UseBufferedResponses` | false | Use buffered responses for streaming | -| `Streaming__MaxConcurrentStreamsPerUser` | - | Maximum concurrent streams per user | - -### Brave Search API Configuration - -| Variable | Default | Description | -|----------|---------|-------------| -| `BRAVE_SEARCH__ENABLED` | false | Enable Brave Search API for images | -| `BRAVE_SEARCH__APIKEY` | - | Brave Search API key | -| `BRAVE_SEARCH__BASEURL` | https://api.search.brave.com | Brave Search API base URL | -| `BRAVE_SEARCH__IMAGESEARCHPATH` | /res/v1/images/search | Image search API path | - -### Plugin Configuration - -Override plugin settings using these environment variables (standard .NET configuration mapping): - -| Variable | Description | -|----------|-------------| -| `Plugins__MetadataProviders__Spotify__Enabled` | Enable/Disable Spotify metadata | -| `Plugins__MetadataProviders__Spotify__ClientId` | Spotify Client ID | -| `Plugins__MetadataProviders__Spotify__ClientSecret` | Spotify Client Secret | -| `Plugins__MetadataProviders__LastFm__Enabled` | Enable/Disable Last.FM metadata | -| `Plugins__MetadataProviders__LastFm__ApiKey` | Last.FM API Key | -| `Plugins__MetadataProviders__LastFm__ApiSecret` | Last.FM Shared Secret | -| `Plugins__MetadataProviders__MusicBrainz__Enabled` | Enable/Disable MusicBrainz (local) | -| `Plugins__MetadataProviders__Itunes__Enabled` | Enable/Disable iTunes metadata | -| `Plugins__MetadataProviders__Deezer__Enabled` | Enable/Disable Deezer metadata | - -## AppSettings Configuration - -### System Configuration - -```json -{ - "System": { - "AppName": "Melodee", - "AppVersion": "1.0.0", - "Environment": "Production" - } -} +This reference separates variables interpreted by the supplied Compose file +from settings read by the Melodee process. Use `__` (double underscore) in +environment variables for nested .NET host configuration. + +## Compose Variables + +These variables are consumed by `compose.yml` on the host: + +| Variable | Default | Purpose | +|----------|---------|---------| +| `DB_PASSWORD` | No safe default | PostgreSQL password and application connection-string value | +| `DB_MIN_POOL_SIZE` | `10` | Minimum Npgsql pool size in the generated connection string | +| `DB_MAX_POOL_SIZE` | `50` | Maximum Npgsql pool size in the generated connection string | +| `MELODEE_PORT` | `8080` | Published host port mapped to container port 8080 | +| `MELODEE_IMAGE` | `localhost/melodee:latest` | Application image name or release tag | +| `DOCKERFILE_PATH` | `Dockerfile` | Dockerfile used by local builds | +| `MELODEE_AUTH_TOKEN` | Insecure placeholder | JWT signing key and legacy Melodee token in the supplied deployment | +| `MELODEE_AUTH_TOKEN_HOURS` | `24` | Legacy token lifetime fallback | +| `JWT_ISSUER` | `MelodeeApi` | Expected JWT issuer | +| `JWT_AUDIENCE` | `MelodeeClient` | Expected JWT audience | + +Replace both secret placeholders before the first start. A signing key should +be random and at least 64 characters in the supplied deployment. + +## Connection Strings + +| Environment variable | Required | Purpose | +|----------------------|----------|---------| +| `ConnectionStrings__DefaultConnection` | Yes | Primary PostgreSQL database | +| `ConnectionStrings__MusicBrainzConnection` | Yes | Local MusicBrainz DecentDB file | +| `ConnectionStrings__ArtistSearchEngineConnection` | Yes | Local artist-search DecentDB file | + +The tracked Compose file supplies all three. For native runs, provide them in an +untracked settings file or environment. DecentDB connection strings use a +`Data Source=...` path; see [DecentDB Usage & Migration](/decentdb/). + +## JWT and Authentication + +| Key / environment variable | Default | Notes | +|----------------------------|---------|-------| +| `Jwt:Key` / `Jwt__Key` | None | Required signing secret | +| `Jwt:Issuer` / `Jwt__Issuer` | `MelodeeApi` | Must match issued tokens | +| `Jwt:Audience` / `Jwt__Audience` | `MelodeeClient` | Must match issued tokens | +| `Auth:SelfRegistrationEnabled` / `Auth__SelfRegistrationEnabled` | `true` | Allows account registration subject to Melodee registration settings | +| `Auth:Google:Enabled` | `false` | Enables Google ID-token authentication | +| `Auth:Google:ClientId` | Empty | Primary Google OAuth client ID | +| `Auth:Google:AdditionalClientIds` | Empty | Additional accepted client IDs | +| `Auth:Google:AllowedHostedDomains` | Empty | Optional hosted-domain allowlist | +| `Auth:Google:AutoLinkEnabled` | `false` | Allows configured account linking behavior | +| `Auth:Google:ClockSkewSeconds` | `300` | Token validation clock skew | +| `Auth:Tokens:AccessTokenLifetimeMinutes` | `15` | JWT access-token lifetime | +| `Auth:Tokens:RefreshTokenLifetimeDays` | `30` | Refresh-token lifetime | +| `Auth:Tokens:MaxSessionDays` | `90` | Maximum session age | +| `Auth:Tokens:RotateRefreshTokens` | `true` | Rotates tokens at refresh | +| `Auth:Tokens:RevokeOnReplay` | `true` | Revokes a token family after replay detection | + +Arrays use numeric environment suffixes, for example: + +```text +Auth__Google__AllowedHostedDomains__0=example.com +Auth__Google__AdditionalClientIds__0=client-id.apps.googleusercontent.com ``` -### Logging Configuration - -```json -{ - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning" - } - }, - "Serilog": { - "Using": ["Serilog.Sinks.Console", "Serilog.Sinks.File"], - "MinimumLevel": "Information", - "WriteTo": [ - { - "Name": "Console", - "Args": { - "outputTemplate": "[{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz}] [{Level:u3}] [{SourceContext}] {Message:lj}{NewLine}{Exception}" - } - }, - { - "Name": "File", - "Args": { - "path": "/app/Logs/melodee-.log", - "rollingInterval": "Day", - "retainedFileCountLimit": 30 - } - } - ] - } -} -``` +Changing the JWT key invalidates existing access tokens. Preserve the key in +backups and rotate it as a planned security operation. -### Cache Configuration - -```json -{ - "Caching": { - "DefaultCacheTimeMinutes": 60, - "MemoryCacheSizeLimit": 100000000, - "Redis": { - "Enabled": false, - "ConnectionString": "localhost:6379" - } - } -} -``` +## CORS and Forwarded Headers -### Job Configuration - -```json -{ - "Jobs": { - "ScanInboundIntervalMinutes": 60, - "MetadataRefreshIntervalHours": 24, - "ArtworkRefreshIntervalHours": 168, - "CleanupIntervalHours": 24 - } -} -``` +| Key | Default | Notes | +|-----|---------|-------| +| `Cors:AllowedOrigins` | Local development origins | Explicit browser origins; production uses an empty allowlist if none are configured | +| `UseForwardedHeaders` | `true` | Processes proxy forwarding headers | -### Transcoding Configuration - -```json -{ - "Transcoding": { - "Enabled": true, - "FFmpegPath": "ffmpeg", - "Presets": { - "LowQuality": "-b:a 128k", - "MediumQuality": "-b:a 256k", - "HighQuality": "-b:a 320k", - "Lossless": "-c:a copy" - } - } -} -``` +Example production origin: -### Plugin Configuration - -```json -{ - "Plugins": { - "MetadataProviders": { - "MusicBrainz": { - "Enabled": true, - "CachePath": "/app/storage/_search-engines/musicbrainz" - }, - "LastFm": { - "Enabled": false, - "ApiKey": "", - "ApiSecret": "" - }, - "Spotify": { - "Enabled": false, - "ClientId": "", - "ClientSecret": "" - }, - "Itunes": { - "Enabled": true - }, - "Deezer": { - "Enabled": true - }, - "MetalApi": { - "Enabled": false - } - } - } -} +```text +Cors__AllowedOrigins__0=https://music.example.com ``` -### Security Configuration - -```json -{ - "Security": { - "RateLimiting": { - "Enabled": true, - "RequestsPerMinute": 100 - }, - "Blacklist": { - "Enabled": true, - "CheckEmail": true, - "CheckIP": true - } - } -} -``` +Origins are scheme, host, and optional port—not URL paths. Do not use `*` with +credentialed browser requests. -## UI Configuration Options +## Rate Limiting -### Theme Configuration +Native API defaults: -The web UI supports various theming options that can be configured through the application settings: +| Key | Default | +|-----|---------| +| `RateLimiting:MelodeeApi:TokenLimit` | `30` | +| `RateLimiting:MelodeeApi:QueueLimit` | `10` | +| `RateLimiting:MelodeeApi:ReplenishmentPeriodSeconds` | `30` | +| `RateLimiting:MelodeeApi:TokensPerPeriod` | `30` | +| `RateLimiting:MelodeeApi:AutoReplenishment` | `true` | +| `RateLimiting:MelodeeAuth:TokenLimit` | `10` | +| `RateLimiting:MelodeeAuth:QueueLimit` | `5` | +| `RateLimiting:MelodeeAuth:ReplenishmentPeriodSeconds` | `60` | +| `RateLimiting:MelodeeAuth:TokensPerPeriod` | `10` | +| `RateLimiting:MelodeeAuth:AutoReplenishment` | `true` | -```json -{ - "Theme": { - "DefaultTheme": "dark", - "Themes": ["light", "dark", "auto"], - "PrimaryColor": "#3498db", - "SecondaryColor": "#2ecc71" - } -} -``` +Authentication limits must remain at least as strict as general API limits; +startup validation rejects inconsistent values. -### Feature Flags - -```json -{ - "Features": { - "EnableAdvancedSearch": true, - "EnableUserPlaylists": true, - "EnableSharing": false, - "EnablePublicAccess": false - } -} -``` +Jellyfin compatibility defaults: -## File-Based Configuration - -In addition to environment variables, Melodee supports configuration through `appsettings.json` files. The configuration hierarchy is: - -1. `appsettings.json` (base configuration) -2. `appsettings.{Environment}.json` (environment-specific overrides) -3. Environment variables (highest priority) - -### Example Configuration File - -```json -{ - "ConnectionStrings": { - "DefaultConnection": "Host=localhost;Port=5432;Database=melodeedb;Username=melodeeuser;Password=yourpassword;Pooling=true;MinPoolSize=10;MaxPoolSize=50;" - }, - "Logging": { - "LogLevel": { - "Default": "Information", - "Microsoft.AspNetCore": "Warning" - } - }, - "AllowedHosts": "*", - "Jwt": { - "Key": "your-256-bit-secret-key-here-must-be-exactly-32-bytes-long", - "Issuer": "melodee", - "Audience": "melodee-clients", - "ExpireMinutes": 15 - }, - "Auth": { - "SelfRegistrationEnabled": true, - "Google": { - "Enabled": false, - "ClientId": "", - "ClientSecret": "", - "AllowedHostedDomains": [], - "AutoLinkEnabled": false - }, - "Tokens": { - "AccessTokenLifetimeMinutes": 15, - "RefreshTokenLifetimeDays": 30, - "MaxSessionDays": 90 - } - }, - "Storage": { - "StoragePath": "/app/storage", - "InboundPath": "/app/inbound", - "StagingPath": "/app/staging", - "UserImagesPath": "/app/user-images", - "PlaylistsPath": "/app/playlists" - } -} -``` +| Key | Default | +|-----|---------| +| `Jellyfin:RateLimit:ApiTokenLimit` | `200` | +| `Jellyfin:RateLimit:ApiPeriodSeconds` | `60` | +| `Jellyfin:RateLimit:AuthTokenLimit` | `10` | +| `Jellyfin:RateLimit:AuthPeriodSeconds` | `60` | +| `Jellyfin:RateLimit:StreamConcurrentLimit` | `10` | -## Configuration Best Practices +## Custom Blocks -### Security +| Key | Default | Description | +|-----|---------|-------------| +| `CustomBlocks:Enabled` | `true` | Loads configured Markdown blocks | +| `CustomBlocks:MaxBytes` | `262144` | Maximum source file size (256 KiB) | +| `CustomBlocks:CacheSeconds` | `30` | In-memory cache duration; `0` disables caching | -- Always use strong passwords for database access -- Store sensitive configuration (like JWT keys) in environment variables or secure vaults -- Use HTTPS in production environments -- Regularly rotate API keys and JWT signing keys +See [Custom Blocks](/custom-blocks/) for valid slots and the HTML sanitizer +allowlist. -### Performance +## Scheduler, Logging, and Host Settings -- Adjust database connection pool sizes based on expected load -- Use SSD storage for database volumes -- Configure appropriate caching strategies -- Set realistic job scheduling intervals based on your library size +| Key | Default | Description | +|-----|---------|-------------| +| `QuartzDisabled` | `false` | Disables all Quartz scheduling when `true` | +| `ASPNETCORE_ENVIRONMENT` | `Production` | Selects environment-specific appsettings | +| `AllowedHosts` | `*` | ASP.NET Core host filtering | +| `Serilog:*` | See `appsettings.json` | Console and rolling compact-JSON file logging | -### Maintenance +Scheduled jobs are individually controlled by database-backed `jobs.*` +settings. Clearing a job's cron expression prevents it from being scheduled. -- Regularly review and update configuration as your library grows -- Monitor log levels to balance debugging information with performance -- Test configuration changes in a staging environment when possible -- Document custom configurations for backup and recovery purposes +## Library-Path Startup Overrides -## Troubleshooting Configuration Issues +The application recognizes these optional variables when they are explicitly +passed to the process: -### Common Issues +| Variable | Default container library path | +|----------|--------------------------------| +| `MELODEE_STORAGE_PATH` | `/app/storage/` | +| `MELODEE_INBOUND_PATH` | `/app/inbound/` | +| `MELODEE_STAGING_PATH` | `/app/staging/` | +| `MELODEE_USER_IMAGES_PATH` | `/app/user-images/` | +| `MELODEE_PLAYLISTS_PATH` | `/app/playlists/` | +| `MELODEE_TEMPLATES_PATH` | `/app/templates/` | + +The supplied Compose file mounts these default paths but does not inject the +optional override variables. + +## Database-Backed Application Settings + +The complete list evolves with the schema and is available from the running +version: + +```bash +./mcli configuration list +./mcli configuration list --filter 'podcast.*' +./mcli configuration get streaming.maxConcurrentStreams.perUser +``` + +To override a database setting with an environment variable, replace periods +with underscores. Matching is case-insensitive: + +```text +system.baseUrl -> SYSTEM_BASEURL +streaming.useBufferedResponses -> STREAMING_USEBUFFEREDRESPONSES +streaming.maxConcurrentStreams.global -> STREAMING_MAXCONCURRENTSTREAMS_GLOBAL +podcast.http.allowHttp -> PODCAST_HTTP_ALLOWHTTP +searchEngine.spotify.apiKey -> SEARCHENGINE_SPOTIFY_APIKEY +``` -1. **Database Connection Issues**: - - Verify `DB_PASSWORD` is set correctly - - Check that PostgreSQL is accessible - - Confirm connection string format +Relevant setting groups and their dedicated guides: -2. **File Path Issues**: - - Ensure all configured paths exist and are writable - - Check volume mounts in containerized deployments - - Verify permissions for configured directories +- `jobs.*`: [Background Jobs](/jobs/) +- `jukebox.*`, `mpv.*`, `mpd.*`: [Jukebox](/jukebox/) +- `podcast.*`: [Podcasts](/podcasts/) +- `scripting.*`: [Event Scripting](/scripting/) +- `theme.*`, `system.defaultTheme`: [Theming](/theming/) +- `userDeviceProfile.enabled`: [User Device Profiles](/user-device-profiles/) -3. **Authentication Issues**: - - Confirm JWT key is properly formatted (32+ bytes) - - Verify OAuth provider settings are correct - - Check that authentication providers are enabled +## CLI-Specific Variables -### Configuration Validation +| Variable | Purpose | +|----------|---------| +| `MELODEE_APPSETTINGS_PATH` | Local-mode CLI settings file | +| `MELODEE_ENVIRONMENT` | CLI environment fallback | +| `MELODEE_SERVER` | Remote-mode server origin | +| `MELODEE_TOKEN` | Remote-mode JWT bearer token | +| `MELODEE_PROFILE` | Remote-mode profile name | -When making configuration changes: -1. Restart the application to apply changes -2. Check application logs for configuration-related errors -3. Test API endpoints to verify functionality -4. Validate that scheduled jobs are running as expected +See [CLI](/cli/) and [CLI Remote Server Mode](/cli-remote-mode/) before storing +tokens in profiles or automation. diff --git a/docs/pages/configuration.md b/docs/pages/configuration.md index 6dbaa2015..4c3b53395 100644 --- a/docs/pages/configuration.md +++ b/docs/pages/configuration.md @@ -1,193 +1,165 @@ --- title: Configuration +description: Configure Melodee host settings, persistent application settings, libraries, providers, jobs, and security. permalink: /configuration/ +tags: + - configuration + - administration + - security --- # Configuration -Melodee exposes configuration through environment variables, the web UI, and (internally) a dynamic settings registry. This page outlines common areas to tune. +Melodee has two configuration layers: -For a comprehensive reference of all configuration options, see the [Configuration Reference](/configuration-reference/) page. +1. **Host configuration** controls startup concerns such as connection strings, + JWT signing, CORS, rate limiting, and logging. It comes from + `appsettings.json` and standard .NET environment variables. +2. **Application settings** control runtime behavior such as ingestion, + providers, jobs, streaming, podcasts, Jukebox, themes, and scripting. They + are stored in PostgreSQL and managed under **Admin > Settings**. -## Configuration Sources +Environment variables take precedence in both layers. A setting supplied by an +environment variable appears read-only in normal administration workflows and +requires a container or service restart to change. -Priority (highest wins): +## First-Run Checklist -1. Environment variables (.env / container env) -2. UI overrides (persisted in database) -3. Default appsettings & internal defaults +After registering the initial administrator: -## Core Categories +1. Set `system.baseUrl` to the origin clients use, including `https://` and any + non-default port. +2. Verify every path under **Admin > Libraries**. +3. Run **Admin > Doctor** and resolve failed required checks. +4. Review registration, download, streaming, and sharing settings before + exposing the server outside your LAN. +5. Configure only the metadata providers you intend to use and supply their + credentials. +6. Review job schedules before importing a large library. -### Server & Network +## Libraries and Paths -- MELODEE_PORT: External port to expose web & API. -- BASE_URL (planned): Canonical base URL for reverse proxy setups. +The default container installation creates Inbound, Staging, Storage, User +Images, Playlist, Templates, Podcasts, and Themes libraries. The tracked +Compose file mounts their default `/app/...` paths. -### Database +You can override these library paths at startup by passing the following +variables into the application container: -Provided via compose. Ensure DB_PASSWORD is strong. For external Postgres, set connection string variables (coming doc expansion). +| Variable | Library | +|----------|---------| +| `MELODEE_INBOUND_PATH` | Inbound | +| `MELODEE_STAGING_PATH` | Staging | +| `MELODEE_STORAGE_PATH` | First Storage library | +| `MELODEE_USER_IMAGES_PATH` | User Images | +| `MELODEE_PLAYLISTS_PATH` | Playlist Data | +| `MELODEE_TEMPLATES_PATH` | Templates | -### Libraries +The default `compose.yml` does not pass these optional variables. Add them to a +Compose override's `environment` section and ensure matching volume mounts +exist. Paths inside the container—not host paths—belong in these variables. -Three logical areas: +## Application Settings -- Inbound: Raw, unprocessed files. -- Staging: Processed, awaiting review & metadata edits. -- Storage: Published canonical library (served to users / APIs). +Use **Admin > Settings** for interactive changes. The CLI provides an auditable +alternative: -Ensure sufficient disk and backup strategy, especially for storage volume. +```bash +# Find settings by wildcard +./mcli configuration list --filter 'streaming.*' -### Metadata & Enrichment - -Providers: MusicBrainz (local cache), Last.FM, Spotify, iTunes, Brave Search. - -Typical options (UI section): - -- Enable/disable provider. -- API keys / tokens (Spotify, Last.FM, Brave Search). -- Artwork size preferences. -- Local MusicBrainz database refresh interval. - -#### Brave Search API Configuration - -To enable image search via Brave Search API, you'll need to: - -1. Get a free API key from [Brave Search API](https://brave.com/search/api/) -2. Set the following environment variables: - ``` - BRAVE_SEARCH__ENABLED=true - BRAVE_SEARCH__APIKEY=your_brave_api_key_here - ``` -3. Optionally configure base URL and search path (defaults are usually fine): - ``` - BRAVE_SEARCH__BASEURL=https://api.search.brave.com - BRAVE_SEARCH__IMAGESEARCHPATH=/res/v1/images/search - ``` - -Brave Search provides high-quality image results for both artist portraits and album covers. - -### Ingestion Rules - -Rule engine applies deterministic transformations: - -- Remove "(feat. X)" from title -> moves featured artist into artist metadata fields. -- Normalize numbering (track 1 -> 01 or 1 based on style). -- Strip stray unicode punctuation / marketing phrases. -- Enforce required tags (Album, Artist, Title, Track, Duration) else item stays in staging. - -### Transcoding & Streaming - -Settings (some forthcoming in UI): - -- Preferred output format (Opus / MP3) for constrained bandwidth clients. -- Max concurrent streams per user (enforced by streaming limiter service). -- Buffered vs direct streaming (SettingRegistry.StreamingUseBufferedResponses). Buffered is safer for some reverse proxies; direct is lower latency. - -### Jobs & Scheduling - -Jobs run on cron‑like schedules (scan inbound, stage promotion, metadata refresh). Adjust intervals balancing freshness vs resource usage. - -### Security - -- First user is admin—create additional non‑admin accounts for daily use. -- API keys are GUIDs associated with users; rotate by regenerating user key if compromised. -- Blacklist service can deny by email or IP (used to mitigate abuse). - -### Logging - -Structured logging via Serilog. Configure sinks (console, file, etc.) in appsettings or environment overrides (documentation forthcoming for custom sinks). - -## Environment Variable Examples +# Read one setting +./mcli configuration get system.baseUrl +# Change one setting +./mcli configuration set system.baseUrl https://music.example.com ``` -DB_PASSWORD=supersecret -MELODEE_PORT=8080 -# FUTURE (illustrative): -# STREAMING_USE_BUFFERED=true -# MAX_CONCURRENT_STREAMS=3 -``` - -## Observability & Metrics - -System statistics endpoint (native API) surfaces counts (songs, albums, artists, etc.) — see /api/ for details. Future metrics (transcoding time, cache hit rates) planned. - -## Homelab Security & Optimization -### Reverse Proxy Setup - -For homelab deployments, it's strongly recommended to put Melodee behind a reverse proxy: - -**Nginx Example:** -```nginx -server { - listen 443 ssl http2; - server_name music.yourdomain.com; +The live Settings page and `mcli configuration list` are the authoritative list +for the installed version. Major categories include: + +| Prefix | Purpose | +|--------|---------| +| `conversion.*`, `transcoding.*` | Ingestion conversion and stream output | +| `imaging.*` | Artwork sizes, limits, and embedded images | +| `jobs.*` | Quartz cron schedules | +| `jukebox.*`, `mpv.*`, `mpd.*` | Server-side playback | +| `magic.*`, `processing.*`, `validation.*` | Ingestion cleanup and validation | +| `podcast.*` | Feed security, download limits, quotas, and retention | +| `scrobbling.*` | Internal and Last.fm scrobbling | +| `scripting.*` | Event-script enablement and assignments | +| `searchEngine.*` | Metadata providers and local search databases | +| `security.*`, `register.*` | Secrets, password reset, registration, and blacklists | +| `streaming.*` | Response buffering and concurrency limits | +| `system.*` | Public URL, site name, downloads, and uploads | +| `theme.*` | Theme library behavior and validation | + +Quote cron expressions and JSON-like values in the shell. Before bulk changes, +create a redacted export: + +```bash +./mcli backup export --output melodee-settings.json --redact-secrets +``` - ssl_certificate /path/to/certificate.crt; - ssl_certificate_key /path/to/private.key; +## Environment Overrides for Application Settings - location / { - proxy_pass http://127.0.0.1:8080; - proxy_set_header Host $host; - proxy_set_header X-Real-IP $remote_addr; - proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; - proxy_set_header X-Forwarded-Proto $scheme; +For database-backed application settings, Melodee maps underscores in an +environment-variable name to periods and compares keys without case +sensitivity. Examples: - # Increase buffer sizes for media streaming - proxy_buffer_size 128k; - proxy_buffers 4 256k; - proxy_busy_buffers_size 256k; - } -} +```text +SYSTEM_BASEURL=https://music.example.com +STREAMING_MAXCONCURRENTSTREAMS_PERUSER=3 +SEARCHENGINE_BRAVE_ENABLED=true +SEARCHENGINE_BRAVE_APIKEY=replace-with-secret ``` -**Traefik Example:** -```yaml -labels: - - "traefik.enable=true" - - "traefik.http.routers.melodee.rule=Host(`music.yourdomain.com`)" - - "traefik.http.routers.melodee.tls=true" - - "traefik.http.routers.melodee.entrypoints=websecure" - - "traefik.http.services.melodee.loadbalancer.server.port=8080" -``` +Compose only sends variables explicitly listed under a service's `environment` +or `env_file`. Values present in the host `.env` are available for Compose +substitution but are not automatically injected into the container. -### Performance Tuning for Homelabs +Example override: -**Database Connection Pool:** -``` -DB_MIN_POOL_SIZE=10 -DB_MAX_POOL_SIZE=50 +```yaml +services: + melodee.blazor: + environment: + - SYSTEM_BASEURL=https://music.example.com + - STREAMING_MAXCONCURRENTSTREAMS_PERUSER=3 + - SEARCHENGINE_BRAVE_ENABLED=true + - SEARCHENGINE_BRAVE_APIKEY=${BRAVE_API_KEY} ``` -**Media Processing:** -- Adjust `MaxConcurrentStreams` based on your hardware capabilities -- Configure transcoding quality settings based on your network bandwidth -- Set appropriate scan intervals to balance freshness with system load +Keep secrets out of tracked YAML. Use a protected `.env`, container secret +facility, or external secret manager appropriate to the deployment. -**Storage Optimization:** -- Mount media volumes to separate drives for performance -- Use SSDs for the database volume -- Consider separate volumes for different library types (lossless vs lossy) +## Reverse Proxy -### Hardening Checklist +When TLS terminates at a reverse proxy: -- Put behind a reverse proxy (nginx / Caddy) with TLS. -- Restrict inbound port to proxy layer only. -- Regularly export DB & storage backups. -- Monitor log volume for anomalous access. +- Set `system.baseUrl` to the external HTTPS origin. +- Preserve `Host`, `X-Forwarded-For`, and `X-Forwarded-Proto`. +- Support connection upgrades for Blazor Server and Party Mode. +- Disable response buffering for long audio streams if the proxy buffers by + default. +- Restrict direct access to the backend port when trusting forwarded headers. -## Updating Configuration +See [Homelab Deployment](/homelab/) for Nginx and Traefik examples. -1. Change value in UI or env. -2. Restart service if an env var (some dynamic settings reload automatically). -3. Observe logs for validation warnings. +## Apply and Validate Changes -## Coming Soon +Database-backed settings generally reset Melodee's configuration cache when +saved. Host settings and environment overrides require a restart: -- Editable YAML/JSON advanced config export/import. -- Live reload for transcoding profiles. -- Per‑user bandwidth caps. +```bash +docker compose restart melodee.blazor +docker compose logs --tail=100 melodee.blazor +curl --fail http://localhost:8080/health +``` -Have a config use‑case not covered? Open an issue so we can expand this section. +Run **Admin > Doctor** after changing paths, connection strings, credentials, or +external tools. Test a representative browse and stream request after changing +authentication, proxy, streaming, or transcoding settings. +See the [Configuration Reference](/configuration-reference/) for startup keys +and defaults. diff --git a/docs/pages/custom-blocks.md b/docs/pages/custom-blocks.md index 07b149f00..d765102ae 100644 --- a/docs/pages/custom-blocks.md +++ b/docs/pages/custom-blocks.md @@ -1,432 +1,152 @@ --- title: Custom Blocks +description: Add sanitized Markdown notices to the Melodee login, registration, password, and dashboard pages. permalink: /custom-blocks/ +tags: + - customization + - administration + - security --- # Custom Blocks -Custom Blocks allow you to inject custom HTML content into specific locations (slots) on Melodee pages. This feature enables you to add announcements, alerts, custom styling, tracking scripts, or any other HTML content without modifying the application code. +Custom Blocks let an administrator place small Markdown notices in predefined +locations without rebuilding Melodee. Files live beneath the Templates library, +are rendered to HTML, and pass through a strict HTML sanitizer before display. -## Overview +They are suitable for welcome text, maintenance notices, support links, and +simple styled banners. Scripts, embedded content, and arbitrary CSS are not +supported. -Custom Blocks are markdown files stored in the Templates library that get rendered as HTML and injected into designated page slots. They support: +## Available Keys -- Full HTML and markdown content -- Per-page customization -- Multiple slots per page -- Caching for performance -- Hot-reload during development +Melodee 2.2.0 renders these keys: -## Quick Start +| Key | Location | +|-----|----------| +| `login.top` | Above the login form | +| `login.bottom` | Below the login form | +| `register.top` | Above the registration form | +| `register.bottom` | Below the registration form | +| `forgot-password.top` | Above the password-reset request form | +| `forgot-password.bottom` | Below the password-reset request form | +| `reset-password.top` | Above the new-password form | +| `reset-password.bottom` | Below the new-password form | +| `dashboard.top` | At the top of the signed-in dashboard | -1. **Ensure Custom Blocks are enabled** in `appsettings.json`: - ```json - "CustomBlocks": { - "Enabled": true, - "MaxBytes": 10240, - "CacheSeconds": 30 - } - ``` +Creating a file for any other key has no effect unless that key is added to the +application in a future release. There are no global, dashboard bottom, or +dashboard announcement slots in 2.2.0. -2. **Create a custom block file** in the Templates library: - ``` - /storage/templates/custom-blocks/{page}/{slot}.md - ``` +## File Locations -3. **Add content** to the file (markdown or HTML) +A dotted key maps to directories and a Markdown filename beneath +`custom-blocks` in the Templates library: -4. **Refresh the page** to see your custom block - -## File Structure - -Custom blocks are stored in the Templates library under the `custom-blocks/` subdirectory: - -``` -/storage/templates/ -└── custom-blocks/ - ├── login/ - │ ├── top.md - │ └── bottom.md - ├── register/ - │ └── top.md - ├── dashboard/ - │ └── announcement.md - └── _global/ - └── header.md +```text +login.top -> custom-blocks/login/top.md +forgot-password.bottom -> custom-blocks/forgot-password/bottom.md +dashboard.top -> custom-blocks/dashboard/top.md ``` -### Naming Convention - -Files follow the pattern: `{page}/{slot}.md` - -- **Page**: The page identifier (e.g., `login`, `register`, `dashboard`) -- **Slot**: The slot identifier where content should appear (e.g., `top`, `bottom`, `announcement`) - -## Available Pages and Slots - -### Authentication Pages - -**Login Page** (`login`) -- `top` - Above the login form -- `bottom` - Below the login form - -**Register Page** (`register`) -- `top` - Above the registration form -- `bottom` - Below the registration form - -**Forgot Password Page** (`forgot-password`) -- `top` - Above the forgot password form -- `bottom` - Below the form - -**Reset Password Page** (`reset-password`) -- `top` - Above the reset password form -- `bottom` - Below the form - -### Application Pages - -**Dashboard** (`dashboard`) -- `top` - Above the dashboard content -- `announcement` - Announcement banner -- `bottom` - Below the dashboard content +The published Compose configuration mounts the Templates library at +`/app/templates/`, so the first example is normally: -### Adding New Slots - -To add custom blocks to other pages: - -1. Add the `` component to the page -2. Specify the page and slot names -3. Create the corresponding markdown file - -Example in a Razor page: -```razor - +```text +/app/templates/custom-blocks/login/top.md ``` -Then create: `/storage/templates/custom-blocks/mypage/top.md` - -## Content Format +If the Templates library path was changed in administration, use that path +instead. Keys and paths are lowercase and case-sensitive on Linux. Key segments +may contain lowercase letters, digits, and hyphens only. -Custom blocks support both Markdown and HTML: +## Create a Block -### Markdown Example +Create the required directories in the Templates volume, then add a `.md` file: ```markdown -## Welcome to Melodee! +## Scheduled maintenance -This is a **custom announcement** for all users. +Playback may be unavailable Saturday from **02:00-03:00 UTC**. -- Feature 1 is now available -- Check out the new [documentation](/docs) +[View service updates](https://status.example.com) ``` -### HTML Example +For the Compose deployment, one way to inspect the effective library is: -```html -
- Maintenance Notice: - System maintenance scheduled for Saturday at 2 AM UTC. -
- - +```bash +docker compose exec melodee.blazor \ + find /app/templates/custom-blocks -maxdepth 3 -type f -name '*.md' -print ``` -### Bootstrap Alerts - -Melodee uses Radzen components, but you can use standard Bootstrap-style classes: - -```html -
- ✅ New feature released! -
- -
- ⚠️ Please update your profile information. -
- -
- 🚨 Critical security update required. -
-``` +Use your normal volume-management workflow to create or edit files. The +application container only needs read access; avoid installing editors or +changing ownership inside a running production container. ## Configuration -Configure Custom Blocks in `appsettings.json`: +Custom Blocks use host configuration in `appsettings.json` or equivalent +environment variables: ```json { "CustomBlocks": { "Enabled": true, - "MaxBytes": 10240, + "MaxBytes": 262144, "CacheSeconds": 30 } } ``` -### Configuration Options - -| Option | Type | Default | Description | -|--------|------|---------|-------------| -| `Enabled` | boolean | `true` | Enable/disable Custom Blocks feature | -| `MaxBytes` | integer | `10240` | Maximum file size in bytes (10KB default) | -| `CacheSeconds` | integer | `30` | How long to cache blocks in memory | - -### Templates Library Path - -Custom blocks are stored in the Templates library. By default, this is `/storage/templates/`. - -To change the Templates library path: - -1. Go to **Settings** → **Libraries** -2. Edit the **Templates** library -3. Update the **Path** field -4. Custom blocks will be read from `{Path}/custom-blocks/` - -## Caching Behavior - -Custom blocks are cached in memory to improve performance: - -- **Cache Duration**: Configured by `CacheSeconds` (default 30 seconds) -- **Cache Key**: Based on page and slot name -- **Not Found**: Files that don't exist are NOT cached, so newly created files appear immediately -- **Updates**: Modified files will appear after the cache expires (wait 30 seconds or restart the application) - -### Development Mode - -During development, set `CacheSeconds` to `0` to disable caching: - -```json -"CustomBlocks": { - "CacheSeconds": 0 -} -``` - -This ensures you see changes immediately without waiting for cache expiration. - -## Examples - -### Login Page Announcement - -**File**: `/storage/templates/custom-blocks/login/top.md` - -```markdown -## 🎉 Welcome Back! - -We've recently updated our platform with exciting new features: - -- **New Music Discovery**: AI-powered recommendations -- **Enhanced Search**: Find music faster than ever -- **Mobile App**: Now available on iOS and Android - -[Learn More →](/changelog) -``` - -### Maintenance Banner +| Option | Default | Effect | +|--------|---------|--------| +| `Enabled` | `true` | Enables loading and rendering | +| `MaxBytes` | `262144` | Maximum size of one source file (256 KiB) | +| `CacheSeconds` | `30` | In-memory lifetime of a successfully loaded block | -**File**: `/storage/templates/custom-blocks/dashboard/announcement.md` +Environment variable names use double underscores, such as +`CustomBlocks__Enabled=false`. Restart the application after changing host +configuration. Missing files are not cached, so a newly created block can be +discovered immediately; an existing cached block can remain visible until its +cache entry expires. -```html -
- ⚠️ Scheduled Maintenance: - The system will undergo maintenance on Saturday, January 15th at 2:00 AM UTC. - Expected downtime: 2 hours. -
-``` +## Allowed Content -### Terms and Conditions Notice +Markdown formatting is supported. Raw HTML is accepted as input only where the +sanitizer permits it. The allowed element set includes headings, paragraphs, +lists, links, block quotes, `div`, `span`, `pre`, `code`, emphasis, rules, and +line breaks. -**File**: `/storage/templates/custom-blocks/register/bottom.md` +Allowed attributes are limited to `class`, `id`, `style`, and `href`. Links may +use `http`, `https`, or `mailto`. A restricted set of visual CSS properties is +allowed, including colors, backgrounds, borders, spacing, text formatting, and +basic dimensions. -```markdown ---- +The sanitizer removes content such as: -By registering, you agree to our [Terms of Service](/terms) and [Privacy Policy](/privacy). +- `script`, `style`, `iframe`, `object`, and `embed` elements; +- event attributes such as `onclick` and `onerror`; +- unsafe URL schemes; +- positioning and stacking properties not on the CSS allowlist. -For questions, contact support@melodee.org -``` - -### Analytics Script - -**File**: `/storage/templates/custom-blocks/dashboard/bottom.md` - -```html - -``` - -## Security Considerations - -⚠️ **Important Security Notes:** - -1. **Trusted Content Only**: Custom blocks render raw HTML. Only allow trusted administrators to create/edit these files. - -2. **No Input Validation**: Content is rendered as-is without sanitization. Malicious scripts can execute. - -3. **File System Access**: The Templates library directory should have restricted permissions. - -4. **Size Limits**: The `MaxBytes` setting prevents extremely large files, but doesn't validate content. - -**Recommendation**: Only enable Custom Blocks if you trust all users with access to the Templates library directory. +Do not use a Custom Block for analytics JavaScript, third-party widgets, +external stylesheets, forms, or executable code. ## Troubleshooting -### Custom Block Not Appearing - -1. **Check if feature is enabled**: - - Verify `CustomBlocks.Enabled` is `true` in `appsettings.json` - -2. **Verify file location**: - - Ensure file is in `/storage/templates/custom-blocks/{page}/{slot}.md` - - Check Templates library path in Settings → Libraries - -3. **Check file name**: - - File name must exactly match the page and slot names - - Names are case-sensitive - - Must have `.md` extension - -4. **Wait for cache expiration**: - - Default cache is 30 seconds - - Wait or restart the application - - Set `CacheSeconds` to `0` during development - -5. **Check file size**: - - File must be under `MaxBytes` limit (default 10KB) - - Large files are rejected silently - -6. **Check logs**: - - Application logs will show errors reading or parsing files - - Look for "CustomBlock" in log entries - -### File Size Too Large - -If you get an error about file size: - -1. Increase `MaxBytes` in configuration: - ```json - "CustomBlocks": { - "MaxBytes": 51200 - } - ``` - -2. Or reduce your content size - -### Content Not Updating - -If changes don't appear: - -1. **Wait for cache to expire** (default 30 seconds) -2. **Restart the application** to clear cache immediately -3. **Check file timestamp** - ensure the file was actually saved -4. **Set cache to 0** during development for instant updates - -### Templates Library Path Issues - -If blocks aren't loading, verify the Templates library: - -1. Go to **Settings** → **Libraries** -2. Find the **Templates** library (Type: Templates) -3. Verify the **Path** is correct -4. Check the path exists and has read permissions -5. Custom blocks should be under `{Path}/custom-blocks/` - -## FAQ - -### Where are custom blocks stored? - -Custom blocks are stored in the Templates library under the `custom-blocks/` subdirectory. By default, this is `/storage/templates/custom-blocks/`. - -### How do I change the storage location? - -Update the Templates library path in **Settings** → **Libraries**. Custom blocks will be read from `{NewPath}/custom-blocks/`. - -### Can I use custom CSS? - -Yes! You can include ` - -
- Welcome to Melodee! -
-``` - -### Can I use JavaScript? - -Yes, you can include `