Store relative paths in the DB to make beets library portable

[snejus]

Migrate Item & Album Paths to Library-Relative Storage

Fixes: #133

Core Problem Solved

Before: Beets stored absolute file system paths in SQLite (e.g. /home/user/Music/Artist/album/track.mp3). This made library databases non-portable — moving the music directory or sharing a database across machines broke all path references.

After: Paths are stored relative to the music directory (e.g. Artist/album/track.mp3), and expanded back to absolute paths transparently on read. The database is now portable.

---

Architecture Changes

1. Context Variable for Music Directory (beets/context.py)

A new contextvars.ContextVar (_music_dir_var) holds the active music directory, set once during Library.__init__ via context.set_music_dir(). This avoids passing library.directory through every call stack.

Library.__init__()
    └─ context.set_music_dir(self.directory)   ← single write point
           ↓
    PathType.to_sql()  / PathType.from_sql()   ← read at DB layer
    pathutils.normalize_path_for_db()
    pathutils.expand_path_from_db()

2. Path Relativization Moved to the DB Layer (beets/dbcore/)

Previously, path conversion lived in Item._setitem / Item.__getitem__ — model-specific overrides. It is now pushed down into PathType.to_sql / PathType.from_sql in dbcore/types.py, through two helpers in the new beets/dbcore/pathutils.py:

Helper	Direction	Behaviour
normalize_path_for_db(path)	write	Strips music dir prefix → relative path
expand_path_from_db(path)	read	Prepends music dir → absolute path

All models using PathType (currently Item and Album) benefit automatically — no per-model overrides required.

3. PathQuery Updated for Relative Storage (beets/dbcore/query.py)

Queries like path:/home/user/Music/Artist now normalize the search term to its relative form before hitting the database, so SQL comparisons match stored values correctly. Both col_clause (SQL path) and match (in-memory path) use normalize_path_for_db.

4. One-Time Database Migration (RelativePathMigration)

Existing absolute paths in path and artpath columns are migrated on startup:

Library startup
    └─ _migrate()
           └─ RelativePathMigration._migrate_data(Item, Album)
                  └─ _migrate_field("path")
                  └─ _migrate_field("artpath")
                         ↓
                  Reads rows where field starts with b"/"
                  Writes os.path.relpath(path, music_dir) in batches

self.directory assignment was moved before super().__init__() in Library.__init__ so the migration can access the music dir when it runs.

5. Context Propagation to Background Threads

The music dir context variable must be available in worker threads (pipeline stages, replaygain pool). Two propagation points were added:

• beets/util/pipeline.py: Pipeline.run_parallel() snapshots the calling context with contextvars.copy_context() and passes a per-thread copy to each PipelineThread. Each stage coroutine is invoked via ctx.run(...).
• beetsplug/replaygain.py: Pool workers and their callbacks are wrapped in ctx.run(...) so expand_path_from_db works correctly inside the process pool.
• beets/util/__init__.py: par_map similarly propagates context into its thread pool workers.

---

Data Flow: Read & Write Path

item.path = "/home/user/Music/Artist/track.mp3"   ← absolute on write
      ↓
PathType.to_sql()
      ↓
normalize_path_for_db()  →  b"Artist/track.mp3"   ← stored in SQLite

item.path                                           ← absolute on read
      ↑
PathType.from_sql()
      ↑
expand_path_from_db()    ←  b"Artist/track.mp3"   ← fetched from SQLite

---

Key Invariants

• Public API unchanged: item.path always returns an absolute bytes path.
• Raw DB value is relative: direct SQL reads return the relative form (tests assert both).
• Paths outside the music dir are stored as-is (e.g. IPFS paths — see beetsplug/ipfs.py fix).
• Migration is idempotent: rows already relative (no leading /) are skipped.

[github-actions]

Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry.

[codecov]

Codecov Report

❌ Patch coverage is 91.60839% with 12 lines in your changes missing coverage. Please review.
✅ Project coverage is 70.67%. Comparing base (948535b) to head (a461ccb).
⚠️ Report is 13 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
beetsplug/ipfs.py	66.66%	2 Missing and 1 partial ⚠️
beets/dbcore/pathutils.py	93.10%	1 Missing and 1 partial ⚠️
beets/dbcore/query.py	80.00%	1 Missing and 1 partial ⚠️
beets/library/models.py	66.66%	1 Missing and 1 partial ⚠️
beets/util/pipeline.py	90.47%	1 Missing and 1 partial ⚠️
beetsplug/replaygain.py	87.50%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #6460      +/-   ##
==========================================
+ Coverage   70.56%   70.67%   +0.11%     
==========================================
  Files         148      150       +2     
  Lines       18812    18916     +104     
  Branches     3065     3078      +13     
==========================================
+ Hits        13274    13369      +95     
- Misses       4891     4896       +5     
- Partials      647      651       +4     

Files with missing lines	Coverage Δ
beets/context.py	100.00% <100.00%> (ø)
beets/dbcore/db.py	94.16% <100.00%> (+<0.01%)	⬆️
beets/dbcore/types.py	96.59% <100.00%> (+0.01%)	⬆️
beets/library/library.py	94.36% <100.00%> (+0.71%)	⬆️
beets/library/migrations.py	96.61% <100.00%> (+0.69%)	⬆️
beets/util/__init__.py	79.10% <100.00%> (+0.08%)	⬆️
beetsplug/replaygain.py	61.33% <87.50%> (+0.31%)	⬆️
beets/dbcore/pathutils.py	93.10% <93.10%> (ø)
beets/dbcore/query.py	91.09% <80.00%> (-0.25%)	⬇️
beets/library/models.py	86.91% <66.66%> (-0.20%)	⬇️
... and 2 more

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Copilot]

Pull request overview

PR make beets store item/album path in DB as library-relative, then expand back to absolute on read. Goal: portable DB when music folder move.

Changes:

• Add context var for active music dir, used by DB path encode/decode.
• Move path normalize/expand into dbcore types + query layer, plus add migration to rewrite old rows.
• Propagate context into thread pools/pipeline; update tests to expect absolute paths on public API.

Reviewed changes

Copilot reviewed 19 out of 19 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
beets/context.py	Add context var for music dir.
beets/library/library.py	Set music dir context early; register new migration.
beets/dbcore/pathutils.py	New helpers to normalize/expand paths for DB.
beets/dbcore/types.py	PathType encode/decode moved into DB type layer.
beets/dbcore/query.py	PathQuery now compares using DB-relative representation.
beets/library/models.py	Adjust non-PathQuery path field queries to match stored relative value.
beets/library/migrations.py	Add RelativePathMigration; add chunk size constant usage.
beets/dbcore/db.py	Add base migration chunk size constant.
beets/util/pipeline.py	Copy context into pipeline worker threads.
beets/util/__init__.py	Propagate context into par_map thread pool.
beetsplug/replaygain.py	Wrap pool work + callbacks in copied context.
beetsplug/ipfs.py	Create temp lib with /ipfs/ dir; store item paths relative to that root.
beets/ui/__init__.py	Override library DB path from BEETS_LIBRARY env var.
test/test_library.py	Update path expectations; assert raw stored relative paths.
test/test_query.py	Add query test for absolute input path.
test/library/test_migrations.py	Add migration test for absolute->relative rewrite.
test/ui/test_ui.py	Update ls -p output expectations to absolute paths.
test/ui/commands/test_list.py	Update $path output expectations to absolute paths.
test/plugins/test_ipfs.py	Normalize expected /ipfs/... path string.
test/test_files.py	Remove one remove/prune test (no replacement in this PR).

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 23 out of 23 changed files in this pull request and generated 4 comments.

[semohr]

I would love to find some time in the following days to look into this in detail!

Have you considered moving a library between operating systems yet? I could see that as one absolutly killer side effects for relative paths. Think portable external drive.

If you havent thought about it yet, we might want to test that paths are normalized correctly between Windows and Posix.

[snejus]

Have you considered moving a library between operating systems yet? I could see that as one absolutly killer side effects for relative paths.

I hope you mean it's a positive side effect! Well, this new logic consistently uses os.path for all operations, so it seems to me this should work fine.

Any ideas how could we possibly test it?

[semohr]

Yeah was meant in a positive way 😅

No idea how to tests it other than writting an example db in either operating system and trying to load it in the other.

Im pretty sure slashes will not be converted/normalized correctly by using os.path. I guess this is one of the reason Pathlib was born.

This is by no means a blocker tho but maybe a good followup enhancement. Just a thing I was wondering as it might be a personal usecase for me which required manually migrating/rewriting paths in the database beforehand.

[snejus]

Someone already responded that they will give it a try because they have access to both file systems.

I do want to get it right here before merging - if that means replacing os.path by pathlib then so be it. The only reason why I used os.path was that I wanted to avoid the conversion from bytes to str.

[snejus]

Now that I think about it, I think we should actually be able to test it - it's just a matter of patching os.system I think.

[snejus]

@semohr the library should now be portable between different file systems.

[snejus]

@semohr would you mind to approve it if you're happy with it?