Implement a portable lockfile. (#743)

Author: tobiasraabe

CHANGELOG.md

@@ -7,6 +7,10 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 
 ## Unreleased
 
+- {pull}`743` adds the `pytask.lock` lockfile as the primary state backend with a
+  portable format and documentation. When no lockfile exists, pytask reads the legacy
+  SQLite state and writes `pytask.lock`; `pytask build` continues updating the legacy
+  database for downgrade compatibility.
 - {pull}`787` makes the `attributes` field mandatory on `PNode` and
   `PProvisionalNode`, and preserves existing node attributes when loading entries from
   the data catalog.

docs/source/how_to_guides/index.md

@@ -13,6 +13,7 @@ maxdepth: 1
 ---
 migrating_from_scripts_to_pytask
 interfaces_for_dependencies_products
+portability
 remote_files
 functional_interface
 capture_warnings

docs/source/how_to_guides/portability.md

@@ -0,0 +1,88 @@
+# Portability
+
+This guide explains what you need to do to move a pytask project between machines and
+why the lockfile is central to that process.
+
+```{seealso}
+The lockfile format and behavior are documented in the
+[reference guide](../reference_guides/lockfile.md).
+```
+
+## How to port a project
+
+Use this checklist when you move a project to another machine or environment.
+
+1. **Update state once on the source machine.**
+
+   Run a normal build so `pytask.lock` is up to date:
+
+   ```console
+   $ pytask build
+   ```
+
+   If you already have a recent lockfile and up-to-date outputs, you can skip this step.
+
+1. **Ship the right files.**
+
+   Commit `pytask.lock` to your repository and move it with the project. In practice,
+   you should move:
+
+   - the project files tracked in version control (source, configuration, data inputs
+     and `pytask.lock`)
+   - the build artifacts you want to reuse (often in `bld/` if you follow the tutorial
+     layout)
+   - the `.pytask` folder in case you are using the data catalog and it manages some of
+     the files
+
+1. **Files outside the project**
+
+   If you have files outside the project root (the folder with the `pyproject.toml`
+   file), you need to make sure that the same relative layout exists on the target
+   machine.
+
+1. **Run pytask on the target machine.**
+
+   When states match, tasks are skipped. When they differ, tasks run and the lockfile is
+   updated.
+
+## What makes a project portable
+
+There are two things that must stay stable across machines:
+
+First, task and node IDs must be stable. An ID is the unique identifier that ties a task
+or node to an entry in `pytask.lock`. pytask builds these IDs from project-relative
+paths anchored at the project root, so most users do not need to do anything. If you
+implement custom nodes, make sure their IDs remain project-relative and stable across
+machines.
+
+Second, state values must be portable. The lockfile stores opaque state strings from
+`PNode.state()` and `PTask.state()`, and pytask uses them to decide whether a task is up
+to date. Content hashes are portable; timestamps or absolute paths are not. This mostly
+matters when you define custom nodes or custom hash functions.
+
+## Tips for stable state values
+
+- Prefer file content hashes over timestamps for custom nodes.
+- For `PythonNode` values that are not natively stable, provide a custom hash function.
+- Avoid machine-specific paths or timestamps in custom `state()` implementations.
+
+```{seealso}
+For custom nodes, see [Writing custom nodes](writing_custom_nodes.md).
+For hashing guidance, see
+[Hashing inputs of tasks](hashing_inputs_of_tasks.md).
+```
+
+## Cleaning up the lockfile
+
+`pytask.lock` is updated incrementally. Entries are only replaced when the corresponding
+tasks run. If tasks are removed or renamed, their old entries remain as stale data and
+are ignored.
+
+To clean up stale entries without deleting the file, run:
+
+```console
+$ pytask build --clean-lockfile
+```
+
+This rewrites the lockfile after a successful build with only the currently collected
+tasks and their current state values.

docs/source/how_to_guides/writing_custom_nodes.md

@@ -89,6 +89,13 @@ Here are some explanations.
   signature is a hash and a unique identifier for the node. For most nodes it will be a
   hash of the path or the name.
 
+- `signature` and lockfile `id` are different concepts.
+
+  - `signature` is the runtime identity in pytask's in-memory DAG.
+  - lockfile `id` is the portable key stored in `pytask.lock`.
+
+  For custom nodes, make sure the lockfile id stays stable and unique within a task.
+
 - The classmethod {meth}`~pytask.PickleNode.from_path` is a convenient method to
   instantiate the class.
 

docs/source/reference_guides/configuration.md

@@ -44,11 +44,13 @@ are welcome to also support macOS.
 
 ````{confval} database_url
 
-pytask uses a database to keep track of tasks, products, and dependencies over runs. By
-default, it will create an SQLite database in the project's root directory called
-`.pytask/pytask.sqlite3`. If you want to use a different name or a different dialect
-[supported by sqlalchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html#backend-specific-urls),
-use either {option}`pytask build --database-url` or `database_url` in the config.
+SQLite is the legacy state format. pytask uses `pytask.lock` as the primary state
+backend for change detection. When no lockfile exists, pytask reads the configured
+database and writes `pytask.lock`. For downgrade compatibility, `pytask build` also
+keeps the legacy database state updated.
+
+The `database_url` option remains for backward compatibility and controls the legacy
+database location and dialect ([supported by sqlalchemy](https://docs.sqlalchemy.org/en/latest/core/engines.html#backend-specific-urls)).
 
 ```toml
 database_url = "sqlite:///.pytask/pytask.sqlite3"

docs/source/reference_guides/index.md

@@ -9,6 +9,7 @@ maxdepth: 1
 ---
 command_line_interface
 configuration
+lockfile
 hookspecs
 api
 ```

docs/source/reference_guides/lockfile.md

@@ -0,0 +1,97 @@
+# The Lock File
+
+`pytask.lock` is the default state backend. It stores task state in a portable,
+git-friendly format so runs can be resumed or shared across machines.
+
+```{note}
+SQLite is the legacy format. When no lockfile exists, pytask reads the legacy database
+state and writes `pytask.lock`. The lockfile remains the primary backend for skip
+decisions, and `pytask build` also keeps the legacy database updated for downgrade
+compatibility.
+```
+
+## Example
+
+```toml
+# This file is automatically @generated by pytask.
+# It is not intended for manual editing.
+
+lock-version = "1"
+
+[[task]]
+id = "src/tasks/data.py::task_clean_data"
+state = "f9e8d7c6..."
+
+[task.depends_on]
+"data/raw/input.csv" = "e5f6g7h8..."
+
+[task.produces]
+"data/processed/clean.parquet" = "m3n4o5p6..."
+```
+
+## Behavior
+
+On each run, pytask:
+
+1. Reads `pytask.lock` (if present).
+1. Compares current dependency/product/task `state()` to stored `state`.
+1. Skips tasks whose states match; runs the rest.
+1. Updates `pytask.lock` after each completed task (atomic write).
+1. Updates `pytask.lock` after skipping unchanged tasks (unless `--dry-run` or
+   `--explain` are active).
+
+## Portability
+
+There are two portability concerns:
+
+1. **IDs**: Lockfile IDs must be project‑relative and stable across machines.
+1. **State values**: `state` is opaque; portability depends on each node’s `state()`
+   implementation. Content hashes are portable; timestamps are not.
+
+## Maintenance
+
+Use `pytask build --clean-lockfile` to rewrite `pytask.lock` with only currently
+collected tasks. The rewrite happens after a successful build and recomputes current
+state values without executing tasks again.
+
+## File Format Reference
+
+### Top-Level
+
+| Field          | Required | Description                      |
+| -------------- | -------- | -------------------------------- |
+| `lock-version` | Yes      | Schema version (currently `"1"`) |
+
+### Task Entry
+
+| Field        | Required | Description                   |
+| ------------ | -------- | ----------------------------- |
+| `id`         | Yes      | Portable task identifier      |
+| `state`      | Yes      | Opaque state string           |
+| `depends_on` | No       | Mapping from node id to state |
+| `produces`   | No       | Mapping from node id to state |
+
+### Dependency/Product Entry
+
+Node entries are stored as key-value pairs inside `depends_on` and `produces`, where the
+key is the node id and the value is the node state string.
+
+### IDs vs Signatures
+
+`id` in the lockfile is a portable identifier used to match entries across runs and
+machines. It is not the same as a node or task `signature` used internally in the DAG.
+
+- `signature`: runtime identity in the in-memory DAG.
+- `id`: portable lockfile key persisted to `pytask.lock`.
+
+When implementing custom nodes, keep lockfile IDs stable and unique within a task.
+
+## Version Compatibility
+
+Only lock-version `"1"` is supported. Older or newer versions error with a clear upgrade
+message.
+
+## Implementation Notes
+
+- The lockfile is encoded/decoded with `msgspec`’s TOML support.
+- Writes are atomic: pytask writes a temporary file and replaces `pytask.lock`.

docs/source/tutorials/making_tasks_persist.md

@@ -9,7 +9,7 @@ In this case, you can apply the {func}`@pytask.mark.persist <pytask.mark.persist
 decorator to the task, which will skip its execution as long as all products exist.
 
 Internally, the state of the dependencies, the source file, and the products are updated
-in the database such that the subsequent execution will skip the task successfully.
+in the lockfile such that the subsequent execution will skip the task successfully.
 
 ## When is this useful?
 

pyproject.toml

@@ -30,6 +30,7 @@ dependencies = [
     "pluggy>=1.3.0",
     "rich>=13.8.0",
     "sqlalchemy>=2.0.31",
+    "msgspec[toml]>=0.18.6",
     'tomli>=1; python_version < "3.11"',
     'typing-extensions>=4.8.0; python_version < "3.11"',
     "universal-pathlib>=0.2.2",
@@ -54,7 +55,7 @@ docs = [
     "matplotlib>=3.5.0",
     "myst-parser>=3.0.0",
     "myst-nb>=1.2.0",
-    "sphinx>=7.0.0",
+    "sphinx>=7.0.0,<9.0.0",
     "sphinx-click>=6.0.0",
     "sphinx-copybutton>=0.5.2",
     "sphinx-design>=0.3",
@@ -138,6 +139,9 @@ ignore = [
 "tests/test_capture.py" = ["T201", "PT011"]
 "tests/*" = ["ANN", "D", "FBT", "PLR2004", "S101"]
 "tests/test_jupyter/*" = ["INP001"]
+"tests/_test_data/*" = ["INP001"]
+"tests/_test_data/*/*" = ["INP001"]
+"tests/_test_data/*/*/*" = ["INP001"]
 "scripts/*" = ["D", "INP001"]
 "docs/source/conf.py" = ["D401", "INP001"]
 "docs_src/*" = ["ARG001", "D", "INP001", "S301"]

src/_pytask/build.py

@@ -72,6 +72,7 @@ def build(  # noqa: PLR0913
     debug_pytask: bool = False,
     disable_warnings: bool = False,
     dry_run: bool = False,
+    clean_lockfile: bool = False,
     editor_url_scheme: Literal["no_link", "file", "vscode", "pycharm"]  # noqa: PYI051
     | str = "file",
     explain: bool = False,
@@ -121,6 +122,8 @@ def build(  # noqa: PLR0913
         Whether warnings should be disabled and not displayed.
     dry_run
         Whether a dry-run should be performed that shows which tasks need to be rerun.
+    clean_lockfile
+        Whether the lockfile should be rewritten to only include collected tasks.
     editor_url_scheme
         An url scheme that allows to click on task names, node names and filenames and
         jump right into you preferred editor to the right line.
@@ -189,6 +192,7 @@ def build(  # noqa: PLR0913
             "debug_pytask": debug_pytask,
             "disable_warnings": disable_warnings,
             "dry_run": dry_run,
+            "clean_lockfile": clean_lockfile,
             "editor_url_scheme": editor_url_scheme,
             "explain": explain,
             "expression": expression,
@@ -305,6 +309,12 @@ def build(  # noqa: PLR0913
     default=False,
     help="Execute a task even if it succeeded successfully before.",
 )
+@click.option(
+    "--clean-lockfile",
+    is_flag=True,
+    default=False,
+    help="Rewrite the lockfile with only currently collected tasks.",
+)
 @click.option(
     "--explain",
     is_flag=True,