Add pytask lock command (#860)

Author: tobiasraabe

.pre-commit-config.yaml

@@ -25,14 +25,21 @@ repos:
     -   id: python-no-log-warn
     -   id: text-unicode-replacement-char
 -   repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.15.11
+    rev: v0.15.12
     hooks:
       - id: ruff-format
       - id: ruff-check
 -   repo: https://github.com/astral-sh/uv-pre-commit
     rev: 0.11.7
     hooks:
       - id: uv-lock
+-   repo: local
+    hooks:
+    -   id: check-termynal-line-lengths
+        name: Check Termynal line lengths
+        entry: python scripts/check_termynal_line_lengths.py
+        language: python
+        files: ^docs/source/_static/md/.*\.md$
 -   repo: https://github.com/executablebooks/mdformat
     rev: 1.0.0
     hooks:
@@ -60,7 +67,7 @@ repos:
     -   id: nbstripout
         exclude: (docs)
 -   repo: https://github.com/crate-ci/typos
-    rev: v1
+    rev: v1.45.1
     hooks:
     -   id: typos
         exclude: (\.ipynb)

.readthedocs.yaml

@@ -19,7 +19,7 @@ build:
       - UV_PROJECT_ENVIRONMENT="${READTHEDOCS_VIRTUALENV_PATH}" uv sync --frozen --group docs
     build:
       html:
-        - UV_PROJECT_ENVIRONMENT="${READTHEDOCS_VIRTUALENV_PATH}" uv run --group docs zensical build
+        - UV_PROJECT_ENVIRONMENT="${READTHEDOCS_VIRTUALENV_PATH}" uvx --from rust-just just docs
     post_build:
       - mkdir -p "${READTHEDOCS_OUTPUT}/html"
       - cp -a docs/build/. "${READTHEDOCS_OUTPUT}/html/"

docs/source/_static/md/commands/build-options.md

@@ -2,7 +2,6 @@
 | ---------------------------------------------------------- | ------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
 | <code>-c, --config FILE</code>                             | -                                                                         | Path to configuration file.                                                                                                                   |
 | <code>--capture [fd\|no\|sys\|tee-sys]</code>              | <code>fd</code>                                                           | Per task capturing method.                                                                                                                    |
-| <code>--clean-lockfile</code>                              | <code>false</code>                                                        | Rewrite the lockfile with only currently collected tasks.                                                                                     |
 | <code>--database-url TEXT</code>                           | -                                                                         | Url to the database.                                                                                                                          |
 | <code>--debug-pytask</code>                                | <code>false</code>                                                        | Trace all function calls in the plugin framework.                                                                                             |
 | <code>--disable-warnings</code>                            | <code>false</code>                                                        | Disables the summary for warnings.                                                                                                            |

docs/source/_static/md/commands/command-list.md

@@ -4,5 +4,6 @@
 | [`clean`](clean.md)     | Clean the provided paths by removing files unknown to pytask. |
 | [`collect`](collect.md) | Collect tasks and report information about them.              |
 | [`dag`](dag.md)         | Create a visualization of the directed acyclic graph.         |
+| [`lock`](lock.md)       | Inspect and update recorded task state in the lockfile.       |
 | [`markers`](markers.md) | Show all registered markers.                                  |
 | [`profile`](profile.md) | Show information about resource consumption.                  |

docs/source/_static/md/lock-accept-dry-run.md

@@ -0,0 +1,16 @@
+<div class="termy">
+
+```console
+$ pytask lock accept -k train --dry-run
+
+<span class="termynal-dim">──────────────────────────</span> Start pytask session <span class="termynal-dim">──────────────────────────</span>
+Platform: win32 -- Python 3.12.0, pytask 0.5.3
+Root: C:\Users\pytask-dev\git\my_project
+Collected 2 tasks.
+<span class="termynal-success">Would accept recorded state for task_train.py::task_train.</span>
+<span class="termynal-success">Would accept recorded state for task_evaluate.py::task_evaluate.</span>
+
+<span class="termynal-dim">─────────────────────────────────────────────────────────────────────────</span>
+```
+
+</div>

docs/source/_static/md/lock-accept-interactive.md

@@ -0,0 +1,16 @@
+<div class="termy">
+
+```console
+$ pytask lock accept -k train
+<span class="termynal-dim">──────────────────────────</span> Start pytask session <span class="termynal-dim">──────────────────────────</span>
+Platform: win32 -- Python 3.12.0, pytask 0.5.3
+Root: C:\Users\pytask-dev\git\my_project
+Collected 2 tasks.
+# Accept recorded state for task_train.py::task_train? [y/N]: $ y
+# Accept recorded state for task_evaluate.py::task_evaluate? [y/N]: $ n
+<span class="termynal-success">Accept recorded state for task_train.py::task_train.</span>
+
+<span class="termynal-dim">─────────────────────────────────────────────────────────────────────────</span>
+```
+
+</div>

docs/source/_static/md/lock-clean.md

@@ -0,0 +1,17 @@
+<div class="termy">
+
+```console
+$ pytask lock clean
+<span class="termynal-dim">──────────────────────────</span> Start pytask session <span class="termynal-dim">──────────────────────────</span>
+Platform: win32 -- Python 3.12.0, pytask 0.5.3
+Root: C:\Users\pytask-dev\git\my_project
+Collected 2 tasks.
+# Remove recorded state for task_old.py::task_train? [y/N]: $ y
+# Remove recorded state for task_old.py::task_evaluate? [y/N]: $ y
+<span class="termynal-success">Remove recorded state for task_old.py::task_train.</span>
+<span class="termynal-success">Remove recorded state for task_old.py::task_evaluate.</span>
+
+<span class="termynal-dim">─────────────────────────────────────────────────────────────────────────</span>
+```
+
+</div>

docs/source/how_to_guides/index.md

@@ -10,6 +10,7 @@ specific tasks with pytask.
 - [Migrating From Scripts To Pytask](migrating_from_scripts_to_pytask.md)
 - [Interfaces For Dependencies Products](interfaces_for_dependencies_products.md)
 - [Portability](portability.md)
+- [Update the Lockfile to Match Project State](reconciling_lockfile_state.md)
 - [Remote Files](remote_files.md)
 - [Functional Interface](functional_interface.md)
 - [Capture Warnings](capture_warnings.md)

docs/source/how_to_guides/portability.md

@@ -17,13 +17,11 @@ Use this checklist when you move a project to another machine or environment.
 Run a normal build with [`pytask build`](../reference_guides/commands.md#pytask-build)
 so `pytask.lock` is up to date:
 
-````
 ```console
 $ pytask build
 ```
 
 If you already have a recent lockfile and up-to-date outputs, you can skip this step.
-````
 
 1. **Ship the right files.**
 
@@ -85,11 +83,10 @@ tasks run. If tasks are removed or renamed, their old entries remain as stale da
 are ignored.
 
 To clean up stale entries without deleting the file, run
-[`pytask build --clean-lockfile`](../reference_guides/commands.md#pytask-build--clean-lockfile):
+[`pytask lock clean`](../reference_guides/commands.md#pytask-lock-clean):
 
 ```console
-$ pytask build --clean-lockfile
+$ pytask lock clean
 ```
 
-This rewrites the lockfile after a successful build with only the currently collected
-tasks and their current state values.
+This removes lockfile entries for tasks which are no longer collected.

docs/source/how_to_guides/reconciling_lockfile_state.md

@@ -0,0 +1,123 @@
+# Update the Lockfile to Match Project State
+
+Use [`pytask lock`](../reference_guides/commands.md#pytask-lock) when the current files
+and outputs in the project are already correct, but the recorded state in `pytask.lock`
+needs to catch up. This can happen after refactoring task files, moving or renaming
+tasks, producing outputs outside of pytask, or deleting tasks.
+
+## Accept current files and outputs
+
+Use [`pytask lock accept`](../reference_guides/commands.md#pytask-lock-accept) when the
+current dependencies, products, and task definition are already correct and should
+become the new recorded state.
+
+Preview the changes without writing them with `--dry-run`:
+
+--8<-- "docs/source/_static/md/lock-accept-dry-run.md"
+
+Then accept the planned changes interactively:
+
+--8<-- "docs/source/_static/md/lock-accept-interactive.md"
+
+Add `--yes` to apply all planned changes without prompting:
+
+```console
+$ pytask lock accept -k train --yes
+```
+
+If no selectors are provided, `pytask lock accept` applies to all collected tasks in the
+provided paths.
+
+If selectors are provided with `-k` or `-m`, `accept` automatically includes the
+ancestors of the selected tasks. This is useful when you target a downstream task and
+want the accepted state to stay consistent with its upstream dependencies.
+
+```console
+$ pytask lock accept -k evaluate
+```
+
+In this example, `pytask` accepts `evaluate` and its ancestors. It does not
+automatically include descendants. If you want to accept a wider part of the DAG, widen
+the task selection yourself.
+
+```console
+$ pytask lock accept -k "train or evaluate"
+```
+
+If a selected task is missing a required dependency or product, the command fails
+instead of accepting incomplete state.
+
+Run a build afterwards to check that unchanged tasks are skipped according to the
+updated lockfile.
+
+```console
+$ pytask build
+```
+
+## Reset state for selected tasks
+
+Use [`pytask lock reset`](../reference_guides/commands.md#pytask-lock-reset) to remove
+recorded state for selected tasks when state was accepted too broadly or when specific
+tasks should be reconsidered from scratch.
+
+```console
+$ pytask lock reset -k train
+```
+
+Unlike `accept`, `reset` with a selector works on the exact selected tasks. It does not
+automatically include ancestors.
+
+Preview the reset with `--dry-run` if you want to check the affected tasks first:
+
+```console
+$ pytask lock reset -k train --dry-run
+```
+
+Add `--yes` to remove all planned entries without prompting:
+
+```console
+$ pytask lock reset -k train --yes
+```
+
+If no selectors are provided, `pytask lock reset` removes the recorded state for all
+collected tasks in the provided paths.
+
+Run a build afterwards so `pytask` determines again whether the selected tasks require
+execution.
+
+```console
+$ pytask build
+```
+
+## Remove stale entries for deleted or moved tasks
+
+Use [`pytask lock clean`](../reference_guides/commands.md#pytask-lock-clean) to remove
+entries from the lockfile which no longer correspond to collected tasks in the current
+project. This is useful after deleting, renaming, or moving tasks when old entries
+should no longer remain in the lockfile.
+
+Preview stale entries without writing them with `--dry-run`:
+
+```console
+$ pytask lock clean --dry-run
+```
+
+Then remove stale entries interactively:
+
+--8<-- "docs/source/_static/md/lock-clean.md"
+
+Add `--yes` to remove all stale entries without prompting:
+
+```console
+$ pytask lock clean --yes
+```
+
+`clean` only removes entries for tasks which are no longer collected. It does not accept
+or update the current state of collected tasks.
+
+## Related
+
+- [`pytask lock`](../reference_guides/commands.md#pytask-lock)
+- [`pytask build`](../reference_guides/commands.md#pytask-build)
+- [Portability](portability.md)
+- [The lockfile](../reference_guides/lockfile.md)