Collect automatically created tasks. (#325)

Author: tobiasraabe

docs/source/changes.md

@@ -1,16 +1,20 @@
 # Changes
 
-This is a record of all past pytask releases and what went into them in reverse
+The document records all past pytask releases and what went into them in reverse
 chronological order. Releases follow [semantic versioning](https://semver.org/) and all
 releases are available on [PyPI](https://pypi.org/project/pytask) and
 [Anaconda.org](https://anaconda.org/conda-forge/pytask).
 
-## 0.2.7 - 2022-xx-xx
+## 0.2.7 - 2022-12-14
 
+- {pull}`307` adds Python 3.11 to the CI.
 - {pull}`308` replaces pydot with pygraphviz.
-- {pull}`318` Clarifies an example on nested dependencies and products.
+- {pull}`311` fixes a link in the documentation.
+- {pull}`311` adds refurb to pre-commit hooks.
+- {pull}`318` clarifies an example on nested dependencies and products.
 - {pull}`321` converts more choice options to enums.
 - {pull}`322` replaces SVGs with animations by termynal.
+- {pull}`325` allows to collect dynamically created tasks.
 
 ## 0.2.6 - 2022-10-27
 
@@ -24,16 +28,16 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`289` shortens the task ids when using `pytask collect`. Fixes {issue}`286`.
 - {pull}`290` implements a dry-run with `pytask --dry-run` to see which tasks would be
   executed.
-- {pull}`296` fixes a bug where the source code of wrapped function could not be
+- {pull}`296` fixes a bug where the source code of the wrapped function could not be
   retrieved.
 
 ## 0.2.4 - 2022-06-28
 
 - {pull}`279` enhances some tutorials with spell and grammar checking.
 - {pull}`282` updates the tox configuration.
 - {pull}`283` fixes an issue with coverage and tests using pexpect + `pdb.set_trace()`.
-- {pull}`285` implements that pytask does not show the traceback of tasks which are
-  skipped because its previous task failed. Fixes {issue}`284`.
+- {pull}`285` implements that pytask does not show the traceback of tasks that are
+  skipped because their previous task failed. Fixes {issue}`284`.
 - {pull}`287` changes that all files that are not produced by a task are displayed in
   the error message. Fixes {issue}`262`.
 
@@ -82,7 +86,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`237` publish more functions.
 - {pull}`238` allows any order of decorators with a `@pytask.mark.task` decorator.
 - {pull}`239` adds a warning on globals for parametrizations and some fixes.
-- {pull}`241` allows to parametrize over single dicts.
+- {pull}`241` allows parametrizing over single dicts.
 - {pull}`242` removes tasks from global {obj}`_pytask.task_utils.COLLECTED_TASKS` to
   avoid re-collection when the programmatic interface is used.
 - {pull}`243` converts choice options to use enums instead of simple strings.
@@ -112,7 +116,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 
 ## 0.1.8 - 2022-02-07
 
-- {pull}`210` allows `__tracebackhide__` to be a callable which accepts the current
+- {pull}`210` allows `__tracebackhide__` to be a callable that accepts the current
   exception as an input. Closes {issue}`145`.
 - {pull}`213` improves coverage and reporting.
 - {pull}`215` makes the help pages of the CLI prettier.
@@ -135,7 +139,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
 - {pull}`199` extends the error message when paths are ambiguous on case-insensitive
   file systems.
 - {pull}`200` implements the {func}`@pytask.mark.task <pytask.mark.task>` decorator to
-  mark functions as tasks regardless whether they are prefixed with `task_` or not.
+  mark functions as tasks regardless of whether they are prefixed with `task_` or not.
 - {pull}`201` adds tests for `_pytask.mark_utils`.
 - {pull}`204` removes internal traceback frames from exceptions raised somewhere in
   pytask.
@@ -149,7 +153,7 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
   in many places.
 - {pull}`185` fix issues with drawing a graph and adds the `--rank-direction` to change
   the direction of the DAG.
-- {pull}`186` enhance live displays by deactivating auto-refresh among other things.
+- {pull}`186` enhance live displays by deactivating auto-refresh, among other things.
 - {pull}`187` allows to enable and disable showing tracebacks and potentially different
   styles in the future with {confval}`show_traceback=True|False`.
 - {pull}`188` refactors some code related to {class}`_pytask.enums.ExitCode`.
@@ -170,10 +174,10 @@ releases are available on [PyPI](https://pypi.org/project/pytask) and
   information.
 - {pull}`173` replaces `ColorCode` with custom rich themes.
 - {pull}`174` restructures loosely defined outcomes to clear `enum.Enum`.
-- {pull}`176` and {pull}`177` implement a summary panel which holds aggregate
-  information about the number of successes, fails and other status.
-- {pull}`178` makes some stylistic changes like reducing tasks ids even more and dims
-  the path part.
+- {pull}`176` and {pull}`177` implement a summary panel that holds aggregate information
+  about the number of successes, fails and other status.
+- {pull}`178` adds stylistic changes like reducing tasks ids even more and dimming the
+  path part.
 - {pull}`180` fixes parsing relative paths from the configuration file.
 - {pull}`181` adds correct formatting of running tasks.
 - {pull}`182` introduces that only the starting year is displayed in the license

src/_pytask/console.py

@@ -172,12 +172,20 @@ def create_url_style_for_task(
     """Create the style to add a link to a task id."""
     url_scheme = _EDITOR_URL_SCHEMES.get(edtior_url_scheme, edtior_url_scheme)
 
-    info = {
-        "path": _get_file(task_function),
-        "line_number": _get_source_lines(task_function),
-    }
+    if not url_scheme:
+        return Style()
+
+    try:
+        info = {
+            "path": _get_file(task_function),
+            "line_number": _get_source_lines(task_function),
+        }
+    except (OSError, TypeError):
+        style = Style()
+    else:
+        style = Style(link=url_scheme.format(**info))
 
-    return Style() if not url_scheme else Style(link=url_scheme.format(**info))
+    return style
 
 
 def create_url_style_for_path(path: Path, edtior_url_scheme: str) -> Style:

src/_pytask/task_utils.py

@@ -52,7 +52,13 @@ def task(
 
     def wrapper(func: Callable[..., Any]) -> None:
         unwrapped = inspect.unwrap(func)
-        path = Path(inspect.getfile(unwrapped)).absolute().resolve()
+
+        raw_path = inspect.getfile(unwrapped)
+        if "<string>" in raw_path:
+            path = Path(unwrapped.__globals__["__file__"]).absolute().resolve()
+        else:
+            path = Path(raw_path).absolute().resolve()
+
         parsed_kwargs = {} if kwargs is None else kwargs
         parsed_name = name if isinstance(name, str) else func.__name__
 

tests/test_execute.py

@@ -351,3 +351,22 @@ def task_second(): pass
     assert ("Task task_example.py::task_second failed" in result.output) is (
         verbose == 2
     )
+
+
+@pytest.mark.end_to_end
+def test_that_dynamically_creates_tasks_are_captured(runner, tmp_path):
+    source = """
+    _DEFINITION = '''
+    def task_example():
+        pass
+    '''
+
+    exec(_DEFINITION)
+    """
+    tmp_path.joinpath("task_module.py").write_text(textwrap.dedent(source))
+
+    result = runner.invoke(cli, [tmp_path.as_posix()])
+
+    assert result.exit_code == ExitCode.OK
+    assert "task_example" in result.output
+    assert "Collect 1 task"

tests/test_task.py

@@ -354,8 +354,9 @@ def func(produces, content):
 
     result = runner.invoke(cli, [tmp_path.as_posix()])
 
-    assert result.exit_code == ExitCode.FAILED
-    assert "TypeError: module, class, method" in result.output
+    assert result.exit_code == ExitCode.OK
+    assert "Collected 1 task." in result.output
+    assert "1  Succeeded" in result.output
 
 
 @pytest.mark.end_to_end
@@ -403,3 +404,27 @@ def task_example():
     assert "task_example[1]" in result.output
     assert "task_example[hello]" in result.output
     assert "Collect 3 tasks"
+
+
+@pytest.mark.end_to_end
+def test_that_dynamically_creates_tasks_are_captured(runner, tmp_path):
+    source = """
+    import pytask
+
+    _DEFINITION = '''
+    @pytask.mark.task
+    def task_example():
+        pass
+    '''
+
+    for i in range(2):
+        exec(_DEFINITION)
+    """
+    tmp_path.joinpath("task_module.py").write_text(textwrap.dedent(source))
+
+    result = runner.invoke(cli, [tmp_path.as_posix()])
+
+    assert result.exit_code == ExitCode.OK
+    assert "task_example[0]" in result.output
+    assert "task_example[1]" in result.output
+    assert "Collect 2 tasks"