Skip to content

Commit 49f8658

Browse files
TTsangSCErotemic
andauthored
ENH: read TOML files for configurations (#335)
* Basic TOML facilities line_profiler/line_profiler_rc.toml Default configuration file line_profiler/toml_config.py[i] New module for reading TOML config files * Packaging updates MANIFEST.in, setup.py Configured to include `line_profiler/line_profiler_rc.toml` in source and wheel distributions requirements/runtime.txt Added dependency `tomli` for Python < 3.11 (to stand in for `tomllib`) * WIP: `kernprof` refactoring kernprof.py - Made all `line_profiler` imports unconditional (the ship has sailed, there's already an unconditional import for `line_profiler.profiler_mixin.ByCountProfilerMixin`) - For each boolean option (e.g. `--view`): - Added a corresponding negative option (e.g. `--no-view`) - Changed the default value from `False` to `None`, so that we can distinguish between cases where the negative option is passed and no option is passed (and in that case read from the config (TODO)) main(), find_module_script(), find_script() Added argument `exit_on_error` to optionally prevent parsing errors from killing the interpretor * `kernprof` refactoring (reading configs) kernprof.py __doc__ Updated with newest `kernprof --help` output short_string_path() New helper function for abbreviating paths _python_command() - Replaced string comparison between paths with `os.path.samefile()` - Updated to use abbreviated paths where possible main() - Updated description to include documentation for the negative options - Added option `--config` for loading config from a specific file instead of going through lookup - Updated `const` value for the bare `-i`/`--output-intereval` option (the old value 0, equivalent to not specifying the option, doesn't really make sense) - Grouped options into argument groups for better organization - Updated help texts for options to be more stylistically consistent and to show the default values - Updated help texts for the `-p`/`--prof-mod` option to show an example - Updated help texts for the `--prof-imports` to be more in line with what it actually does (see docstring of `line_profiler.autoprofile.ast_tree_profiler.AstTreeProfiler`) - Added option resolution: values of un-specified flags now taken from the specified/looked-up config file * Feature: read config file from the env line_profiler/toml_config.py[i] find_and_read_config_file() New argument `env_var` for controlling whether/whence to read the config from the environment if not specified via `config` get_config() New arguemnt `read_env` for controlling whether to read the config from the environment variable `${LINE_PROFILER_RC}` if specified via `config` * Reorganized code: `line_profiler.cli_utils` kernprof.py Moved common utilities to `line_profiler/cli_utils.py` line_profiler/cli_utils.py[i] New module for common utilities shared by `kernprof` and `python -m line_profiler` * Made `line_profiler.line_profiler` configurable kernprof.py::main() Now passing the received `-c`/`--config` onto `LineProfiler.print_stats()` line_profiler/line_profiler.py[i] <Overall> Formatting changes (avoid using hanging indents where suitable, to be more consistent with the rest of the codebase) LineProfiler.print_stats(), show_func(), show_text() - Added optional argument `config` to allow for specifying the config file - Now reading output column widths from the `tool.line_profiler.show.column_widths` table in config files main() - Refactored to use the `.cli_utils` - Added description for the CLI application - Added negative options to the boolean options - Added option `-c`/`--config` to read default values for the options from the `tool.line_profiler.cli` table line_profiler/line_profiler_rc.toml - Added documentation for the environment variable `${LINE_PROFILER_RC}` - Added subtables `tool.line_profiler.cli` and `tool.line_profiler.show.column_widths` * Refactored `line_profiler.toml_config` line_profiler.toml_config.py[i] get_subtable() - Added doctest - Added type check that the returned object is a mapping get_headers() New function for getting the subtable headers from a table get_config() - Updated docs with reference to the new `tool.line_profiler.cli` and `.show.column_widths` tables - Added check for subtable existence - Fixed traceback and error message when the table is malformed * Moved code around kernprof.py::_python_command Migrated definition to `line_profiler/cli_utils.py::get_python_executable()` line_profiler/cli_utils.py::get_python_executable() New function used by both `kernprof` and `line_profiler.explicit_profiler` * Made `.explicit_profiler` configurable line_profiler/explicit_profiler.py GlobalProfiler __doc__ Updated __init__() - Added argument `config` to allow for explicitly providing a config file - Now reading `.{setup,write,show}_config` and `.output_prefix` from the config file, instead of using hard-coded values show() Minor refatoring and reformatting _python_command() Now using `.cli_utils.get_python_executable()` line_profiler/line_profiler_rc.toml::[tool.line_profiler.write] Added item `output_prefix`, corresponding to `line_profiler.explicit_profiler.GlobalProfiler.output_prefix` * TOML tests line_profiler/toml_config.py::get_config() - Now promoting `config` to `pathlib.Path` objects early so as to catch bad arg types - Now raising `ValueError` or `FileNotFoundError` if a `config` is specified and loading fails tests/test_toml_config.py New test module for tests related to `line_profiler.toml_config`: - test_environment_isolation() Test that the fixture we use suffices to isolate the tests from the environment - test_default_config_deep_copy() Test that `get_default_config()` returns fresh, deep copies - test_table_normalization() Test that `get_config()` always normalizes the config entires to supply missing entires and remove spurious ones - test_malformed_table() Test that we get a `ValueError` from malformed TOML files - test_config_lookup_hierarchy() Test the hierarchy according to which we resolve which TOML to read the configs from * Config test in `tests/test_explicit_profile.py` tests/test_explicit_profile.py test_*() Updated to use `tempfile.TemporaryDirectory()` instead of `tempfile.mkdtemp()` so that the tmpdirs are cleaned up regardless of the test outcome test_explicit_profile_with_customized_config() New test for customizing explicit profiling with a user-supplied TOML config file * Config test in `tests/test_autoprofile.py` tests/test_autoprofile.py test_*() Updated to use `tempfile.TemporaryDirectory()` instead of `tempfile.mkdtemp()` so that the tmpdirs are cleaned up regardless of the test outcome test_autoprofile_with_customized_config() New test for customizing `kernprof` auto-profiling and `python -m line_profiler` output formatting with a user-supplied TOML config file * Added changelog entry * CI fixes * `line_profiler_rc.toml` -> `line_profiler.toml` line_profiler/explicit_profiler.py::GlobalProfiler Updated docstring line_profiler/toml_config.py::targets Changed lookup target from `line_profiler_rc.toml` -> `line_profiler.toml` tests/test_toml_config.py::test_config_lookup_hierarchy() Updated test to use `line_profiler.toml` instead of `line_profiler_rc.toml` * Updated comments in TOML file * Reduce code run during import-time line_profiler/line_profiler.py[i] minimum_column_widths Removed global constant get_minimum_column_widths() New cached callable for getting the above value * Added the `--no-config` flag kernprof.py __doc__ Updated with new `kernprof --help` output main() Added a `--no-config` flag for disabling the loading of non-default configs line_profiler/cli_utils.py[i]::get_cli_config() - Added the explicit named argument `config` - Added processing for boolean values of `config` (true -> default behavior, false -> fall back for `get_default_config()`) line_profiler/line_profiler.py[i] LineProfiler.print_stats(), show_func(), show_text() Added handling for boolean values of `config` main() Added a `--no-config` flag for disabling the loading of non-default configs tests/test_autoprofile.py test_autoprofile_with_customized_config() Fixed malformed indentation test_autoprofile_with_no_config() New test for disabling config lookup * Centralized code for `config = <bool>` line_profiler/cli_utils.py[i]::get_cli_config() Rolled back last commit line_profiler/explicit_profiler.py[i]::GlobalProfiler - Updated docstring - Added missing `config` argument to `.__init__()` in the stub file line_profiler/line_profiler.py Removed wrapper function around `line_profiler.toml_config.get_config()` line_profiler/toml_config.py[i]::get_config() Added handling for `config = <bool>`: - `False`: don't look up or load any user-supplied config and just use the default - `True`: same as `None` (default behavior) tests/test_toml_config.py::test_config_lookup_hierarchy() Now also testing `get_config(True)` and `get_config(False)` * Fixed MANIFEST * Better boolean-option parsing line_profiler/cli_utils.py[i] <General> Updated docstring to use double backticks for inlined code to be friendlier towards sphinx/RST add_argument() - Updated call signature: - `parser_like` now positional-only - Added `arg` between `parser_like` and `*args` to indicate that at least one positional argument should be passed to `parser_like.add_argument()` - Now adding separate actions for long and short boolean flags so that the long option can be specified in both no-arg and single-arg forms - Now skipping the generation of the complementary falsy action instead of raising an `AssertionError` when the boolean flag doesn't have a long form boolean() New function for parsing strings into booleans * `line_profiler` minor refactoring line_profiler/explicit_profiler.py::GlobalProfiler._implicit_setup() Updated parsing of environment variables to use `line_profiler.cli_utils.boolean()` line_profiler/line_profiler.py::main() - Removed note on boolean options in the parser description - Removed parenthetical remark "boolean option" from option help texts * Refactoring in `kernprof.py` kernprof.py __doc__ Updated with the latest `kernprof --help` output, and with a narrower window so that the lines aren't too long __doc__ RepeatedTimer.__doc__ Added linter-friendly `noqa` comment for long lines in docstrings meant for `sphinx` consumption RepeatedTimer.start() main() Wrapped certain long lines of code main() - Removed note on boolean options in parser description - Removed parenthetical remark "boolean option" in option help texts - Removed redundant instantiation of `RepeatedTimer` - Refactored chained if-elif-else when executing code in non-autoprofiling mode * Tests for `line_profiler.cli_utils.add_argument` tests/test_cli.py parser() New fixture (example parser) test_boolean_argument_help_text() New test for the help texts for boolean options generated by `line_profiler.cli_utils.add_argument()` test_boolean_argument_parsing() New test for the parsing of long and short boolean options * Lint * . * Bug fixes kernprof.py __doc__ Updated _add_core_parser_arguments() Updated help text of `-p`/`--prof-mod` _parse_arguments() - Fixed bug where the config file is stil looked up despite passing `--no-config` - Added logging output for the loading of configs _normalize_profiling_targets() Now allow for an empty string to invalidate previous targets (so that e.g. `--prof-mod ''` can be used to drop profiling targets specified in the `--config`) main(), _write_tempfile(), _write_preimports(), _pre_profile() Now using `short_string_path()` to abbreviate filenames in output * Add example of using TOML config to docs * Fix malformed f-string Co-authored-by: Jon Crall <erotemic@gmail.com> * Use `global` for accessing global vars Co-authored-by: Jon Crall <erotemic@gmail.com> * format and install fixes * small tweak * Refactoring of `line_profiler.toml_config` kernprof.py Updated implementations line_profiler/cli_utils.py[i] <General> Reformatted docstrings get_cli_config() Now returning a `ConfigSource` boolean() Added doctest line_profiler/explicit_profiler.py::GlobalProfiler - Reformatted docstring - Updated implementation line_profiler/line_profiler.py[i] get_column_widths() Refactored from `get_minimum_column_widths()` show_text() main() Updated implementations line_profiler/toml_config.py[i] <General> Reformatted docstrings NAMESPACE, TARGETS, ENV_VAR Renamed from lowercased constants ConfigSource New data class refactored from `get_config()` and `get_default_config()` tests/test_toml_config.py Updated implementations * Doc fixes - Fixed typos and links - Added doc pages for `line_profiler.toml_config` and `line_profiler.cli_utils` * CLI fixes kernprof.py Added flag `--summarize` as an analog for `python -m line_profiler --summarize` line_profiler/line_profiler.py Fixed bug where the wrong defaults are shown in the help texts of the `--sort` and `--summarize` flags line_profiler/rc/line_profiler.toml::[tool.line_profiler.kernprof] Added new boolean value `summarize` * `kernprof` fixes kernprof.py::_post_profile() - Added diagnostic debug message for the call to `LineProfiler.print_stats()` - Fixed bug where `options.summarize` is not passed to `.print_stats()` --------- Co-authored-by: joncrall <erotemic@gmail.com>
1 parent 9e09196 commit 49f8658

24 files changed

Lines changed: 2752 additions & 837 deletions

CHANGELOG.rst

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,7 @@ Changes
1111
* FIX: Fixed explicit profiling of class methods; added handling for profiling static, bound, and partial methods, ``functools.partial`` objects, (cached) properties, and async generator functions
1212
* FIX: Fixed namespace bug when running ``kernprof -m`` on certain modules (e.g. ``calendar`` on Python 3.12+).
1313
* FIX: Fixed ``@contextlib.contextmanager`` bug where the cleanup code (e.g. restoration of ``sys`` attributes) is not run if exceptions occurred inside the context
14-
* ENH: Added CLI arguments ``-c`` to ``kernprof`` for (auto-)profiling inline-script execution instead of that of script files; passing ``'-'`` as the script-file name now also reads from and profiles ``stdin``
14+
* ENH: Added CLI arguments ``-c`` to ``kernprof`` for (auto-)profiling module/package/inline-script execution instead of that of script files; passing ``'-'`` as the script-file name now also reads from and profiles ``stdin``
1515
* ENH: In Python >=3.11, profiled objects are reported using their qualified name.
1616
* ENH: Highlight final summary using rich if enabled
1717
* ENH: Made it possible to use multiple profiler instances simultaneously
@@ -43,6 +43,11 @@ Changes
4343
callbacks during profiling
4444
* Now allowing switching back to the "legacy" trace system on Python 3.12+,
4545
controlled by an environment variable
46+
* ENH: Added capability to parse TOML config files for defaults (#335):
47+
48+
* ``kernprof`` and ``python -m line_profiler`` CLI options
49+
* ``GlobalProfiler`` configurations, and
50+
* profiler output (e.g. ``LineProfiler.print_stats()``) formatting
4651

4752
4.2.0
4853
~~~~~

MANIFEST.in

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,4 @@
1+
graft line_profiler/rc
12
include *.md
23
include *.rst
34
include *.py
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
line\_profiler.cli\_utils module
2+
================================
3+
4+
.. automodule:: line_profiler.cli_utils
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:

docs/source/auto/line_profiler.rst

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,11 +17,13 @@ Submodules
1717

1818
line_profiler.__main__
1919
line_profiler._line_profiler
20+
line_profiler.cli_utils
2021
line_profiler.explicit_profiler
2122
line_profiler.ipython_extension
2223
line_profiler.line_profiler
2324
line_profiler.profiler_mixin
2425
line_profiler.scoping_policy
26+
line_profiler.toml_config
2527

2628
Module contents
2729
---------------
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
line\_profiler.toml\_config module
2+
================================
3+
4+
.. automodule:: line_profiler.toml_config
5+
:members:
6+
:undoc-members:
7+
:show-inheritance:
Lines changed: 221 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,221 @@
1+
Using the line-profiler TOML configuration
2+
------------------------------------------
3+
4+
This tutorial walks the user through setting up a toy Python project and then
5+
interacting with it via the new line-profiler TOML configuration.
6+
7+
First, we need to setup a small project, for which we will use ``uv``. We will
8+
also use the ``tomlkit`` package to edit the config file programatically. If
9+
you don't have these installed, first run:
10+
11+
.. code:: bash
12+
13+
pip install uv tomlkit
14+
15+
16+
Next, we are going to setup a small package for this demonstration.
17+
18+
.. code:: bash
19+
20+
TEMP_DIR=$(mktemp -d --suffix=demo_pkg)
21+
mkdir -p $TEMP_DIR
22+
cd $TEMP_DIR
23+
24+
uv init --lib --name demo_pkg
25+
26+
# helper to prevent indentation errors
27+
codeblock(){
28+
echo "$1" | python -c "import sys; from textwrap import dedent; print(dedent(sys.stdin.read()).strip('\n'))"
29+
}
30+
31+
codeblock "
32+
import time
33+
from demo_pkg.utils import leq
34+
from demo_pkg import utils
35+
36+
def fib(n):
37+
if leq(n, 1):
38+
return n
39+
part1 = fib(n - 1)
40+
part2 = fib(n - 2)
41+
result = utils.add(part1, part2)
42+
return result
43+
44+
def sleep_loop(n):
45+
for _ in range(n):
46+
time.sleep(0.01)
47+
" > src/demo_pkg/core.py
48+
49+
codeblock "
50+
def leq(a, b):
51+
return a <= b
52+
53+
def add(a, b):
54+
return a + b
55+
" > src/demo_pkg/utils.py
56+
57+
codeblock "
58+
from demo_pkg import core
59+
import uuid
60+
61+
def main():
62+
run_uuid = uuid.uuid4()
63+
print('The UUID of this run is', run_uuid)
64+
print('compute fib 10')
65+
result = core.fib(10)
66+
print('result', result)
67+
print('sleeping 5')
68+
core.sleep_loop(5)
69+
print('done')
70+
71+
if __name__ == '__main__':
72+
main()
73+
" > src/demo_pkg/__main__.py
74+
75+
# Run `uv pip install -e .` to install the project locally:
76+
uv pip install -e .
77+
78+
79+
Test that the main entrypoint works.
80+
81+
.. code:: bash
82+
83+
python -m demo_pkg
84+
85+
86+
Running kernprof with a main script that uses your package behaves as in 4.x in that no defaults are modified.
87+
88+
.. code:: bash
89+
90+
kernprof -m demo_pkg
91+
92+
93+
However, you can modify pyproject.toml to specify new defaults. After doing
94+
this, running kernprof will use defaults specified in your pyproject.toml (You
95+
may also pass ``--config`` to tell kernprof to use a different file to load the
96+
default config).
97+
98+
.. code:: bash
99+
100+
# Edit the `pyproject.toml` file to modify default behavior
101+
update_pyproject_toml(){
102+
python -c "if 1:
103+
import pathlib
104+
import tomllib
105+
import tomlkit
106+
import sys
107+
config_path = pathlib.Path('pyproject.toml')
108+
config = tomllib.loads(config_path.read_text())
109+
110+
# Add in new values
111+
from textwrap import dedent
112+
new_text = dedent(sys.argv[1])
113+
114+
new_parts = tomllib.loads(new_text)
115+
config.update(new_parts)
116+
117+
new_text = tomlkit.dumps(config)
118+
config_path.write_text(new_text)
119+
" "$1"
120+
}
121+
122+
update_pyproject_toml "
123+
# New Config
124+
[tool.line_profiler.kernprof]
125+
line-by-line = true
126+
rich = true
127+
verbose = true
128+
skip-zero = true
129+
prof-mod = ['demo_pkg']
130+
"
131+
132+
# Now, running kernprof uses the new defaults
133+
kernprof -m demo_pkg
134+
135+
136+
You will now see how long each function took, and what the line-by line breakdown is
137+
138+
.. code::
139+
140+
# line-by-line breakdown omitted here
141+
142+
0.05 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/__main__.py:4 - main
143+
0.00 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/core.py:5 - fib
144+
0.05 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/core.py:13 - sleep_loop
145+
0.00 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/utils.py:1 - leq
146+
0.00 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/utils.py:4 - add
147+
148+
149+
Note that by specifying ``prof-mod``, every function within the package is
150+
automatically profiled without any need for the ``@profile`` decorator.
151+
152+
It is worth noting, there is no requirement that the module you are profiling
153+
is part of your package. You can specify any module name as part of
154+
``prof-mod``. For example, lets profile the stdlib uuid module.
155+
156+
157+
.. code:: bash
158+
159+
update_pyproject_toml "
160+
# New Config
161+
[tool.line_profiler.kernprof]
162+
line-by-line = true
163+
rich = true
164+
verbose = 0
165+
skip-zero = true
166+
prof-mod = ['uuid']
167+
"
168+
169+
# Now, running kernprof uses the new defaults
170+
kernprof -m demo_pkg
171+
python -m line_profiler -rmtz demo_pkg.lprof
172+
173+
174+
This results in only showing calls in the uuid package:
175+
176+
.. code::
177+
178+
# line-by-line breakdown omitted here
179+
180+
0.00 seconds - .pyenv/versions/3.13.2/lib/python3.13/uuid.py:142 - UUID.__init__
181+
0.00 seconds - .pyenv/versions/3.13.2/lib/python3.13/uuid.py:283 - UUID.__str__
182+
0.00 seconds - .pyenv/versions/3.13.2/lib/python3.13/uuid.py:277 - UUID.__repr__
183+
0.00 seconds - .pyenv/versions/3.13.2/lib/python3.13/uuid.py:710 - uuid4
184+
185+
186+
You can list exact functions to profile as long as they are addressable by
187+
dotted names. The above only profiles the ``fib`` function in our package:
188+
189+
.. code:: bash
190+
191+
update_pyproject_toml "
192+
# New Config
193+
[tool.line_profiler.kernprof]
194+
line-by-line = true
195+
rich = true
196+
verbose = 0
197+
skip-zero = true
198+
prof-mod = ['demo_pkg.core.fib']
199+
"
200+
201+
# Now, running kernprof uses the new defaults
202+
kernprof -m demo_pkg
203+
python -m line_profiler -rmtz demo_pkg.lprof
204+
205+
206+
The output is:
207+
208+
.. code::
209+
210+
Line # Hits Time Per Hit % Time Line Contents
211+
==============================================================
212+
5 def fib(n):
213+
6 177 145.1 0.8 42.5 if leq(n, 1):
214+
7 89 29.7 0.3 8.7 return n
215+
8 88 29.1 0.3 8.5 part1 = fib(n - 1)
216+
9 88 27.7 0.3 8.1 part2 = fib(n - 2)
217+
10 88 78.0 0.9 22.8 result = utils.add(part1, part2)
218+
11 88 32.2 0.4 9.4 return result
219+
220+
221+
0.00 seconds - /tmp/tmp.vKpODQr6wndemo_pkg/src/demo_pkg/core.py:5 - fib

docs/source/manual/examples/index.rst

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,10 +5,12 @@ Examples of line profiler usage:
55

66
+ `Basic Usage <../../index.html#line-profiler-basic-usage>`_
77

8-
+ `kernprof invocations <example_kernprof.rst>`_
8+
+ `Kernprof Usage <example_kernprof.rst>`_
99

1010
+ `Auto Profiling <../../auto/line_profiler.autoprofile.html#auto-profiling>`_
1111

1212
+ `Explicit Profiler <../../auto/line_profiler.explicit_profiler.html#module-line_profiler.explicit_profiler>`_
1313

1414
+ `Timing Units <example_units.rst>`_
15+
16+
+ `TOML Config Usage <example_toml_config.rst>`_

0 commit comments

Comments
 (0)