Implement PEP 799 – A dedicated profiling package for organizing Python profiling tools

[pablogsal]

Issue to collect all PRs for the implementation of PEP-799

Linked PRs

• gh-138122: Implement PEP 799 #138142
• gh-138122: Use profiling.sampling in the documentation (PEP 799) #138389
• gh-138122: Allow to filter by thread in tachyon's flamegraph #139216
• gh-138122: Improve the profiling section in the 3.15 what's new document #140156
• gh-138122: Refactor the CLI of profiling.sampling into subcommands #141813
• gh-138122: Fix permission error detection in tachyon's test #141897
• gh-138122: Add thread status statistics to flamegraph profiler #141900
• gh-138122: Don't sample partial frame chains #141912
• gh-138122: Split Modules/_remote_debugging_module.c into multiple files #141934
• gh-138122: New Tachyon UI #142116
• gh-138122: Implement frame caching in RemoteUnwinder to reduce memory reads #142137
• gh-138122: Small fixes to the new tachyon UI #142157
• gh-138122: Add inverted flamegraph #142288
• gh-138122: Integrate live profiler TUI with _colorize theming system #142360
• gh-138122: Make sampling profiler integration tests more resilient #142382
• gh-138122: Make the tachyon profiler opcode-aware #142394
• gh-138122: Fix unused variable warning in threads.c #142425
• gh-138122: Add exception profiling mode to the sampling profiler #142561
• gh-138122: Make Tachyon flamegraph and heatmap output more similar #142590
• gh-138122: Tachyon heatmap: Make toggle switches keyboard accessible #142601
• gh-138122: Add code examples to the profiling.sampling ddocs #142609
• gh-138122: Add docs button to Tachyon heatmap and flamegraph #142614
• gh-138122: Add --subprocesses flag to profile child processes in tachyon #142636
• GH-141565: Add async code awareness to Tachyon #141533
• gh-138122: Tachyon Flamegraph: Make toggle keyboard accesible and adjust sidebar collapse CSS #142638
• GH-142646: Update Tachyon to use backtick formatting in CLI help #142647
• GH-142591: Tachyon does not handle non-existent file/module #142592
• gh-138122: Add some screenshots to the profiling.sampling docs #142676
• gh-138122: Fix sample counting for filtered profiling modes #142677
• gh-138122: Allow tachyon to write and read binary output #142730
• gh-138122: Add sampling profiler visualisation to docs #142772
• gh-138122: Update Tachyon dark theme colors #142841
• gh-138122: Validate base frame before caching in remote debugging frame cache #142852
• gh-138122: Improve bytecode panel #142910
• gh-138122: Add blocking mode for accurate stack traces in Tachyon #142998
• gh-138122: Replace --interval with --sampling-rate #143085
• gh-138122: Extend binary profiling format with full source location and opcode #143088
• gh-138122: Remove default duration for statistical profiling #143174
• gh-138122: Convert GIL/GC/exception stats from tiles to progress bars #143177
• gh-142927: Auto-open HTML output in browser after generation #143178
• gh-138122: fix AC warnings in Modules/_remote_debugging/module.c #143218
• gh-138122: Move local imports to module level in sampling profiler #143257
• gh-138122: Skip threads on EPERM in blocking mode profiler #143368
• gh-138122: Default highlight to sh #143782
• gh-138122: Add differential flame graph #145785

[gaogaotiantian]

Not sure if this is the best place to discuss the name, but I'm not sure if "tracing" is the best new name for cProfile.

First of all, I think it's a great idea to organize all of our profilers and make it a new place to host other kinds of profilers, but "tracing" actually has its own meaning (as far as I understand).

The more accurate concept against "sampling" is probably "deterministic" (and I know it's long and difficult to spell), which means we record at all function entries and exits.

"tracing" on the other hand, often implies that we have not only the accumulated data for function calls, but also the information of each call - so that we can visualize it on a timeline. If you google tracing profiler, Perfetto might be one of the tools that catch your eyes - that's a real tracing tool (or chrome tracing which is its predecessor). VizTracer is a tracing profiler for Python. It generates a tracing graph, which looks like a flamegraph but it's not.

cProfile does not have the capability. It can only accumulate data and summarize it. Calling it a tracing profiler might confuse people (and might take the position of an actual tracing profiler we want to add in the future).

From another perpective, cProfile does not use sys.settrace, which is another "trace" concept that exists in CPython already.

Do you think we should come up with another name for cProfile in the new package?

[pablogsal]

Not sure if this is the best place to discuss the name, but I'm not sure if "tracing" is the best new name for cProfile.

First of all, I think it's a great idea to organize all of our profilers and make it a new place to host other kinds of profilers, but "tracing" actually has its own meaning (as far as I understand).

The more accurate concept against "sampling" is probably "deterministic" (and I know it's long and difficult to spell), which means we record at all function entries and exits.

"tracing" on the other hand, often implies that we have not only the accumulated data for function calls, but also the information of each call - so that we can visualize it on a timeline. If you google tracing profiler, Perfetto might be one of the tools that catch your eyes - that's a real tracing tool (or chrome tracing which is its predecessor). VizTracer is a tracing profiler for Python. It generates a tracing graph, which looks like a flamegraph but it's not.

cProfile does not have the capability. It can only accumulate data and summarize it. Calling it a tracing profiler might confuse people (and might take the position of an actual tracing profiler we want to add in the future).

From another perpective, cProfile does not use sys.settrace, which is another "trace" concept that exists in CPython already.

Do you think we should come up with another name for cProfile in the new package?

I completely understand your concerns about the naming, and thank you for taking the time to provide this detailed technical feedback. You make good points about the distinction between “tracing” and what cProfile actually does.

Although I kind of disagree I think you are also right that “tracing” can be ambiguous here and indeed it may imply that we are referring to timeline-based profiling that captures individual call information The potential confusion with sys.settrace is also a valid concern although I think it doesn't apply a lot here as is an implementation aspect in any case. But is an interesting point.

My personal opinion here is that although what you say is true the distinction is too nuanced to matter. Deterministic profiler is also not a great name IMHO as if we are a real pure tracing profiler that would also be deterministic, which would also be confusing.

I’m definitely open to considering this feedback and think it merits some discussion. The technical accuracy of our naming is important, both for clarity and to avoid blocking future actual tracing profiler implementations.

However, as you noted, the timing is challenging since PEP 799 has already been accepted, which would mean we’d need to go through the process of modifying the PEP if we decide to change the name. That said, getting the naming right is important enough that it may be worth that effort.

I think this needs input from the docs working group at least. We can keep the discussion here or maybe open another one in discuss. Do you have any preference?

[pablogsal]

CC @AA-Turner @hugovk @StanFromIreland

[AA-Turner]

Thanks for the ping @pablogsal.

Not sure if this is the best place to discuss the name, but I'm not sure if "tracing" is the best new name for cProfile.
...

I completely understand your concerns about the naming, and thank you for taking the time to provide this detailed technical feedback. You make good points about the distinction between “tracing” and what cProfile actually does.

My personal opinion here is that although what you say is true the distinction is too nuanced to matter. Deterministic profiler is also not a great name IMHO as if we are a real pure tracing profiler that would also be deterministic, which would also be confusing.

From what I can see from a quick search, tracing, instrumen{ted,ation}, deterministic, and call graph seem to be used as near-synonyms. This contrasts with sampling profiling, which seems to use the same term. There doesn't seem to be an obvious and consistent bright-line distinction I can find in third-party documents for what distinguishes a 'proper' tracing profiler.

Although I kind of disagree I think you are also right that “tracing” can be ambiguous here and indeed it may imply that we are referring to timeline-based profiling that captures individual call information The potential confusion with sys.settrace is also a valid concern although I think it doesn't apply a lot here as is an implementation aspect in any case. But is an interesting point.

The technical accuracy of our naming is important, both for clarity and to avoid blocking future actual tracing profiler implementations.

If a future Python version implemented support for tracing profiling in the VizTracer/Perfetto form, would we still keep the current (cProfile) implementation of profiling.tracing? A possible/likely scenario would be that we expose the higher fidelity information under the existing profiling.tracing name, possibly with a compat layer. If so, I don't think there is a risk to using the tracing name now, as it doesn't prevent future improvements/features.

A

[ZeroIntensity]

I propose the name profiling.boring for cProfile.

[pablogsal]

I propose the name profiling.boring for cProfile.

[gaogaotiantian]

I think it's possible that people are not that familiar with tracers so they think tracers are very similar to profilers. From my point of view, a tracer is distinctly different than a profiler. A tracer can be used for profiling (or, as a profiler), but it normally provides much more detailed information.

OpenTelemetry is probably the most popular cross-language tracing framework. One of the core concepts is a "span", meaning an event with a start/end pair timestamp - which a profiler normally does not provide.

All the results from my search on Google for "tracing profiler" are some kinds of profiler(tool) that provides a real tracing capability(Android, Google Cloud, Perfetto, dotNet, torch, etc.). I did not find any counter-example with my limited efforts.

A tracer (tracing profiler) has its own limitation - because it records too much information, it has to compromise on either the functions it records, or the length of the program. So we can't replace cProfile with a tracing profiler like VizTracer.

In my opinion, profiler.boring is a slightly better name than profiler.tracing :).

What about a similar but more official name profiler.basic? cProfile is not only instrumented, but also provides less information on the call frame compared to our new sampling profiler. It only has information of it caller (not the full frame). basic might be a good name to drive people away from using it when they need more advanced features.

[AA-Turner]

a tracer is distinctly different than a profiler

In this case, would a (hypothetical) future stdlib tracing implementation be considered for the profiling package/namespace?

I would be -1 on a name like profiling.basic, because it doesn't describe what the profiler does, only what we have decided its capabilities are. As Pablo/PEP 799 said, there remain valid use cases for the cProfile based implementation. I would also hope that in the future, we will have improved profiling.tracing to have more information such that is is no longer 'basic'.

A