Tachyon Oracle or Solaris (was: Bye ABBA and friends!)#5
Open
maurycy wants to merge 29 commits into
Open
Conversation
Fixes python#151613. The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack. This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.
This reverts commit 6024fc7.
Owner
Author
|
Comparison between https://github.com/python/cpython/pull/151614/changes: maurycy@gimel cpython (maurycy/bye-abba 981433d?) % time sudo ~/work/cpython-pablo/python.exe Tools/inspection/oracle_external_inspection.py --duration 3 --runs 8
3.16.0a0 (heads/gh-151613-remote-debugging-frame-cache-aba:fc9fafd7699, Jun 20 2026, 00:31) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False
async_running_task (get_async_stack_trace) ref_keys=35
mode samples stacks empty errors% impossible impossible%
blocking-nocache 41229 82458 1760834 0.00±0.00 0 0.00±0.00
live-nocache 492594 984981 20066 3.60±0.22 19908 2.02±0.16
code_object_reuse (get_stack_trace) ref_keys=22
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1373761 0.01±0.01 0 0.00±0.00 ref -
blocking-cache 1809613 0.00±0.00 0 0.00±0.00 0.011±0.007 0.002
live-cache 2399991 0.00±0.00 0 0.00±0.00 0.693±0.031 0.002
live-nocache 1692980 0.02±0.01 0 0.00±0.00 0.019±0.004 0.002
deep_recursion (get_stack_trace) ref_keys=1214
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 479506 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 428103 0.00±0.00 0 0.00±0.00 0.063±0.018 0.061
live-cache 391620 50.67±1.63 90004 22.99±0.51 0.504±0.020 0.068
live-nocache 385514 53.70±0.78 100363 26.04±0.99 0.501±0.032 0.069
flat_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1356368 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1477297 0.00±0.00 0 0.00±0.00 0.012±0.008 0.002
live-cache 1487656 13.69±0.29 10785 0.72±0.07 0.052±0.010 0.002
live-nocache 1648031 13.84±0.41 11503 0.70±0.07 0.050±0.010 0.002
gen_alternating (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1540029 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1779310 0.00±0.00 2 0.00±0.00 0.024±0.015 0.004
live-cache 1635477 17.18±0.26 383810 23.43±1.45 0.241±0.009 0.005
live-nocache 1758238 16.95±0.98 295768 16.83±2.70 0.222±0.003 0.005
nested_alternating (get_stack_trace) ref_keys=20
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1072793 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1529883 0.00±0.00 0 0.00±0.00 0.005±0.004 0.003
live-cache 2385360 0.00±0.00 76 0.00±0.00 0.161±0.012 0.003
live-nocache 1535487 0.04±0.01 3 0.00±0.00 0.173±0.011 0.003
oversized_chunk (get_stack_trace) ref_keys=17
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1538581 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 2198736 0.00±0.00 0 0.00±0.00 0.013±0.003 0.003
live-cache 2398517 0.06±0.08 18449 0.77±1.35 0.124±0.016 0.003
live-nocache 2035354 5.13±0.34 19 0.00±0.00 0.114±0.009 0.003
shared_leaf (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1334732 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1860127 0.00±0.00 0 0.00±0.00 0.003±0.002 0.001
live-cache 2400005 0.00±0.00 1403 0.06±0.08 0.146±0.014 0.001
live-nocache 1855932 0.02±0.00 0 0.00±0.00 0.140±0.012 0.001v. (this branch) maurycy@gimel cpython (maurycy/bye-abba 981433d?) % time sudo ~/work/cpython/python.exe Tools/inspection/oracle_external_inspection.py --duration 3 --runs 8
3.16.0a0 (heads/maurycy/bye-abba:981433d86dd, Jun 21 2026, 21:23:12) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False
async_running_task (get_async_stack_trace) ref_keys=49
mode samples stacks empty errors% impossible impossible%
blocking-nocache 179565 359130 969030 0.00±0.00 0 0.00±0.00
live-nocache 519182 1037570 32370 3.52±0.30 13508 1.30±0.12
code_object_reuse (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 948961 0.00±0.01 0 0.00±0.00 ref -
blocking-cache 1250076 0.01±0.03 0 0.00±0.00 0.057±0.029 0.002
live-cache 2399939 0.00±0.00 0 0.00±0.00 0.076±0.030 0.002
live-nocache 1548763 0.02±0.01 0 0.00±0.00 0.074±0.031 0.002
deep_recursion (get_stack_trace) ref_keys=1201
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 305322 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 261432 0.00±0.00 0 0.00±0.00 0.050±0.018 0.081
live-cache 293174 51.16±0.86 67580 23.03±1.47 0.445±0.023 0.086
live-nocache 309822 52.07±1.35 69699 22.43±1.61 0.446±0.031 0.084
flat_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 929811 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1037779 0.00±0.00 0 0.00±0.00 0.157±0.019 0.003
live-cache 1290342 13.14±0.41 11110 0.86±0.07 0.277±0.017 0.003
live-nocache 1345912 13.82±0.46 9636 0.71±0.13 0.271±0.017 0.003
gen_alternating (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1102219 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1344287 0.00±0.00 0 0.00±0.00 0.030±0.022 0.005
live-cache 1348941 16.90±0.38 380953 28.25±2.53 0.243±0.007 0.006
live-nocache 1382138 17.26±0.61 273412 19.72±2.83 0.228±0.008 0.006
nested_alternating (get_stack_trace) ref_keys=21
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 914757 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1214234 0.00±0.00 0 0.00±0.00 0.007±0.007 0.003
live-cache 2393961 0.00±0.00 0 0.00±0.00 0.161±0.009 0.003
live-nocache 1477038 0.03±0.00 6 0.00±0.00 0.180±0.009 0.003
oversized_chunk (get_stack_trace) ref_keys=17
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1592552 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 2233183 0.00±0.00 0 0.00±0.00 0.026±0.008 0.003
live-cache 2399683 0.01±0.00 1 0.00±0.00 0.123±0.005 0.003
live-nocache 2008073 5.05±0.09 13 0.00±0.00 0.110±0.004 0.003
shared_leaf (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 1278516 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 1759760 0.00±0.00 0 0.00±0.00 0.003±0.002 0.002
live-cache 2399988 0.00±0.00 2 0.00±0.00 0.146±0.006 0.002
live-nocache 1813803 0.02±0.00 0 0.00±0.00 0.143±0.005 0.002 |
Owner
Author
|
I was able to nicely reproduce python#151424 with 56.35±3.21% impossible rate, that directionally matches that finding: [130] 2026-06-21T21:36:22.319345000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-old-main/python.exe Tools/inspection/oracle_external_inspection.py --case flat_alternating --rate 1 --duration 3 --runs 8
3.16.0a0 (heads/main:6679ac07d88) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False
flat_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 24008 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 24008 0.00±0.00 0 0.00±0.00 0.021±0.008 0.017
live-cache 20536 14.46±0.91 11569 56.35±3.21 0.290±0.043 0.023
live-nocache 20631 14.07±0.79 64 0.31±0.12 0.029±0.013 0.018
2026-06-21T21:38:21.522363000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % |
This reverts commit 3207d12.
Owner
Author
|
The full run with all the current repros. 1KHz: 2026-06-21T21:40:25.861867000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --rate 1 --duration 3 --runs 4
3.16.0a0 (heads/main:1fb874cc076, Jun 21 2026, 00:51:43) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=4 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False
async_running_task (get_async_stack_trace) ref_keys=20
mode samples stacks empty errors% impossible impossible%
blocking-nocache 10608 21216 1396 0.00±0.00 0 0.00±0.00
live-nocache 10800 21593 741 3.86±0.65 608 2.82±0.39
code_object_reuse (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 -0.004±0.003 0.012
live-cache 12002 0.02±0.02 0 0.00±0.00 0.000±0.002 0.012
live-nocache 12001 0.02±0.03 0 0.00±0.00 -0.008±0.002 0.012
deep_recursion (get_stack_trace) ref_keys=754
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 0.024±0.006 0.232
live-cache 6335 47.23±1.51 3772 59.45±4.24 0.300±0.002 0.394
live-nocache 5441 54.67±0.74 2735 50.25±2.13 0.309±0.008 0.380
flat_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 0.001±0.011 0.016
live-cache 10206 14.98±0.40 50 0.49±0.04 0.074±0.019 0.017
live-nocache 10260 14.53±0.61 35 0.34±0.18 0.075±0.014 0.017
gen_alternating (get_stack_trace) ref_keys=18
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 0.004±0.003 0.034
live-cache 9842 18.01±0.73 3556 36.12±1.74 0.238±0.022 0.041
live-nocache 9853 17.92±1.02 3484 35.36±0.86 0.232±0.021 0.041
nested_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 -0.000±0.005 0.018
live-cache 11996 0.07±0.04 0 0.00±0.00 0.113±0.005 0.018
live-nocache 11998 0.05±0.04 0 0.00±0.00 0.131±0.002 0.018
oversized_chunk (get_stack_trace) ref_keys=10
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12004 0.00±0.00 0 0.00±0.00 0.002±0.007 0.021
live-cache 11126 7.31±0.37 1 0.01±0.02 0.030±0.016 0.021
live-nocache 11135 7.24±0.39 0 0.00±0.00 0.026±0.014 0.021
shared_leaf (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 12004 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 12003 0.00±0.00 0 0.00±0.00 0.001±0.005 0.011
live-cache 12004 0.00±0.00 23 0.19±0.09 0.110±0.021 0.011
live-nocache 12003 0.01±0.02 0 0.00±0.00 0.104±0.017 0.011100KHz, 8 runs, Poisson interval: 2026-06-21T21:38:21.522363000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --rate 100 --duration 3 --runs 8 --poisson
3.16.0a0 (heads/main:1fb874cc076, Jun 21 2026, 00:51:43) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=True
async_running_task (get_async_stack_trace) ref_keys=48
mode samples stacks empty errors% impossible impossible%
blocking-nocache 291822 583644 13490 0.00±0.00 0 0.00±0.00
live-nocache 283700 567210 16062 3.40±0.38 14565 2.60±0.37
code_object_reuse (get_stack_trace) ref_keys=18
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 415911 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 519756 0.00±0.00 0 0.00±0.00 0.003±0.002 0.003
live-cache 762223 0.01±0.00 0 0.00±0.00 0.503±0.033 0.003
live-nocache 512079 0.02±0.00 0 0.00±0.00 0.008±0.003 0.003
deep_recursion (get_stack_trace) ref_keys=1192
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 204941 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 208733 0.00±0.00 0 0.00±0.00 0.006±0.007 0.094
live-cache 118025 46.32±2.45 38763 32.87±1.18 0.477±0.014 0.124
live-nocache 140715 54.12±0.95 32335 22.96±2.33 0.494±0.009 0.112
flat_alternating (get_stack_trace) ref_keys=6
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 409186 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 397940 0.00±0.00 0 0.00±0.00 0.009±0.006 0.004
live-cache 467688 14.97±0.33 1036 0.22±0.02 0.054±0.010 0.004
live-nocache 439633 14.82±0.26 1526 0.35±0.05 0.054±0.008 0.004
gen_alternating (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 439443 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 458007 0.00±0.00 1 0.00±0.00 0.010±0.004 0.008
live-cache 467073 17.42±0.32 174754 37.65±4.90 0.271±0.018 0.009
live-nocache 481223 16.99±0.50 95638 20.97±11.38 0.233±0.015 0.008
nested_alternating (get_stack_trace) ref_keys=19
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 436887 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 576923 0.00±0.00 0 0.00±0.00 0.006±0.004 0.004
live-cache 689881 0.00±0.00 3578 0.52±0.02 0.139±0.009 0.004
live-nocache 552035 0.03±0.01 1 0.00±0.00 0.153±0.018 0.004
oversized_chunk (get_stack_trace) ref_keys=17
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 703071 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 763936 0.00±0.00 0 0.00±0.00 0.000±0.002 0.004
live-cache 913365 1.32±0.03 40178 4.40±0.12 0.067±0.002 0.004
live-nocache 805437 7.59±0.05 13 0.00±0.00 0.063±0.003 0.004
shared_leaf (get_stack_trace) ref_keys=15
mode samples errors% impossible impossible% tvd_excess tvd_floor
blocking-nocache 502934 0.00±0.00 0 0.00±0.00 ref -
blocking-cache 669466 0.00±0.00 0 0.00±0.00 0.004±0.003 0.002
live-cache 840010 0.00±0.00 1557 0.19±0.06 0.133±0.005 0.002
live-nocache 655902 0.02±0.01 0 0.00±0.00 0.127±0.004 0.002 |
This reverts commit e6d5f5c.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Removed the old description, but it should be available under edits here.)
The branch, relative to python#151614:
last_profiled_frame_seqon every stack pop, not onlylast_profiled_frame(when sampling), assuming that the churn can happen anywhere,Tools/inspection/oracle_external_inspection.pywith reproductions inspired by gh-151613: Fix remote debugging frame cache ABA python/cpython#151614 (callednested_alternating), theshared_leafwe spoke about and some others.Most of this should be obvious. The oracle is ~250-300 LOC except the repros, and lots of it is multiline getopt.
There's very important usage gotcha:
--rate 0is a freerunning version, which does not fully match the-m profiling.samplingbehaviour. I have my doubts whether we should use it at all (that's how python#151613 repro worked, though.)