Skip to content

Tachyon Oracle or Solaris (was: Bye ABBA and friends!)#5

Open
maurycy wants to merge 29 commits into
mainfrom
maurycy/bye-abba
Open

Tachyon Oracle or Solaris (was: Bye ABBA and friends!)#5
maurycy wants to merge 29 commits into
mainfrom
maurycy/bye-abba

Conversation

@maurycy

@maurycy maurycy commented Jun 19, 2026

Copy link
Copy Markdown
Owner

(Removed the old description, but it should be available under edits here.)

The branch, relative to python#151614:

  • increases last_profiled_frame_seq on every stack pop, not only last_profiled_frame (when sampling), assuming that the churn can happen anywhere,
  • adds the Tools/inspection/oracle_external_inspection.py with reproductions inspired by gh-151613: Fix remote debugging frame cache ABA python/cpython#151614 (called nested_alternating), the shared_leaf we spoke about and some others.

Most of this should be obvious. The oracle is ~250-300 LOC except the repros, and lots of it is multiline getopt.

There's very important usage gotcha: --rate 0 is a freerunning version, which does not fully match the -m profiling.sampling behaviour. I have my doubts whether we should use it at all (that's how python#151613 repro worked, though.)

maurycy@gimel cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --help                        
usage: oracle_external_inspection.py [-h]
                                     [--case {async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf}]
                                     [--mode {blocking-cache,blocking-nocache,live-cache,live-nocache}]
                                     [--reference-mode {blocking-cache,blocking-nocache,live-cache,live-nocache}] [--duration DURATION] [--runs RUNS]
                                     [--rate-khz RATE_KHZ] [--warmup WARMUP] [--poisson-sampling]

options:
  -h, --help            show this help message and exit
  --case {async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf}
                        case to run; may be passed more than once; omit to run all cases (default: None)
  --mode {blocking-cache,blocking-nocache,live-cache,live-nocache}
                        mode to run; may be passed more than once; omit to run all modes (default: None)
  --reference-mode {blocking-cache,blocking-nocache,live-cache,live-nocache}
                        mode used as the distribution reference (default: blocking-nocache)
  --duration DURATION   seconds to sample each mode (default: 3.0)
  --runs RUNS           number of independent runs per case (default: 1)
  --rate-khz RATE_KHZ   target sampling rate in kHz; 0 samples as fast as possible (default: 100.0)
  --warmup WARMUP       seconds to let the target run before sampling (default: 0.7)
  --poisson-sampling    sample with exponential inter-arrival times instead of a fixed period (default: False)

pablogsal and others added 2 commits June 17, 2026 23:06
Fixes python#151613.

The remote debugging frame cache previously used only the last_profiled_frame address as its cache anchor. If a frame returned and a later frame reused the same _PyInterpreterFrame address, the profiler could accept a stale cache entry and splice parent frames from a different call chain into the current stack.

This adds a last_profiled_frame_seq counter next to last_profiled_frame, increments it when the anchor advances, stores it in frame cache entries, and validates cache hits against both the frame address and the sequence. Cache miss walks now copy stack chunks before storing new cache entries so stored continuations come from a stable snapshot. The new regression test exercises alternating call chains and checks that cached stacks never contain frames from both branches.
@maurycy maurycy self-assigned this Jun 19, 2026
@maurycy maurycy changed the title Bye, ABBA! Bye, AB(B)A! Jun 19, 2026
@maurycy maurycy changed the title Bye, AB(B)A! Bye, AB(B)A and friends! Jun 19, 2026
@maurycy maurycy changed the title Bye, AB(B)A and friends! Bye AB(B)A and friends! Jun 19, 2026
This reverts commit 6024fc7.
@maurycy maurycy marked this pull request as ready for review June 20, 2026 08:38
@maurycy

maurycy commented Jun 20, 2026

Copy link
Copy Markdown
Owner Author

Comparison between https://github.com/python/cpython/pull/151614/changes:

maurycy@gimel cpython (maurycy/bye-abba 981433d?) % time sudo ~/work/cpython-pablo/python.exe Tools/inspection/oracle_external_inspection.py --duration 3 --runs 8
3.16.0a0 (heads/gh-151613-remote-debugging-frame-cache-aba:fc9fafd7699, Jun 20 2026, 00:31) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=35
mode                 samples    stacks     empty      errors% impossible  impossible%
blocking-nocache       41229     82458   1760834    0.00±0.00          0    0.00±0.00
live-nocache          492594    984981     20066    3.60±0.22      19908    2.02±0.16

code_object_reuse (get_stack_trace) ref_keys=22
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1373761    0.01±0.01          0    0.00±0.00          ref          -
blocking-cache       1809613    0.00±0.00          0    0.00±0.00  0.011±0.007      0.002
live-cache           2399991    0.00±0.00          0    0.00±0.00  0.693±0.031      0.002
live-nocache         1692980    0.02±0.01          0    0.00±0.00  0.019±0.004      0.002

deep_recursion (get_stack_trace) ref_keys=1214
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      479506    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        428103    0.00±0.00          0    0.00±0.00  0.063±0.018      0.061
live-cache            391620   50.67±1.63      90004   22.99±0.51  0.504±0.020      0.068
live-nocache          385514   53.70±0.78     100363   26.04±0.99  0.501±0.032      0.069

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1356368    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1477297    0.00±0.00          0    0.00±0.00  0.012±0.008      0.002
live-cache           1487656   13.69±0.29      10785    0.72±0.07  0.052±0.010      0.002
live-nocache         1648031   13.84±0.41      11503    0.70±0.07  0.050±0.010      0.002

gen_alternating (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1540029    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1779310    0.00±0.00          2    0.00±0.00  0.024±0.015      0.004
live-cache           1635477   17.18±0.26     383810   23.43±1.45  0.241±0.009      0.005
live-nocache         1758238   16.95±0.98     295768   16.83±2.70  0.222±0.003      0.005

nested_alternating (get_stack_trace) ref_keys=20
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1072793    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1529883    0.00±0.00          0    0.00±0.00  0.005±0.004      0.003
live-cache           2385360    0.00±0.00         76    0.00±0.00  0.161±0.012      0.003
live-nocache         1535487    0.04±0.01          3    0.00±0.00  0.173±0.011      0.003

oversized_chunk (get_stack_trace) ref_keys=17
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1538581    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       2198736    0.00±0.00          0    0.00±0.00  0.013±0.003      0.003
live-cache           2398517    0.06±0.08      18449    0.77±1.35  0.124±0.016      0.003
live-nocache         2035354    5.13±0.34         19    0.00±0.00  0.114±0.009      0.003

shared_leaf (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1334732    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1860127    0.00±0.00          0    0.00±0.00  0.003±0.002      0.001
live-cache           2400005    0.00±0.00       1403    0.06±0.08  0.146±0.014      0.001
live-nocache         1855932    0.02±0.00          0    0.00±0.00  0.140±0.012      0.001

v. (this branch)

maurycy@gimel cpython (maurycy/bye-abba 981433d?) % time sudo ~/work/cpython/python.exe Tools/inspection/oracle_external_inspection.py --duration 3 --runs 8      
3.16.0a0 (heads/maurycy/bye-abba:981433d86dd, Jun 21 2026, 21:23:12) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=49
mode                 samples    stacks     empty      errors% impossible  impossible%
blocking-nocache      179565    359130    969030    0.00±0.00          0    0.00±0.00
live-nocache          519182   1037570     32370    3.52±0.30      13508    1.30±0.12

code_object_reuse (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      948961    0.00±0.01          0    0.00±0.00          ref          -
blocking-cache       1250076    0.01±0.03          0    0.00±0.00  0.057±0.029      0.002
live-cache           2399939    0.00±0.00          0    0.00±0.00  0.076±0.030      0.002
live-nocache         1548763    0.02±0.01          0    0.00±0.00  0.074±0.031      0.002

deep_recursion (get_stack_trace) ref_keys=1201
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      305322    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        261432    0.00±0.00          0    0.00±0.00  0.050±0.018      0.081
live-cache            293174   51.16±0.86      67580   23.03±1.47  0.445±0.023      0.086
live-nocache          309822   52.07±1.35      69699   22.43±1.61  0.446±0.031      0.084

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      929811    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1037779    0.00±0.00          0    0.00±0.00  0.157±0.019      0.003
live-cache           1290342   13.14±0.41      11110    0.86±0.07  0.277±0.017      0.003
live-nocache         1345912   13.82±0.46       9636    0.71±0.13  0.271±0.017      0.003

gen_alternating (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1102219    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1344287    0.00±0.00          0    0.00±0.00  0.030±0.022      0.005
live-cache           1348941   16.90±0.38     380953   28.25±2.53  0.243±0.007      0.006
live-nocache         1382138   17.26±0.61     273412   19.72±2.83  0.228±0.008      0.006

nested_alternating (get_stack_trace) ref_keys=21
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      914757    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1214234    0.00±0.00          0    0.00±0.00  0.007±0.007      0.003
live-cache           2393961    0.00±0.00          0    0.00±0.00  0.161±0.009      0.003
live-nocache         1477038    0.03±0.00          6    0.00±0.00  0.180±0.009      0.003

oversized_chunk (get_stack_trace) ref_keys=17
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1592552    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       2233183    0.00±0.00          0    0.00±0.00  0.026±0.008      0.003
live-cache           2399683    0.01±0.00          1    0.00±0.00  0.123±0.005      0.003
live-nocache         2008073    5.05±0.09         13    0.00±0.00  0.110±0.004      0.003

shared_leaf (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache     1278516    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache       1759760    0.00±0.00          0    0.00±0.00  0.003±0.002      0.002
live-cache           2399988    0.00±0.00          2    0.00±0.00  0.146±0.006      0.002
live-nocache         1813803    0.02±0.00          0    0.00±0.00  0.143±0.005      0.002

@maurycy maurycy changed the title Bye AB(B)A and friends! ABBA Oracle Jun 20, 2026
@maurycy maurycy changed the title ABBA Oracle Tachyon Oracle (was: Bye ABBA and friends!) Jun 20, 2026
@maurycy

maurycy commented Jun 20, 2026

Copy link
Copy Markdown
Owner Author

I was able to nicely reproduce python#151424 with 56.35±3.21% impossible rate, that directionally matches that finding:

[130] 2026-06-21T21:36:22.319345000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-old-main/python.exe Tools/inspection/oracle_external_inspection.py --case flat_alternating --rate 1 --duration 3 --runs 8
3.16.0a0 (heads/main:6679ac07d88) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=flat_alternating runs=8 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       24008    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         24008    0.00±0.00          0    0.00±0.00  0.021±0.008      0.017
live-cache             20536   14.46±0.91      11569   56.35±3.21  0.290±0.043      0.023
live-nocache           20631   14.07±0.79         64    0.31±0.12  0.029±0.013      0.018
2026-06-21T21:38:21.522363000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % 

@maurycy

maurycy commented Jun 20, 2026

Copy link
Copy Markdown
Owner Author

The full run with all the current repros.

1KHz:

2026-06-21T21:40:25.861867000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --rate 1 --duration 3 --runs 4
3.16.0a0 (heads/main:1fb874cc076, Jun 21 2026, 00:51:43) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=4 duration=3.0 rate_khz=1.0 warmup=0.7 poisson_sampling=False

async_running_task (get_async_stack_trace) ref_keys=20
mode                 samples    stacks     empty      errors% impossible  impossible%
blocking-nocache       10608     21216      1396    0.00±0.00          0    0.00±0.00
live-nocache           10800     21593       741    3.86±0.65        608    2.82±0.39

code_object_reuse (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00 -0.004±0.003      0.012
live-cache             12002    0.02±0.02          0    0.00±0.00  0.000±0.002      0.012
live-nocache           12001    0.02±0.03          0    0.00±0.00 -0.008±0.002      0.012

deep_recursion (get_stack_trace) ref_keys=754
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00  0.024±0.006      0.232
live-cache              6335   47.23±1.51       3772   59.45±4.24  0.300±0.002      0.394
live-nocache            5441   54.67±0.74       2735   50.25±2.13  0.309±0.008      0.380

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00  0.001±0.011      0.016
live-cache             10206   14.98±0.40         50    0.49±0.04  0.074±0.019      0.017
live-nocache           10260   14.53±0.61         35    0.34±0.18  0.075±0.014      0.017

gen_alternating (get_stack_trace) ref_keys=18
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00  0.004±0.003      0.034
live-cache              9842   18.01±0.73       3556   36.12±1.74  0.238±0.022      0.041
live-nocache            9853   17.92±1.02       3484   35.36±0.86  0.232±0.021      0.041

nested_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00 -0.000±0.005      0.018
live-cache             11996    0.07±0.04          0    0.00±0.00  0.113±0.005      0.018
live-nocache           11998    0.05±0.04          0    0.00±0.00  0.131±0.002      0.018

oversized_chunk (get_stack_trace) ref_keys=10
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12004    0.00±0.00          0    0.00±0.00  0.002±0.007      0.021
live-cache             11126    7.31±0.37          1    0.01±0.02  0.030±0.016      0.021
live-nocache           11135    7.24±0.39          0    0.00±0.00  0.026±0.014      0.021

shared_leaf (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache       12004    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache         12003    0.00±0.00          0    0.00±0.00  0.001±0.005      0.011
live-cache             12004    0.00±0.00         23    0.19±0.09  0.110±0.021      0.011
live-nocache           12003    0.01±0.02          0    0.00±0.00  0.104±0.017      0.011

100KHz, 8 runs, Poisson interval:

2026-06-21T21:38:21.522363000+0200 maurycy@gimel /Users/maurycy/work/cpython (maurycy/bye-abba 981433d?) % sudo ~/work/cpython-main/python.exe Tools/inspection/oracle_external_inspection.py --rate 100 --duration 3 --runs 8 --poisson
3.16.0a0 (heads/main:1fb874cc076, Jun 21 2026, 00:51:43) [Clang 21.0.0 (clang-2100.1.1.101)]
cases=async_running_task,code_object_reuse,deep_recursion,flat_alternating,gen_alternating,nested_alternating,oversized_chunk,shared_leaf runs=8 duration=3.0 rate_khz=100.0 warmup=0.7 poisson_sampling=True

async_running_task (get_async_stack_trace) ref_keys=48
mode                 samples    stacks     empty      errors% impossible  impossible%
blocking-nocache      291822    583644     13490    0.00±0.00          0    0.00±0.00
live-nocache          283700    567210     16062    3.40±0.38      14565    2.60±0.37

code_object_reuse (get_stack_trace) ref_keys=18
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      415911    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        519756    0.00±0.00          0    0.00±0.00  0.003±0.002      0.003
live-cache            762223    0.01±0.00          0    0.00±0.00  0.503±0.033      0.003
live-nocache          512079    0.02±0.00          0    0.00±0.00  0.008±0.003      0.003

deep_recursion (get_stack_trace) ref_keys=1192
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      204941    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        208733    0.00±0.00          0    0.00±0.00  0.006±0.007      0.094
live-cache            118025   46.32±2.45      38763   32.87±1.18  0.477±0.014      0.124
live-nocache          140715   54.12±0.95      32335   22.96±2.33  0.494±0.009      0.112

flat_alternating (get_stack_trace) ref_keys=6
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      409186    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        397940    0.00±0.00          0    0.00±0.00  0.009±0.006      0.004
live-cache            467688   14.97±0.33       1036    0.22±0.02  0.054±0.010      0.004
live-nocache          439633   14.82±0.26       1526    0.35±0.05  0.054±0.008      0.004

gen_alternating (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      439443    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        458007    0.00±0.00          1    0.00±0.00  0.010±0.004      0.008
live-cache            467073   17.42±0.32     174754   37.65±4.90  0.271±0.018      0.009
live-nocache          481223   16.99±0.50      95638  20.97±11.38  0.233±0.015      0.008

nested_alternating (get_stack_trace) ref_keys=19
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      436887    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        576923    0.00±0.00          0    0.00±0.00  0.006±0.004      0.004
live-cache            689881    0.00±0.00       3578    0.52±0.02  0.139±0.009      0.004
live-nocache          552035    0.03±0.01          1    0.00±0.00  0.153±0.018      0.004

oversized_chunk (get_stack_trace) ref_keys=17
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      703071    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        763936    0.00±0.00          0    0.00±0.00  0.000±0.002      0.004
live-cache            913365    1.32±0.03      40178    4.40±0.12  0.067±0.002      0.004
live-nocache          805437    7.59±0.05         13    0.00±0.00  0.063±0.003      0.004

shared_leaf (get_stack_trace) ref_keys=15
mode                 samples      errors% impossible  impossible%   tvd_excess  tvd_floor
blocking-nocache      502934    0.00±0.00          0    0.00±0.00          ref          -
blocking-cache        669466    0.00±0.00          0    0.00±0.00  0.004±0.003      0.002
live-cache            840010    0.00±0.00       1557    0.19±0.06  0.133±0.005      0.002
live-nocache          655902    0.02±0.01          0    0.00±0.00  0.127±0.004      0.002

@maurycy maurycy changed the title Tachyon Oracle (was: Bye ABBA and friends!) Tachyon Oracle or Solaris (was: Bye ABBA and friends!) Jun 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants