gh-124503: simplified ast.literal_eval by kmagusiak · Pull Request #137010 · python/cpython

kmagusiak · 2025-07-22T15:32:38Z

The implementation does not create anymore local functions which reduces the overhead for small inputs. Some other calls are inlined into a single _convert_literal function.
We have a gain of 10-20% for small inputs and only 1-2% for bigger inputs.

Issue: Stack-based ast.literal_eval Implementation #124503

bedevere-app · 2025-07-22T15:32:42Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

python-cla-bot · 2025-07-22T15:32:42Z

All commit authors signed the Contributor License Agreement.

serhiy-storchaka

LGTM. 👍

kmagusiak · 2025-07-30T07:41:02Z

LGTM. 👍

Is there anything that needs to be done to merge this PR?

serhiy-storchaka · 2025-07-30T09:45:26Z

I think that it it is worth to add a small NEWS entry.

bedevere-app · 2025-07-30T11:23:41Z

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

The implementation does not create anymore local functions which reduces the overhead for small inputs. Some other calls are inlined into a single `_convert_literal` function. We have a gain of 10-20% for small inputs and only 1-2% for bigger inputs.

kmagusiak · 2025-07-30T11:26:21Z

I think that it it is worth to add a small NEWS entry.

Done

Eclips4

LGTM. Thank you!

serhiy-storchaka · 2025-07-31T09:56:08Z

Thank you for your contribution, @kmagusiak.

bedevere-bot · 2025-07-31T11:56:17Z

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Hi! The buildbot ARM Raspbian Linux Asan 3.x (no tier) has failed when building commit b723c8b.

What do you need to do:

Don't panic.
Check the buildbot page in the devguide if you don't know what the buildbots are or how they work.
Go to the page of the buildbot that failed (https://buildbot.python.org/#/builders/1811/builds/58) and take a look at the build logs.
Check if the failure is related to this commit (b723c8b) or if it is a false positive.
If the failure is related to this commit, please, reflect that on the issue and make a new Pull Request with a fix.

You can take a look at the buildbot page here:

https://buildbot.python.org/#/builders/1811/builds/58

Summary of the results of the build (if available):

Click to see traceback logs

remote: Enumerating objects: 9, done.        
remote: Counting objects:  12% (1/8)        
remote: Counting objects:  25% (2/8)        
remote: Counting objects:  37% (3/8)        
remote: Counting objects:  50% (4/8)        
remote: Counting objects:  62% (5/8)        
remote: Counting objects:  75% (6/8)        
remote: Counting objects:  87% (7/8)        
remote: Counting objects: 100% (8/8)        
remote: Counting objects: 100% (8/8), done.        
remote: Compressing objects:  14% (1/7)        
remote: Compressing objects:  28% (2/7)        
remote: Compressing objects:  42% (3/7)        
remote: Compressing objects:  57% (4/7)        
remote: Compressing objects:  71% (5/7)        
remote: Compressing objects:  85% (6/7)        
remote: Compressing objects: 100% (7/7)        
remote: Compressing objects: 100% (7/7), done.        
remote: Total 9 (delta 1), reused 1 (delta 1), pack-reused 1 (from 1)        
From https://github.com/python/cpython
 * branch                    main       -> FETCH_HEAD
Note: switching to 'b723c8be071afcf3f865c55a5efb6da54f7695a0'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by switching back to a branch.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -c with the switch command. Example:

  git switch -c <new-branch-name>

Or undo this operation with:

  git switch -

Turn off this advice by setting config variable advice.detachedHead to false

HEAD is now at b723c8be071 gh-124503: Optimize ast.literal_eval() for small input (GH-137010)
Switched to and reset branch 'main'

configure: WARNING: no system libmpdec found; falling back to pure-Python version for the decimal module

In file included from ./Include/internal/pycore_dict.h:11,
                 from Objects/typeobject.c:7:
In function ‘Py_DECREF_MORTAL’,
    inlined from ‘PyStackRef_XCLOSE’ at ./Include/internal/pycore_stackref.h:730:9,
    inlined from ‘_PyThreadState_PopCStackRef’ at ./Include/internal/pycore_stackref.h:810:5,
    inlined from ‘vectorcall_maybe’ at Objects/typeobject.c:3103:9:
./Include/internal/pycore_object.h:481:8: warning: array subscript 0 is outside array bounds of ‘PyObject[0]’ {aka ‘struct _object[]’} [-Warray-bounds]
  481 |     if (--op->ob_refcnt == 0) {
      |        ^

Timeout (0:05:00)!
Thread 0x0000007f7aaef100 [Thread-2] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2056 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007f7b2ff100 [Thread-1] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/setup.py", line 83 in _test_audit_hook
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 2057 in _wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/subprocess.py", line 1277 in wait
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 194 in _run_process
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 299 in run_tmp_files
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 363 in _runtest
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 403 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1074 in _bootstrap_inner
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/threading.py", line 1036 in _bootstrap

Thread 0x0000007f870d16c0 [python] (most recent call first):
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/utils.py", line 61 in format_duration
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 464 in get_running
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 551 in _get_result
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/run_workers.py", line 610 in run
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 455 in _run_tests_mp
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 561 in _run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 598 in run_tests
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 767 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/libregrtest/main.py", line 775 in main
  File "/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/Lib/test/__main__.py", line 2 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main
make: *** [Makefile:2517: buildbottest] Error 1

Cannot open file '/home/buildbot/buildarea/3.x.pablogsal-rasp.asan/build/test-results.xml' for upload

kmagusiak · 2025-07-31T12:21:34Z

Seems like an unrelated issue. Should I do anything?

serhiy-storchaka · 2025-07-31T12:40:17Z

Ignore, it is unrelated.

…H-137010) The implementation does not create anymore local functions which reduces the overhead for small inputs. Some other calls are inlined into a single `_convert_literal` function. We have a gain of 10-20% for small inputs and only 1-2% for bigger inputs.

bedevere-app Bot mentioned this pull request Jul 22, 2025

Stack-based ast.literal_eval Implementation #124503

Closed

kmagusiak marked this pull request as ready for review July 22, 2025 15:35

kmagusiak requested review from Eclips4, JelleZijlstra, isidentical and tomasr8 as code owners July 22, 2025 15:35

bedevere-app Bot added the awaiting review label Jul 22, 2025

serhiy-storchaka approved these changes Jul 23, 2025

View reviewed changes

bedevere-app Bot added awaiting merge and removed awaiting review labels Jul 23, 2025

kmagusiak force-pushed the gh-124503 branch from 37c7b81 to 5799eef Compare July 30, 2025 11:23

kmagusiak force-pushed the gh-124503 branch from 5799eef to 01afd3d Compare July 30, 2025 11:24

serhiy-storchaka reviewed Jul 30, 2025

View reviewed changes

Comment thread Misc/NEWS.d/next/Library/2025-07-30-11-12-22.gh-issue-124503.d4hc7b.rst Outdated

pythongh-124503: update news

9e2f796

Eclips4 approved these changes Jul 31, 2025

View reviewed changes

serhiy-storchaka approved these changes Jul 31, 2025

View reviewed changes

serhiy-storchaka merged commit b723c8b into python:main Jul 31, 2025
41 checks passed

bedevere-app Bot removed the awaiting merge label Jul 31, 2025

kmagusiak deleted the gh-124503 branch July 31, 2025 09:56

Uh oh!

Conversation

kmagusiak commented Jul 22, 2025 • edited by bedevere-app Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bedevere-app Bot commented Jul 22, 2025

Uh oh!

python-cla-bot Bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

serhiy-storchaka left a comment

Choose a reason for hiding this comment

Uh oh!

kmagusiak commented Jul 30, 2025

Uh oh!

serhiy-storchaka commented Jul 30, 2025

Uh oh!

bedevere-app Bot commented Jul 30, 2025

Uh oh!

kmagusiak commented Jul 30, 2025

Uh oh!

Uh oh!

Eclips4 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

serhiy-storchaka commented Jul 31, 2025

Uh oh!

bedevere-bot commented Jul 31, 2025

⚠️⚠️⚠️ Buildbot failure ⚠️⚠️⚠️

Uh oh!

kmagusiak commented Jul 31, 2025

Uh oh!

serhiy-storchaka commented Jul 31, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

kmagusiak commented Jul 22, 2025 •

edited by bedevere-app Bot

Loading

python-cla-bot Bot commented Jul 22, 2025 •

edited

Loading