fix(mutation): keep canonicalization output parseable#5
Open
GrigoryEvko wants to merge 4 commits into
Open
Conversation
When `strip_comments_and_docstrings=True` and a function/class body contains only a docstring, popping it leaves the body empty and `ast.unparse` emits `def f():` with no indented suite — re-parsing fails. Append `ast.Pass()` for function/async/class nodes; module bodies are allowed to be empty.
ast.unparse does not validate its output. If any future AST transformer emits unparseable code (the same class of bug the previous commit fixed in _DocstringRemover), `_canonicalize_code` would return invalid Python to its caller silently. Re-parse the result and fall back to the original on round-trip failure.
- _DocstringRemover: copy_location on the inserted Pass so callers that compile() the transformed tree directly (without an ast.unparse round-trip) don't crash on missing lineno/col_offset. - _canonicalize_code: broaden the fallback to catch ValueError and RecursionError from ast.unparse, not just SyntaxError.
Decorated function and class-with-bases preserve their decorators/bases after the Pass insertion. Non-string leading Expr is not misidentified as a docstring. ast.unparse raising ValueError or RecursionError both fall back to the original code.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
_DocstringRemoverpops a function or class's only statement (its docstring) and leaves an empty body;ast.unparsethen emitsdef f():with no suite — invalid Python returned to the canonicalization caller silently.Reproducer against current main:
Fix: insert
ast.Pass()(withcopy_location) when the body becomes empty (function/class only; modules can stay empty). Plus a round-trip check in_canonicalize_code— re-parse the unparse result, fall back to original onSyntaxError/ValueError/RecursionError.