Skip to content

Fix parser translator during string escaping with invalid utf-8#3583

Merged
kddnewton merged 1 commit into
ruby:mainfrom
Earlopain:parser-translator-escape-invalid-encoding
Jun 11, 2025
Merged

Fix parser translator during string escaping with invalid utf-8#3583
kddnewton merged 1 commit into
ruby:mainfrom
Earlopain:parser-translator-escape-invalid-encoding

Conversation

@Earlopain

Copy link
Copy Markdown
Collaborator

byte.ord can raise when it is not valid in its encoding. Instead, prefer scan_byte over get_byte since that already returns the byte as an integer, sidestepping conversion issues.

Fixes #3582

@Earlopain
Earlopain force-pushed the parser-translator-escape-invalid-encoding branch from 154ed98 to 1bcf087 Compare June 11, 2025 11:31
@Earlopain

Copy link
Copy Markdown
Collaborator Author

Right, no scan_byte, it's much to recent. Just make the character binary then, seems good enough.

@kddnewton kddnewton left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's use scan_byte and add a polyfill for when it's necessary

@Earlopain
Earlopain force-pushed the parser-translator-escape-invalid-encoding branch from 1bcf087 to f2a2ebc Compare June 11, 2025 12:58
Instead, prefer `scan_byte` over `get_byte` since that already returns the byte as an integer, sidestepping conversion issues.

Fixes ruby#3582
@Earlopain
Earlopain force-pushed the parser-translator-escape-invalid-encoding branch from f2a2ebc to 7f3008b Compare June 11, 2025 13:28
@Earlopain

Copy link
Copy Markdown
Collaborator Author

Right, makes sense. I changed it.

@kddnewton
kddnewton merged commit 615e726 into ruby:main Jun 11, 2025
56 checks passed
@Earlopain
Earlopain deleted the parser-translator-escape-invalid-encoding branch October 3, 2025 10:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parser Translation: tokenization error

2 participants