[SwiftParser] Intern the whole source buffer for a full parse#3384
Open
rintaro wants to merge 1 commit into
Open
[SwiftParser] Intern the whole source buffer for a full parse#3384rintaro wants to merge 1 commit into
rintaro wants to merge 1 commit into
Conversation
748dde4 ([SwiftParser] Intern source text per parsed token) changed parsing to intern each token's text individually instead of copying the whole source buffer once, so a parsed tree no longer retains the source buffer. That is important for incremental reparsing memory, but it regressed full-parse performance: a full parse now copies the source in as many pieces as there are tokens, each with its own bump-allocation and memcpy, rather than in a single bulk copy. For a full (non-incremental) parse, copy the whole source into the arena once and lex over that copy. Parsed tokens then point directly into arena-owned memory, so `internParsedTokenText` returns their text unchanged instead of copying it, and the tree stays self-contained. An incremental reparse keeps interning each re-lexed token individually, so its new tree still references only the re-lexed spans and not the whole buffer. `ParsingRawSyntaxArena.internSourceBuffer` copies the buffer, records its bounds, and returns the copy for `Parser` to lex over on a full parse; `internParsedTokenText` returns text that lies within the copy unchanged and copies anything else (e.g. synthesized tokens). Because the copy is owned by the arena, a full parse needs no parser-owned source copy, and nothing points into the caller's buffer once parsing returns.
rintaro
force-pushed
the
parser-fullparse-bulk-intern
branch
from
c044d1d to
1f6c623
Compare
Member
Author
|
@swift-ci Please test |
Member
Author
|
@swift-ci Please test Linux |
2 similar comments
Member
Author
|
@swift-ci Please test Linux |
Member
Author
|
@swift-ci Please test Linux |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
748dde4 stopped copying the whole source buffer by interning each token's text individually — good for incremental reparsing memory, but it regressed full parses, which now copy the source in as many small pieces as there are tokens instead of one bulk copy.
This recovers most of that while keeping the self-contained tree. For a full (non-incremental) parse, the whole source is copied into the arena once and the lexer runs over that copy, so tokens point into arena-owned memory and
internParsedTokenTextreturns their text unchanged instead of copying it. Incremental reparsing is unchanged (still per-token), so its new tree references only the re-lexed spans, not the whole buffer.ParsingRawSyntaxArena.internSourceBuffercopies the buffer and returns the copy forParserto lex over;internParsedTokenTextreturns text within the copy as-is and copies anything else (e.g. synthesized tokens). Since the arena owns the copy, a full parse no longer needs a separate parser-owned source copy and nothing points into the caller's buffer after parsing.Measurements: instructions per full parse of a 178 KB source (release, whole-package cross-module optimization, RawSyntax validation off, deterministic instruction counts):
So this recovers roughly 80% of the regression, with wall-clock back at the pre-748 level; the small remainder is the per-token
internParsedTokenTextcall itself, which still runs (returning the text unchanged) but no longer copies.