Skip to content

Commit ddec3a7

Browse files
committed
Optimize ripper translator token sorting
With the benchmark from 2ea8139 Prism is: * 1.33x slower before * 1.07x slower after
1 parent 61e1207 commit ddec3a7

1 file changed

Lines changed: 7 additions & 2 deletions

File tree

lib/prism/lex_compat.rb

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -802,8 +802,13 @@ def result
802802
# Drop the EOF token from the list
803803
tokens = tokens[0...-1]
804804

805-
# We sort by location to compare against Ripper's output
806-
tokens.sort_by!(&:location)
805+
# We sort by location because `Ripper.lex` sorts. `Ripper::Lexer.new("foo").parse` emits tokens
806+
# in the same order that prism does. Since that method is not part of the public API, its behavior
807+
# is not replicated. Manually implemented instead of `sort_by!(&:location)` for performance.
808+
tokens.sort_by! do |token|
809+
line, column = token.location
810+
source.line_to_byte_offset(line) + column
811+
end
807812

808813
# Add :on_sp tokens
809814
tokens = add_on_sp_tokens(tokens, source, result.data_loc, bom, eof_token)

0 commit comments

Comments
 (0)