Parser fails to tokenize ERB with two subsequent returning expressions

[leoarnold]

Using Prism 1.4.0 on Ruby 3.4.3 I get

3.4.3 :002 > buffer = Parser::Source::Buffer.new('foo.erb')
 => #<Parser::Source::Buffer foo.erb> 
3.4.3 :003 > buffer.source = "<%= a %>"
 => "<%= a %>" 
3.4.3 :004 > Prism::Translation::Parser34.new.tokenize(buffer)
 => [nil, [], [[:tLT, ["<", #<Parser::Source::Range foo.erb 0...1>]], [:tSTRING_BEG, ["%=", #<Parser::Source::Range foo.erb 1...3>]], [:tSTRING_CONTENT, [" a %>", #<Parser::Source::Range foo.erb 3...8>]]]] 
3.4.3 :005 > buffer = Parser::Source::Buffer.new('bar.erb')
 => #<Parser::Source::Buffer bar.erb> 
3.4.3 :006 > buffer.source = "<%= a %><%= b ? c : d %>"
 => "<%= a %><%= b ? c : d %>" 
3.4.3 :007 > Prism::Translation::Parser34.new.tokenize(buffer)
(smart-workspaces):7:in '<main>': undefined method '[]' for nil (NoMethodError)

              next_next_token = lexed[index + 1][0]
                                                ^^^

[Earlopain]

Thanks for the report! I openend #3551 for a different issue, but it will fix your problem as well.

[leoarnold]

@Earlopain Awesome!