Ripper translation layer relies on upstream Ripper and is not fully compatible

[eregon]

https://github.com/ruby/prism/blob/main/docs/ripper_translation.md mentions

require "prism/translation/ripper/shim"

However that does not work:

$ ruby -v -rprism -rprism/translation/ripper/shim -e 'p Ripper.lex("1 + 2")'
ruby 4.0.0 (2025-12-25 revision 553f1675f3) +PRISM [x86_64-linux]
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/translation/ripper/shim.rb:5: warning: already initialized constant Ripper
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/parse_result.rb:776:in 'Prism::LexResult#initialize': stack level too deep (SystemStackError)
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/lex_compat.rb:631:in 'Prism.lex'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/lex_compat.rb:631:in 'Prism::LexCompat#result'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism.rb:69:in 'Prism.lex_compat'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/translation/ripper.rb:74:in 'Prism::Translation::Ripper.lex'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/lex_compat.rb:639:in 'Prism::LexCompat#result'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism.rb:69:in 'Prism.lex_compat'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/translation/ripper.rb:74:in 'Prism::Translation::Ripper.lex'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/lex_compat.rb:639:in 'Prism::LexCompat#result'
	 ... 7850 levels...
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/lex_compat.rb:639:in 'Prism::LexCompat#result'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism.rb:69:in 'Prism.lex_compat'
	from /home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/translation/ripper.rb:74:in 'Prism::Translation::Ripper.lex'
	from -e:1:in '<main>'

Also while

$ ruby -rprism -rprism/translation/ripper -e 'p Prism::Translation::Ripper.lex("1 + 2"); puts $".grep(/ripper/)'
[[[1, 0], :on_int, "1", END], [[1, 2], :on_op, "+", BEG], [[1, 4], :on_int, "2", END]]
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/x86_64-linux/ripper.so
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/ripper/core.rb
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/ripper/lexer.rb
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/ripper/filter.rb
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/ripper/sexp.rb
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/ripper.rb
/home/eregon/.rubies/ruby-4.0.0/lib/ruby/4.0.0/prism/translation/ripper.rb

works it relies on upstream Ripper.
Which means it does not work without another implementation of Ripper/without parse.y (and there is no much point to it if you need to load the original Ripper anyway).

It also returns a different result than standard Ripper:

$ ruby -rripper -e 'p Ripper.lex("1 + 2")'                        
[[[1, 0], :on_int, "1", END], [[1, 1], :on_sp, " ", END], [[1, 2], :on_op, "+", BEG], [[1, 3], :on_sp, " ", BEG], [[1, 4], :on_int, "2", END]]

(the :on_sp are missing in Prism::Translation::Ripper)

For context we'd like to use Prism::Translation::Ripper to implement Ripper on TruffleRuby, since the ripper extension is causing endless problems, every single time we update the Ruby version and along with it the ripper extension: truffleruby/truffleruby#3481

Which of these 3 issues do you think could be solved?
@andrykonchin mostly worked through the first 2 issues on a branch at some point, would fixes for that be welcome in Prism?

[kddnewton]

For sure, anything that makes that layer better is more than welcome. I'm much less keen to support the on_sp event inside the parser itself since it seems like it would require a lot of code, but maybe we could deduce that by looking at adjacent tokens and inserting on_sp when necessary.

All of that is to say, I'm open to any changes here to make this work.

[Earlopain]

Keep in mind that prism only exposes the public api of ripper. I know that irb for example uses Ripper::Lexer::State which will not be present. Basically irb break immediatly after you require the shim inside the console.

[eregon]

Keep in mind that prism only exposes the public api of ripper. I know that irb for example uses Ripper::Lexer::State which will not be present. Basically irb break immediatly after you require the shim inside the console.

Right, I recall we ran into this, and not sure anymore what was the workaround/solution, I'll have to look.
Is it something that the translation layer could support?
How do you define public vs private API for Ripper? Here it seems Ripper#state and it is a public method though undocumented.

[eregon]

I've rebased @andrykonchin's branch and this is the changes we had: https://github.com/truffleruby/truffleruby/pull/4078/files

[Earlopain]

How do you define public vs private API for Ripper?

Not sure tbh. It looks like anything in ripper is fair game really. The entirety of Ripper::Lexer is marked #:nodoc: internal use only but you can see how well that worked out.

this is the changes we had

Yeah, that looks about what I would expect. Thanks for sharing. Many things are luckily defined in ruby so it wouldn't be too difficult to just move here.

Overall i'm not sure if prism should accomodate usage of private api. I'm tending to yes, if people use it then it wouldn't be a very good compatibility layer. I would like common gems like irb work at least (speaking for the shim that is). At the very least Ripper::EXPR_* since that seems to be somewhat necessary when working with ripper (irb, rdoc, syntax_suggest all make use of it) and it is returned by a public method.

But I will leave that to @kddnewton. irb and rdoc are already in the process of moving away from ripper so it may become moot in the not so distant future.

[eregon]

irb and rdoc are already in the process of moving away from ripper so it may become moot in the not so distant future.

Unfortunately that will still be too late for TruffleRuby, which would ship irb/rdoc from the same version as a CRuby release, and those rely on Ripper.
I raised the issue about a year ago to hope making it moot by now but it hasn't: ruby/irb#1024.
TruffleRuby could try to ship a newer irb/rdoc, but it can cause compatibility issues.

RSpec also relies on Ripper (at last some versions).
Deprecation might also take a while so gems might keep using Ripper for a long time.
So I think it will be quite useful if we can replicate the entire Ripper API or most of it with the translation layer.

[Earlopain]

1 and 2 are basically done. I don't plan to add other methods unless it becomes clear that someone actually uses them (Ripper.slice, Ripper.token_match, etc.)

on_sp is still not emitted and it looks like rdoc relies on that in some way. Maybe syntax highlighting/something related to indentation, not entirely sure yet.

maybe we could deduce that by looking at adjacent tokens and inserting on_sp when necessary.

This would be my prefered solution. I don't want to litter code into the parser itself only to support something that people are supposed to move on from.

[eregon]

Looks like we're going to use this soon in TruffleRuby because of truffleruby/truffleruby#4094 (comment) (too messy to use the Ripper extension)