Feature discussion: true machine platform gems for JRuby (java platform) ffi-based extensions

[byteit101]

Currently, the major Ruby implementations all support FFI, but there isn't a convenient way to distribute pre-built binaries via rubygems in an engine-agnostic way. CRuby and TruffleRuby can download the correct pre-built binary gems (like ffi-libarchive-binary) for their host platform (ie. x86-linux, etc), while JRuby, has the platform of java being undifferentiated by the underlying platform. As such, to package pre-built binaries for all of the 3 major Ruby implementations, one must make separate loading logic and archives for native platforms (just the correct binary), and the java platform (all binaries of all underlying platforms must be included)

I would like to propose a new feature to differentiate pre-built or underlying platforms from the current system of differentiating post-install built platforms. I see a few ways this can be done:

1. Add a new platform prefix of ffi- for gems that don't build or differentiate based on the Ruby platform, typically as they use FFI to access any included artifacts. Thus CRuby would try, for example x86-linux and ffi-x86-linux, while JRuby would try java and ffi-x86-linux. Post-install gems can use this too if they want to only install on some platform, regardless of being JRuby or not. This is the simplest to do today.
2. Add a new property engine_invariance or such that has the same affect as option 1
3. Introduce a new api property that is separate from the platform, and change how platform works. This involves a large migration, but I think is the most sane when taking a big picture approach. Native extension gems that use CRuby's C API would thus move to, for example, {api: "cruby", platform: "x86-linux"}, native extension gems that use JRuby's API would be {api: "java", platform: "x86-linux"}, and FFI gems would be something like {api: "ruby", platform: "x86-linux"} (unsure if "ruby" or nil or something else is better here). This would also work towards solving this issue, as prebuilt CRuby extensions could be api: "cruby-bin" which TruffleRuby could reject while allowing api: "cruby" in source form. Note that this IS NOT a proposal to encode the engine, but the extension API's that are required of the engine, hence why MRI and TruffleRuby both would support downloading similar gems. Install-time building of FFI binaries, like what sassc does, would be something like {api: "ruby", platform: nil }

I think option 3 is the most cogent and logically defensible for the long term, but it involves breaking a lot of compatibility and assumptions in gems along the way. Thoughts on these or any other options?

cc @headius @eregon @hsbt

[eregon]

I think option 3 is indeed the way to go, and would nicely solve this and related issues.

The mapping from existing Gem platforms would be (IIUC):

• binary platform-specific gem => { api: "cruby-bin", platform: "arch-os" }
• java platform gem => { api: "jruby", platform: nil } (because it's typically platform-independent), maybe { api: "jruby", platform: arch-os } for some cases where a native part is shipped.
• other gem, currently using the "ruby" platform (so either pure-Ruby or C extension in source form)
  => { api: nil, platform: nil }

For FFI gems or also sorbet-runtime, where the precompiled binaries depend on the native platform but does not use an extension API, they would opt-in to { api: nil, platform: "arch-os" }.

{api: "cruby", platform: "x86-linux"} would need to be opt-in (seems somewhat hard/hacky to guess), I'm not sure we need it, in practice existing extensions (in source form) would use { api: nil, platform: nil }.

How do we fit musl in this? Part of the platform string or a third component being the libc (with values: nil (any), glibc, musl)?

Related: #2945

[byteit101]

binary platform-specific gem => { api: "cruby-bin", platform: "arch-os" }

If we want to be lazy, then yes, though if we wanted to be clever we could scan the binaries for references to the CRuby Extension API and if missing, then mark them as FFI {api: nil, platform: "arch-os"}, though this is likely a minority of gems today (sorbet, ffi-libarchive-binary, etc)

java platform gem => { api: "jruby", platform: nil } (because it's typically platform-independent), maybe { api: "jruby", platform: arch-os } for some cases where a native part is shipped.

Yes, though I just realized that for consistency, unless they are shipping java sources to be compiled, it should probably be "jruby-bin". Unsure if that's a useful distinction for JRuby. In contrast, for CRuby (cruby, cruby-bin) vs TruffleRuby (only cruby) it does make sense, though.

other gem, currently using the "ruby" platform (so either pure-Ruby or C extension in source form)
=> { api: nil, platform: nil }

Well, I proposed "ruby" or nil as pure-ruby, or compiled but no extension API in use. But that is otherwise the correct target for that group of gems.

in practice existing extensions (in source form) would use { api: nil, platform: nil }

If you are talking about source C extensions, then no, that would be { api: "cruby", platform: nil } as the same gem can be downloaded on all platforms, but it uses the CRuby extension API when built.

How do we fit musl in this? Part of the platform string or a third component being the libc (with values: nil (any), glibc, musl)?

That's a good point, I conceptually think of it as part of the ABI, At the very least, GCC tacks it on like the ABI: aarch64-linux-musl, arm-linux-gnueabihf, etc. As A) musl is much less common than glibc, and B) binary gems are less common than source gems, I'm tempted to not worry about it too much. I tried to find out what rubygems does for the multitude of ARM ABI's (hf, el, v6, v7, etc), but I am unsure where to look. If ARM ABI's are in the platform, I'm inclined to put it in the platform as well. If ARM ABI's are not in the platform, I'm slightly inclined to put it in a third component: {api: nil, platform: "arm-linux", abi: "armhf-musl"}. Might also be useful to think of what to do to the other "fragmented" ISA's like ARM and RISC-V where ABI's change dramatically within one architecture. I don't really have any strong opinions here other than that it should be with the ABI.

Another key thing I didn't mention about option 3 is that we would need to really focus on updating docs and learning/getting started material as it would change the way gem developers need to think about the platform.

[eregon]

though if we wanted to be clever we could scan the binaries for references to the CRuby Extension API

I'm unsure if we can reliably detect that. For example the gemspec.extensions = does not on its own imply CRuby Extension API.

Yes, though I just realized that for consistency, unless they are shipping java sources to be compiled, it should probably be "jruby-bin".

I guess it could be useful if e.g. it's necessary to recompile the Java sources, e.g., due to using older/newer JDK or so.
I was also thinking there could be an api: "java" for https://github.com/jruby/jruby/wiki/CallingJavaFromJRuby where it's more of a portable java API (written in Ruby) vs the JRuby internals API used by Java code.

We might also want the possibility to have api: "truffleruby" which would be for Ruby files which use TruffleRuby-specific APIs (https://github.com/oracle/truffleruby/blob/master/doc/user/truffleruby-additions.md). This would also be useful if e.g. the dependencies of a gem should be different on TruffleRuby, this is currently the case for mini_racer for instance, where the libv8-node dependency is unnecessary on TruffleRuby and the TruffleRuby APIs are used instead. The current workaround is to install libv8-node as binary on truffleruby, which is rather messy in RubyGems & Bundler, and has caused issues.

Well, I proposed "ruby" or nil as pure-ruby, or compiled but no extension API in use. But that is otherwise the correct target for that group of gems.

I forgot to mention, I think the current RubyGems "ruby" platform is pretty confusing (e.g., it's also used for FFI gems and source C ext gems), so this seems a good opportunity to not use that and use the clearer nil (or maybe "any" if it must be a string).

If you are talking about source C extensions, then no, that would be { api: "cruby", platform: nil } as the same gem can be downloaded on all platforms, but it uses the CRuby extension API when built.

Right, but that depends on whether we can reliably detect that. Currently such gems are platform=ruby, the same platform used for pure-Ruby gems (and FFI gems like sassc).
If we can detect it reliably agreed more precise is best.

musl

There is some handling of musl in recent RubyGems, I don't know the details. I think a 3rd component is much cleaner and easier to reason about than some string suffix somewhere which has to be parsed, and there might be new values, etc.

By (native) platform I often see that as os-arch/arch-os and I don't expect other system configurations to end up there.
Also I'd think it's possible to have a system which has both glibc and musl, so IMHO better be a separate component.

Another key thing I didn't mention about option 3 is that we would need to really focus on updating docs and learning/getting started material as it would change the way gem developers need to think about the platform.

Right of course the documentation would need to be updated to reflect the changes. Although for almost all gems nothing would need to change in their gemspec or anything. Probably only binary FFI gems would need to specify api.

I'd love to have @deivid-rodriguez's opinion on this proposal.

[headius]

I also like proposal 3, if we can make it happen. This question has been lingering for a decade at least with little progress.

The main problem that has always blocked us from improving the platform system is the encoding of platform in the gem name. The platforms are selected by looking at the gem filename, to avoid opening each file and looking for platform metadata... so gems prebuilt for Windows are blah-gem-0.1-mswin32.gem and gems for JRuby are blah-gem-0.1-java.gem. In order to have more dimensions to the platform, we may have to abandon having all platform information encoded in the filename.

I guess it could be useful if e.g. it's necessary to recompile the Java sources

We have never supported this and I consider it pretty unnecessary, but it raises the question of JRuby gems that depend on a specific level of JDK... should the platform include a version number? java-17 as an API? But it also needs JRuby, and perhaps a specific version... jruby-9.4-java-17?

Obviously this quickly becomes unworkable.

Our recommendation in general is that your precompiled JRuby extension should support all Java versions that the target JRuby supports. For 9.3 and 9.4, this is Java 8, 11, and 17. For JRuby.next, we may jump to 17 only.

I will also point out there are parallel discussions about adding metadata to gems to define system-level dependencies (C libraries, OS packages, etc).