Feature: gem aliases/provides/replaces

[ioquatix]

I'd like to kindly suggest a feature and discuss some ideas.

My current problem is I am using the "mail" gem, and I'd like to replace the "mime-types" gem with a more efficient, API compatible gem, say, called "mime-types-native". I have several other gems along with "mail" which all depend on "mime-types" and I'd like to replace "mime-types" project wide, i.e. in all dependencies of my project too.

Additionally, some times this problem shows up in a more limited form when people have published a gem with a very restrictively versioned dependency. For example, where they might have meant to say "> 2.2" they actually wrote "> 2.2.1". This is actually a relatively common problem, and I'll try to add some references to issues bringing this up.

This issue is related to installing a library and dependency resolution.

There are a couple of ways to address this. I'll use Arch's pacman package manager as one example.

Firstly, a pacman package has several fields: provides, conflicts and depends. These are independent of the package name. For example, there are several packages which provide java.

Here is the JRE 7 package "Gemfile": https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/java7-openjdk

Here is the JRE 8 package "Gemfile": https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/java8-openjdk

Note the lines starting with provides (there are several discrete package built from this PKGBUILD file).

Essentially, with RubyGems, the only way to require functionality is to specify the exact name of the gem.

Therefore, it's not possible to make another gem, which replaces the functionality of an existing gem, and include that as a replacement. This limitation becomes an issue when using a tool such as bundler which does dependency/version management and enforces strict rules on what can and can't be installed.

I'd like to add support, without any specific requirement on syntax, to replace a package with a different package.

I'd like to suggest some possible syntax for this:

# Idea 1: Simply ignores this gem as a dependency, then we specify another one later on:
gem "mime-types", ignore: true
gem "mime-types-native", "~> 1.0"

# Idea 2: Override this gem with different version requirements
gem "mime-types", override: "~> 3.0"

# Idea 3: Mark a gem to be replaced by another which is specified elsewhere.
gem "mime-types", provided_by: "mime-types-native"
gem "mime-types-native", "~> 1.0"

In the last case, provided_by is functionally similar to git: but of course it's not pulling the gem from git but just another gem. The ideas above are not concrete but meant to be a starting point for discussion.

I think the core problem here is the inability to specify "I depend on this specific gem" vs "I depend on someone providing this API".

I will abide by the code of conduct.

[qrush]

That's a neat idea! However this sounds more like a Bundler feature. Bundler does consume rubygems.org's API - but I wonder if this could all be implemented on the client.

Any thoughts @indirect @arthurnn ?

[ioquatix]

@qrush My feeling is that @indirect is not keen on this idea in general being implemented directly in bundler. I sort of agree with this stance as I think it needs to be implemented in RubyGems because it would modify what is possible to express in a Gemfile.

Some related issues on Bundler:

Closed issue relating to changing version requirements: rubygems/bundler#1549, rubygems/bundler-features#20 (both closed) and finally rubygems/bundler#4552 (still open)

This issue discusses having multiple gems of the same name, but for different platforms: rubygems/bundler#751

There might be other issues which surround this feature request but I'm not sure of them. If possible, it would be good to link to them from here so that whatever proposal we come up with correctly solves the problems people are having.

[ioquatix]

Another related issue: rubygems/bundler#1549

I found this comment made an interesting proposal:

gem 'sass-rails' do
  override 'railties', '4.0.0'
end

In addition, the reply from @indirect

This request is fundamentally requesting the ability to monkeypatch a gem's code (specifically the gemspec) inside your Gemfile. The entire reason we added the :git option to Bundler was to allow a way to fix gems without monkeypatching

This is an interesting position. I'd agree that monkey patching (e.g. by somehow overriding the Gemspec data structure for that gem) is a bad idea.

Instead I'd suggest providing a layer of indirection in the top level Gemfile data structure so that the gem, say in my example case, gem "mime-types" actually has an alias which says "rather than installing this gem to satisfy this requirement, either 1/ ignore it or 2/ install this other gem (which might be the same gem but with a different version). Therefore, we avoid any kind of monkey patching.

[andy-twosticks]

My 10c, kinda devils advocate thing:

First, anyone making use of this feature is giving themselves a lot of hard work. Any gem you declare as a replacement for another gem is going to have to be 100% compatible. It will have to behave exactly the same (Bear in mind, also, the grey line between public/private interface, and how much Ruby code steps over that line.) And when the gem you are replacing changes, you will have to change too.

Maybe this is fine ... until someone packages the resulting code as a gem, and other people use it...

Second: same deal, other way around. Right now you can have a gem Foo that depends on gem Bar. All the tests in Foo can safely assume that it is the actual Bar gem. If Bar isn't clear about what is a public method and what is a private method, that's almost entirely a problem for the developer of Foo. With this modification, does she have to assume that some other gem might replace Bar in the actual implementation? If so, the job of coding Foo just got way harder.

Again, maybe this is okay. You can take the view that if you decide to override Bar with Baz, it's your lookout if Foo now does not work as designed. ...at least, so long as Foo isn't a gem that other code has as a dependancy, at which point it becomes, well, quite confusing for everyone...

[ioquatix]

@andy-twosticks That is a well thought out contribution to the discussion. However, those arguments apply, perhaps even more-so, to the git source feature that is already supported. Therefore, the liability of my proposal in this specific regard, has already been accepted to some extent.

[duckinator]

Just want to say I like this idea. I'm not entirely sure how your first two ideas are supposed to work, but your third suggestion (gem "mime-types", provided_by: "mime-types-native") makes sense to me, and seems rather straightforward.

---

However, I want to verify I completely understand your suggestion. So:

Assuming that gem foo has the following Gemfile:

gem 'mime-types'

and gem bar has the following Gemfile:

gem "mime-types", provided_by: "mime-types-native"
gem "mime-types-native", "~> 1.0"

and gem baz has the following Gemfile:

gem 'foo'
gem 'bar'

Then baz would use mime-types-native, not mime-types, correct? I'd expect that, because it's the most specific of the two provided, and you're explicitly declaring them as API-compatible.

[duckinator]

Also:

An alternative syntax could be:

gem "mime-types-native", "~> 1.0", provides: "mime-types"

And I just noticed a potential issue: How do you specify what version a gem is equivalent to?

E.g., say version 1.0 of mime-types-native is API-compatible with version 3.0 of mime-types. I've seen this kind of thing before, and it'd need to be handled.

And having both the version and provides: is already messy. Adding a second version could get very confusing.

Extending my alternative syntax above, perhaps something like,

gem "mime-types-native", "~> 1.0" do
  provides "mime-types", "~> 3.0"
end

(EDIT to add:)

or, extending yours:

gem "mime-types", "~> 3.0" do
  provided_by "mime-types-native", "~> 1.0"
end

[ioquatix]

@duckinator Yes that's in line with what I'm thinking.

I welcome discussion about how this could work or alternatives.

With regards to the ideas 1 and 2: they are not direct alternatives, but complementing features/ideas. I could go into more details but at this point I'd rather not try to be too specific with my own idea but invite everyone to contribute what they think would make sense and see if we can form a consensus.

[ioquatix]

I like your proposed syntax:

gem "mime-types-native", "~> 1.0", provides: "mime-types"

And I just noticed a potential issue: How do you specify what version a gem is equivalent to?

I think the simplest answer to this is that you don't need to. You assume if the user is specifying a different gem that they are taking full responsibility for the versioning.

Alternatively you could be explicit, perhaps something like

gem "mime-types-native", "~> 1.0", provides: "mime-types", equivalent_to: "3.0.0"

[duckinator]

Assuming mime-types-native 1.0 is compatible with mime-types 3.0, but mime-types-native 0.9 is not:

gem foo Gemfile:

gem "mime-types", "~> 3.0"

gem bar Gemfile:

gem "mime-types-native", "~> 0.9", provides: "mime-types"
gem "foo"

This should raise an error during bundle install because the versions are incompatible, but it can't because you have not provided that information. That's what I was trying to resolve with that last comment.

[ioquatix]

@duckinator do you mean for "gem foo Gemfile":

gem "mime-types", "~> 3.0"

[duckinator]

@ioquatix ah, yeah! Updated my comment accordingly. Thanks.

[tonobo]

+1

[deivid-rodriguez]

Hi! I'm 👎 on this proposal. Essentially because of the reasons given in the answered quoted from @indirect:

This request is fundamentally requesting the ability to monkeypatch a gem's code (specifically the gemspec) inside your Gemfile. The entire reason we added the :git option to Bundler was to allow a way to fix gems without monkeypatching

I feel this is a very niche use case, and the :git source already gives you the ability to temporarily patch a gemspec (or any other code in a dependency).

Supporting this directly by extending the Gemfile DSL feels like encouraging the wrong thing to me. Gemspecs should be fixed at their source, not at end users Gemfiles. If a gem needs to relax a dependency, the gem should change its gemspec. If a gem needs to replace a dependency with a better alternative, the gem should change its gemspec too. Giving people a way of changing this directly in their Gemfiles feels like discouraging people from actually fixing things in the right place, so that they benefit the whole community.

Plus, I think we have enough things to deal with so I don't want to maintain this new feature.

[ioquatix]

I'm fine with your position, but generally speaking, many modern package managers do support some kind of aliases, e.g. pacman is the first one that comes to mind with depends and provides per package, so it's definitely not a niche feature. I disagree that it's only about "monkey patching" because in practice, you can see things like variants of code bases, e.g. mariadb provides mysql. It's fair enough that one gem could provide the interface of another. The depends, provides and conflicts model of pacman is very useful and well used in practice.

[deivid-rodriguez]

@duckinator Are you still on board with this? I'd like to gather more maintainers opinions and make a decision. As I explained, I'm negative on adding this.

[duckinator]

Revisiting this, I just had a realization: there is a fundamental difference between how pacman and such approaches it and how this would work.

Let's assume we have these 3 gems to work with:

• foo: Your project.
• bar: The original dependency, but it's abandoned.
• bar-new: A new, API-compatible replacement.

With pacman, this would be solved by bar-new saying it provides bar. In other words, bar-new says "I am API compatible with bar. If someone depends on bar, I satisfy that dependency."

With this proposal, it would be foo saying that bar-new provides bar. In other words, foo says "Hey, I know neither of them say this, but bar-new is API compatible with bar, just use that."

---

The idea is fundamentally different, because of who is declaring it. So I agree with @indirect and @deivid-rodriguez — this feels significantly closer to monkeypatching than what pacman and other package managers provide, and I don't think it's a good solution to the problem.

[ioquatix]

@duckinator I have no problem with what you proposed (i.e. bar-new declaring what it supports using provides).

[Fryguy]

@ioquatix I stumbled upon this via the old mini_mime issue. To solve some similar use cases, we built a gem we called bundler-inject that allows you to specify overrides of gems. Would this be useful for you? https://github.com/ManageIQ/bundler-inject

[ioquatix]

Another data point / relevant discussion: https://www.reddit.com/r/ruby/comments/1ctcs5m/when_maintainers_abandon_ignore_their_gemscan/

[duckinator]

I think this problem has become prominent enough that it should be looked into.

Based on all the input various people (including my past self) have had, I came up with an option I think might be good.

Example situation

For this example, we'll use one of my gems, okay.

Okay v12.0.4 exists. Let's say, for whatever reason, I stop maintaining it.

It's 3 years later, and something breaks.

So someone forks it, and renames it fine.

Gem implementing the API

The fork (fine) can say it provides an equivalent API:

Gem::Specification.new do |spec|
  # [...]
  
  s.provides_api "okay", "12.0.4"
  
  # [...]
end

Gem using the API

Let's say some gem used okay originally, like this:

gem "okay", "~> 12.0"

But then they run into problems, and learn about the fine fork.

Now they can expand that gem call into this:

api "okay", "~> 12.0 do
  provided_by "fine", "~> 1.0"
end

Rationale

1. Puts upstream projects (the fork) in charge of the provided API.
2. Relatively straightforward conceptually: We treat the provided_by gem/version as if it had the api gem/version, for the Gemfile it's specified in.

[duckinator]

@ioquatix @deivid-rodriguez before I start pulling other people in for opinions, what do you two think of my proposal?^

[simi]

@duckinator IMHO this is too complex. With namespaced gems, this should be fixed (if implemented as different sources). You can provide same gem name, just from different source (aka your namespace). Introducing the provided_by and api seems too much and it will not be easy for resolver to adopt. Namespaced gems (on rubygems.org) will work nicely with current resolver already.

[simi]

Also looking at this again the problem is mostly with abandoned gems. But if I understand it well, provides_api will need to be present in abandoned gem. If you can control gemspec and publish gem, you can fix it. If not, this feature will not solve your problem.

[deivid-rodriguez]

I agree with @simi, namespaced gems make the most sense to me right now. Should we close this and keep the discussion at rubygems/rfcs#54? We can eventually move back here once we agree on what to implement.

[duckinator]

this is too complex

I've not seen a simpler solution that actually addresses the problem of "foo 5.3 is abandoned, but bar 1.x is API-compatible".

Introducing the provided_by and api seems too much and it will not be easy for resolver to adopt.

The simplest implementation of this would be, given:

api "okay", "~> 12.0" do
  provided_by "fine", "~> 1.0"
end

... it would stick a fake okay 12.0 gem into the resolver that does nothing but depend on fine ~> 1.0.

It does not even need server-side accommodation, even when you add validation of it.

In fact, if we don't want to add s.provides_api "okay", "12.0.4" we could go with s.metadata["provides_api"] = ["okay", "12.0.4"] in practice. That would limit the change entirely to adding the api/provided_by combination to Bundler.

But if I understand it well, provides_api will need to be present in abandoned gem.

I'm not sure how you came to this conclusion. The original is not involved at all aside from the name being mentioned. Giving the fork (not the upstream, nor the end-user) the power to declare what they're compatible with is the entire point of my proposal — as I stated, "The fork can say it provides an equivalent API".

With namespaced gems, this should be fixed

namespaced gems make the most sense to me right now

What are y'all referring to, here? How do namespaces of any sort solve this?

[simi]

@duckinator let's say you'll be able to push gem into rubygems.org custom namespace and namespace will be custom source, you'll be able to do following

source 'https://rubygems.org/namespace/simi' do
  gem 'abandoned_gem_in_at_rubygems_org'
end

You'll be able to publish own version of the gem, but in your namespace. Since dependencies are just name + version constraints, any source will be eligible to provide the gem.

[duckinator]

@simi is there somewhere I can keep track of the current state of namespaces?

[simi]

@duckinator the discussion is sadly spread across multiple places (issue tracker, discussions, RFCs issues, ... and also Slack). IMHO it is good time to open RFC and unify us in there. I'll work on that.

[ioquatix]

While I believe namespaces are useful and acknowledge that they address some of the same issues as provides, they are fundamentally different. Namely, namespaces focus on managing ownership and security, whereas provides deals with dependency resolution.

In particular, I aim to create "composite gems," which are essentially amalgamations of multiple individual libraries (gems). These "composite gems" should provides every dependency they encapsulate. Namespaces do not address this need.

The value of "composite gems" lies in their ability to allow me, as an author of numerous discrete components, to release a single package that combines multiple compatible libraries. This approach is beneficial for various audiences, including businesses that prefer to manage fewer dependencies by using a single, well-tested package.

[duckinator]

I have to agree with @ioquatix here. I think namespaces solve some of the problems raised in this issue, but not all of them.

It solves the hard-fork problem. It doesn't solve "composite" gems, and it doesn't solve new gems that implement compatibility APIs.

For hard forks without a rename, I think namespaces are the better option. For renames, composite gems, or compatible-but-different gems, I don't think they help.

In terms of composite gems: A good example would actually be Rails. It exposes a lot of APIs it doesn't implement itself.

[simi]

Can you show some real example of what are you looking to cover with those new concepts @ioquatix and @duckinator?

[duckinator]

Composite gems: Rails treats a lot of its dependencies as part of itself. Many smaller gems also do this.

Renames: I don't have one off-hand.

Compatible-but-different gems: json vs json_pure. Same API, same devs, but different implementation (C vs Ruby).

[simi]

Composite gems: Rails treats a lot of its dependencies as part of itself. Many smaller gems also do this.

I have never heard this being problem for rails. What problem will be solved by this? Is it related to ActiveRecord and individual adapters having different dependencies?

Compatible-but-different gems: json vs json_pure. Same API, same devs, but different implementation (C vs Ruby).

I understand this in theory, but is this causing any real troubles? I remember https://github.com/intridea/multi_json to solve this partially, but your whole bundle needs to support this and that's usually not happening. Any details on this?

[ioquatix]

Composite gems - I have never heard this being problem for rails.

I think the assertion is that there could be one rails v8.0.0 gem that provides "activerecord", "8.0.0" and all the related components etc. I could imagine that reducing load path overhead too.

Compatible-but-different gems - I understand this in theory, but is this causing any real troubles?

Regarding the "compatible-but-different" there is a good example here: mime-types/ruby-mime-types#123. In short, we wanted to provide a native implementation called mime-types-mini of the mime-types gem with a compatible interface.

Composite gems

I'd like to release a bundled version of falcon that includes all the dependencies.

[simi]

I think the assertion is that there could be one rails v8.0.0 gem that provides "activerecord", "8.0.0" and all the related components etc. I could imagine that reducing load path overhead too.

I don't follow. What kind of problem is this going solve? Is there any discussion in activerecord by its maintainers looking for this feature?

Regarding the "compatible-but-different" there is a good example here: mime-types/ruby-mime-types#123. In short, we wanted to provide a native implementation called mime-types-mini of the mime-types gem with a compatible interface.

Is this simlar to tzinfo-data gem? That one is optional and added only for related platforms.

I'd like to release a bundled version of falcon that includes all the dependencies.

What's the blocker on this? You can vendor all your deps and release in isolation, even outside of RubyGems/RubyGems.org like other projects did in history (to disconnect them from Ruby ecosystem) like Chef.

[ioquatix]

Is this simlar to tzinfo-data gem? That one is optional and added only for related platforms.

I have no idea about tzinfo-data. What we wanted was to provide a different gem, with the same interface as the mime-types gem, with a completely different internal implementation.

I'd like to release a bundled version of falcon that includes all the dependencies. - What's the blocker on this?

It's simply not possible to release a single gem which is an amalgamation of other gems without breaking dependency resolution currently?

[ioquatix]

I don't follow. What kind of problem is this going solve?

From the angle you are looking, it doesn't solve any problem at all. Every problem is already solved by the explicit DAG. However, some people look at that complexity and see an ergonomic and optics problem.

I know from discussions with businesses, that they see a large number of implicit dependencies as a problem - even if it's not, technically speaking.

A single dependency like gem "falcon" pulls in this:

GEM
  remote: https://rubygems.org/
  specs:
    async (2.11.0)
      console (~> 1.25, >= 1.25.2)
      fiber-annotation
      io-event (~> 1.5, >= 1.5.1)
      timers (~> 4.1)
    async-container (0.18.2)
      async (~> 2.10)
    async-http (0.66.3)
      async (>= 2.10.2)
      async-pool (>= 0.6.1)
      io-endpoint (~> 0.10, >= 0.10.3)
      io-stream (~> 0.4)
      protocol-http (~> 0.26.0)
      protocol-http1 (~> 0.19.0)
      protocol-http2 (~> 0.17.0)
      traces (>= 0.10.0)
    async-http-cache (0.4.3)
      async-http (~> 0.56)
    async-pool (0.6.1)
      async (>= 1.25)
    async-service (0.12.0)
      async
      async-container (~> 0.16)
    console (1.25.2)
      fiber-annotation
      fiber-local (~> 1.1)
      json
    falcon (0.47.6)
      async
      async-container (~> 0.18)
      async-http (~> 0.66, >= 0.66.3)
      async-http-cache (~> 0.4.0)
      async-service (~> 0.10)
      bundler
      localhost (~> 1.1)
      openssl (~> 3.0)
      process-metrics (~> 0.2.0)
      protocol-rack (~> 0.5)
      samovar (~> 2.3)
    fiber-annotation (0.2.0)
    fiber-local (1.1.0)
      fiber-storage
    fiber-storage (0.1.1)
    io-endpoint (0.10.3)
    io-event (1.5.1)
    io-stream (0.4.0)
    json (2.7.2)
    localhost (1.3.1)
    mapping (1.1.1)
    openssl (3.2.0)
    process-metrics (0.2.1)
      console (~> 1.8)
      samovar (~> 2.1)
    protocol-hpack (1.4.3)
    protocol-http (0.26.5)
    protocol-http1 (0.19.1)
      protocol-http (~> 0.22)
    protocol-http2 (0.17.0)
      protocol-hpack (~> 1.4)
      protocol-http (~> 0.18)
    protocol-rack (0.5.1)
      protocol-http (~> 0.23)
      rack (>= 1.0)
    rack (3.0.11)
    samovar (2.3.0)
      console (~> 1.0)
      mapping (~> 1.0)
    timers (4.3.5)
    traces (0.11.1)

PLATFORMS
  arm64-darwin-23
  ruby

DEPENDENCIES
  falcon

BUNDLED WITH
   2.5.5

Using composite gems would be one strategy to reduce the size of this list and improve the ergonomics and optics for companies who see such a large list as a risk.

There is a second advantage, which is managing compatibility. A company may feel concern about knowing whether the above list of gems is compatible with each other. As a developer, I can release composite gems which are tested together and provide assurances around compatibility.

The same is probably true for rails, e.g.

    rails (7.1.3.3)
      actioncable (= 7.1.3.3)
      actionmailbox (= 7.1.3.3)
      actionmailer (= 7.1.3.3)
      actionpack (= 7.1.3.3)
      actiontext (= 7.1.3.3)
      actionview (= 7.1.3.3)
      activejob (= 7.1.3.3)
      activemodel (= 7.1.3.3)
      activerecord (= 7.1.3.3)
      activestorage (= 7.1.3.3)
      activesupport (= 7.1.3.3)
      railties (= 7.1.3.3)

All of these could just be one composite gem. It greatly simplifies what people are exposed to and the optics associated with dependency management. There is no reason for this code to be split out like this, except that sometimes people depend on specific parts of it (which won't change in my proposal).

Some of the other ancillary advantages I can think of:

• Probably faster to install (less dependencies).
• Probably faster to load (less load paths).
• Easier for users to do the right thing (just install one gem, and have one logical entry in the lock file).

[ntkme]

Another real world example of compatible-but-different is sassc vs sassc-embedded.

sassc has been deprecated for more than 3 years. In order to help the community migrate to sass-embedded (dart-sass) without huge effort, I maintain an API bridge called sassc-embedded, which emulates the SassC public API using the sass-embedded as a backend. However, because lots of rails projects have gem dependencies that declare sassc as a transitive dependency (e.g. font-awesome). They would still force the user to install sassc, which takes ~1 minutes to compile native extension, even if users no longer want to use it.

Also, if a user has both sassc and sassc-embedded installed, they currently need to use some dirty workaround to replace sassc with sassc-embedded, which is painful: https://github.com/sass-contrib/sassc-embedded-shim-ruby?tab=readme-ov-file#troubleshooting

In contrast, this can be easily done with npm in node.js world, for example: npm i -D sass@npm:sass-embedded@latest, and it would replace sass npm package with sass-embedded npm package, which provides the same API with different implementation.