Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Bundler 2.4 blog post and release notes #978

Merged
merged 2 commits into from
Jan 31, 2023
Merged

Add Bundler 2.4 blog post and release notes #978

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
bf423e8
Add 'bundle gem --ext=rust' introduction blog post.
simi Dec 24, 2022
9395eb9
Add Bundler 2.4 to documentation
deivid-rodriguez Dec 22, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
167 changes: 167 additions & 0 deletions source/blog/2023-01-31-bundler-v2-4.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,167 @@
---
title: "Bundler v2.4: new resolver, gems with Rust extensions, and more"
date: 2023-01-31 14:43 UTC
tags:
author: David Rodríguez
author_url: https://github.com/deivid-rodriguez
category: release
---

2022 has been a busy year for the Bundler team, and we're glad to present
several improvements that we hope will make our users happy :)

## A better, PubGrub based, resolver

Bundler now uses the most advanced algorithm to resolve versions, PubGrub.
Kudos to Natalie Weizenbaum for [inventing
it](https://nex3.medium.com/pubgrub-2fb6470504f) and to John Hawthorn for
[porting it to Ruby](https://github.com/jhawthorn/pub_grub)!

Our previous resolver, [Molinillo](https://github.com/CocoaPods/Molinillo), worked pretty well, but it really got in the
middle when it didn't.

This may sound familiar for some:

~~~
$ bundle
Fetching gem metadata from https://rubygems.org/............
Resolving dependencies....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C
$ # Ok, that was enough waiting
~~~

Our new resolver, PubGrub, is usually much faster, because it learns from the
resolution conflicts it finds during the resolution process to avoid redoing the
same work over and over again. You can find more about this "conflict-driven
clause learning" techniques in its [presentation blog
post](https://nex3.medium.com/pubgrub-2fb6470504f) back from 2018.

Molinillo sometimes took too long to resolve because it would try the same
things, backtrack, and run into the same conflicts over and over again, having
to traverse very inefficiently a huge search space. But that was a relatively
rare case in the real world.

What's probably more common is specifying version requirements in your Gemfile,
that can't be all satisfied at the same time. This is when the version solving
problem does not have a solution, and when it becomes crucial to explain to users
_why_, so that they can fix their set of version requirements to become
solvable.

Molinillo run into trouble here, and in cases with many moving parts, like
upgrading Rails for example, it could end up printing a lot of conflicts, not
easy to understand and solve. This is an old example from a public ticket:

~~~
Bundler could not find compatible versions for gem "actionpack":
In Gemfile:
inherited_resources (= 1.6.0) was resolved to 1.6.0, which depends on
actionpack (>= 3.2, < 5)
rails (= 4.2.0) was resolved to 4.2.0, which depends on
actionpack (= 4.2.0)
Bundler could not find compatible versions for gem "activesupport":
In Gemfile:
inherited_resources (= 1.6.0) was resolved to 1.6.0, which depends on
has_scope (~> 0.6.0.rc) was resolved to 0.6.0, which depends on
activesupport (>= 3.2, < 5)
rails (= 4.2.0) was resolved to 4.2.0, which depends on
activesupport (= 4.2.0)
Bundler could not find compatible versions for gem "railties":
In Gemfile:
inherited_resources (= 1.6.0) was resolved to 1.6.0, which depends on
railties (>= 3.2, < 5)
rails (= 4.2.0) was resolved to 4.2.0, which depends on
railties (= 4.2.0)
inherited_resources (= 1.6.0) was resolved to 1.6.0, which depends on
responders was resolved to 1.1.2, which depends on
railties (>= 3.2, < 4.2)
~~~

Not easy to know what to do about it.

With PubGrub, you should now get human-readable explanations of failures. The
most complex cases may are still, well... complex. But explanations should
always make sense and point to the root cause of resolution failures.

Here's an example from our test suite:

~~~
Because every version of c depends on a < 1
and every version of b depends on a >= 2,
every version of c is incompatible with b >= 0.
So, because Gemfile depends on b >= 0
and Gemfile depends on c >= 0,
version solving has failed.
~~~

We tried to make this migration as backwards-compatible as possible, but
there's a chance of experiencing some different solutions to the ones found by
Molinillo, since the version solving problem does not have unique solutions.
Please report any issues you find with the new resolver.


### Easily generate gems with Rust extensions using `bundle gem`

It's now easier than ever to get started using Rust inside your gems. Check out
[this blog post](rust-gem-skeleton.html) to learn how to generate a gem with all
the boilerplate necessary with just a few commands.

### Faster git sources

In the Bundler world, it's common to point to git repositories when there's
no version released to rubygems.org that includes the changes that you need.
This works fine, but it can get slow and use a lot of disk space when dealing
with very big repositories.

We have improved the way we clone these repositories to be faster and use less
disk space. For example, something like

```ruby
gem "rails", github: "rails/rails"
```

in your Gemfile could previously take ~30s and use up to 1Gb of disk space,
because we would clone the full Rails repository, which has a large history.

Now we just clone what's strictly necessary for Bundler to work, resulting in
big disk space savings, and much faster bundling.

### New CLI features

We added a few small CLI features, such as a new `--pre` flag to `bundle update`
and `bundle lock` to explicitly opt-in to prereleases of selected (of all) gems
without having to explictly change your Gemfile with pre-release requirements
such as `>= 7.1.0.beta`.

### Some minor breaking changes

We took new year's release to move on and get rid of some stuff that was causing
maintenance burden for us:

* Ruby 2.3, 2.4, and 2.5 are no longer supported.
* RubyGems 2.5, 2.6, and 2.7 are no longer supported.

In general, this support drop should not break anything because RubyGems should
be able to choose the latest supported Bundler on the Ruby version that you're
using. But there are still some old RubyGems out there that don't have this
feature, and the `gem install bundler` command could break there. We have
warned using Bundler on those old rubies for a year now, so we believe it's time
to move on.

We also completely removed a controversial (mis-)feature from the Bundler code
base, where Bundler would automatically acquire sudo permissions when not having
the proper access rights. A great majority of users considered this feature harmful and
hardly useful, so we decided to get rid of it.

### And bug fixes

As always, we continue to smooth the experience of using Bundler, so that it
gets the job done and does not get in the middle other than that. And we're also
shipping a bunch of bug fixes to keep moving towards that goal.

That's all from the Bundler team. Have a happy new year, and enjoy using Bundler
2.4! 🎉
158 changes: 158 additions & 0 deletions source/blog/2023-01-31-rust-gem-skeleton.html.markdown
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,158 @@
---
title: "Generate gem skeleton with Rust extension"
date: 2023-01-31 14:42 UTC
tags:
author: Josef Šimánek
author_url: https://github.com/simi
category: feature
---

Do you think [dynamically typed](https://en.wikipedia.org/wiki/Dynamic_programming_language) [interpreted](https://en.wikipedia.org/wiki/Interpreter_(computing)) [Ruby language](https://www.ruby-lang.org/) and [statically typed](https://en.wikipedia.org/wiki/Type_system#Static_type_checking) [compiled](https://en.wikipedia.org/wiki/Compiled_language) [Rust language](https://www.rust-lang.org/) could be friends? Yes, they can! And actually, they are!

Officially it all started when [YJIT](https://github.com/ruby/ruby/blob/d5635dfe36588b04d3dd6065ab4e422f51629b11/doc/yjit/yjit.md) was [ported to Rust](https://bugs.ruby-lang.org/issues/18481) and [Ruby codebase](https://github.com/ruby/ruby) has officially [onboarded Rust code](https://github.com/ruby/ruby/tree/master/yjit/src). This friendship matured when RubyGems [3.3.11](https://rubygems.org/gems/rubygems-update/versions/3.3.11) (with a new [*Add cargo builder for rust extensions*](https://github.com/rubygems/rubygems/pull/5175) feature) [was released](https://blog.rubygems.org/2022/04/07/3.3.11-released.html) capable of compiling Rust-based extensions during gem installation process (similar to well-known C-based gem extensions like nokogiri, pg or puma).

And now, with Bundler 2.4, `bundle gem` skeleton generator can provide all the glue you need to start using Rust inside your gems thanks to the new `--ext=rust` parameter!

## What's new?

Thanks to new parameter it is possible to generate simple Rust-based gem extension.

*Make sure to use RubyGems 3.4.6 or higher for the best experience.*

*Notice I already have `bundle gem` command configured. Your output can differ. When running `bundle gem` for the first time, it will interactively ask you few questions.*

~~~
$ bundle gem --ext=rust hello_rust
Creating gem 'hello_rust'...
MIT License enabled in config
Initializing git repo in /home/retro/code/hello_rust
create hello_rust/Gemfile
create hello_rust/lib/hello_rust.rb
create hello_rust/lib/hello_rust/version.rb
create hello_rust/sig/hello_rust.rbs
create hello_rust/hello_rust.gemspec
create hello_rust/Rakefile
create hello_rust/README.md
create hello_rust/bin/console
create hello_rust/bin/setup
create hello_rust/.gitignore
create hello_rust/test/test_helper.rb
create hello_rust/test/test_hello_rust.rb
create hello_rust/LICENSE.txt
create hello_rust/Cargo.toml
create hello_rust/ext/hello_rust/Cargo.toml
create hello_rust/ext/hello_rust/extconf.rb
create hello_rust/ext/hello_rust/src/lib.rs
Gem 'hello_rust' was successfully created. For more information on making a RubyGem visit https://bundler.io/guides/creating_gem.html
~~~

For Rust-based extension last 4 entries are interesting.

- `hello_rust/Cargo.toml`
- Top-level `Cargo.toml` is just pointing to "nested" `Cargo.toml` in `ext` folder.
- It is useful to be able to run all `cargo` commands in top-level directory (next to `bundle`, `gem`, ...).
- It is also useful for your IDE to be able to recognize there is Rust code in this folder, but not in standard path for Rust crate.
- `hello_rust/ext/hello_rust/Cargo.toml`
- Actual `Cargo.toml` as known from Rust crates. It includes package metadata, configuration and dependencies. You can think of this file as a "gemspec for Rust packages".
- `hello_rust/ext/hello_rust/extconf.rb`
- Config file responsible for configuration of compilation of your Rust code in Ruby world (for example during gem installation).
- Currently based on [rb_sys gem](https://github.com/oxidize-rb/rb-sys/tree/main/gem#the-rb_sys-gem). Check [project README](https://github.com/oxidize-rb/rb-sys/tree/main/gem#create_rust_makefile) for more info.
- `hello_rust/ext/hello_rust/src/lib.rs`
- Yes, the holy grail of Rust-based extension - the Rust code!

## Hello from Rust!

Generated `hello_rust/ext/hello_rust/src/lib.rs` contains hello world example method defined at base class of extension. In my case it is `HelloRust#hello` with 1 string argument returning string as well. It is using [magnus](https://github.com/matsadler/magnus) Rust bindings to Ruby for super smooth developer experience.

~~~rust
# hello_rust/ext/hello_rust/src/lib.rs
use magnus::{define_module, function, prelude::*, Error};

fn hello(subject: String) -> String {
format!("Hello from Rust, {}!", subject)
}

#[magnus::init]
fn init() -> Result<(), Error> {
let module = define_module("HelloRust")?;
module.define_singleton_method("hello", function!(hello, 1))?;
Ok(())
}
~~~

That is equivalent to following Ruby code, including some boilerplate code, to enable Rust extension to communicate with Ruby.

~~~ruby
module HelloRust
def self.hello(subject)
"Hello from Rust, #{subject}!"
end
end
~~~

## Let's compile and run some Rust!

To be able to test this boilerplate code, you need to run `bundle install` first (to install all Ruby dependencies) followed by `bundle exec rake compile` compiling Rust code.

*Notice generated gemspec is not valid by default and running `bundle install` can break. In that case it is needed to update gemspec first and replace all TODO values with some real ones.*

~~~
$ bundle install
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies...
Using rake 13.0.6
Using bundler 2.4.0
Using hello_rust 0.1.0 from source at `.`
Using minitest 5.16.3
Using rake-compiler 1.2.1
Using rb_sys 0.9.52
Bundle complete! 5 Gemfile dependencies, 6 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
~~~

At this stage, everything is ready to compile Rust code and glue it with Ruby.

*You need to have Rust already installed on your system. See [rustup](https://rustup.rs/) for a simple installation experience.*

~~~
$ bundle exec rake compile
mkdir -p tmp/x86_64-linux/hello_rust/3.1.2
cd tmp/x86_64-linux/hello_rust/3.1.2
/home/retro/.rubies/ruby-3.1.2/bin/ruby -I. -r.rake-compiler-siteconf.rb ../../../../ext/hello_rust/extconf.rb
cd -
cd tmp/x86_64-linux/hello_rust/3.1.2
/usr/bin/gmake
generating target/release/libhello_rust.so (release)
cargo rustc --target-dir target --manifest-path ../../../../ext/hello_rust/Cargo.toml --lib --release -- -C linker=gcc -L native=/home/retro/.rubies/ruby-3.1.2/lib -C link-arg=-lm
Updating crates.io index
... shortened
Compiling magnus-macros v0.2.0
Compiling rb-sys-build v0.9.52
Compiling rb-sys v0.9.52
Compiling hello_rust v0.1.0 (/home/retro/code/hello_rust/ext/hello_rust)
Finished release [optimized] target(s) in 1m 03s
cd -
mkdir -p tmp/x86_64-linux/stage/lib/hello_rust
/usr/bin/gmake install target_prefix=
generating target/release/libhello_rust.so (release)
cargo rustc --target-dir target --manifest-path ../../../../ext/hello_rust/Cargo.toml --lib --release -- -C linker=gcc -L native=/home/retro/.rubies/ruby-3.1.2/lib -C link-arg=-lm
Finished release [optimized] target(s) in 0.09s
installing hello_rust.so to /home/retro/code/hello_rust/lib/hello_rust
/usr/bin/install -c -m 0755 hello_rust.so /home/retro/code//hello_rust/lib/hello_rust
cp tmp/x86_64-linux/hello_rust/3.1.2/hello_rust.so tmp/x86_64-linux/stage/lib/hello_rust/hello_rust.so
~~~

And finally, it is possible to call `hello` method defined in Rust returning a string and printing it to the console.

~~~
$ bundle exec ruby -rhello_rust -e 'puts HelloRust.hello("Josef")'
"Hello from Rust, Josef!"
~~~

*Feel free to try to break this extension. For example you can try to pass different types of argument (like number or symbol). [magnus](https://github.com/matsadler/magnus) is doing a great job [automatically converting](https://github.com/matsadler/magnus#defining-methods) all those mistakes with friendly error messages.*

## Summary

Starting Bundler 2.4, you can generate gem skeleton with all boilerplate code needed to start using Rust. But it is not only about your custom Rust code you can easily integrate into gems now. Thanks to integration with [cargo](https://doc.rust-lang.org/cargo/) (Rust package manager) you can use any of [Rust crates](https://crates.io/) available. Rust ecosystem is well known for highly optimized and memory safe libraries. Thanks to [magnus](https://github.com/matsadler/magnus) and `bundle gem` command, it is possible to glue those Rust libraries into Ruby world smoothly. Sky is the limit ;-)

*To see real-life example how powerful could be Rust for data processing, I recommend to check [kirby project](https://github.com/rubytogether/kirby) parsing logs for [rubygems.org](http://rubygems.org/).*
33 changes: 32 additions & 1 deletion source/v2.4/whats_new.html.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,35 @@
# What's New in v2.4

As always, a detailed list of every change is provided in
The [Bundler 2.4 announcement](/blog/2022/12/22/bundler-v2-4.html)
includes context and a more detailed explanation of the changes in this version. This is a summary of the biggest changes. As always, a detailed list of every change is provided in
[the changelog](https://github.com/rubygems/rubygems/blob/3.4/bundler/CHANGELOG.md).

### A new PubGrub based resolver

Bundler now uses [pub_grub](https://github.com/jhawthorn/pub_grub) under the
hood to resolve versions. The most advance algorithm to approach the version
solving problem! 💪

### Generate gems with Rust extensions

`bundle gem` now supports Rust! Pass the `--ext=rust` flag to generate a gem
with a Rust extension.

### Faster git sources in Gemfile

Git sources in Gemfile now work faster and use less disk space.

### Old Ruby and RubyGems no longer supported

Support for Ruby 2.3, 2.4, and 2.5, and RubyGems 2.5, 2.6, and 2.7 has been dropped.

### No more auto-sudo

Bundler no longer tries to use `sudo` to upgrade privileges under any circumstances.

### Other improvements

Bundler 2.4 also includes other improvements like a new `--pre` flag
to `bundle lock` and `bundle update` to explicitly opt-in to prereleases.

<a href="https://github.com/rubygems/rubygems/blob/3.4/bundler/CHANGELOG.md" class="btn btn-primary">Full 2.4 changelog</a>