Use Narwhals to support Polars and Modin DataFrames

[mwouts]

This PR add tests for Modin dataframes. It is on top of @DeaMariaLeon's PR to add support for more DataFrame types using Narwhals at #339 .

From this quick experiment I would say that the nbytes function (estimate the size of the dataframe in bytes) and the downsample functions (keep left and right most columns, top and bottom rows) could benefit a bit more from Narwhals.

[github-actions]

Thank you for making this pull request.

Did you know? You can try it on Binder: .

Also, the version of ITables developed in this PR can be installed with pip:

pip install git+https://github.com/mwouts/itables.git@use_narwhals

(this requires nodejs, see more at Developing ITables)

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 84.47205% with 25 lines in your changes missing coverage. Please review.

Project coverage is 91.95%. Comparing base (a3dfd99) to head (f3d81a2).

Files with missing lines	Patch %	Lines
tests/test_ibis.py	27.77%	13 Missing ⚠️
src/itables/sample_dfs.py	79.31%	12 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #341      +/-   ##
==========================================
- Coverage   94.76%   91.95%   -2.81%     
==========================================
  Files          24       26       +2     
  Lines        1280     1343      +63     
==========================================
+ Hits         1213     1235      +22     
- Misses         67      108      +41     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mwouts]

@DeaMariaLeon @MarcoGorelli I hope to finalize this PR in the coming days (sorry for not being able to find time for that earlier!)

I plan to

• update the docs to mention the fact that we use narwhals to support the other types of dataframes
• test a few more types of dataframes
• see if I can do something about the performance¹ when displaying Modin dataframes

Footnotes

1. Running the tests on the modin dataframes takes twice longer, and some dataframes take a long time to display, see e.g. the "wide" dataframe in the documentation that takes 318ms when using Pandas, 68ms when using Polars (through Narwhals), and 3.54s when using Modin!!
   ↩

[mwouts]

I can't seem to be able to use Narwhals for rendering Ibis dataframes - at least I can't call len(nw.from_native(...)) on either

t = ibis.table(dict(one="string", two="float", three="int32"), name="my_data")

or

t = ibis.memtable(pd.DataFrame(
    [["a", 1, 2], ["b", 3, 4]],
    columns=["one", "two", "three"],
    index=[5, 6],
), name="t")

The error I get is the following:

NotImplementedError: Attribute __len__ is not supported for metadata-only dataframes.

If you would like to see this kind of object better supported in Narwhals, please open a feature request at https://github.com/narwhals-dev/narwhals/issues.

[MarcoGorelli]

hey @mwouts - at the moment, our support for Ibis is only "interchange-level" (https://narwhals-dev.github.io/narwhals/extending/#interchange-only-support). This means you can only do the following:

• .schema, hence column names via .schema.names() and column types via .schema.dtypes()
  -.columns
• .to_pandas() and .to_arrow(), for converting to Pandas and Arrow, respectively.
• .select(names) (Ibis and DuckDB), where names is a list of (string) column names. This is useful for selecting columns before converting to another library.

The plan is to implement lazy support for them relatively soon in early 2025: narwhals-dev/narwhals#1657. We've got something cooking for PySpark and DuckDB, but we'll do the same for Ibis too. However, even once we have Ibis supported, calling len on a narwhals.LazyFrame won't be supported. Indeed, it's not even supported for ibis.Table, as they follow a deferred execution model

What you could do now (and this is what Marimo do) is:

df = nw.from_native(df_native)
if nw.get_level(df) == 'interchange':
    df = df.to_pandas()  # or df.to_arrow(), if you support PyArrow

and then continue with the rest of the logic