Release Minor DataFusion 43.1.0 release

[alamb]

Is your feature request related to a problem or challenge?

• We released Release DataFusion 43.0.0 #12470
• Delta rs is still having trouble updating: chore: upgrade to datafusion 43 delta-io/delta-rs#2886

@timsaucer mentioned he has have found several items while working on delta table integration and would like to get a small incremental release to fix them

List of items to include

• Simplify expressions swallows a cast expression #13481

[alamb]

I am happy to run the release process to make this release

[timsaucer]

Thank you. Right now #13481 is a blocker for delta-rs

[gruuya]

I thought #13468 resolves #13481 (again only keeping the second assert in test_simplify_case_cast_list)? If so can we proceed with merging it?

EDIT: Maybe it doesn't resolve the proposed tests case completely but I think that shouldn't be a blocker for merging.

[rtyler]

Thanks for pushing forward here @timsaucer ! I believe that I have resolved the last of our issues in delta-rs but in order to avoid problems we have needed to disable some default optimizers. The changes you're working on would allow us to resume usage of the default optimizers.

Either way, I'm hoping to land our pull request and cut a release soon either way 😈

[timsaucer]

Ok, we do still have a handful of uft8view issues to address in #13504 but I think we've gotten all the blockers for delta-rs and datafusion-python once #13481 goes in. Then I think it would make sense to go ahead with this minor release and we'll get all the rest of the remaining utf8view issues in DF44.

[timsaucer]

@alamb is there anything I can do to help with this?

[alamb]

Hi @timsaucer -- if you could create PRs with the necessary changes that target the branch-43 branch that would be awesome: https://github.com/apache/datafusion/tree/branch-43

The idea is that I will then create a release candidate from that branch

This is described here: https://github.com/apache/datafusion/blob/18fc103a403ab0efe5245dd4352f3f3b93c2a4fe/dev/release/README.md#branching-policy

In the past, what I have done is something like

git checkout branch-43
git checkout -b backport_to_43
git cherry-pick <sha_from_master>
git push -u alamb

And then create a PR that targets branch-43 rather than main

Does that make sense?

[alamb]

I also made a PR with this example

• Minor: Add example of backporting / cherry-picking to release branch #13565

[timsaucer]

Ok, I can do that. In the past did you cherry pick everything from the point of the past release up to current in main? Or did you only grab specific commits you wanted?

[simonvandel]

Would it be possible to include the following PRs in the point-release?

• Fix redundant data copying in unnest #13441
• [bug]: Fix wrong order by removal from plan #13497

Please let me know if I can in any way assist

[timsaucer]

I'm going back through to add in the utf8view commits (and will try to grab those two Simon) but it's been 3 weeks since some of them went in. I'm starting to wonder if it would be just better to wait for a DF44 release instead. What do you think?

[simonvandel]

I'm starting to wonder if it would be just better to wait for a DF44 release instead. What do you think?

I can for sure also wait for the 44 release - no stress

[alamb]

Ok, I can do that. In the past did you cherry pick everything from the point of the past release up to current in main? Or did you only grab specific commits you wanted?

Normally I would suggest we only cherry-pick the commits we want.

[alamb]

Is anyone waiting on this release? If not let's focus our efforts on

• Release DataFusion 44.0.0 #13334

And ideally

• [DISCUSS] More extensive pre-release testing #13661

[gruuya]

Is anyone waiting on this release?

I don't think so; delta-rs has a temp work-around for #13481, and 44 is just around the corner so we might as well go for that instead.

[alamb]

I agree -- it would be great to get as much effort as possible into making 44 a stable, easy to use release

[alamb]

Closing for now, please comment / let me know if anyone disagrees