Preserve field name when casting List #13468

timsaucer · 2024-11-18T15:45:53Z

Which issue does this PR close?

This also pertains to delta-io/delta-rs#2886

Rationale for this change

During a cast operation, the field name for a list can be changed to "item" from whatever it is previously. This change adds an option to set the field name.

What changes are included in this PR?

Adds an option for array_into_list_array and the other variants to pass in the field name for the schema.

Are these changes tested?

Unit tests added and tested against the example in #13481

Are there any user-facing changes?

Internal change only. No user facing changes.

… field name during cast

gruuya · 2024-11-18T16:40:33Z

Thanks for looking into this!

FWIW, I've also been trying to see what the problem is, and the following is a minimal repro (test case) along the lines of the delta-rs test I've been able to come up with

    #[tokio::test]
    async fn test_list_item() -> Result<()> {
        use datafusion_functions_nested::expr_fn::make_array;

        let element_field = Arc::new(Field::new("element", DataType::Int32, true));
        let items_field = Field::new(
            "items",
            DataType::List(element_field.clone()),
            true,
        );
        let schema = Schema::new(vec![items_field.clone()]);

        let mut items_builder =
            ListBuilder::new(Int32Builder::new()).with_field(element_field.clone());
        items_builder.append_value([Some(1)]);
        let batch = RecordBatch::try_new(Arc::new(schema), vec![Arc::new(items_builder.finish())])?;

        let ctx = SessionContext::new();
        let df = ctx.read_batch(batch).expect("source DataFrame")
            .with_column("condition", lit(false))?
            .select(vec![case(col("condition")).when(lit(false), make_array(vec![lit(2), lit(3)])).otherwise(col("items"))?.alias("items")])?;


        let _ = df.create_physical_plan().await?;

        Ok(())
    }

This might be a red herring, but I'm also looking at whether the issue arises somewhere in TypeCoercion analyzer, specifically in coerce_case_expression. This is because the CASE clause here is crucial for the repro, as without it the outermost plan doesn't get coerced to the child plan's list type (i.e. the one with items named element), but instead retains the types of the expression itself (with the generic item names).

findepi · 2024-11-18T17:12:10Z

During a cast operation, the field name for a list can be changed to "item" from whatever it is previously. This change adds an option to set the field name.

option, or is it always the case?

btw in #12622 we probably would want to have all List(x) to not be discern by their field name. cc @notfilippo @alamb

timsaucer · 2024-11-18T17:17:41Z

@findepi I put it as an option because we also use these same functions to build arrays from things like iterators that don't have an associated field name.

gruuya · 2024-11-19T10:42:07Z

I'm also looking at whether the issue arises somewhere in TypeCoercion analyzer

Ok, looks like TypeCoercion works as intended, and it seems like SimplifyExpressions is the culprit behind the particular problem in delta-rs. From the new issue #13481 I've opened:

Basically TypeCoercion injects the CAST, while SimplifyExpressions omits it. Consequently OptimizeProjections folds the two projections, but the schema is computed from the outermost projection expressions, which now lack the proper type info (i.e. the non-standard list item field names).

EDIT: Oh, I see now that the proposed fix in this PR resolves the bug with SimplifyExpressions described in #13481

timsaucer · 2024-11-19T14:08:43Z

Ok, this should be good to go. I did test it against the examples shown in #13481 also. It does resolve the test_list_item test but not the test_simplify_case_cast_list. I will try to investigate that under a different PR.

gruuya · 2024-11-19T14:19:51Z

but not the test_simplify_case_cast_list

Just a note that it should only really make the second assert pass (the one asserting on the output data type), not the first (i.e. the cast should be transmuted into the output data type).

gruuya · 2024-11-19T15:28:08Z

datafusion/common/src/utils/mod.rs

+pub fn array_into_list_array(
+    arr: ArrayRef,
+    nullable: bool,
+    field_name: Option<&str>,


Maybe make this Option<impl Into<String>> to align with the type in Field::new?

We can but that means we'd now have to add type annotations to everywhere we call this, including those that are going to pass None because they don't have field elements. I think this would add bloat that doesn't really add much value. If you feel strongly, I'll make the adjustment.

Yeah fair point, let's keep it as is.

Since this function is public I don't think we can change its signature in a minor release

https://docs.rs/datafusion/latest/datafusion/common/utils/fn.array_into_list_array.html

Thus I suggest we keep the existing functions but mark them as deprecated and add a new API

Maybe we can do something builder style to make future modifications easier

let list_array = ListArrayWrapper::new(arr) .with_nullable(false) .with_field(field)

Or something -- that would also give us a way to take the FieldRef directly as well as document what the defaults are 🤔

gruuya · 2024-11-19T15:30:03Z

datafusion/common/src/utils/mod.rs

    let offsets = OffsetBuffer::from_lengths([arr.len()]);
    ListArray::new(
-        Arc::new(Field::new_list_field(arr.data_type().to_owned(), nullable)),
+        Arc::new(Field::new(
+            field_name.unwrap_or("item"),


I know this is not a concern of this PR, but looks as if this naming convention should be formalized/centralized somewhere (at least a constant), since otherwise there's a whole lot of "item" literals throughout the code.

100% agree -- I think it should be done in arrow-rs

Maybe we could add some comments on https://docs.rs/arrow/latest/arrow/datatypes/struct.Field.html#method.new_list or something

I think it should be done in arrow-rs

Yeah, makes sense—I can pick that up if there's no objection, maybe something like this

impl Field { + /// Default list member field name + pub const LIST_FIELD_DEFAULT_NAME: &'static str = "item"; + /// Creates a new field with the given name, type, and nullability pub fn new(name: impl Into<String>, data_type: DataType, nullable: bool) -> Self { Field { @@ -144,7 +147,7 @@ impl Field { /// ); /// ``` pub fn new_list_field(data_type: DataType, nullable: bool) -> Self { - Self::new("item", data_type, nullable) + Self::new(Self::LIST_FIELD_DEFAULT_NAME, data_type, nullable) }

could add some comments

I think new_list_field already has an adequate comment (Field::new_list probably doesn't need one since the name is explicitly passed there).

at the very least having a default constant exposed would be a nice improvement I think

timsaucer · 2024-11-20T19:45:54Z

Are we okay to merge?

alamb

The code looks good and makes sense to me -- than you @timsaucer and @gruuya

My only concern is that we won't be able to release this change to 43.0.0 as it would be a breaking API change.

alamb · 2024-11-20T21:00:02Z

datafusion/common/src/utils/mod.rs

    let offsets = OffsetBuffer::from_lengths([arr.len()]);
    ListArray::new(
-        Arc::new(Field::new_list_field(arr.data_type().to_owned(), nullable)),
+        Arc::new(Field::new(
+            field_name.unwrap_or("item"),


at the very least having a default constant exposed would be a nice improvement I think

alamb · 2024-11-20T21:02:46Z

datafusion/common/src/utils/mod.rs

+pub fn array_into_list_array(
+    arr: ArrayRef,
+    nullable: bool,
+    field_name: Option<&str>,


Since this function is public I don't think we can change its signature in a minor release

https://docs.rs/datafusion/latest/datafusion/common/utils/fn.array_into_list_array.html

Thus I suggest we keep the existing functions but mark them as deprecated and add a new API

Maybe we can do something builder style to make future modifications easier

let list_array = ListArrayWrapper::new(arr) .with_nullable(false) .with_field(field)

Or something -- that would also give us a way to take the FieldRef directly as well as document what the defaults are 🤔

…d in a parallel function for the small cases where we want to explicitly set the field name

timsaucer · 2024-11-23T19:09:51Z

Since it would be a breaking change and there is only one place that we internally need this feature, I just switched it to have a different function. I can change over to a builder method if you like, but I wasn't convinced this small use needed it. Happy to support if you think that's valuable.

alamb · 2024-11-24T22:53:44Z

datafusion/common/src/utils/mod.rs

@@ -342,6 +342,20 @@ pub fn array_into_list_array(arr: ArrayRef, nullable: bool) -> ListArray {
    )
 }

+pub fn array_into_list_array_with_field_name(


So is the idea that we will backport this to 43.0.0 and then consolidate the other functions on main? That makes sense to me if so

alamb · 2024-11-24T22:54:06Z

Thank you very much @timsaucer

* Add option to pass in field name to create array to support retaining field name during cast * add unit tests for list casting round trip * Documentation example was missing parameter * Rather than deprecate an existing function or change pub signature add in a parallel function for the small cases where we want to explicitly set the field name

alamb · 2024-12-02T20:32:05Z

Since it would be a breaking change and there is only one place that we internally need this feature, I just switched it to have a different function. I can change over to a builder method if you like, but I wasn't convinced this small use needed it. Happy to support if you think that's valuable.

Here is my attempt to migrate to a builder (and maybe avoid these kinds of issues in the future 🤔 ): #13623

Add option to pass in field name to create array to support retaining…

a68ef01

… field name during cast

github-actions bot added common Related to common crate functions labels Nov 18, 2024

timsaucer mentioned this pull request Nov 18, 2024

chore: upgrade to datafusion 43 delta-io/delta-rs#2886

Merged

findepi mentioned this pull request Nov 18, 2024

Coerce Array inner types #13452

Merged

timsaucer added 2 commits November 19, 2024 08:54

add unit tests for list casting round trip

f0292d3

Documentation example was missing parameter

bf61504

timsaucer marked this pull request as ready for review November 19, 2024 14:03

gruuya approved these changes Nov 19, 2024

View reviewed changes

This was referenced Nov 20, 2024

Release Minor DataFusion 43.1.0 release #13499

Closed

Simplify expressions swallows a cast expression #13481

Closed

alamb reviewed Nov 20, 2024

View reviewed changes

alamb mentioned this pull request Nov 23, 2024

fix: Ignore names of technical inner fields (of List and Map types) when comparing datatypes for logical equivalence #13522

Merged

Rather than deprecate an existing function or change pub signature ad…

9564d79

…d in a parallel function for the small cases where we want to explicitly set the field name

github-actions bot removed the functions label Nov 23, 2024

timsaucer mentioned this pull request Nov 24, 2024

Documentation updates: simplify examples and add section on data sources apache/datafusion-python#955

Merged

alamb approved these changes Nov 24, 2024

View reviewed changes

timsaucer merged commit 2e05648 into apache:main Nov 25, 2024
26 checks passed

gruuya mentioned this pull request Nov 25, 2024

Formalize the name of the nested Field in a list apache/arrow-rs#6784

Closed

findepi mentioned this pull request Nov 28, 2024

Preserve field name when casting List (#13468) sdf-labs/arrow-datafusion#77

Merged

alamb mentioned this pull request Dec 2, 2024

Create ArrayScalarBuilder for creating single element List arrays #13623

Merged

gruuya mentioned this pull request Dec 3, 2024

[minor] Consolidate construction of the list field #13627

Merged

alamb mentioned this pull request Dec 3, 2024

Dec 3. 2024: This week in DataFusion #13630

Open

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preserve field name when casting List #13468

Preserve field name when casting List #13468

timsaucer commented Nov 18, 2024 •

edited

Loading

gruuya commented Nov 18, 2024

findepi commented Nov 18, 2024

timsaucer commented Nov 18, 2024

gruuya commented Nov 19, 2024 •

edited

Loading

timsaucer commented Nov 19, 2024

gruuya commented Nov 19, 2024

gruuya Nov 19, 2024

timsaucer Nov 20, 2024

gruuya Nov 20, 2024

alamb Nov 20, 2024

gruuya Nov 19, 2024 •

edited

Loading

alamb Nov 19, 2024

gruuya Nov 20, 2024

alamb Nov 20, 2024

timsaucer commented Nov 20, 2024

alamb left a comment

alamb Nov 20, 2024

alamb Nov 20, 2024

timsaucer commented Nov 23, 2024

alamb Nov 24, 2024

alamb commented Nov 24, 2024

alamb commented Dec 2, 2024

Preserve field name when casting List #13468

Preserve field name when casting List #13468

Conversation

timsaucer commented Nov 18, 2024 • edited Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

gruuya commented Nov 18, 2024

findepi commented Nov 18, 2024

timsaucer commented Nov 18, 2024

gruuya commented Nov 19, 2024 • edited Loading

timsaucer commented Nov 19, 2024

gruuya commented Nov 19, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gruuya Nov 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timsaucer commented Nov 20, 2024

alamb left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

timsaucer commented Nov 23, 2024

Choose a reason for hiding this comment

alamb commented Nov 24, 2024

alamb commented Dec 2, 2024

timsaucer commented Nov 18, 2024 •

edited

Loading

gruuya commented Nov 19, 2024 •

edited

Loading

gruuya Nov 19, 2024 •

edited

Loading