How to explode and select structs in cells

[CrashLaker]

Hi all,

amazing project!
i've being trying to read a nested JSON on it. can anyone help me with what am i doing wrong?

test.json

{
    "Records": [
        {"a": 10, "b": 2},
        {"a": 11, "b": 3},
        {"a": 12, "b": 4, "c": 3},
        {"a": 13, "b": "aa", "c": 3}
    ]
}

test2.json

{ "Records": [ {"a": 10, "b": 2}, {"a": 11, "b": 3}, {"a": 12, "b": 4, "c": 3}, {"a": 13, "b": "aa", "c": 3} ] }

import daft
from daft import col, lit
from daft.datatype import DataType


df = daft.read_json(
    "test2.json",
    schema_hints={
        "Records": DataType.list(DataType.struct({
            "a": DataType.string(),
            "b": DataType.string(),
            "c": DataType.string(),
            "d": DataType.string(),
        }))
    }
)
print(df.show())
"""
╭──────────────────────────────────────────────────╮
│ Records                                          │
│ ---                                              │
│ List[Struct[a: Utf8, b: Utf8, c: Utf8, d: Utf8]] │
╞══════════════════════════════════════════════════╡
│ [{a: 10,                                         │
│ b: 2,                                            │
│ c: None,                                         │
│ d: No…                                           │
╰──────────────────────────────────────────────────╯
"""

print(df.explode(col("Records")).show())
"""
╭────────────────────────────────────────────╮
│ Records                                    │
│ ---                                        │
│ Struct[a: Utf8, b: Utf8, c: Utf8, d: Utf8] │
╞════════════════════════════════════════════╡
│ {a: 10,                                    │
│ b: 2,                                      │
│ c: None,                                   │
│ d: Non…                                    │
├╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌╌┤
│ {a: 11,                                    │
"""

print(df.explode(col("Records")).select(col("Records.*")).show())

last command outputs the following error

Traceback (most recent call last):
  File "/root/test2.py", line 46, in <module>
    print(df.explode(col("Records")).select(col("Records.*")).show())
  File "/usr/local/lib64/python3.9/site-packages/daft/api_annotations.py", line 31, in _wrap
    return timed_method(*args, **kwargs)
  File "/usr/local/lib64/python3.9/site-packages/daft/analytics.py", line 189, in tracked_method
    result = method(*args, **kwargs)
  File "/usr/local/lib64/python3.9/site-packages/daft/dataframe/dataframe.py", line 545, in select
    builder = self._builder.project(self.__column_input_to_expression(columns))
  File "/usr/local/lib64/python3.9/site-packages/daft/logical/builder.py", line 112, in project
    builder = self._builder.project(projection_pyexprs, custom_resource_request)
ValueError: DaftError::External Unable to create logical plan node.
Due to: DaftError::FieldNotFound Column "Records.*" not found in schema: ["Records"]

expected behavior was to have them as columns

  a   b     c     d
 10   2  None  None
 11   3  None  None
 12   4     3  None
 13  aa     3  None

regards,c.

[samster25]

Hi @CrashLaker,

Thanks for trying out daft! So the issue here is that we currently don't support selector expressions with regex col("Records.*") but we will be adding that soon!

In the mean time, we do provide a struct accessor.

Another idea of what we can do is add a flatten() operation for struct columns (like a horizontal explode) that will give you the output you're looking for.

Workaround example!

In [75]: df = df.explode("Records")

In [76]: df
Out[76]: 
╭────────────────────────────────────────────╮
│ Records                                    │
│ ---                                        │
│ Struct[a: Utf8, b: Utf8, c: Utf8, d: Utf8] │
╰────────────────────────────────────────────╯

(No data to display: Dataframe not materialized)

In [77]: df.select(*[col("Records").struct.get(c) for c in ["a", "b", "c", "d"]]).show()
╭──────┬──────┬──────┬──────╮
│ a    ┆ b    ┆ c    ┆ d    │                                                                                                                                               
│ ---  ┆ ---  ┆ ---  ┆ ---  │
│ Utf8 ┆ Utf8 ┆ Utf8 ┆ Utf8 │
╞══════╪══════╪══════╪══════╡
│ 10   ┆ 2    ┆ None ┆ None │
├╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ 11   ┆ 3    ┆ None ┆ None │
├╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ 12   ┆ 4    ┆ 3    ┆ None │
├╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┼╌╌╌╌╌╌┤
│ 13   ┆ aa   ┆ 3    ┆ None │
╰──────┴──────┴──────┴──────╯
(Showing first 4 of 4 rows)

[samster25]

Here's the issue I made to track for the selector expression
#1965