Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Simplify schema modification of child tables #1647

Open
akelad opened this issue Jul 29, 2024 · 3 comments · May be fixed by #1855 or #2165
Open

Simplify schema modification of child tables #1647

akelad opened this issue Jul 29, 2024 · 3 comments · May be fixed by #1855 or #2165
Assignees
Labels
enhancement New feature or request sprint Marks group of tasks with core team focus at this moment support This issue is monitored by Solution Engineer
Milestone

Comments

@akelad
Copy link
Contributor

akelad commented Jul 29, 2024

Feature description

You should be able to use apply_hints on child tables just like you can on the parent tables. Right now there's no way to modify the schema of child tables without explicitly doing so in a schema.yml

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

When child tables are generated all of the field types are text . This makes them essentially unusable for analysis/aggregation without further transformations. I also can't partition them.

Slack conversation: https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1721944874168369

Proposed solution

No response

Related issues

No response

@akelad akelad added the enhancement New feature or request label Jul 29, 2024
@akelad akelad added community This issue came from slack community workspace support This issue is monitored by Solution Engineer labels Jul 29, 2024
@sh-rp
Copy link
Collaborator

sh-rp commented Jul 29, 2024

What might work is to add a path parameter of type List[str] to apply_table_hints which would enable support for setting hints on subtables. The DltResource (or rather the DltResourceHints class) would have to be able to hold multiple TTableSchemas which then will be merged into the full schema at the end of the extraction or during normalization.

Example:

@dlt.resource(table_name="items")
def my_resource():
   yield [{
      "id": 1,
      "sub_items": [
         {"id": 11}, 
         {"id": 12}
      ]
      }]
   
# applies primary key hint on main table
my_resource.apply_hints(primary_key="id")

# applies primary key hin on sub_items table
my_resource.apply_hints(path=["sub_items"], primary_key="id")

# ever further nested subtable
my_resource.apply_hints(path=["sub_items", "subsub_items"], primary_key="id")

@rudolfix rudolfix moved this from Todo to Planned in dlt core library Aug 5, 2024
@burnash burnash added this to the 1.0 release milestone Aug 8, 2024
@rudolfix rudolfix self-assigned this Aug 25, 2024
@rudolfix rudolfix added sprint Marks group of tasks with core team focus at this moment and removed community This issue came from slack community workspace labels Aug 25, 2024
@rudolfix
Copy link
Collaborator

rudolfix commented Sep 18, 2024

Interface proposal (based on path idea above).

  1. extend the ResourceHints to include nested_hints dictionary that represent nested hints. Note that DltResource derives from hints so what I show below
resource = my_resource()

resource.nested_hints["sub_items"] = make_hints(primary_key="id")
resource.nested_hints["sub_items"]["subsub_items"] = make_hints(primary_key="id")

# works like filesystem.walk, where path is a tuple ie ("sub_items", "subsub_items") that may be converted into path
# the operation is recursive
for path, hint in resource.nested_hints.items():
   ...
  1. extend the @dlt.resource to accept nested_hints ie as dictionary {path: hints} or as ResourceHints instance

Some implementation details:

  1. There's just one resource here - the root one. All the others are hints that share data with the root. Mind that only root table(s) created may have resource name attached (again: there's just one resource name here)
  2. There are a few places where we compute_table_schema. Now we need to do this but also we need to walk over nested hints and generate a table chain that we'll later add to schema
  3. Mind that table schemas are dynamically created. And nested tables must know the name of parent table. You have two options here: or you store the parent hint and compute the parent table name recursively or you pass the previously generated parent table schema to compute_table_schema (where I think the latter is way more efficient)
  4. You still may allow users to specify table_name on the nested hint. If you do so, you'll need to modify the normalizer so it maps paths to those names. IMO this is for another ticket and bigger overhaul of the schema
  5. prevent following to be set on nested table:
parent_table_name: TTableHintTemplate[str] = None,
write_disposition: TTableHintTemplate[TWriteDispositionConfig] = None,
table_format: TTableHintTemplate[TTableFormat] = None,
incremental: TIncrementalConfig = None,

@joscha
Copy link
Contributor

joscha commented Nov 26, 2024

I believe this would also solve the issue described here: https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1732199697666119

Background:

I have a data source that has this data in rows:

{   
    "id": 1,
    "links": [
        {
            "link": "https://linkedin.com/in/some-person-1234",
            "text": "Some Person (Co Founder)"
        }
    ],
    "event": {
        "id": 3715858079,
        "type": "meeting",
        "title": "Meeting title",
        "allDay": false,
        "startTime": "2023-05-09T16:00:00Z",
        "endTime": "2023-05-09T17:30:00Z",
        "attendees": [
            {
                "emailAddress": "[email protected]",
                "person": {
                    "id": 147862259,
                    "firstName": "Will",
                    "lastName": "Smith",
                    "primaryEmailAddress": "[email protected]",
                    "type": "external"
                }
            },
            {
                "emailAddress": "[email protected]",
                "person": {
                    "id": 147862258,
                    "firstName": "Joscha",
                    "lastName": "Feth",
                    "primaryEmailAddress": "[email protected]",
                    "type": "internal"
                }
            },
            {
                "emailAddress": "[email protected]",
                "person": null
            }
        ]
    }
}

What I would like to get from this is:

  • a main table containing the row ID as the primary key
  • a events table, containing all meetings, using the event.id as the primary key and referencing the row ID as the parent
  • an attendees table, with the custom fields and a link to an existing person in the persons table that I provide
  • a links table, containing all links using a synthesized primary key and referencing thw row ID as the parent

E.g. in the above example, I'd end up with:

main table:

ID ...
1 ...

links table:

_synth_ID parent ID link text
1234 1 https://linkedin.com/in/some-person-1234 Some Person (Co Founder)

events table:

ID type title all_day ...
3715858079 meeting Meeting title FALSE ...

attendees table:

_synth_ID parent ID email_address person_id
1234 3715858079 [email protected] 147862259
4567 3715858079 [email protected] 147862258
8910 3715858079 [email protected] NULL

persons table:

ID first_name last_name primary_email_address type
147862259 Will Smith [email protected] external
147862258 Joscha Feth [email protected] internal

Marking nested structures with hints about how to split them would work for this I believe.

@rudolfix rudolfix assigned steinitzu and unassigned rudolfix Dec 3, 2024
@rudolfix rudolfix moved this from Planned to In Progress in dlt core library Dec 3, 2024
@steinitzu steinitzu linked a pull request Dec 19, 2024 that will close this issue
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request sprint Marks group of tasks with core team focus at this moment support This issue is monitored by Solution Engineer
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

6 participants