-
Notifications
You must be signed in to change notification settings - Fork 190
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Simplify schema modification of child tables #1647
Comments
What might work is to add a Example:
|
Interface proposal (based on path idea above).
resource = my_resource()
resource.nested_hints["sub_items"] = make_hints(primary_key="id")
resource.nested_hints["sub_items"]["subsub_items"] = make_hints(primary_key="id")
# works like filesystem.walk, where path is a tuple ie ("sub_items", "subsub_items") that may be converted into path
# the operation is recursive
for path, hint in resource.nested_hints.items():
...
Some implementation details:
|
I believe this would also solve the issue described here: https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1732199697666119 Background: I have a data source that has this data in rows: {
"id": 1,
"links": [
{
"link": "https://linkedin.com/in/some-person-1234",
"text": "Some Person (Co Founder)"
}
],
"event": {
"id": 3715858079,
"type": "meeting",
"title": "Meeting title",
"allDay": false,
"startTime": "2023-05-09T16:00:00Z",
"endTime": "2023-05-09T17:30:00Z",
"attendees": [
{
"emailAddress": "[email protected]",
"person": {
"id": 147862259,
"firstName": "Will",
"lastName": "Smith",
"primaryEmailAddress": "[email protected]",
"type": "external"
}
},
{
"emailAddress": "[email protected]",
"person": {
"id": 147862258,
"firstName": "Joscha",
"lastName": "Feth",
"primaryEmailAddress": "[email protected]",
"type": "internal"
}
},
{
"emailAddress": "[email protected]",
"person": null
}
]
}
} What I would like to get from this is:
E.g. in the above example, I'd end up with: main table:
links table:
events table:
attendees table:
persons table:
Marking nested structures with hints about how to split them would work for this I believe. |
Feature description
You should be able to use
apply_hints
on child tables just like you can on the parent tables. Right now there's no way to modify the schema of child tables without explicitly doing so in a schema.ymlAre you a dlt user?
Yes, I'm already a dlt user.
Use case
Slack conversation: https://dlthub-community.slack.com/archives/C04DQA7JJN6/p1721944874168369
Proposed solution
No response
Related issues
No response
The text was updated successfully, but these errors were encountered: