Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Databricks is not able to recognize complex datatypes (struct, map, array) #10808

Open
Shinnnyshinshin opened this issue Dec 21, 2024 · 0 comments
Assignees

Comments

@Shinnnyshinshin
Copy link
Contributor

Describe the bug

Currently the databricks sql connector is not able to expose a STRUCT(), MAP(), or ARRAY() type. This means if there is an ARRAY of STRING values, then GX will read in the column as a STRING. Although this sort of casting will allow for Expectations that depend on types (like ExpectColumnToBeOfType) to run, the casting of the complex types can cause confusion.

This is already an issue in the databricks-sql-python repo here, and we will monitor this ticket until it is resolved

To Reproduce
Please include your great_expectations.yml config, the code you’re executing that causes the issue, and the full stack trace of any error(s).

Expected behavior

  • Databricks is able to detect complex datatypes properly and not cast them to a basic type.

Environment (please complete the following information):

  • Operating System: Linux
  • Great Expectations Version: 1.3.0
  • Data Source: Databricks
  • Cloud environment: Databricks

Additional context

@Shinnnyshinshin Shinnnyshinshin self-assigned this Dec 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant