You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the databricks sql connector is not able to expose a STRUCT(), MAP(), or ARRAY() type. This means if there is an ARRAY of STRING values, then GX will read in the column as a STRING. Although this sort of casting will allow for Expectations that depend on types (like ExpectColumnToBeOfType) to run, the casting of the complex types can cause confusion.
This is already an issue in the databricks-sql-python repo here, and we will monitor this ticket until it is resolved
To Reproduce
Please include your great_expectations.yml config, the code you’re executing that causes the issue, and the full stack trace of any error(s).
Expected behavior
Databricks is able to detect complex datatypes properly and not cast them to a basic type.
Environment (please complete the following information):
Operating System: Linux
Great Expectations Version: 1.3.0
Data Source: Databricks
Cloud environment: Databricks
Additional context
The text was updated successfully, but these errors were encountered:
Describe the bug
Currently the databricks sql connector is not able to expose a
STRUCT()
,MAP()
, orARRAY()
type. This means if there is anARRAY
ofSTRING
values, then GX will read in the column as aSTRING
. Although this sort of casting will allow for Expectations that depend on types (likeExpectColumnToBeOfType
) to run, the casting of the complex types can cause confusion.This is already an issue in the
databricks-sql-python
repo here, and we will monitor this ticket until it is resolvedTo Reproduce
Please include your great_expectations.yml config, the code you’re executing that causes the issue, and the full stack trace of any error(s).
Expected behavior
Environment (please complete the following information):
Additional context
The text was updated successfully, but these errors were encountered: