Add Option to Disable Persistence in Spark Expectations for Serverless Compatibility

[vlad-luca-colibri]

Describe the bug
I am encountering an issue while working with serverless compute in Databricks, which does not support any form of persistence. Specifically, when using the following call:

result: ExpectationValidationResult = df.expect_column_values_to_not_be_null(
    column=column, meta=self.meta, catch_exceptions=False
)

The method expect_column_values_to_not_be_null internally calls:

col_df = self.spark_df.select(F.col(eval_col))  # pyspark.sql.DataFrame

# A couple of tests indicate that caching here helps performance
col_df.persist()

This is located in .../great_expectations/dataset/sparkdf_dataset.py.
Since col_df.persist() is not supported on serverless compute, it results in a failure.

To Reproduce

1. Start a serverless compute environment in Databricks.
2. Run the expect_column_values_to_not_be_null method on a Spark DataFrame.

Expected behavior
I would expect there to be an option (e.g., a parameter) to enable or disable persistence, allowing compatibility with environments that do not support persistence.

Environment (please complete the following information):

• Operating System: MacOS
• Great Expectations Version: 0.18.18
• Data Source: Spark
• Cloud environment: DataBricks

[adeola-ak]

Hi there! Thank you for submitting this feature request. I've noted it and will pass it along to the appropriate team. Please check back on the issue for any updates.