Remove test checks for Spark versions before 3.2.0 #11316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 5 commits
August 7, 2024 06:17
Signed-off-by: Sameer Raheja <[email protected]>
Signed-off-by: Sameer Raheja <[email protected]>
Signed-off-by: Sameer Raheja <[email protected]>
Signed-off-by: Sameer Raheja <[email protected]>
…into branch-24.10-remove_3_1_test_checks
jlowe
reviewed
Aug 12, 2024
| @@ -118,7 +105,7 @@ def test_cast_string_date_invalid_ansi(invalid): | |||
|
|
|||
|
|
|||
| # test try_cast in Spark versions >= 320 and < 340 | |||
| @pytest.mark.skipif(is_before_spark_320() or is_spark_340_or_later() or is_databricks113_or_later(), reason="try_cast only in Spark 3.2+") | |||
| @pytest.mark.skipif(is_before_spark_340() or is_databricks113_or_later(), reason="try_cast only in Spark 3.2+") | |||
There was a problem hiding this comment.
Isn't this supposed to be is_spark_340_or_later instead of is_before_spark_340?
| @@ -224,8 +220,7 @@ def test_dpp_reuse_broadcast_exchange_cpu_scan(spark_tmp_table_factory): | |||
| @pytest.mark.parametrize('s_index', list(range(len(_statements))), ids=idfn) | |||
| @pytest.mark.parametrize('aqe_enabled', [ | |||
| 'false', | |||
| pytest.param('true', marks=pytest.mark.skipif(is_before_spark_320() and not is_databricks_runtime(), | |||
| reason='Only in Spark 3.2.0+ AQE and DPP can be both enabled')) | |||
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), reason='AQE+DPP not supported on Databricks')) | |||
There was a problem hiding this comment.
Previous condition was AND not OR and would only trigger on Spark 3.1.x
Suggested change
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), reason='AQE+DPP not supported on Databricks')) |
Comment on lines
+247
to
+248
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), | ||
| reason='AQE+DPP not supported on Databricks')) |
There was a problem hiding this comment.
Suggested change
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), | |
| reason='AQE+DPP not supported on Databricks')) |
Comment on lines
+269
to
+270
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), | ||
| reason='AQE+DPP not supported on Databricks')) |
There was a problem hiding this comment.
Suggested change
| pytest.param('true', marks=pytest.mark.skipif(not is_databricks_runtime(), | |
| reason='AQE+DPP not supported on Databricks')) |
Comment on lines
-786
to
-787
| # special constant values | ||
| pytest.param("\"" + optional_whitespace_regex + "(now|today|tomorrow|epoch)" + optional_whitespace_regex + "\"", marks=pytest.mark.xfail(condition=is_before_spark_320(), reason="https://github.com/NVIDIA/spark-rapids/issues/9724")), |
There was a problem hiding this comment.
This parameter should not have been deleted, just the xfail marker removed, e.g.:
# special constant values
"\"" + optional_whitespace_regex + "(now|today|tomorrow|epoch)" + optional_whitespace_regex + "\"",| @@ -192,8 +192,7 @@ def query_map_scalar(spark): | |||
|
|
|||
|
|
|||
| @allow_non_gpu('WindowLocalExec') | |||
| @datagen_overrides(seed=0, condition=is_before_spark_314() | |||
| or (not is_before_spark_320() and is_before_spark_323()) | |||
| @datagen_overrides(seed=0, condition=is_before_spark_322() | |||
There was a problem hiding this comment.
Suggested change
| @datagen_overrides(seed=0, condition=is_before_spark_322() | |
| @datagen_overrides(seed=0, condition=is_before_spark_323() |
There was a problem hiding this comment.
is_neg_dec_scale_bug_version should also be updated, as it's using spark_version() comparisons directly against 3.1.1 and 3.1.3.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Spark versions before 3.2.0 are no longer supported. This PR removes the following test conditions
Fixes #11160