From b883a4c32e51508dca64b9fe8b2ef5fa83b032a0 Mon Sep 17 00:00:00 2001 From: yikaifei Date: Thu, 18 Jan 2024 11:32:01 +0800 Subject: [PATCH] [SPARK-45593][BUILD] Building a runnable distribution from master code running spark-sql raise error MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ### What changes were proposed in this pull request? Fix a build issue, when building a runnable distribution from master code running spark-sql raise error: ``` Caused by: java.lang.ClassNotFoundException: org.sparkproject.guava.util.concurrent.internal.InternalFutureFailureAccess at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520) ... 58 more ``` the problem is due to a gauva dependency in spark-connect-common POM that **conflicts** with the shade plugin of the parent pom. - the spark-connect-common contains `connect.guava.version` version of guava, and it is relocation as `${spark.shade.packageName}.guava` not the `${spark.shade.packageName}.connect.guava`; - The spark-network-common also contains guava related classes, it has also been relocation is `${spark.shade.packageName}.guava`, but guava version `${guava.version}`; - As a result, in the presence of different versions of the classpath org.sparkproject.guava.xx; In addition, after investigation, it seems that module spark-connect-common is not related to guava, so we can remove guava dependency from spark-connect-common. ### Why are the changes needed? Building a runnable distribution from master code is not runnable. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? I ran the build command output a runnable distribution package manually for the tests; Build command: ``` ./dev/make-distribution.sh --name ui --pip --tgz -Phive -Phive-thriftserver -Pyarn -Pconnect ``` Test result: image I also checked the `org.sparkproject.guava.cache.LocalCache` from jars dir; Before: ``` ➜ jars grep -lr 'org.sparkproject.guava.cache.LocalCache' ./ .//spark-connect_2.13-4.0.0-SNAPSHOT.jar .//spark-network-common_2.13-4.0.0-SNAPSHOT.jar .//spark-connect-common_2.13-4.0.0-SNAPSHOT.jar ``` Now: ``` ➜ jars grep -lr 'org.sparkproject.guava.cache.LocalCache' ./ .//spark-network-common_2.13-4.0.0-SNAPSHOT.jar ``` ### Was this patch authored or co-authored using generative AI tooling? No Closes #43436 from Yikf/SPARK-45593. Authored-by: yikaifei Signed-off-by: yangjie01 --- assembly/pom.xml | 6 +++++ connector/connect/client/jvm/pom.xml | 8 +------ connector/connect/common/pom.xml | 34 ++++++++++++++++++++++++++++ connector/connect/server/pom.xml | 25 -------------------- 4 files changed, 41 insertions(+), 32 deletions(-) diff --git a/assembly/pom.xml b/assembly/pom.xml index 47b54729bbd2d..6e4b3f6944614 100644 --- a/assembly/pom.xml +++ b/assembly/pom.xml @@ -159,6 +159,12 @@ org.apache.spark spark-connect_${scala.binary.version} ${project.version} + + + org.apache.spark + spark-connect-common_${scala.binary.version} + + org.apache.spark diff --git a/connector/connect/client/jvm/pom.xml b/connector/connect/client/jvm/pom.xml index 91f041db33f61..34813184d6283 100644 --- a/connector/connect/client/jvm/pom.xml +++ b/connector/connect/client/jvm/pom.xml @@ -51,15 +51,9 @@ ${project.version} - - com.google.guava - guava - ${connect.guava.version} - compile - com.google.protobuf protobuf-java diff --git a/connector/connect/common/pom.xml b/connector/connect/common/pom.xml index 0bbb54e88fc80..33d22bb5816ac 100644 --- a/connector/connect/common/pom.xml +++ b/connector/connect/common/pom.xml @@ -47,6 +47,11 @@ com.google.protobuf protobuf-java + com.google.guava guava @@ -152,6 +157,35 @@ + + org.apache.maven.plugins + maven-shade-plugin + + false + + + org.spark-project.spark:unused + com.google.guava:guava + com.google.guava:failureaccess + org.apache.tomcat:annotations-api + + + + + com.google.common + ${spark.shade.packageName}.connect.guava + + + + + + package + + shade + + + + diff --git a/connector/connect/server/pom.xml b/connector/connect/server/pom.xml index 1cdb52febb19f..bf476a2bd6bd6 100644 --- a/connector/connect/server/pom.xml +++ b/connector/connect/server/pom.xml @@ -51,12 +51,6 @@ org.apache.spark spark-connect-common_${scala.binary.version} ${project.version} - - - com.google.guava - guava - - org.apache.spark @@ -158,17 +152,6 @@ scala-parallel-collections_${scala.binary.version} --> - - com.google.guava - guava - ${connect.guava.version} - compile - - - com.google.guava - failureaccess - ${guava.failureaccess.version} - com.google.protobuf protobuf-java @@ -289,7 +272,6 @@ false - com.google.guava:* io.grpc:*: com.google.protobuf:* @@ -309,13 +291,6 @@ - - com.google.common - ${spark.shade.packageName}.connect.guava - - com.google.common.** - - com.google.thirdparty ${spark.shade.packageName}.connect.guava