Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[question] Is there an option on connector to manage/configure inserts to multiple partitions? #368

Closed
ShishkovSpace opened this issue Nov 8, 2024 · 5 comments
Closed

[question] Is there an option on connector to manage/configure inserts to multiple partitions? #368

ShishkovSpace opened this issue Nov 8, 2024 · 5 comments

Comments

@ShishkovSpace
Copy link

ShishkovSpace commented Nov 8, 2024 •
edited
Loading

Hi,

In our process, we are using connector to transfer data from HDFS to Clickhouse via Spark. And this process overwrites a large number of partitions (there is not much data in each of them).
The target table in Clickhouse is a Distributed table, under which there are ReplicatedMergeTree tables on 6 nodes.
Lately, we are increasingly encountering a problem where tasks in Spark crash with an error

xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@any_id
    at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121)
    at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86)
    at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
    at scala.util.Try$.apply(Try.scala:213)
    at xenon.clickhouse.Utils$.retry(Utils.scala:99)
    at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230)
    at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220)
    at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192)
    at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39)
    at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442)
    at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538)
    at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480)
    at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381)
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
    at org.apache.spark.scheduler.Task.run(Task.scala:136)
    at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
    at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

We are using the following connector configuration in SparkConf

.config("spark.task.maxFailures", "6")
.config("spark.sql.catalog.clickhouse", "xenon.clickhouse.ClickHouseCatalog")
.config("spark.sql.catalog.clickhouse.host", "host")
.config("spark.sql.catalog.clickhouse.protocol", "http")
.config("spark.sql.catalog.clickhouse.http_port", "8123")
.config("spark.sql.catalog.clickhouse.user", "user")
.config("spark.sql.catalog.clickhouse.password", "password")
.config("spark.sql.catalog.clickhouse.database", "database")
.config("spark.sql.catalog.clickhouse.option.socket_keepalive", "true")
.config("spark.sql.catalog.clickhouse.option.session_timeout", "1200")
.config("spark.sql.catalog.clickhouse.option.socket_timeout", "600000")
.config("spark.clickhouse.write.format", "arrow")
.config("spark.clickhouse.write.distributed.convertLocal", "false")
.config("spark.clickhouse.write.batchSize", "100000")
.config("spark.clickhouse.write.maxRetry", "5")
.config("spark.clickhouse.write.retryInterval", 60)
.config("spark.clickhouse.write.repartitionByPartition", "false")
.config("spark.clickhouse.write.distributed.useClusterNodes", "true")
.config("spark.clickhouse.ignoreUnsupportedTransform", "true")
.config("spark.clickhouse.read.format", "binary")
.config("spark.clickhouse.read.distributed.convertLocal", "false")

Spark version - 3.3.0
Connector version - 0.7.3 (jdbc - 0.4.6)

One method we tried to fix it was to play with the batch_size per insert. But it is still not always possible to achieve stability and reliability for jobs.

We didn't find any descriptions of options and connector configurations for cases like this.
There is only a config at the source code which we were used for configuration - https://github.com/ClickHouse/spark-clickhouse-connector/blob/v0.7.3/spark-3.3/clickhouse-spark/src/main/scala/org/apache/spark/sql/clickhouse/ClickHouseSQLConf.scala
Could you please guid if there are any connector options/configs through which we can manage the above scenarios? Or jdbc driver options which we could configure to achieve stability per inserts?
@pan3793
Copy link
Collaborator

pan3793 commented Nov 11, 2024

the error code 1002 means UNKNOWN_EXCEPTION, have you checked the clickhouse server's log? is there any clue for the error?

@pan3793
Copy link
Collaborator

pan3793 commented Nov 11, 2024 •
edited
Loading

I'm not sure if you have awared this article, which introduces the generic idea of writing optimization. the key idea is

So, to achieve better performance, we should re-organize the DataFrame to fit ClickHouse data layout before inserting.

I noticed you have disabled spark.clickhouse.write.repartitionByPartition, then how did you organize your data before inserting? could you please paste your Spark's DataFrame DAG and the writing stage info?

@ShishkovSpace
Copy link
Author

ShishkovSpace commented Nov 12, 2024 •
edited
Loading

the error code 1002 means UNKNOWN_EXCEPTION, have you checked the clickhouse server's log? is there any clue for the error?

Thanks for fast reply.
Yes, we have checked server logs and there were no any evident related error traces in it.

All i have is error trace from failed Spark tasks

logs 

18:49:16.403 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.util.Utils - Aborting task
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:16.422 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 30 (task 891, attempt 1, stage 7.0)
18:49:16.423 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 30 (task 891, attempt 1, stage 7.0)
18:49:16.427 [Executor task launch worker for task 30.1 in stage 7.0 (TID 891)] ERROR org.apache.spark.executor.Executor - Exception in task 30.1 in stage 7.0 (TID 891)
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:17.657 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.util.Utils - Aborting task
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:17.659 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 5 (task 890, attempt 4, stage 7.0)
18:49:17.660 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 5 (task 890, attempt 4, stage 7.0)
18:49:17.661 [Executor task launch worker for task 5.4 in stage 7.0 (TID 890)] ERROR org.apache.spark.executor.Executor - Exception in task 5.4 in stage 7.0 (TID 890)
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-821203095
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:17.812 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.util.Utils - Aborting task
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:17.814 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 0 (task 887, attempt 4, stage 7.0)
18:49:17.814 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 0 (task 887, attempt 4, stage 7.0)
18:49:17.815 [Executor task launch worker for task 0.4 in stage 7.0 (TID 887)] ERROR org.apache.spark.executor.Executor - Exception in task 0.4 in stage 7.0 (TID 887)
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:220) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:192) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.write(ClickHouseWriter.scala:39) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:442) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://2.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@-412378870
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:79) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
	Suppressed: java.io.IOException: Error writing request body to server
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
		at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
		at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:34) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.flush(Lz4OutputStream.java:82) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.ClickHouseOutputStream.close(ClickHouseOutputStream.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.data.stream.Lz4OutputStream.close(Lz4OutputStream.java:92) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:266) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
		at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
		at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
18:49:18.122 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.util.Utils - Aborting task
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:218) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.commit(ClickHouseWriter.scala:263) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:453) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:51) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:80) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
18:49:18.124 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborting commit for partition 18 (task 895, attempt 0, stage 7.0)
18:49:18.124 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask - Aborted commit for partition 18 (task 895, attempt 0, stage 7.0)
18:49:18.125 [Executor task launch worker for task 18.0 in stage 7.0 (TID 895)] ERROR org.apache.spark.executor.Executor - Exception in task 18.0 in stage 7.0 (TID 895)
xenon.clickhouse.exception.CHServerException: [HTTP][email protected]:8123}/clustername [1002] Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at xenon.clickhouse.client.NodeClient.syncInsert(NodeClient.scala:121) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.client.NodeClient.syncInsertOutputJSONEachRow(NodeClient.scala:86) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.$anonfun$doFlush$1(ClickHouseWriter.scala:232) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23) ~[scala-library-2.12.15.jar:?]
	at scala.util.Try$.apply(Try.scala:213) ~[scala-library-2.12.15.jar:?]
	at xenon.clickhouse.Utils$.retry(Utils.scala:99) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.doFlush(ClickHouseWriter.scala:230) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.flush(ClickHouseWriter.scala:218) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at xenon.clickhouse.write.ClickHouseWriter.commit(ClickHouseWriter.scala:263) ~[clickhouse-spark-runtime-3.3_2.12-0.7.3.jar:?]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.$anonfun$run$1(WriteToDataSourceV2Exec.scala:453) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1538) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.DataWritingSparkTask$.run(WriteToDataSourceV2Exec.scala:480) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.sql.execution.datasources.v2.V2TableWriteExec.$anonfun$writeWithV2$2(WriteToDataSourceV2Exec.scala:381) ~[spark-sql_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.scheduler.Task.run(Task.scala:136) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551) ~[spark-core_2.12-3.3.0.jar:3.3.0]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_102]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_102]
	at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_102]
Caused by: com.clickhouse.client.ClickHouseException: Error writing request body to server, server ClickHouseNode [uri=http://1.clickhouse.host:8123/clustername, options={socket_timeout=600000,socket_keepalive=true,session_timeout=1200}]@251648836
	at com.clickhouse.client.ClickHouseException.of(ClickHouseException.java:168) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:275) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more
Caused by: java.io.IOException: Error writing request body to server
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3479) ~[?:1.8.0_102]
	at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3462) ~[?:1.8.0_102]
	at com.clickhouse.data.stream.Lz4OutputStream.flushBuffer(Lz4OutputStream.java:51) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.stream.AbstractByteArrayOutputStream.writeBuffer(AbstractByteArrayOutputStream.java:80) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:889) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.data.ClickHouseInputStream.pipe(ClickHouseInputStream.java:504) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpConnection.postData(ClickHouseHttpConnection.java:318) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.HttpUrlConnectionImpl.post(HttpUrlConnectionImpl.java:225) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.http.ClickHouseHttpClient.send(ClickHouseHttpClient.java:124) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.sendAsync(AbstractClient.java:161) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at com.clickhouse.client.AbstractClient.lambda$execute$0(AbstractClient.java:273) ~[clickhouse-jdbc-0.4.6.jar:clickhouse-jdbc 0.4.6 (revision: dd91e17)]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1590) ~[?:1.8.0_102]
	... 3 more

@ShishkovSpace
Copy link
Author

I'm not sure if you have awared this article, which introduces the generic idea of writing optimization. the key idea is

So, to achieve better performance, we should re-organize the DataFrame to fit ClickHouse data layout before inserting.

I noticed you have disabled spark.clickhouse.write.repartitionByPartition, then how did you organize your data before inserting? could you please paste your Spark's DataFrame DAG and the writing stage info?

Yes, sure, i disabled spark.clickhouse.write.repartitionByPartition because data already organized by partiton key. I've made some tests in which try to investigate impact of connector parameters and dataframe reorganization.
Here are options which i have played:

  • spark.clickhouse.write.repartitionByPartition set to True
  • explicit call of partitionBy(cols)
  • explicit call of repartition(cols)

In all of these cases i have faced approximately same results on writing stage
df_connector_option_rapartitionByPartition_true

Maybe i missed something but i didn't face any impact of spark.clickhouse.write.repartitionByPartition in my scenario, but i suppose that my connector's configuration is not correct for these cases.

@ShishkovSpace
Copy link
Author

I'm not sure if you have awared this article, which introduces the generic idea of writing optimization. the key idea is

So, to achieve better performance, we should re-organize the DataFrame to fit ClickHouse data layout before inserting.

I noticed you have disabled spark.clickhouse.write.repartitionByPartition, then how did you organize your data before inserting? could you please paste your Spark's DataFrame DAG and the writing stage info?

Well, after additional investigation I have achieved necessary performance.
Implicit repartition and orderBy calls helped.
Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pan3793 @ShishkovSpace