Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

It seems to me that a connection pool is being created and shut down for every transaction in NiFi. #1817

Open
com-exem-route7 opened this issue Sep 10, 2024 · 14 comments
Open

It seems to me that a connection pool is being created and shut down for every transaction in NiFi. #1817

com-exem-route7 opened this issue Sep 10, 2024 · 14 comments
Labels
bug
Milestone
Priority Backlog

Comments

@com-exem-route7
Copy link

com-exem-route7 commented Sep 10, 2024 •
edited
Loading

Description

Hello Team.
I am using NiFi and clickhouse-jdbc to collect data and load it into ClickHouse.
However, over time, NiFi's JVM Old Generation memory usage keeps increasing.
After generating and analyzing a heap dump, I suspect that the issue might be related to clickhouse-jdbc.
Based on the logs, it seems to me that a connection pool is being created and shut down for every transaction.
Could you help me determine if this is a problem with NiFi or with clickhouse-jdbc?

JConsole G1 Old Generation Heap

Spec

  • ClickHouse Version: 24.5.1.1763 (2R-1S)
  • ClickHouse JDBC Version: clickhouse-jdbc-0.6.2-all.jar
  • JDK Version: openjdk version "1.8.0_392" (Zulu 8.74.0.17-CA-linux64)
  • NiFi Connection Configuration
    • URL: jdbc:ch:http://{CH1}:8123,{CH2}:8123/{DB}?load_balancing_policy=firstAlive&health_check_interval=5000&check_all_nodes=true&failover=2&retry=2&connect_timeout=10000
    • Max Wait Time: 10000 millis
    • Max Total Connections: 30
    • Minimum Idle Connections: 5
    • Max Idle Connections: 15
    • Minimum Evictable Idle Time: 30 minutes

Heap Dump (Eclipse MAT)

Leak Suspects

image

Dominator Tree

8

Leak Hunter

image
image
image
image
image
image

+)
I enabled debug logs for the com.clickhouse.client and com.clickhouse.jdbc packages in NiFi. I have a question: the Connection pool shut down log is being printed for every transaction. Is this normal behavior? It doesn't seem like the connections in the connection pool are being reused. PoolingHttpClientConnectionManager Connection pool shut down was logged 6125 times over a period of 20 seconds.

......
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.cli.ClickHouseCommandLineClient@201bfc92] against [HTTP]...
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.grpc.ClickHouseGrpcClient@68ad8553] against [HTTP]...
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@51a45d80] against [HTTP]...
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,connect_timeout=10000}]@-1200753860
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.ApacheHttpConnectionImpl@764ba5ed
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015016 preparing request execution
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015016 target auth state: UNCHALLENGED
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015016 proxy auth state: UNCHALLENGED
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015016 acquiring connection with route {}->http://10.10.10.1:8123
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015016 acquiring endpoint (3 MINUTES)
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015016 endpoint lease request (3 MINUTES) [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 0 of 10; total allocated: 0 of 10]
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015016 endpoint leased [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015016 acquired ep-0211015016
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015016 acquired endpoint ep-0211015016
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015016 opening connection {}->http://10.10.10.1:8123
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015016 connecting endpoint (null)
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 connecting endpoint to http://10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolving remote address
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolved to [/10.10.10.1]
2024-09-12 15:16:43,566 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connecting null->/10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567461 set socket timeout to 10000 MILLISECONDS
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connected null->/10.10.10.1:8123 as http-outgoing-202567461
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 connected http-outgoing-202567461
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015016 endpoint connected
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015016 executing POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.p.RequestAddCookies ex-0211015016 Cookie spec selected: strict
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015016 start execution ex-0211015016
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 executing exchange ex-0211015016 over http-outgoing-202567461
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> accept: */*
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> x-clickhouse-user: nifi
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> x-clickhouse-key: password
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> x-clickhouse-database: default
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> x-clickhouse-format: TabSeparated
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> accept-encoding: lz4
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> Content-Type: text/plain; charset=UTF-8
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> Transfer-Encoding: chunked
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> Host: 10.10.10.1:8123
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 >> Connection: keep-alive
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "POST /?compress=1&extremes=0 HTTP/1.1[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "accept: */*[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "x-clickhouse-user: nifi[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "x-clickhouse-key: password[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "x-clickhouse-database: default[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "x-clickhouse-format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "accept-encoding: lz4[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "Host: 10.10.10.1:8123[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "Connection: keep-alive[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "1c[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "SELECT 1 FORMAT TabSeparated[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "0[\r][\n]"
2024-09-12 15:16:43,567 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 >> "[\r][\n]"
2024-09-12 15:16:43,569 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "HTTP/1.1 200 OK[\r][\n]"
2024-09-12 15:16:43,569 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "Date: Thu, 12 Sep 2024 06:16:43 GMT[\r][\n]"
2024-09-12 15:16:43,569 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "Connection: Keep-Alive[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "X-ClickHouse-Server-Display-Name: ch1[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "X-ClickHouse-Query-Id: d9e4febc-39c5-423b-bdf7-a3a9cd49e830[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "X-ClickHouse-Format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "X-ClickHouse-Timezone: Asia/Seoul[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "Keep-Alive: timeout=10[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"2198710"}[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "1C[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "[0xffffffde]y[0xffffffcf][0x8][0x7f][0xffffffb6]5[0x4][0xffffff9d][0xffffffb8][0x16][0xffffffdf][0x19][[0x1]k[0xffffff82][0xc][0x0][0x0][0x0][0x2][0x0][0x0][0x0] 1[\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << HTTP/1.1 200 OK
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << Date: Thu, 12 Sep 2024 06:16:43 GMT
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << Connection: Keep-Alive
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << Content-Type: text/tab-separated-values; charset=UTF-8
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << X-ClickHouse-Server-Display-Name: ch1
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << Transfer-Encoding: chunked
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << X-ClickHouse-Query-Id: d9e4febc-39c5-423b-bdf7-a3a9cd49e830
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << X-ClickHouse-Format: TabSeparated
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << X-ClickHouse-Timezone: Asia/Seoul
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << Keep-Alive: timeout=10
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567461 << X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"2198710"}
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015016 connection can be kept alive for 10 SECONDS
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015016 Cannot retry non-repeatable request
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.HttpRequestRetryExec ex-0211015016 cannot retry non-repeatable request
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "0[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567461 << "[\r][\n]"
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015016 releasing valid endpoint
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 releasing endpoint
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 connection http-outgoing-202567461 can be kept alive for 10 SECONDS
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015016 connection released [route: {}->http://10.10.10.1:8123][total available: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Shutdown connection pool GRACEFUL
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567461 close connection GRACEFUL
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Connection pool shut down
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.cli.ClickHouseCommandLineClient@55d639f4] against [HTTP]...
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.grpc.ClickHouseGrpcClient@55d142d1] against [HTTP]...
2024-09-12 15:16:43,570 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@759988e2] against [HTTP]...
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,connect_timeout=10000}]@-1167212923
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.ApacheHttpConnectionImpl@7c2fc198
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015017 preparing request execution
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015017 target auth state: UNCHALLENGED
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015017 proxy auth state: UNCHALLENGED
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015017 acquiring connection with route {}->http://10.10.10.2:8123
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015017 acquiring endpoint (3 MINUTES)
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015017 endpoint lease request (3 MINUTES) [route: {}->http://10.10.10.2:8123][total available: 0; route allocated: 0 of 10; total allocated: 0 of 10]
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015017 endpoint leased [route: {}->http://10.10.10.2:8123][total available: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015017 acquired ep-0211015017
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015017 acquired endpoint ep-0211015017
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015017 opening connection {}->http://10.10.10.2:8123
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015017 connecting endpoint (null)
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 connecting endpoint to http://10.10.10.2:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2 resolving remote address
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2 resolved to [/10.10.10.2]
2024-09-12 15:16:43,571 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2:8123 connecting null->/10.10.10.2:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567462 set socket timeout to 10000 MILLISECONDS
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2:8123 connected null->/10.10.10.2:8123 as http-outgoing-202567462
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 connected http-outgoing-202567462
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015017 endpoint connected
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015017 executing POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.p.RequestAddCookies ex-0211015017 Cookie spec selected: strict
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015017 start execution ex-0211015017
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 executing exchange ex-0211015017 over http-outgoing-202567462
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> accept: */*
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> x-clickhouse-user: nifi
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> x-clickhouse-key: password
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> x-clickhouse-database: default
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> x-clickhouse-format: TabSeparated
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> accept-encoding: lz4
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> Content-Type: text/plain; charset=UTF-8
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> Transfer-Encoding: chunked
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> Host: 10.10.10.2:8123
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 >> Connection: keep-alive
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "POST /?compress=1&extremes=0 HTTP/1.1[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "accept: */*[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "x-clickhouse-user: nifi[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "x-clickhouse-key: password[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "x-clickhouse-database: default[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "x-clickhouse-format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "accept-encoding: lz4[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "Host: 10.10.10.2:8123[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "Connection: keep-alive[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "1c[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "SELECT 1 FORMAT TabSeparated[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "0[\r][\n]"
2024-09-12 15:16:43,599 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 >> "[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "HTTP/1.1 200 OK[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "Date: Thu, 12 Sep 2024 06:16:43 GMT[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "Connection: Keep-Alive[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "X-ClickHouse-Server-Display-Name: ch2[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "X-ClickHouse-Query-Id: 62885e2b-ef62-49d6-b9ff-15082c0a2d91[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "X-ClickHouse-Format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "X-ClickHouse-Timezone: Asia/Seoul[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "Keep-Alive: timeout=10[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1825171"}[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "1C[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "[0xffffffde]y[0xffffffcf][0x8][0x7f][0xffffffb6]5[0x4][0xffffff9d][0xffffffb8][0x16][0xffffffdf][0x19][[0x1]k[0xffffff82][0xc][0x0][0x0][0x0][0x2][0x0][0x0][0x0] 1[\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "0[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567462 << "[\r][\n]"
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << HTTP/1.1 200 OK
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << Date: Thu, 12 Sep 2024 06:16:43 GMT
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << Connection: Keep-Alive
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << Content-Type: text/tab-separated-values; charset=UTF-8
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << X-ClickHouse-Server-Display-Name: ch2
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << Transfer-Encoding: chunked
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << X-ClickHouse-Query-Id: 62885e2b-ef62-49d6-b9ff-15082c0a2d91
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << X-ClickHouse-Format: TabSeparated
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << X-ClickHouse-Timezone: Asia/Seoul
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << Keep-Alive: timeout=10
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567462 << X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1825171"}
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015017 connection can be kept alive for 10 SECONDS
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015017 Cannot retry non-repeatable request
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.HttpRequestRetryExec ex-0211015017 cannot retry non-repeatable request
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015017 releasing valid endpoint
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 releasing endpoint
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 connection http-outgoing-202567462 can be kept alive for 10 SECONDS
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015017 connection released [route: {}->http://10.10.10.2:8123][total available: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Shutdown connection pool GRACEFUL
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567462 close connection GRACEFUL
2024-09-12 15:16:43,631 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Connection pool shut down

2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.cli.ClickHouseCommandLineClient@4fd83056] against [HTTP]...
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.grpc.ClickHouseGrpcClient@4e52126a] against [HTTP]...
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@3eb9a96e] against [HTTP]...
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,connect_timeout=10000}]@-1200753860
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.ApacheHttpConnectionImpl@6b55cff6
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015018 preparing request execution
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015018 target auth state: UNCHALLENGED
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015018 proxy auth state: UNCHALLENGED
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015018 acquiring connection with route {}->http://10.10.10.1:8123
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015018 acquiring endpoint (3 MINUTES)
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015018 endpoint lease request (3 MINUTES) [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 0 of 10; total allocated: 0 of 10]
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015018 endpoint leased [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0211015018 acquired ep-0211015018
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0211015018 acquired endpoint ep-0211015018
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0211015018 opening connection {}->http://10.10.10.1:8123
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015018 connecting endpoint (null)
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 connecting endpoint to http://10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolving remote address
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolved to [/10.10.10.1]
2024-09-12 15:16:43,632 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connecting null->/10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567463 set socket timeout to 10000 MILLISECONDS
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connected null->/10.10.10.1:8123 as http-outgoing-202567463
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 connected http-outgoing-202567463
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015018 endpoint connected
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015018 executing POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.p.RequestAddCookies ex-0211015018 Cookie spec selected: strict
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015018 start execution ex-0211015018
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 executing exchange ex-0211015018 over http-outgoing-202567463
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> POST /?compress=1&extremes=0 HTTP/1.1
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> accept: */*
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> x-clickhouse-user: nifi
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> x-clickhouse-key: password
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> x-clickhouse-database: default
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> x-clickhouse-format: TabSeparated
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> accept-encoding: lz4
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> Content-Type: text/plain; charset=UTF-8
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> Transfer-Encoding: chunked
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> Host: 10.10.10.1:8123
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 >> Connection: keep-alive
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "POST /?compress=1&extremes=0 HTTP/1.1[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "accept: */*[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "user-agent: ClickHouse-JavaClient/0.6.0 (Linux/3.10.0-1160.102.1.el7.x86_64; OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1; rv:cb8e198)[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "x-clickhouse-user: nifi[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "x-clickhouse-key: password[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "x-clickhouse-database: default[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "x-clickhouse-format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "accept-encoding: lz4[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "Host: 10.10.10.1:8123[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "Connection: keep-alive[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "1c[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "SELECT 1 FORMAT TabSeparated[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "0[\r][\n]"
2024-09-12 15:16:43,633 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 >> "[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "HTTP/1.1 200 OK[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "Date: Thu, 12 Sep 2024 06:16:43 GMT[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "Connection: Keep-Alive[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "X-ClickHouse-Server-Display-Name: ch1[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "Transfer-Encoding: chunked[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "X-ClickHouse-Query-Id: 222ae225-6da9-4cac-a6d5-0275cd5e78fe[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "X-ClickHouse-Format: TabSeparated[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "X-ClickHouse-Timezone: Asia/Seoul[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "Keep-Alive: timeout=10[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1721033"}[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "1C[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "[0xffffffde]y[0xffffffcf][0x8][0x7f][0xffffffb6]5[0x4][0xffffff9d][0xffffffb8][0x16][0xffffffdf][0x19][[0x1]k[0xffffff82][0xc][0x0][0x0][0x0][0x2][0x0][0x0][0x0] 1[\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "[\r][\n]"
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << HTTP/1.1 200 OK
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << Date: Thu, 12 Sep 2024 06:16:43 GMT
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << Connection: Keep-Alive
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << Content-Type: text/tab-separated-values; charset=UTF-8
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << X-ClickHouse-Server-Display-Name: ch1
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << Transfer-Encoding: chunked
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << X-ClickHouse-Query-Id: 222ae225-6da9-4cac-a6d5-0275cd5e78fe
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << X-ClickHouse-Format: TabSeparated
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << X-ClickHouse-Timezone: Asia/Seoul
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << Keep-Alive: timeout=10
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-202567463 << X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1721033"}
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0211015018 connection can be kept alive for 10 SECONDS
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0211015018 Cannot retry non-repeatable request
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.HttpRequestRetryExec ex-0211015018 cannot retry non-repeatable request
2024-09-12 15:16:43,635 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "0[\r][\n]"
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-202567463 << "[\r][\n]"
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0211015018 releasing valid endpoint
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 releasing endpoint
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 connection http-outgoing-202567463 can be kept alive for 10 SECONDS
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0211015018 connection released [route: {}->http://10.10.10.1:8123][total available: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Shutdown connection pool GRACEFUL
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-202567463 close connection GRACEFUL
2024-09-12 15:16:43,636 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Connection pool shut down
......
@com-exem-route7 com-exem-route7 changed the title It seems that a memory leak is occurring when using clickhouse-jdbc in NiFi. It seems that a memory leak is occurring when using clickhouse-jdbc in NiFi Sep 10, 2024
@com-exem-route7 com-exem-route7 changed the title It seems that a memory leak is occurring when using clickhouse-jdbc in NiFi It seems to me that a connection pool is being created and shut down for every transaction in NiFi. Sep 12, 2024
@mzitnik
Copy link
Contributor

mzitnik commented Sep 12, 2024

Can you update to the latest version of the JDBC driver

@chernser
Copy link
Contributor

@com-exem-route7 thank you for reporting the issue and providing so many details!

@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 13, 2024 •
edited
Loading

I tried applying clickhouse-jdbc-0-6-5-all.jar to NiFi, but encountered an error.
Upon comparing it with another version of the jar, I found that the class files in the com.clickhouse.client package were missing. As a result, I applied clickhouse-jdbc-0.6.5-shaded.jar to NiFi instead.
Is it okay if I use shaded jar?

image

image

@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 13, 2024 •
edited
Loading

After appling clickhouse-jdbc-0.6.5-shaded.jar, (I used JDK 1.8, but I tried JDK 21 and result was same)
it seems like nothing has changed except for the addition of two INFO log lines. 😭

Could there be an issue with the ClickHouse connection settings in NiFi?

  • NiFi Connection Configuration (The other parameters are set to their default values)
    • URL: jdbc:ch:http://{CH1}:8123,{CH2}:8123/{DB}?load_balancing_policy=firstAlive&health_check_interval=5000&check_all_nodes=true&failover=2&retry=2&connect_timeout=10000
    • Max Wait Time: 10000 millis
    • Max Total Connections: 30
    • Minimum Idle Connections: 5
    • Max Idle Connections: 15
    • Minimum Evictable Idle Time: 30 minutes
......

2024-09-13 12:42:40,044 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@4dd9f376] against [HTTP]...
2024-09-13 12:42:40,044 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,connect_timeout=10000}]@499695906
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-13 12:42:40,045 INFO [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl Connection TTL: 0 ms
2024-09-13 12:42:40,045 INFO [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl Connection reuse strategy: LIFO
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.ApacheHttpConnectionImpl@53c4a193
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl HTTP request attempt 0
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570488 preparing request execution
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570488 target auth state: UNCHALLENGED
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570488 proxy auth state: UNCHALLENGED
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0006570488 acquiring connection with route {}->http://10.10.10.2:8123
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570488 acquiring endpoint (3 MINUTES)
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570488 endpoint lease request (3 MINUTES) [route: {}->http://10.10.10.2:8123][total available: 0; route allocated: 0 of 10; total allocated: 0 of 0]
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570488 endpoint leased [route: {}->http://10.10.10.2:8123][total available: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570488 acquired ep-0006570488
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570488 acquired endpoint ep-0006570488
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0006570488 opening connection {}->http://10.10.10.2:8123
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570488 connecting endpoint (null)
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 connecting endpoint to http://10.10.10.2:8123 (10000 MILLISECONDS)
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2 resolving remote address
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2 resolved to [/10.10.10.2]
2024-09-13 12:42:40,045 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2:8123 connecting null->/10.10.10.2:8123 (10000 MILLISECONDS)
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-6181225 set socket timeout to 10000 MILLISECONDS
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.2:8123 connected null->/10.10.10.2:8123 as http-outgoing-6181225
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 connected http-outgoing-6181225
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570488 endpoint connected
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0006570488 executing POST /?compress=1&extremes=0 HTTP/1.1
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.p.RequestAddCookies ex-0006570488 Cookie spec selected: strict
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570488 start execution ex-0006570488
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 executing exchange ex-0006570488 over http-outgoing-6181225
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> POST /?compress=1&extremes=0 HTTP/1.1
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> accept: */*
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> user-agent: ClickHouse-JavaClient/0.6.5 (OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1)
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> x-clickhouse-user: nifi
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> x-clickhouse-key: password
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> x-clickhouse-database: default
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> x-clickhouse-format: TabSeparated
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> accept-encoding: lz4
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> Content-Type: text/plain; charset=UTF-8
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> Transfer-Encoding: chunked
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> Host: 10.10.10.2:8123
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 >> Connection: keep-alive
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "POST /?compress=1&extremes=0 HTTP/1.1[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "accept: */*[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "user-agent: ClickHouse-JavaClient/0.6.5 (OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1)[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "x-clickhouse-user: nifi[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "x-clickhouse-key: password[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "x-clickhouse-database: default[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "x-clickhouse-format: TabSeparated[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "accept-encoding: lz4[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "Transfer-Encoding: chunked[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "Host: 10.10.10.2:8123[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "Connection: keep-alive[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "1c[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "SELECT 1 FORMAT TabSeparated[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "0[\r][\n]"
2024-09-13 12:42:40,059 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 >> "[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "HTTP/1.1 200 OK[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "Date: Fri, 13 Sep 2024 03:42:40 GMT[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "Connection: Keep-Alive[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "X-ClickHouse-Server-Display-Name: CH2[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "Transfer-Encoding: chunked[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "X-ClickHouse-Query-Id: 736f218c-613c-45fd-aa29-e583bc979e9b[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "X-ClickHouse-Format: TabSeparated[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "X-ClickHouse-Timezone: Asia/Seoul[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "Keep-Alive: timeout=10[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1524766"}[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "1C[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "[0xffffffde]y[0xffffffcf][0x8][0x7f][0xffffffb6]5[0x4][0xffffff9d][0xffffffb8][0x16][0xffffffdf][0x19][[0x1]k[0xffffff82][0xc][0x0][0x0][0x0][0x2][0x0][0x0][0x0] 1[\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "0[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181225 << "[\r][\n]"
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << HTTP/1.1 200 OK
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << Date: Fri, 13 Sep 2024 03:42:40 GMT
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << Connection: Keep-Alive
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << Content-Type: text/tab-separated-values; charset=UTF-8
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << X-ClickHouse-Server-Display-Name: CH2
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << Transfer-Encoding: chunked
2024-09-13 12:42:40,073 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << X-ClickHouse-Query-Id: 736f218c-613c-45fd-aa29-e583bc979e9b
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << X-ClickHouse-Format: TabSeparated
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << X-ClickHouse-Timezone: Asia/Seoul
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << Keep-Alive: timeout=10
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181225 << X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"1524766"}
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0006570488 connection can be kept alive for 10 SECONDS
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570488 Cannot retry non-repeatable request
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.HttpRequestRetryExec ex-0006570488 cannot retry non-repeatable request
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570488 releasing valid endpoint
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 releasing endpoint
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 connection http-outgoing-6181225 can be kept alive for 10 SECONDS
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570488 connection released [route: {}->http://10.10.10.2:8123][total available: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Shutdown connection pool GRACEFUL
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-6181225 close connection GRACEFUL
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Connection pool shut down

2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@26993194] against [HTTP]...
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,connect_timeout=10000}]@466154969
2024-09-13 12:42:40,074 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-13 12:42:40,074 INFO [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl Connection TTL: 0 ms
2024-09-13 12:42:40,074 INFO [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl Connection reuse strategy: LIFO
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.ApacheHttpConnectionImpl@6096f378
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.client.http.ApacheHttpConnectionImpl HTTP request attempt 0
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570498 preparing request execution
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570498 target auth state: UNCHALLENGED
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570498 proxy auth state: UNCHALLENGED
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0006570498 acquiring connection with route {}->http://10.10.10.1:8123
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570498 acquiring endpoint (3 MINUTES)
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570498 endpoint lease request (3 MINUTES) [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 0 of 10; total allocated: 0 of 0]
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570498 endpoint leased [route: {}->http://10.10.10.1:8123][total available: 0; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ex-0006570498 acquired ep-0006570498
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ex-0006570498 acquired endpoint ep-0006570498
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.impl.classic.ConnectExec ex-0006570498 opening connection {}->http://10.10.10.1:8123
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570498 connecting endpoint (null)
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 connecting endpoint to http://10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolving remote address
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1 resolved to [/10.10.10.1]
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connecting null->/10.10.10.1:8123 (10000 MILLISECONDS)
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-6181226 set socket timeout to 10000 MILLISECONDS
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultHttpClientConnectionOperator 10.10.10.1:8123 connected null->/10.10.10.1:8123 as http-outgoing-6181226
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 connected http-outgoing-6181226
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570498 endpoint connected
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0006570498 executing POST /?compress=1&extremes=0 HTTP/1.1
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.p.RequestAddCookies ex-0006570498 Cookie spec selected: strict
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570498 start execution ex-0006570498
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 executing exchange ex-0006570498 over http-outgoing-6181226
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> POST /?compress=1&extremes=0 HTTP/1.1
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> accept: */*
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> user-agent: ClickHouse-JavaClient/0.6.5 (OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1)
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> x-clickhouse-user: nifi
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> x-clickhouse-key: password
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> x-clickhouse-database: default
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> x-clickhouse-format: TabSeparated
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> accept-encoding: lz4
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> Content-Type: text/plain; charset=UTF-8
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> Transfer-Encoding: chunked
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> Host: 10.10.10.1:8123
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 >> Connection: keep-alive
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "POST /?compress=1&extremes=0 HTTP/1.1[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "accept: */*[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "user-agent: ClickHouse-JavaClient/0.6.5 (OpenJDK 64-Bit Server VM/25.392-b08; Apache-HttpClient/5.2.1)[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "x-clickhouse-user: nifi[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "x-clickhouse-key: password[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "x-clickhouse-database: default[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "x-clickhouse-format: TabSeparated[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "accept-encoding: lz4[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "Transfer-Encoding: chunked[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "Host: 10.10.10.1:8123[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "Connection: keep-alive[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "1c[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "SELECT 1 FORMAT TabSeparated[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "0[\r][\n]"
2024-09-13 12:42:40,075 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 >> "[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "HTTP/1.1 200 OK[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "Date: Fri, 13 Sep 2024 03:42:40 GMT[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "Connection: Keep-Alive[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "X-ClickHouse-Server-Display-Name: CH1[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "Transfer-Encoding: chunked[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "X-ClickHouse-Query-Id: 9e68c0d5-2c6c-4d81-b8fc-0528c8363740[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "X-ClickHouse-Format: TabSeparated[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "X-ClickHouse-Timezone: Asia/Seoul[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "Keep-Alive: timeout=10[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"995199"}[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "1C[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "[0xffffffde]y[0xffffffcf][0x8][0x7f][0xffffffb6]5[0x4][0xffffff9d][0xffffffb8][0x16][0xffffffdf][0x19][[0x1]k[0xffffff82][0xc][0x0][0x0][0x0][0x2][0x0][0x0][0x0] 1[\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "0[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.wire http-outgoing-6181226 << "[\r][\n]"
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << HTTP/1.1 200 OK
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << Date: Fri, 13 Sep 2024 03:42:40 GMT
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << Connection: Keep-Alive
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << Content-Type: text/tab-separated-values; charset=UTF-8
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << X-ClickHouse-Server-Display-Name: CH1
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << Transfer-Encoding: chunked
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << X-ClickHouse-Query-Id: 9e68c0d5-2c6c-4d81-b8fc-0528c8363740
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << X-ClickHouse-Format: TabSeparated
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << X-ClickHouse-Timezone: Asia/Seoul
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << Keep-Alive: timeout=10
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.apache.hc.client5.http.headers http-outgoing-6181226 << X-ClickHouse-Summary: {"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"128","elapsed_ns":"995199"}
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.MainClientExec ex-0006570498 connection can be kept alive for 10 SECONDS
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.classic.ProtocolExec ex-0006570498 Cannot retry non-repeatable request
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.HttpRequestRetryExec ex-0006570498 cannot retry non-repeatable request
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.c.InternalHttpClient ep-0006570498 releasing valid endpoint
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 releasing endpoint
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 connection http-outgoing-6181226 can be kept alive for 10 SECONDS
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager ep-0006570498 connection released [route: {}->http://10.10.10.1:8123][total available: 1; route allocated: 1 of 10; total allocated: 1 of 10]
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Shutdown connection pool GRACEFUL
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.DefaultManagedHttpClientConnection http-outgoing-6181226 close connection GRACEFUL
2024-09-13 12:42:40,077 DEBUG [ClickHouseScheduler-1] c.c.c.i.a.h.c.h.i.i.PoolingHttpClientConnectionManager Connection pool shut down

......

@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 13, 2024 •
edited
Loading

I've also looked at the heap dump, but since NiFi was recently restarted, there isn't much significant information yet. I expect that after about a week, with Full Garbage Collection (FGC), we should be able to see updates. I’d like to reduce the heap size and increase throughput to get faster results, but the data generated by the pipeline is being used by other teams, so I can't make those changes. I will provide updates as soon as there's new information.

@chernser
Copy link
Contributor

@com-exem-route7 thank you for so detailed information!

@chernser
Copy link
Contributor

@com-exem-route7

@chernser chernser added this to the 0.7.0 milestone Sep 13, 2024
@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 14, 2024 •
edited
Loading

@chernser

using shaded jar is fine, but we would need to fix -all one. #1823

OK. I will use shaded jar until the issue you shared is resolved.

JDBC seems doesn't accept all passed parameters #1665

Thank you for sharing the issue. I think I need to test the parameters of the jdbc url I set.

what connection type in NiFi do you use? Connection pool on top of apache http client would work not well and another client I would recommend, but it require to pass params thru URL.
it should not be normal for client WHEN it is using apache http client to create own connection object each time. I will use this issue to figure out why it happens. I suspect it is because something wrong in load-balancing feature.

I set up the connection by referring to this document.
https://clickhouse.com/docs/en/integrations/nifi#4-add-dbcpconnectionpool-controller-service-and-configure-its-properties
So I am using DBCPConnectionPool.
If you tell me the recommended connection type and the parameters to set, I will test it.

I will also test it without the load balancing parameter(s).

Until next Wednesday, Korea will be observing a holiday period, so it might be difficult to conduct tests or updates.
I would appreciate your understanding.

Thank you for reviewing this issue.

@com-exem-route7
Copy link
Author

@chernser
I encountered a heap memory issue in another NiFi system, and after analyzing the heap dump, I found the same results as the first case I posted about. So, once version 0.7.0 of clickhouse-jdbc is released, I plan to test it.

What connection type in NiFi do you use? Connection pool on top of Apache HTTP client would not work well, and I would recommend another client, but it requires passing parameters through the URL.

I didn't quite understand this statement. For reference, I am using DBCPConnectionPool to configure ClickHouse connection settings, and I am using this configuration in the PutDatabaseRecord or ExecuteSQL processor.
If there’s anything wrong with my NiFi configuration or anything I should be aware of, please let me know. Otherwise, I’ll just wait for version 0.7.0.

Thank you again. 😊

@chernser chernser modified the milestones: 0.7.0, 0.7.1 Sep 24, 2024
@chernser
Copy link
Contributor

Good day, @com-exem-route7 !
There is nothing wrong of using connection pool.

When the client is reused - the internal connection pool would work. But it is true only for Apache Http Client as the provider.
When the client is used by JDBC and DBCP pool then external pool (DBCP) will ask JDBC driver to create as many connections as pool is configured. It mean that multiple Apache http clients will be created internally and each will have own connection pool. It is a big overhead. In this case I would recommend to use Java HTTP Client from JDK. For that we need to make sure all parameters are parsed correctly when they are in the connection URL.

@com-exem-route7
Copy link
Author

@chernser
Thank you for the detailed explanation. 👍
Here’s what I understood from your explanation. Could you please confirm if I’ve understood it correctly?

  1. If this issue is resolved, the problem of increasing memory usage in NiFi will be addressed.
  2. However, even if the issue is resolved, using DBCPConnectionPool with clickhouse-jdbc in NiFi will result in an overhead. This is because, for each connection pool size configured, a separate clickhouse-client with its own connection pool will be created.
  3. Then, it is recommended to use the Java HTTP Client. But when doing so, it is necessary to test and ensure that parameters are correctly applied.

@chernser
Copy link
Contributor

Good day, @com-exem-route7 !
Would you please try adding this configuration http_connection_provider=HTTP_URL_CONNECTION to the URL?
It should work with any version already.
If you can migrate to a newer Java (11, 17 or 21) you may use http_connection_provider=HTTP_CLIENT - it is improved implementation of the built-in client.

Thanks!

@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 28, 2024 •
edited
Loading

Hello, @chernser
Oh, so you were referring to the contents of this document!
I didn’t know there was such a parameter.

I now understand what you were talking about above. I will test with JDK 17+ and http_connection_provider=HTTP_CLIENT next week.

Thank you for kindly explaining it.

@com-exem-route7
Copy link
Author

com-exem-route7 commented Sep 30, 2024 •
edited
Loading

Hello, @chernser .

I applied the configuration to the development NiFi and monitored it alongside the production NiFi for comparison.
Both NiFi instances are almost same and were restarted simultaneously.

Configuration (Dev)

  1. Changed to JDK 17: 
    > java --version
java 17.0.11 2024-04-16 LTS
Java(TM) SE Runtime Environment (build 17.0.11+7-LTS-207)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.11+7-LTS-207, mixed mode, sharing)
  2. Added the http_connection_provider=HTTP_CLIENT parameter to the JDBC URL: 
    jdbc:ch:http://{CH1}:8123,{CH2}:8123/{DATABASE}?load_balancing_policy=firstAlive&health_check_interval=5000&check_all_nodes=true&failover=2&retry=2&http_connection_provider=HTTP_CLIENT

Observations:

  1. No shut down messages appear in the logs. I'm not sure if logging is not working, or if the shut down process isn’t happening at all.

    ......
2024-09-30 10:15:55,073 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@6057bf18] against [HTTP]...
2024-09-30 10:15:55,073 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1459524160
2024-09-30 10:15:55,073 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,074 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@4071005d
2024-09-30 10:15:55,074 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,078 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,079 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@32ef4a38] against [HTTP]...
2024-09-30 10:15:55,079 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1493065097
2024-09-30 10:15:55,079 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,079 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@1851d78f
2024-09-30 10:15:55,079 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,083 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,083 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@6b248ee3] against [HTTP]...
2024-09-30 10:15:55,083 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1459524160
2024-09-30 10:15:55,083 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,084 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@2fe42276
2024-09-30 10:15:55,084 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,089 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,207 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@345fa37b] against [HTTP]...
2024-09-30 10:15:55,207 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1493065097
2024-09-30 10:15:55,207 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,208 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@1834d421
2024-09-30 10:15:55,208 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@627efa5a] against [HTTP]...
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1459524160
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@2916138b
2024-09-30 10:15:55,211 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,215 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,215 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@3cf44117] against [HTTP]...
2024-09-30 10:15:55,215 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1493065097
2024-09-30 10:15:55,215 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,216 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@7d2f1950
2024-09-30 10:15:55,216 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,218 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,219 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@55ad192a] against [HTTP]...
2024-09-30 10:15:55,219 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.2:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1459524160
2024-09-30 10:15:55,219 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.2:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,219 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@8ce942d
2024-09-30 10:15:55,219 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,223 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
2024-09-30 10:15:55,259 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseNodeSelector Checking [com.clickhouse.client.http.ClickHouseHttpClient@208c1930] against [HTTP]...
2024-09-30 10:15:55,259 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connecting to: ClickHouseNode [uri=http://10.10.10.1:8123/default, options={retry=2,http_connection_provider=HTTP_CLIENT}]@-1493065097
2024-09-30 10:15:55,259 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpConnection url [http://10.10.10.1:8123/?compress=1&extremes=0]
2024-09-30 10:15:55,260 DEBUG [ClickHouseScheduler-1] com.clickhouse.client.AbstractClient Connection established: com.clickhouse.client.http.HttpClientConnectionImpl@44938c22
2024-09-30 10:15:55,260 DEBUG [ClickHouseScheduler-1] c.c.client.http.ClickHouseHttpClient Query: SELECT 1 FORMAT TabSeparated
2024-09-30 10:15:55,263 DEBUG [ClickHouseScheduler-1] c.c.client.ClickHouseStreamResponse 2 bytes skipped before closing input stream
......

  2. Threads are continuously increasing. (The left side shows the development NiFi where the configuration was applied, and the right side shows the production NiFi)
    A large number of threads were created on the left side, and they were named HttpClient-######-SelectorManager. When clicking on a thread from the thread list, the following information is displayed.
    3
    4

    Name: HttpClient-1302528-SelectorManager
State: RUNNABLE
Total blocked: 0  Total waited: 0

Stack trace: 
[email protected]/sun.nio.ch.EPoll.wait(Native Method)
[email protected]/sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:118)
[email protected]/sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:129)
    - locked sun.nio.ch.Util$2@235a6ff4
    - locked sun.nio.ch.EPollSelectorImpl@4fc283ba
[email protected]/sun.nio.ch.SelectorImpl.select(SelectorImpl.java:141)
platform/[email protected]/jdk.internal.net.http.HttpClientImpl$SelectorManager.run(HttpClientImpl.java:889)

  3. While writing the report, the number of threads increased significantly and then decreased. An error occurred in the NiFi process, so I had no choice but to revert the changes. 😭
    7

Thank you.

@chernser chernser modified the milestones: 0.7.0, 0.7.1, Priority Backlog Sep 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
Priority Backlog
Development

No branches or pull requests
3 participants
@mzitnik @chernser @com-exem-route7