Problem with clickhouse after upgrading to 24.4.0

[lraphael]

Self-Hosted Version

24.4.0

CPU Architecture

x86_64

Docker Version

26.0.1

Docker Compose Version

2.26.1

Steps to Reproduce

1. Upgrade to 24.4.0
2. docker compose up -d

Expected Result

All services start properly

Actual Result

docker logs dockerID:
Processing configuration file '/etc/clickhouse-server/config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
Logging warning to /var/log/clickhouse-server/clickhouse-server.log
Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
Logging warning to console
2024.04.16 17:29:09.980602 [ 43 ] {} Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:8123 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.04.16 17:29:10.032086 [ 43 ] {} Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9000 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.04.16 17:29:10.032457 [ 43 ] {} Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9009 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.04.16 17:29:10.040725 [ 43 ] {} Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9004 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.04.16 17:29:10.041053 [ 43 ] {} Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9005 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
2024.04.16 17:29:14.566128 [ 52 ] {} ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
9. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
   (version 21.8.13.1.altinitystable (altinity build))
   2024.04.16 17:29:15.509705 [ 53 ] {} ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):

docker compose logs:
snuba-outcomes-consumer_1 | {"timestamp":"2024-04-16T17:19:30.040060Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
snuba-generic-metrics-sets-consumer_1 | {"timestamp":"2024-04-16T17:19:30.084673Z","level":"INFO","fields":{"message":"skipping write of 0 rows"},"target":"rust_snuba::strategies::clickhouse"}
clickhouse_1 | 2024.04.16 17:19:30.086090 [ 327 ] {} ServerErrorHandler: Poco::Exception. Code: 1000, e.code() = 107, e.displayText() = Net Exception: Socket is not connected, Stack trace (when copying this message, always include the lines below):
clickhouse_1 |
clickhouse_1 | 0. Poco::Net::SocketImpl::error(int, std::__1::basic_string<char, std::__1::char_traits, std::__1::allocator > const&) @ 0x13c4ee8e in /usr/bin/clickhouse
clickhouse_1 | 1. Poco::Net::SocketImpl::peerAddress() @ 0x13c510d6 in /usr/bin/clickhouse
clickhouse_1 | 2. DB::HTTPServerRequest::HTTPServerRequest(std::__1::shared_ptr<DB::Context const>, DB::HTTPServerResponse&, Poco::Net::HTTPServerSession&) @ 0x110e6f0b in /usr/bin/clickhouse
clickhouse_1 | 3. DB::HTTPServerConnection::run() @ 0x110e5d6e in /usr/bin/clickhouse
clickhouse_1 | 4. Poco::Net::TCPServerConnection::start() @ 0x13c5614f in /usr/bin/clickhouse
clickhouse_1 | 5. Poco::Net::TCPServerDispatcher::run() @ 0x13c57bda in /usr/bin/clickhouse
clickhouse_1 | 6. Poco::PooledThread::run() @ 0x13d89e59 in /usr/bin/clickhouse
clickhouse_1 | 7. Poco::ThreadImpl::runnableEntry(void*) @ 0x13d860ea in /usr/bin/clickhouse
clickhouse_1 | 8. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
clickhouse_1 | 9. clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
clickhouse_1 | (version 21.8.13.1.altinitystable (altinity build))

---

Event ID

No response

[lraphael]

I tried to replace all rust-consumer with consumer and start it and now only the first part of the error appears and after that no traceback every second.

docker compose logs clickhouse
WARN[0000] /var/docker_env/sentry_onpremise/docker-compose.prod.yml: `version` is obsolete
sentry-self-hosted_clickhouse_1  | Processing configuration file '/etc/clickhouse-server/config.xml'.
sentry-self-hosted_clickhouse_1  | Merging configuration file '/etc/clickhouse-server/config.d/docker_related_config.xml'.
sentry-self-hosted_clickhouse_1  | Merging configuration file '/etc/clickhouse-server/config.d/sentry.xml'.
sentry-self-hosted_clickhouse_1  | Logging warning to /var/log/clickhouse-server/clickhouse-server.log
sentry-self-hosted_clickhouse_1  | Logging errors to /var/log/clickhouse-server/clickhouse-server.err.log
sentry-self-hosted_clickhouse_1  | Logging warning to console
sentry-self-hosted_clickhouse_1  | 2024.04.16 17:44:34.210284 [ 43 ] {} <Warning> Application: Listen [0.0.0.0]:8123 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:8123 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
sentry-self-hosted_clickhouse_1  | 2024.04.16 17:44:34.211033 [ 43 ] {} <Warning> Application: Listen [0.0.0.0]:9000 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9000 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
sentry-self-hosted_clickhouse_1  | 2024.04.16 17:44:34.211454 [ 43 ] {} <Warning> Application: Listen [0.0.0.0]:9009 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9009 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
sentry-self-hosted_clickhouse_1  | 2024.04.16 17:44:34.211835 [ 43 ] {} <Warning> Application: Listen [0.0.0.0]:9004 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9004 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>
sentry-self-hosted_clickhouse_1  | 2024.04.16 17:44:34.212226 [ 43 ] {} <Warning> Application: Listen [0.0.0.0]:9005 failed: Poco::Exception. Code: 1000, e.code() = 98, e.displayText() = Net Exception: Address already in use: 0.0.0.0:9005 (version 21.8.13.1.altinitystable (altinity build)). If it is an IPv6 or IPv4 address and your host has disabled IPv6 or IPv4, then consider to specify not disabled IPv4 or IPv6 address to listen in <listen_host> element of configuration file. Example for disabled IPv6: <listen_host>0.0.0.0</listen_host> . Example for disabled IPv4: <listen_host>::</listen_host>

[aldy505]

This is a known issue. See getsentry/snuba#5707 for tracking. I'm going to close this as a duplicate.