SSL handshake failed on sinks.kafka

[lavis11]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

`/etc/vector/bin/vector validate /etc/vector/vector.toml

√ Loaded ["/etc/vector/vector.toml"]
√ Component configuration
2024-05-17T07:23:50.609338Z ERROR rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://kafka-1-external.kafka:30001/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 21ms in state SSL_HANDSHAKE)
2024-05-17T07:23:51.594908Z ERROR rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://kafka-0-external.kafka:30000/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 9ms in state SSL_HANDSHAKE)
2024-05-17T07:23:52.590942Z ERROR rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://kafka-2-external.kafka:30002/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 7ms in state SSL_HANDSHAKE)
2024-05-17T07:23:52.591050Z ERROR rdkafka::client: librdkafka: Global error: AllBrokersDown (Local: All broker connections are down): 3/3 brokers are down

2024-05-17T07:23:53.586483Z ERROR vector::topology::builder: msg="Healthcheck failed." error=Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure) component_kind="sink" component_type="kafka" component_id=kafka
x Health check for "kafka" failed: Meta data fetch error: BrokerTransportFailure (Local: Broker transport failure)
2024-05-17T07:23:53.592329Z ERROR rdkafka::client: librdkafka: Global error: SSL (Local: SSL error): ssl://kafka-1-external.kafka:30001/bootstrap: SSL handshake failed: error:0A000086:SSL routines::certificate verify failed: broker certificate could not be verified, verify that ssl.ca.location is correctly configured or root CA certificates are installed (install ca-certificates package) (after 9ms in state SSL_HANDSHAKE, 1 identical error(s) suppressed)`

Configuration

[sinks.kafka]
type = "kafka"
inputs = [ "filter_comments"]
#Configure the list of kafka servers along with port numbers
bootstrap_servers = "kafka-0-external.kafka:30000,kafka-1-external.kafka:30001,kafka-2-external.kafka:30002"
topic = "data_8_1"
compression = "none"
key_field = "host"
encoding.codec = "json"
batch = {max_events=50000}
buffer = {max_events=50000, type="memory", when_full="block"}
librdkafka_options = {"linger.ms"="1000","queue.buffering.max.kbytes"="1048576","security.protocol" = "ssl","ssl.ca.location" = "/home/ubuntu/kafka24/client/root.crt","ssl.certificate.location" = "/home/ubuntu/kafka24/client/client.crt","ssl.key.location" = "/home/ubuntu/kafka24/client/client.key","enable.ssl.certificate.verification" = "true","ssl.endpoint.identification.algorithm" = "none"}

Version

0.34.2

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[lavis11]

Also tried with this configuration. But same error pops
tls.enabled = true tls.ca_file = "/home/ubuntu/kafka24/client/root.crt" tls.crt_file = "/home/ubuntu/kafka24/client/client.crt" tls.key_file = "/home/ubuntu/kafka24/client/client.key"

[jszwedko]

This feels like a misconfiguration issue since we have integration tests which test connecting to Kafka over SSL. I know from experience that debugging these sorts of issues is difficult though 😓