Getting folowing error log "broker unavailable: produce: Broker: Not leader for partition" when replica is greater than one

[thrineshactian]

Also find details here : https://stackoverflow.com/questions/72560170/fluent-bit-sent-messages-are-not-consistently-delivered-to-redpanda-kafka-topic

I am using redpanda v23.1.3. below is my cluster configuration :

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
name: redpanda-kafka
spec:
image: "vectorized/redpanda"
version: "latest"
enableSasl: true
superUsers:
- username: admin
replicas: 3
resources:
requests:
cpu: 1
memory: 2Gi
limits:
cpu: 1
memory: 2Gi
configuration:
rpcServer:
port: 33145
kafkaApi:
- port: 9092
- external:
enabled: true
subdomain: "sub-domain-name.com"
tls:
enabled: true
issuerRef:
name: letsencrypt
kind: ClusterIssuer
adminApi:
- port: 9644
developerMode: true
nodeSelector:
node: public
tolerations:

• key: "redpanda-node"
  operator: "Equal"
  value: "true"
  effect: "NoSchedule"

I have created the topic "engine-log-events" with 5 partitions and a default replica and replication factor that is 3.

using fluent-bit to forward my application logs to rpk topic. below are my fluent-bit output config.

outputs: |
  [OUTPUT]
      Name        kafka
      Match       *
      Message_Key_Field       filename
      Brokers       0.<sub-domain-name>:<port>,1.<sub-domain-name>:<port>,2.<sub-domain-name>:<port>
      Topics        engine-log-events
      rdkafka.security.protocol       sasl_ssl
      rdkafka.sasl.mechanism        SCRAM-SHA-256
      rdkafka.sasl.username       <user>
      rdkafka.sasl.password       <password>
      rdkafka.debug       all

fluent-bit can forward the logs successfully when it is connected to 0.:.
It frequently changed the broker to 1.:,2.: when it is trying to forward the message to above broker i am getting below error.

[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (0 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1]: Produce MessageSet with 14 message(s) (5900 bytes, ApiVersion 7, MsgVersion 2, MsgId 0, BaseSeq -1, PID{Invalid}, uncompressed)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Sent ProduceRequest (v7, 5967 bytes @ 0, CorrId 9489)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Received ProduceResponse (v7, 61 bytes, CorrId 9489, rtt 2.18ms)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1]: MessageSet with 14 message(s) (MsgId 0, BaseSeq -1) encountered error: Broker: Not leader for partition (actions Refresh,MsgNotPersisted)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: engine-log-events [1]: broker unavailable: produce: Broker: Not leader for partition
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (14 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (0 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1]: Produce MessageSet with 14 message(s) (5900 bytes, ApiVersion 7, MsgVersion 2, MsgId 0, BaseSeq -1, PID{Invalid}, uncompressed)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Sent ProduceRequest (v7, 5967 bytes @ 0, CorrId 9490)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Received ProduceResponse (v7, 61 bytes, CorrId 9490, rtt 2.24ms)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1]: MessageSet with 14 message(s) (MsgId 0, BaseSeq -1) encountered error: Broker: Not leader for partition (actions Refresh,MsgNotPersisted)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: engine-log-events [1]: broker unavailable: produce: Broker: Not leader for partition
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (14 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Received ProduceResponse (v7, 61 bytes, CorrId 9490, rtt 2.24ms)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: ProduceRequest failed: Broker: Not leader for partition: actions Refresh,MsgNotPersisted
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1]: MessageSet with 14 message(s) (MsgId 0, BaseSeq -1) encountered error: Broker: Not leader for partition (actions Refresh,MsgNotPersisted)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: engine-log-events [1]: broker unavailable: produce: Broker: Not leader for partition
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (14 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: Requesting metadata for 1/1 topics: partition leader query
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Request metadata for 1 topic(s): partition leader query
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Sent MetadataRequest (v4, 48 bytes @ 0, CorrId 9491)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (0 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Received MetadataResponse (v4, 410 bytes, CorrId 9491, rtt 1.24ms)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:sasl_ssl://2.kafka-redpanda.dev..com:31824/boots]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: engine-log-events [1] 14 message(s) in xmit queue (0 added from partition queue)
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: ===== Received metadata (for 1 requested topics): partition leader query =====
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: ClusterId: , ControllerId: 0
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: 3 brokers, 1 topics
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Broker #0/3: 1.kafka-redpanda.dev..com:31824 NodeId 1
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Broker #1/3: 2.kafka-redpanda.dev..com:31824 NodeId 2
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Broker #2/3: 0.kafka-redpanda.dev..com:31824 NodeId 0
[2022/06/14 08:10:06] [debug] [output:kafka:kafka.0] fluent-bit#producer-1: [thrd:main]: sasl_ssl://2.kafka-redpanda.dev..com:31824/2: Topic #0/1: engine-log-events with 5 partitions

Am i missing any configuration here ?

[emaxerrno]

@thrineshactian - try setting topic auto create to true. see docs for details.

[thrineshactian]

Thanks emaxerrno, for me issue is with external subdomain created by redpanda was not correct, so i had to create R53 entries manually and map the external IP. it worked for me.

[emaxerrno]

boom! awesome. 😎 👏 🤩 👍 great to hear.