Application not starting when Kafka Servers down

[seasuresh]

I'm using Spring boot and danielwegener kafka appender to publish application logs to Kafka but the spring boot application is not starting when Kafka server are down.

Can you please advise to resolve this issue?

<root level="INFO">
            <appender-ref ref="kafkaLogAppenderAsync"/>
    </root>

 <appender name="kafkaLogAppenderAsync" class="ch.qos.logback.classic.AsyncAppender">
        <appender-ref ref="logAppender"/>
    </appender>

<appender name="logAppender" class="com.github.danielwegener.logback.kafka.KafkaAppender">
        <encoder class="com.github.danielwegener.logback.kafka.encoding.LayoutKafkaMessageEncoder">
            <layout class="ch.qos.logback.classic.PatternLayout">
                <pattern>concise-local - %msg</pattern>
                <pattern>${CONSOLE_LOG_PATTERN}</pattern>
            </layout>
        </encoder>
        <topic>app-logs</topic>
        <keyingStrategy class="com.github.danielwegener.logback.kafka.keying.RoundRobinKeyingStrategy"/>
        <deliveryStrategy class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"/>
        <producerConfig>bootstrap.servers=${kafka.bootstrap.servers}</producerConfig>
        <producerConfig>acks=0</producerConfig>
        <producerConfig>compression.type=gzip</producerConfig>
        <producerConfig>retries=2</producerConfig>
        <producerConfig>request.timeout.ms=10</producerConfig>
        <producerConfig>max.block.ms=800</producerConfig>
        <producerConfig>metadata.fetch.timeout.ms=10</producerConfig>
        <appender-ref ref="CONSOLE"/>
    </appender>

[danielwegener]

This behaviour matches what ch.qos.logback.classic.net.SocketAppender would do (except an additional buffer). What would you expect to happen if the broker is unreachable? Where should the unlogged messages go (dropped/delegated to fallback-appenders or dropped)? How to decide if a broker is unreachable of simply slow?

[seasuresh]

Can we log the messages to console if brokers are not reachable? If primary is not available it will go to secondary (fail-over) broker and if both are not reachable then write it to console.

[danielwegener]

Yeah, thats what fallback-appenders are meant to be for (see the Readme). But still - the Producer will not instantly be ready when the application starts, so the first n messages might end up in the fallback appender. Would that be a valid behaviour for you?

[seasuresh]

Yes Daniel, All we need is application should not be impacted if primary and fallback brokers are down or unreachable.

…

On Thu, Nov 23, 2017 at 5:26 PM, Daniel Wegener ***@***.***> wrote: Yeah, thats what fallback-appenders are meant to be for (see the Readme). But still - the Producer will not *instantly* be ready when the application starts, so the first *n* messages might end up in the fallback appender. Would that be a valid behaviour for you? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#53 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/APsQ-bcGAIBr4dy9vhc_Zjk9RZgSghjxks5s5fEWgaJpZM4QKFTT> .

[michaelandrepearce]

+1 on this, also needed.

[michaelandrepearce]

@danielwegener please see #56 we believe the issue is that the lazy producer fails to initialise a producer, thus returns null on lazyproducer.get this is then passed to and then invoked within the delegating deliveryStrategy causing an NPE

[danielwegener]

@michaelandrepearce Thanks for the PR. Your fix will still not fix the issue that the broker start may block the logging thread during metadata exchange (and since the start is synchronized possibly all other threads that start to log). But when the broker is not available at all (no route to host or other "fast" cancellations) - this fix can at least make sure that "some" logs arrive in the fallback appender. This is still a very unfortunate situation because the failed producer start will automatically be retried for every following logging attempt (and each time issue possibly expensive kafka-producer start attempts).
I think the whole topic "what happens if the broker is down" needs to be properly addressed as a whole. As mentioned above - it is not always easy to decide between "not reachable" and "reachable but slow" - or even "we don't know yet".

[michaelandrepearce]

@danielwegener agreed. Just NPE fix sorts as you say the most trivial and easiest to hit issue.

[sudheerbabu]

+1 for the issue ran to this issue and impacted our current application availability, where we have fallback mechanism to write messages in case error to another file store, not sure how this is behaving with Rabbit MQ

[piyush2409]

This issue is old but do we have resolution for it.

[piyush2409]

java.lang.IllegalStateException: Logback configuration error detected:
ERROR in com.github.danielwegener.logback.kafka.KafkaAppender[kafkaAppender] - Appender [kafkaAppender] failed to append. org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
ERROR in com.github.danielwegener.logback.kafka.KafkaAppender[kafkaAppender] - Appender [kafkaAppender] failed to append. org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
ERROR in com.github.danielwegener.logback.kafka.KafkaAppender[kafkaAppender] - Appender [kafkaAppender] failed to append. org.apache.kafka.common.errors.InterruptException: java.lang.InterruptedException
at org.springframework.boot.logging.logback.LogbackLoggingSystem.loadConfiguration(LogbackLoggingSystem.java:162)
at org.springframework.boot.logging.AbstractLoggingSystem.initializeWithConventions(AbstractLoggingSystem.java:81)
at org.springframework.boot.logging.AbstractLoggingSystem.initialize(AbstractLoggingSystem.java:59)
at org.springframework.boot.logging.logback.LogbackLoggingSystem.initialize(LogbackLoggingSystem.java:115)
at org.springframework.boot.logging.LoggingApplicationListener.initializeSystem(LoggingApplicationListener.java:304)
at org.springframework.boot.logging.LoggingApplicationListener.initialize(LoggingApplicationListener.java:277)
at org.springframework.boot.logging.LoggingApplicationListener.onApplicationEnvironmentPreparedEvent(LoggingApplicationListener.java:240)
at org.springframework.boot.logging.LoggingApplicationListener.onApplicationEvent(LoggingApplicationListener.java:213)
at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:172)
at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:165)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:139)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:122)
at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:74)
at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:54)
at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:325)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:296)