Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem with latency #540

Closed
Arfey opened this issue Sep 16, 2021 · 5 comments
Closed

Problem with latency #540

Arfey opened this issue Sep 16, 2021 · 5 comments

Comments

@Arfey
Copy link

Arfey commented Sep 16, 2021

Hi ✋

The vast majority of kafka bridge requests have good response times (less than 200ms), but some have more than 2-5 seconds. I tried to find the problem and check the latency from the pod for the /ready handler (to rule out kafka and networking issues), but also some requests had response times more than 1-2 seconds.

Without a load, this problem is not present (produce 2-3 messages per second).

repository: strimzi/kafka-bridge
tag: "0.19.0"
jvmOptions: -Xmx512m -Xms512m

i guess i have not problem with gc or heap (even when i increase the memory it does not give any result). Also i have not problem with cpu:

cpu usage: ~1%
memory: ~400mb

i have discover problem (1-3%) with throttling and increase count of cores for bridge from 1 to 4 (throttling 0%) and it have decreased some count of slow requests but not fully solved my problem

Someone knows what might be wrong or how i can solve a current problem? Thx!

@scholzj
Copy link
Member

scholzj commented Sep 16, 2021

@sknot-rh @ppatierno Any ideas?

@Torbik
Copy link

Torbik commented Sep 16, 2021

I'd like to add a few additions to this case:

  • the bridge is used as a producer
  • thread count (jvm_threads_live_threads) was increasing over time:
    image

I suspect so many threads give significant overhead that affects overall performance.

@sknot-rh
Copy link
Member

Hi. Are you sure you do use bridge just as a producer? I did some testing and when the bridge is just producing records, it won't get over 54 threads.

@Arfey
Copy link
Author

Arfey commented Sep 21, 2021

Sure. The current instance is only used to produce messages

@ppatierno
Copy link
Member

@Arfey I am going to close this issue, not hearing from you since years. Feel free to re-open if the problem seems to be still in place for you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants