High number of threads

[dinegri]

Hi,

I have running a big Kafka Connect cluster using Sink Lambda connector, we have 450 connectors and 2700 tasks and 12 workers (22GB reserved to xmx) and we are facing issues of "OutOfMemoryError: Failed to create a thread message occurs when the system does not have enough resources to create a new thread" after long running, for example, after 4 weeks

Look into implementation I saw that both sync and async lambda request are using Futures

Checking metrics I have seen that the number of threads is always increasing, Is it may a thread leak?

Should the number of threads decrease when there is not activity for a while?

Any tips about that?