How to scale Up Netflix Conductor ?

[jegadjame]

How to scale Up Netflix Conductor ? If we have multiple request for same task, How to make the request process parallelly and process quick instead of sequentially

[erikbrgr]

Good question. The easiest way is to simply run multiple instances of the workerhost. On 25 Mar 2022, at 10:50, jegadjame ***@***.***> wrote:  How to scale Up Netflix Conductor ? If we have multiple request for same task, How to make the request process parallelly and process quick instead of sequentially — Reply to this email directly, view it on GitHub<#13>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AESRNUBDMHSUPKW4ZX5WKSLVBWD4RANCNFSM5RTVFPYQ>. You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[jegadjame]

Hi @erikbrgr ,
Thank you for the answer.

In the below image, How to increase the worker from 1-2? I tried to run 2 Testworker.exe, still i see worker as 1. Any idea how to increase to more than 1

[erikbrgr]

That’s a totally different worker, though I understand the confusion. This is the worker that processes the internal Conductor queue. Pretty sure there is a setting in a properties file you can use, but I would have to direct you to the Conductor docs for that. By adding a second TestWorker, tasks on the queue will be picked up in parallel, where possible. So when running two workflows simultaneously, for example, or a workflow with a FORK/JOIN. It won’t show in the Conductor UI, but it will be quicker. We run 8 workers in our Production environment, so longer-running workflows don’t hold up others. On 25 Mar 2022, at 12:11, jegadjame ***@***.***> wrote:  Hi @erikbrgr<https://github.com/erikbrgr> , Thank you for the answer. In the below image, How to increase the worker from 1-2? I tried to run 2 Testworker.exe, still i see worker as 1. Any idea how to increase to more than 1 [image]<https://user-images.githubusercontent.com/5119603/160109930-1def72ef-dd93-46fd-84a0-7f82a4eafb50.png> — Reply to this email directly, view it on GitHub<#13 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AESRNUACNNFYLGI5SKIKDBLVBWNNZANCNFSM5RTVFPYQ>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[jegadjame]

Thank you @erikbrgr

I tried to run 5 instances of testworker, It have picked 5 records and started processing parallelly.

[jegadjame]

If there is separate project for each task instead of all tasks in single file, Workflows are executing parellely