Question: Can tasks be scheduled in distributed systems with multiple workers on different machines.

[FSMaxB]

Looking at the documentation and blog-post as well as skimming through the code, I wasn't able to answer the question if fang allows scheduling tasks across distributed systems, with multiple workers on different machines.

Is there some synchronization to make this work and to prevent duplicate work across multiple workers, as well as a mechanism for another worker to pick up a task if the worker that started it dies before reporting back the task as finished?

[ayrat555]

Looking at the documentation and blog-post as well as skimming through the code, I wasn't able to answer the question if fang allows scheduling tasks across distributed systems, with multiple workers on different machines.

Fang uses a PostgreSQL db as a task queue. As long as all workers access the same PostgreSQL instance, workload will be distributed across all running workers

Each worker fetches and locks a task, so tasks are executed only once.

Is there some synchronization to make this work and to prevent duplicate work across multiple workers, as well as a mechanism for another worker to pick up a task if the worker that started it dies before reporting back the task as finished?

The task will stay forever in in_progress state. I think it's a developer's responsibility to make sure there are no panics in application logic. You can change the state of a failed "in progress" task manually to re-execute it

If a task returns Err(FangError), it will be retried

[FSMaxB]

Thanks for the response. With that and looking at the code again, I was able to gather the information I need.

Each worker fetches and locks a task, so tasks are executed only once.

Ok, so if I understand that correctly, the only locking happens in fetch_task_type.sql with the FOR UPDATE and this is also the only place where SKIP LOCKED happens. The subsequent update_task_state.sql that sets it in_progress still has the row locked, which guarantees that no second worker can pick up this task.

I think it's a developer's responsibility to make sure there are no panics in application logic.

This is not about panic. Workers can die for any reason at any time. For example when the number of containers running the workers is reduced or any of the nodes running any of the workers is shut down. See also Diposability in the Twelve-Factor-App. Note that in the absence of graceful shutdown (which is quite hard to achieve with async rust), the behavior is indistinguishable from the sudden death scenario described there. e.g. if you cancel a tokio task, it just stops polling it, therefore a task cannot report back its error or finished state. Same thing if there is a network connection issue, then the task cannot be set to finished or error either.

You can change the state of a failed "in progress" task manually to re-execute it

Ok so one way to do that would be to look for in_progress tasks with updated_at older than some timeout and set it to retried so that it can be picked up again. Something like UPDATE fang_tasks SET state = 'retried', retries = retries + 1, updated_at = now WHERE state = 'in_progress' AND now - updated_at > $timeout AND type = $type (half pseudo-code).
This needs to happen in one UPDATE.

You only change the database layout in breaking releases I presume?

If I end up using fang and that rescheduling of timed out tasks works out, would you consider taking some of that back upstream? One thing to consider there is adding a new state timed_out instead of having to set it to retried or new.

[FSMaxB]

Update: The problem I ran into is that it's impossible to schedule a background job in the same transaction as another database change, making this inherently racy. Therefore I'll have to implement distributed tasks myself after all.

[ayrat555]

You only change the database layout in breaking releases I presume?

Of course

If I end up using fang and that rescheduling of timed out tasks works out, would you consider taking some of that back upstream? One thing to consider there is adding a new state timed_out instead of having to set it to retried or new.

You can always submit a pr with your changes. Why don't you want to use failed state for timed out tasks? You can set max_retries to 0 if you don't want to retry it