Expose some identifier for Worker and a way to query its status

[felipesere]

Feature Request

Is your feature request related to a problem? Please describe.

In my app I can upload (large) images. For the frontend I use the image crate to resize a much smaller thumbnail, but this can take a few seconds. I would like for this work to be scheduled into the background from the requst.

Describe the solution you'd like

Scheduling in the background is possible with a Worker, which is awesome.
Ideally, there would be an identifier generated when I call let job_id = ImageResizer::perform_later().
I would then expose that ID to the frontend and let it poll on some endpoint to check when the job is done.

That way I can tell the user that the image has been uploaded, and then pop an extra notification when the background job is done and the image is available to use.

Describe alternatives you've considered

I have considered some kind of ID tracking in the job itself where I track jobs for a given resource (recipe, in my case) in a separate table but that sounds awfully annoying.

[jondot]

Hi,
A common solution for this workflow is to decouple a job from "item at work".
Why does this decoupling matter? well, for example, imagine you're running via Async job backend, this means you cannot poll from anywhere else other than the current server on which a thread is performing the job (a "thread job id" would not mean anything anywhere else).

So what to do? In general, have an "images" or "conversions" table:

1. create a new conversion record with: [id, resource url, status]
2. spawn a new job, carrying the conversion record ID
3. when the job finishes, update the conversion record by ID, set status to 'done'

Your polling happens against the "conversions" table.

This way lets you:

a) switch backens
b) move from ad-hoc background jobs to bulk job processing (if you wish)
c) have as much tracking of resources as you wish (keep this table with the URLs, be able to refresh the table if URLs change and spawn new jobs to re-resize, etc)

See more in discussions here: https://www.reddit.com/r/rails/comments/naszfk/getting_job_status_in_rails/

Having said that, there are instances where a "job ID" exists, specifically job workflow coordination. We do carry job IDs in each individual implementation, so I'd be more than willing to expose this, because I see no reason not to.

[felipesere]

🤦 Jup makes total sense. I’ll pretty much build a “jobs” table to track them. Thank you for the reminder!
If I wanted to extract this into a loco plugin, how would I share the schema?

(I say this, yet at the moment I am super time-limited)

[jondot]

To provide the enhancement you originally wanted, it should be feasible.
if you look here https://github.com/loco-rs/loco/blob/master/src/bgworker/pg.rs you will see we already have such a thing as a "jobs" table (it exists in the sqlite provider too).
And if you look here: https://github.com/loco-rs/loco/blob/master/src/bgworker/mod.rs#L52 you will see that at least the SQL based providers do return a job id (which is eventually a String). I haven't looked but from my head I think also the Redis based provider has a concept of job id but dont know if it returns a Job Id.
So it would probably require:

1. validate that all providers return an id
2. "bubble up" the id to the main enqueue method
3. make the method return the ID in the result
4. adjust any breaking interface (I dont think there will be because we move from "nothing" to "something")
5. done

NOTE: if the Redis provider cannot supply a job ID, then we can return a Result<Option>, and providers which do not support this return None

[anhnmt]

@jondot

Hello, I use workers to process my post-production videos and I have some problems when they suddenly shut down and when restarted, the workers with processing status do not continue to work, what should I do? so I can retry it.

I am using postgres queue