Support for horizonally scaled installs

[Thomas-Ganter]

Context

As provided in this bug report there are issues when horizontally scaling paperless-ngx, and the solution very much seems like an ugly fix. The solution for Mail tasks is even more ugly and involves putting fake DNS entries into all but one install.

As this is not really a bug, I put forward the things I feel are required in this feature request.

Changes requested

• Do a duplicate check on the filename before adding entries to the work queue.
  This should catch parallel consumers from picking up the same files independently.
  I full path/ filename check, potentially including size or timestamp, should suffice to catch this.
• Do a task locking to prevent multiple instances of checking / fetching from the same inbox.
  Again this should ensure that only at most one task is running at the same time, preventing the same Mail document from being picked up multiple times.
• Add a check for „dead“ processings and recycle them back into the work queue.
  I did have instances in which a container was killed due to an OOM situation on the node and then the item was listed as being „in progress“ indefinitely and had to be manually cleaned out.

These three things I became aware of and needed to find workarounds for. There may be more I am so far unaware of.

[peschmae]

I think enabling scaling for kubernetes shouldn't be done using the current setup, where one pod is running all the parts of the application.

The 'main' django app, mostly serves the UI and doesn't generate much load. Handling the documents is done using tasks that are executed through celery, and we should be able to scale the celery workers to handle more tasks needing to be processed.

Ideally celery would run as the only process in it's own pod, and that can then be scaled if needed. Since celery will schedule the queued tasks by itself this should make it easier to scale, as the app consuming documents only runs once

[Thomas-Ganter]

Happy to work with you on this.
Or, if you give me a pointer how to achieve this — happy to give it a try.

[girlpunk]

I'd agree with this, to fully scale in an appropriate manner each instance of the container should run one single process, be it gunicorn, celery, celery beat, document consumer, or flower. In theory that's possible at the moment, though some configuration changes may be needed depending on how the different processes communicate with each other.

[peschmae]

I had some time to play around with a distributed setup for paperless-ngx and got it working quite well. I've published my kubernetes configuration here: https://github.com/peschmae/paperless-distributed

Sometimes if a new document is added through the consumer, I had errors on the celery-workers that they can't find the file, but adding the same file a second time often resolved this issue. I'm not entirely sure why that happened, as I mount a shared PVC into the /tmp/paperless path, but I have yet to find time to follow up on this.

I'm using the official container images for this setup, since all the processes seem to communicate through redis, this seems to have worked fine as is.

[girlpunk]

If each pod is running a single process (as designed), why bother keeping supervisord?

[peschmae]

I did ths as a proof of concecpt to see if the application can be split into multiple pods or if there are some "hidden" dependencies, and to figure out which volumes need to be shared between each component.

Additionally, I'm relying on renovate bot to udpate the versions in my cluster, so my aim is to use the official images anyways and not rebuild them myself. Using supervisord allowed me to just change one config file, and not bother with the commands run in each pod.

In an ideal world, where the aim is to provide a proper distributed setup, I would of course advise against using supervisord, but as long as the default containers are a one-stop shop, I'm fine with reusing as much as possible from the original setup.

[girlpunk]

Using supervisord allowed me to just change one config file, and not bother with the commands run in each pod

Except you have, because that config file container the commands run by each pod. For reference, I've done a similar proof of concept without supervisord which seems to work too.

[peschmae]

Nowhere did I change the command executed by the image, they still execute supervisord.

Ofc changing the supervisord config changed what processes are running in the container...

While I could replace the command run by the container, to the one run by supervisord I wouldn't gain much, as the image would still be bloated, and I could no longer just reuse the configs provided by the project

If the project intends to go towards a distributed/kubernetes native approach, I absolutely agree that supervisord should be removed from the image.

[github-actions[bot]]

This discussion has been automatically closed due to lack of community support. Please see our contributing guidelines for more details.

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.