Use mounted OneDrive folder for consume and media folder

[henrik-sc]

Hi all!
I would like to mount OneDrive folders via rclone and use that folders as consume and media folder. However, it do not get it to work. Once the consume folder ist mounted, Paperless does not notice the files therein, does not consume them if I upload new ones. If I upload a file via the web interface it is stored but not visible in the media folder. Only after I restart the machine (and the OneDrive folders are not mounted anymore, I can see the media directory and subdirectories.

Does anyone know, how I can use my local Paperless-ngx with OneDrive directories mounted via rclone?

Many thanks in advance!

I have attached my config files und the command I use for mounting the OneDrive directories ("consume" as example)

paperless@paperless:/HDD/paperless/paperless-ngx$ cat docker-compose.env 
USERMAP_UID=1001
USERMAP_GID=1001
PAPERLESS_TIME_ZONE=Europe/Berlin
PAPERLESS_OCR_LANGUAGE=deu
PAPERLESS_SECRET_KEY='$`mykey

paperless@paperless:/HDD/paperless/paperless-ngx$ cat docker-compose.yml 
# Docker Compose file for running paperless from the docker container registry.
# This file contains everything paperless needs to run.
# Paperless supports amd64, arm and arm64 hardware.
#
# All compose files of paperless configure paperless in the following way:
#
# - Paperless is (re)started on system boot, if it was running before shutdown.
# - Docker volumes for storing data are managed by Docker.
# - Folders for importing and exporting files are created in the same directory
#   as this file and mounted to the correct folders inside the container.
# - Paperless listens on port 8000.
#
# In addition to that, this Docker Compose file adds the following optional
# configurations:
#
# - Instead of SQLite (default), PostgreSQL is used as the database server.
# - Apache Tika and Gotenberg servers are started with paperless and paperless
#   is configured to use these services. These provide support for consuming
#   Office documents (Word, Excel, Power Point and their LibreOffice counter-
#   parts.
#
# To install and update paperless with this file, do the following:
#
# - Copy this file as 'docker-compose.yml' and the files 'docker-compose.env'
#   and '.env' into a folder.
# - Run 'docker compose pull'.
# - Run 'docker compose run --rm webserver createsuperuser' to create a user.
# - Run 'docker compose up -d'.
#
# For more extensive installation and update instructions, refer to the
# documentation.

version: "3.4"
services:
  broker:
    image: docker.io/library/redis:7
    restart: unless-stopped
    volumes:
      - redisdata:/data

  db:
    image: docker.io/library/postgres:15
    restart: unless-stopped
    volumes:
      - /HDD/paperless/paperless-ngx/database:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: paperless
      POSTGRES_USER: paperless
      POSTGRES_PASSWORD: mypsswd

  webserver:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest
    restart: unless-stopped
    depends_on:
      - db
      - broker
      - gotenberg
      - tika
    ports:
      - "8000:8000"
    volumes:
      - /HDD/paperless/paperless-ngx/data:/usr/src/paperless/data
      - /HDD/paperless/paperless-ngx/media:/usr/src/paperless/media
      - ./export:/usr/src/paperless/export
      - /HDD/paperless/paperless-ngx/consume:/usr/src/paperless/consume
    env_file: docker-compose.env
    environment:
      PAPERLESS_REDIS: redis://broker:6379
      PAPERLESS_DBHOST: db
      PAPERLESS_TIKA_ENABLED: 1
      PAPERLESS_TIKA_GOTENBERG_ENDPOINT: http://gotenberg:3000
      PAPERLESS_TIKA_ENDPOINT: http://tika:9998

  gotenberg:
    image: docker.io/gotenberg/gotenberg:7.10
    restart: unless-stopped

    # The gotenberg chromium route is used to convert .eml files. We do not
    # want to allow external content like tracking pixels or even javascript.
    command:
      - "gotenberg"
      - "--chromium-disable-javascript=true"
      - "--chromium-allow-list=file:///tmp/.*"

  tika:
    image: ghcr.io/paperless-ngx/tika:latest
    restart: unless-stopped

volumes:
  redisdata:

And here is the command for mounting the folder:

paperless@paperless:/HDD/paperless/paperless-ngx$ rclone --vfs-cache-mode writes mount onedrive:/paperless/consume/ /HDD/paperless/paperless-ngx/consume &

I run paperless in a Proxmox VM with Ubuntu Server installed. Paperless was installed via the script from the website's setup section.

BR
Henrik

[github-Jan]

I would expect that inotify is not working on the onedrive thingy.. have you tried polling?
https://docs.paperless-ngx.com/configuration/#polling
(i do not use onedrive or rclone but paperless not noticing hints to such issues with inotify

[henrik-sc]

Hi github-Jan!
Many thanks for your reply!

Yes, you are right. inotify does not work for mounted OneDrive folders.
I changed the docker-compose.env and restarted paperless-ngx

paperless@paperless:/HDD/paperless/paperless-ngx$ docker compose restart
WARN[0000] /HDD/paperless/paperless-ngx/docker-compose.yml: `version` is obsolete 
[+] Restarting 5/5
 ✔ Container paperless-broker-1     Started                                                                                                                                                        0.9s 
 ✔ Container paperless-webserver-1  Started                                                                                                                                                        0.9s 
 ✔ Container paperless-db-1         Started                                                                                                                                                        1.0s 
 ✔ Container paperless-gotenberg-1  Started                                                                                                                                                       10.5s 
 ✔ Container paperless-tika-1       Started                                                                                                                                                        1.0s 
paperless@paperless:/HDD/paperless/paperless-ngx$ rclone --vfs-cache-mode writes mount onedrive:/paperless/consume/ /HDD/paperless/paperless-ngx/consume/ &
[1] 2541
paperless@paperless:/HDD/paperless/paperless-ngx$ cd consume/
paperless@paperless:/HDD/paperless/paperless-ngx/consume$ ls
2024-03-27_Kurzmitteilung.pdf
paperless@paperless:/HDD/paperless/paperless-ngx/consume$ 

My docker-compose.env looks like this:

paperless@paperless:/HDD/paperless/paperless-ngx$ cat docker-compose.env 
USERMAP_UID=1001
USERMAP_GID=1001
PAPERLESS_TIME_ZONE=Europe/Berlin
PAPERLESS_OCR_LANGUAGE=deu
PAPERLESS_SECRET_KEY='mykey'

#MEINE ANPASSUNGEN
PAPERLESS_CONSUMER_POLLING=5
PAPERLESS_FILENAME_FORMAT='{created_year}-{created_month}-{created_day} - {correspondent} {title} {tag_list}'
paperless@paperless:/HDD/paperless/paperless-ngx$ 

But still, it does not take away the file from the mounted directory. The file was copy to OneDrive directly.

[github-Jan]

What is the log inside paperless saying.
Is the consume directory named in the docker-compose.yml? Is paperless having write rights?
[..]
volumes:
- data:/usr/src/paperless/data
- media:/usr/src/paperless/media
- ./export:/usr/src/paperless/export
- /home/shares/scans4paperless/:/usr/src/paperless/consume
env_file: docker-compose.env
[..]

my env file contains:
PAPERLESS_OCR_LANGUAGE=deu
PAPERLESS_TRASH_DIR=../media/trash
PAPERLESS_FILENAME_FORMAT={owner_username}/{created_year}/{correspondent}/{document_type}/{title}
PAPERLESS_CONSUMER_RECURSIVE=true

need this for different users/consume folders

[henrik-sc]

Hi!

Is paperless having write rights? Think so? Result of ls-lisa
525595 4 drwxr-xr-x 2 paperless paperless 4096 Apr 29 16:04 consume

Is the volume in the docker-compose.yml? Yes (see also above)

volumes:
      - /HDD/paperless/paperless-ngx/data:/usr/src/paperless/data
      - /HDD/paperless/paperless-ngx/media:/usr/src/paperless/media
      - ./export:/usr/src/paperless/export
      - /HDD/paperless/paperless-ngx/consume:/usr/src/paperless/consume
    env_file: docker-compose.env

[github-Jan]

So what is the log inside paperless telling you
(or docker-compose logs -f )
Do you mount before starting the docker (so is the directory existing at launch ?

[henrik-sc]

I mount after starting docker.
Rclone does something to the folders. I am unable to restart (via docker compose restart). It stops telling me that the consume folder already exists and cannot be created.

The log looks like this:

[2024-04-29 11:05:00,117] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 11:05:00,139] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 11:05:00,142] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 12:05:00,110] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 12:05:00,132] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 12:05:00,135] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 13:05:00,112] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 13:05:00,134] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 13:05:00,137] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 14:05:00,110] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 14:05:00,135] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 14:05:00,138] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 15:05:00,112] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 15:05:00,133] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 15:05:00,136] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 16:05:00,112] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 16:05:00,134] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 16:05:00,137] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 17:05:00,111] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 17:05:00,132] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 17:05:00,136] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 17:22:58,307] [INFO] [paperless.management.consumer] Received SIGINT, stopping inotify
[2024-04-29 17:22:58,324] [DEBUG] [paperless.management.consumer] Consumer exiting.
[2024-04-29 17:30:11,955] [INFO] [paperless.management.consumer] Polling directory for changes: /usr/src/paperless/consume
[2024-04-29 17:46:11,344] [DEBUG] [paperless.management.consumer] Consumer exiting.
[2024-04-29 18:03:27,177] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/Test1.txt to the task queue.
[2024-04-29 18:03:27,368] [INFO] [paperless.management.consumer] Polling directory for changes: /usr/src/paperless/consume
[2024-04-29 18:03:28,494] [DEBUG] [paperless.tasks] Skipping plugin CollatePlugin
[2024-04-29 18:03:28,494] [DEBUG] [paperless.tasks] Skipping plugin BarcodePlugin
[2024-04-29 18:03:28,495] [DEBUG] [paperless.tasks] Executing plugin WorkflowTriggerPlugin
[2024-04-29 18:03:28,498] [INFO] [paperless.tasks] WorkflowTriggerPlugin completed with: 
[2024-04-29 18:03:28,508] [INFO] [paperless.consumer] Consuming Test1.txt
[2024-04-29 18:03:28,517] [DEBUG] [paperless.consumer] Detected mime type: text/plain
[2024-04-29 18:03:28,545] [DEBUG] [paperless.consumer] Parser: TextDocumentParser
[2024-04-29 18:03:28,549] [DEBUG] [paperless.consumer] Parsing Test1.txt...
[2024-04-29 18:03:28,549] [DEBUG] [paperless.consumer] Generating thumbnail for Test1.txt...
[2024-04-29 18:03:28,893] [DEBUG] [paperless.consumer] Saving record to database
[2024-04-29 18:03:28,894] [DEBUG] [paperless.consumer] Creation date from st_mtime: 2024-04-29 17:58:36.320262+02:00
[2024-04-29 18:03:29,520] [INFO] [paperless.handlers] Assigning document type Versicherungsunterlage to 2024-04-29 Test1
[2024-04-29 18:03:29,532] [INFO] [paperless.handlers] Tagging "2024-04-29 Test1" with "Versicherung"
[2024-04-29 18:03:29,629] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-ngxuvltm40o/Test1.txt
[2024-04-29 18:03:29,631] [DEBUG] [paperless.parsing.text] Deleting directory /tmp/paperless/paperless-5xvrs8ug
[2024-04-29 18:03:29,632] [INFO] [paperless.consumer] Document 2024-04-29 Test1 consumption finished
[2024-04-29 18:04:07,373] [DEBUG] [paperless.management.consumer] Waiting for file /usr/src/paperless/consume/test2.txt to remain unmodified
[2024-04-29 18:04:12,374] [INFO] [paperless.management.consumer] Adding /usr/src/paperless/consume/test2.txt to the task queue.
[2024-04-29 18:04:12,450] [DEBUG] [paperless.tasks] Skipping plugin CollatePlugin
[2024-04-29 18:04:12,450] [DEBUG] [paperless.tasks] Skipping plugin BarcodePlugin
[2024-04-29 18:04:12,450] [DEBUG] [paperless.tasks] Executing plugin WorkflowTriggerPlugin
[2024-04-29 18:04:12,452] [INFO] [paperless.tasks] WorkflowTriggerPlugin completed with: 
[2024-04-29 18:04:12,462] [INFO] [paperless.consumer] Consuming test2.txt
[2024-04-29 18:04:12,469] [DEBUG] [paperless.consumer] Detected mime type: text/plain
[2024-04-29 18:04:12,473] [DEBUG] [paperless.consumer] Parser: TextDocumentParser
[2024-04-29 18:04:12,477] [DEBUG] [paperless.consumer] Parsing test2.txt...
[2024-04-29 18:04:12,477] [DEBUG] [paperless.consumer] Generating thumbnail for test2.txt...
[2024-04-29 18:04:12,613] [DEBUG] [paperless.consumer] Saving record to database
[2024-04-29 18:04:12,613] [DEBUG] [paperless.consumer] Creation date from st_mtime: 2024-04-29 18:04:04.317678+02:00
[2024-04-29 18:04:13,004] [INFO] [paperless.handlers] Assigning document type Versicherungsunterlage to 2024-04-29 test2
[2024-04-29 18:04:13,012] [INFO] [paperless.handlers] Tagging "2024-04-29 test2" with "Versicherung"
[2024-04-29 18:04:13,072] [DEBUG] [paperless.consumer] Deleting file /tmp/paperless/paperless-ngxknlft_b5/test2.txt
[2024-04-29 18:04:13,075] [DEBUG] [paperless.parsing.text] Deleting directory /tmp/paperless/paperless-7wo0jdiy
[2024-04-29 18:04:13,075] [INFO] [paperless.consumer] Document 2024-04-29 test2 consumption finished
[2024-04-29 18:05:00,133] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 18:05:00,163] [DEBUG] [paperless.classifier] 8 documents, 1 tag(s), 0 correspondent(s), 1 document type(s). 0 storage path(es)
[2024-04-29 18:05:00,163] [DEBUG] [paperless.classifier] Vectorizing data...
[2024-04-29 18:05:00,575] [DEBUG] [paperless.classifier] Training tags classifier...
[2024-04-29 18:05:00,901] [DEBUG] [paperless.classifier] There are no correspondents. Not training correspondent classifier.
[2024-04-29 18:05:00,901] [DEBUG] [paperless.classifier] Training document type classifier...
[2024-04-29 18:05:01,203] [DEBUG] [paperless.classifier] There are no storage paths. Not training storage path classifier.
[2024-04-29 18:05:01,206] [INFO] [paperless.tasks] Saving updated classifier model to /usr/src/paperless/data/classification_model.pickle...
[2024-04-29 18:56:05,051] [DEBUG] [paperless.management.consumer] Consumer exiting.
[2024-04-29 22:58:43,100] [INFO] [paperless.management.consumer] Polling directory for changes: /usr/src/paperless/consume
[2024-04-29 22:58:44,635] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 22:58:44,670] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 22:58:44,673] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-29 23:05:00,116] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-29 23:05:00,153] [INFO] [paperless.classifier] No updates since last training
[2024-04-29 23:05:00,157] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 00:05:00,120] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 00:05:00,147] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 00:05:00,150] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 01:05:00,111] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 01:05:00,137] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 01:05:00,140] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 02:05:00,123] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 02:05:00,153] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 02:05:00,156] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 03:05:00,115] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 03:05:00,142] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 03:05:00,145] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 04:05:00,109] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 04:05:00,134] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 04:05:00,137] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 05:05:00,212] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 05:05:00,244] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 05:05:00,248] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 06:05:00,111] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 06:05:00,137] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 06:05:00,140] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 07:05:00,119] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 07:05:00,148] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 07:05:00,150] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 08:05:00,112] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 08:05:00,138] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 08:05:00,141] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 09:05:00,118] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 09:05:00,144] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 09:05:00,147] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 10:05:00,116] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 10:05:00,144] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 10:05:00,148] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 11:05:00,122] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 11:05:00,149] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 11:05:00,152] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 12:05:00,116] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 12:05:00,143] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 12:05:00,146] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 13:05:00,126] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 13:05:00,163] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 13:05:00,167] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 14:05:00,112] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 14:05:00,138] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 14:05:00,141] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 15:05:00,126] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 15:05:00,155] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 15:05:00,158] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 16:05:00,113] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 16:05:00,139] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 16:05:00,142] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 17:05:00,114] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 17:05:00,142] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 17:05:00,144] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 18:05:00,110] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 18:05:00,135] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 18:05:00,138] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 19:05:00,118] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 19:05:00,145] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 19:05:00,148] [DEBUG] [paperless.tasks] Training data unchanged.
[2024-04-30 20:05:00,114] [DEBUG] [paperless.classifier] Gathering data from database...
[2024-04-30 20:05:00,139] [INFO] [paperless.classifier] No updates since last training
[2024-04-30 20:05:00,142] [DEBUG] [paperless.tasks] Training data unchanged.

The two test files that were consumed were created in the folder in a SSH session.