Proper way to add custom root CA certificates ?

[pandaem0nium]

Hello,

I'm trying to make OIDC working for Paperless with Keycloak and Smallstep's step-ca (a private CA that can act as an ACME server). These services are deployed through Docker, and Traefik handles the proxying and the certificate requests.

I've made a custom Paperless image to add my root CA certificate to its trust store.

FROM ghcr.io/paperless-ngx/paperless-ngx:2.6.3

COPY ./root_ca.crt /usr/local/share/ca-certificates/

RUN /usr/sbin/update-ca-certificates

I've added the following variables to my docker-compose.yml :

PAPERLESS_APPS="allauth.socialaccount.providers.openid_connect"
PAPERLESS_SOCIALACCOUNT_PROVIDERS='
{"openid_connect": {"APPS": [{"provider_id": "keycloak","name": "Keycloak","client_id": "paperless","secret": "somesecret","settings": { "server_url": "https://auth.somedomain/realms/Local.domain/.well-known/openid-configuration"}}]}}'

I'm able to curl on the given url from the paperless container without issues, but Django does not seem to be aware of this certificate. I got a Server Error (500) when I try to sign in to Paperless:

paperless  | [2024-04-02 14:42:13,759] [ERROR] [django.request] Internal Server Error: /accounts/oidc/keycloak/login/
paperless  | Traceback (most recent call last):
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 467, in _make_request
paperless  |     self._validate_conn(conn)
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
paperless  |     conn.connect()
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 653, in connect
paperless  |     sock_and_verified = _ssl_wrap_socket_and_match_hostname(
paperless  |                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
paperless  |     ssl_sock = ssl_wrap_socket(
paperless  |                ^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
paperless  |     ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
paperless  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
paperless  |     return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/ssl.py", line 517, in wrap_socket
paperless  |     return self.sslsocket_class._create(
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/ssl.py", line 1104, in _create
paperless  |     self.do_handshake()
paperless  |   File "/usr/local/lib/python3.11/ssl.py", line 1382, in do_handshake
paperless  |     self._sslobj.do_handshake()
paperless  | ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
paperless  | 
paperless  | During handling of the above exception, another exception occurred:
paperless  | 
paperless  | Traceback (most recent call last):
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 793, in urlopen
paperless  |     response = self._make_request(
paperless  |                ^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 491, in _make_request
paperless  |     raise new_e
paperless  | urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)
paperless  | 
paperless  | The above exception was the direct cause of the following exception:
paperless  | 
paperless  | Traceback (most recent call last):
paperless  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 486, in send
paperless  |     resp = conn.urlopen(
paperless  |            ^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/connectionpool.py", line 847, in urlopen
paperless  |     retries = retries.increment(
paperless  |               ^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/urllib3/util/retry.py", line 515, in increment
paperless  |     raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
paperless  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  | urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='auth.somedomain', port=443): Max retries exceeded with url: /realms/Local.domain/.well-known/openid-configuration (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))
paperless  | 
paperless  | During handling of the above exception, another exception occurred:
paperless  | 
paperless  | Traceback (most recent call last):
paperless  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
paperless  |     raise exc_info[1]
paperless  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/exception.py", line 42, in inner
paperless  |     response = await get_response(request)
paperless  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 534, in thread_handler
paperless  |     raise exc_info[1]
paperless  |   File "/usr/local/lib/python3.11/site-packages/django/core/handlers/base.py", line 253, in _get_response_async
paperless  |     response = await wrapped_callback(
paperless  |                ^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 479, in __call__
paperless  |     ret: _R = await loop.run_in_executor(
paperless  |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/asgiref/current_thread_executor.py", line 40, in run
paperless  |     result = self.fn(*self.args, **self.kwargs)
paperless  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/asgiref/sync.py", line 538, in thread_handler
paperless  |     return func(*args, **kwargs)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/openid_connect/views.py", line 69, in login
paperless  |     return view(request)
paperless  |            ^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 87, in view
paperless  |     return self.dispatch(request, *args, **kwargs)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/base/mixins.py", line 10, in dispatch
paperless  |     return self.login(request, *args, **kwargs)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 116, in login
paperless  |     client = self.get_client(request, app)
paperless  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/oauth2/views.py", line 102, in get_client
paperless  |     self.adapter.access_token_url,
paperless  |     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/openid_connect/views.py", line 39, in access_token_url
paperless  |     return self.openid_config["token_endpoint"]
paperless  |            ^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/allauth/socialaccount/providers/openid_connect/views.py", line 23, in openid_config
paperless  |     resp = get_adapter().get_requests_session().get(server_url)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
paperless  |     return self.request("GET", url, **kwargs)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
paperless  |     resp = self.send(prep, **send_kwargs)
paperless  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
paperless  |     r = adapter.send(request, **kwargs)
paperless  |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
paperless  |   File "/usr/local/lib/python3.11/site-packages/requests/adapters.py", line 517, in send
paperless  |     raise SSLError(e, request=request)
paperless  | requests.exceptions.SSLError: HTTPSConnectionPool(host='auth.somedomain', port=443): Max retries exceeded with url: /realms/Local.domain/.well-known/openid-configuration (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)')))

I suppose it should be possible, as I've been able to make OIDC working for Forgejo with the same method.

I've also tried PAPERLESS_EMAIL_CERTIFICATE_LOCATION as discussed in #5114, but no luck.

Is there a proper way to make Paperless / Django aware of a private root CA certificate ?

[stumpylog]

RUN /usr/sbin/update-ca-certificates

What does this output? Does it recognize a new certificate was added?

You might also want to ask the allauth users, https://github.com/pennersr/django-allauth/discussions

[pandaem0nium]

I've updated the command to show some output.

RUN /usr/sbin/update-ca-certificates --verbose

Which gives :

#7 [3/3] RUN /usr/sbin/update-ca-certificates --verbose
#7 0.251 Updating certificates in /etc/ssl/certs...
#7 1.244 rehash: warning: skipping ca-certificates.crt,it does not contain exactly one certificate or CRL
#7 1.246 Doing .
#7 1.246 link D-TRUST_Root_Class_3_CA_2_EV_2009.pem -> d4dae3dd.0
[...]
#7 1.246 link root_ca.pem -> 7405e5a0.0
[...]
#7 1.248 1 added, 0 removed; done.
#7 1.248 Running hooks in /etc/ca-certificates/update.d...
#7 1.249 done.
#7 DONE 1.3s

This is consistent with the fact that I cannot curl on this url if I use the base image.

root@paperless:/usr/src/paperless/src# curl https://auth.somedomain/realms/Local.domain/.well-known/openid-configuration
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

I'll ask the allauth users, thanks @stumpylog.

[pandaem0nium]

The django-allauth maintainer pointed me in the right direction #3716.

The key was to add the REQUESTS_CA_BUNDLE to my docker-compose.yml.

services:
  paperless:
    [...]
    environment:
      [...]
      REQUESTS_CA_BUNDLE:  /path/to/ca_cert.crt

There may be side effects, as mentionned in the link he gave me.

[athiosh]

First time setting up paperless today, setup SSO with Authentik and had the same issue. I tried adding the new variable in my docker compose but nothing happen. Are you building your own image or pulling the latest? Any help would be appreciated

[github-actions[bot]]

This discussion has been automatically locked since there has not been any recent activity after it was closed. Please open a new discussion for related concerns. See our contributing guidelines for more details.