LDAP implementation, certificate not found

[shyreneching]

Hello, I'm trying to implement LDAP in Stackstorm but for some reason, I am receiving an error from the st2auth not being able to read the certificate file. Would anyone know how to address this issues?

ValueError: Unable to find the cacert file "/opt/stackstorm/st2/certificate/cacert.pem" for the LDAP connection.

When I cat my cert file, it is in this format

cat /opt/stackstorm/st2/certificate/cacert.pem
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

[nzlosh]

Please show the full command used to cat the contents of the file.

[guzzijones]

could you post the full stack trace. This could very well be a bug.

[shyreneching]

Here is the logs for the /var/log/st2/st2auth.log file. I replaced the backend_kwargs details

[2024-01-31 07:29:29 +0000] [1118] [INFO] Starting gunicorn 20.1.0
[2024-01-31 07:29:29 +0000] [1118] [INFO] Listening at: http://127.0.0.1:9103 (1118)
[2024-01-31 07:29:29 +0000] [1118] [INFO] Using worker: eventlet
[2024-01-31 07:29:29 +0000] [1736] [INFO] Booting worker with pid: 1736
[2024-01-31 07:29:59 +0000] [1118] [CRITICAL] WORKER TIMEOUT (pid:1736)
[2024-01-31 07:29:59 +0000] [1736] [INFO] Worker exiting (pid: 1736)
[2024-01-31 07:30:00 +0000] [1118] [WARNING] Worker with pid 1736 was terminated due to signal 9
[2024-01-31 07:30:00 +0000] [2088] [INFO] Booting worker with pid: 2088
[2024-01-31 07:30:30 +0000] [1118] [CRITICAL] WORKER TIMEOUT (pid:2088)
[2024-01-31 07:30:30 +0000] [2088] [INFO] Worker exiting (pid: 2088)
[2024-01-31 07:30:31 +0000] [1118] [WARNING] Worker with pid 2088 was terminated due to signal 9
[2024-01-31 07:30:31 +0000] [2455] [INFO] Booting worker with pid: 2455
[2024-01-31 07:31:01 +0000] [1118] [CRITICAL] WORKER TIMEOUT (pid:2455)
[2024-01-31 07:31:01 +0000] [2455] [INFO] Worker exiting (pid: 2455)
[2024-01-31 07:31:02 +0000] [1118] [WARNING] Worker with pid 2455 was terminated due to signal 9
[2024-01-31 07:31:02 +0000] [2798] [INFO] Booting worker with pid: 2798
[2024-01-31 07:31:23 +0000] [2798] [ERROR] Exception in worker process
Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/backends/__init__.py", line 53, in get_backend_instance
    cls_instance = cls(**kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth_ldap/ldap_backend.py", line 99, in __init__
    (cacert))
ValueError: Unable to find the cacert file "/opt/stackstorm/st2/certificate/cacert.pem" for the LDAP connection.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
    worker.init_process()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/geventlet.py", line 134, in init_process
    super().init_process()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base.py", line 134, in init_process
    self.load_wsgi()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/workers/base.py", line 146, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 58, in load
    return self.load_wsgiapp()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 48, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/gunicorn/util.py", line 359, in import_app
    mod = importlib.import_module(module)
  File "/opt/stackstorm/st2/lib64/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/wsgi.py", line 39, in <module>
    application = app.setup_app(config)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/app.py", line 80, in setup_app
    router.add_spec(spec, transforms=transforms)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/router.py", line 262, in add_spec
    __import__(module_name)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/controllers/v1/auth.py", line 92, in <module>
    token_controller = TokenController()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/controllers/v1/auth.py", line 61, in __init__
    self.handler = HANDLER_MAPPINGS[cfg.CONF.auth.mode]()
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/handlers.py", line 150, in __init__
    self._auth_backend = get_auth_backend_instance(name=cfg.CONF.auth.backend)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/backends/__init__.py", line 63, in get_backend_instance
    raise exc_cls(msg)
ValueError: Failed to instantiate auth backend "ldap" (class LDAPAuthenticationBackend) with backend settings "{"bind_dn": "CN=st2admin,ou=[my_ou]", "bind_password": "st2poc", "base_ou": "ou=[my_ou]", "group_dns": ["CN=[common_name_of_my_group],ou=[my_ou]"], "host": "[ldap_server]", "port": 636, "use_ssl": true, "cacert": "/opt/stackstorm/st2/certificate/cacert.pem"}": Unable to find the cacert file "/opt/stackstorm/st2/certificate/cacert.pem" for the LDAP connection.

Traceback (most recent call last):
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth/backends/__init__.py", line 53, in get_backend_instance
    cls_instance = cls(**kwargs)
  File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2auth_ldap/ldap_backend.py", line 99, in __init__
    (cacert))
ValueError: Unable to find the cacert file "/opt/stackstorm/st2/certificate/cacert.pem" for the LDAP connection.

[2024-01-31 07:31:26 +0000] [2798] [INFO] Worker exiting (pid: 2798)
[2024-01-31 07:31:30 +0000] [1118] [INFO] Shutting down: Master
[2024-01-31 07:31:30 +0000] [1118] [INFO] Reason: Worker failed to boot.

[guzzijones]

Lets mitigate the gunicorn worker timeouts first.
Extend the timeout on the gunicorn worker for the st2auth service in the st2auth systemctl unit file.

[guzzijones]

Also , what are the permissions on the folder for the cacert file and what is the size of the cacert file?

[shyreneching]

I have change the permission of the folder, which seems to be the problem as to why it can't find the cert. I missed this and just modified the permission of the actual cert. However, even after extending the timeout to 600 secs, I still receive the worker timeout error.

[2024-02-02 10:31:53 +0000] [27498] [INFO] Starting gunicorn 20.1.0
[2024-02-02 10:31:53 +0000] [27498] [INFO] Listening at: http://127.0.0.1:9103 (27498)
[2024-02-02 10:31:53 +0000] [27498] [INFO] Using worker: eventlet
[2024-02-02 10:31:54 +0000] [27663] [INFO] Booting worker with pid: 27663
[2024-02-02 10:32:24 +0000] [27498] [CRITICAL] WORKER TIMEOUT (pid:27663)
[2024-02-02 10:32:24 +0000] [27663] [INFO] Worker exiting (pid: 27663)
[2024-02-02 10:32:25 +0000] [27498] [WARNING] Worker with pid 27663 was terminated due to signal 9
[2024-02-02 10:32:25 +0000] [27985] [INFO] Booting worker with pid: 27985
[2024-02-02 10:33:41 +0000] [27498] [INFO] Handling signal: term
[2024-02-02 10:33:42 +0000] [27985] [INFO] Worker exiting (pid: 27985)
[2024-02-02 10:33:42 +0000] [27498] [INFO] Shutting down: Master
[2024-02-02 10:33:56 +0000] [28635] [INFO] Starting gunicorn 20.1.0
[2024-02-02 10:33:56 +0000] [28635] [INFO] Listening at: http://127.0.0.1:9103 (28635)
[2024-02-02 10:33:56 +0000] [28635] [INFO] Using worker: eventlet
[2024-02-02 10:33:56 +0000] [28808] [INFO] Booting worker with pid: 28808
[2024-02-02 10:34:26 +0000] [28635] [CRITICAL] WORKER TIMEOUT (pid:28808)
[2024-02-02 10:34:26 +0000] [28808] [INFO] Worker exiting (pid: 28808)
[2024-02-02 10:34:27 +0000] [28635] [WARNING] Worker with pid 28808 was terminated due to signal 9
[2024-02-02 10:34:27 +0000] [29102] [INFO] Booting worker with pid: 29102

Not sure if it is working as expected as I am still unable to login using the correct credentials. I am receiving an error when loging in:

ERROR: Failed to log in as [username]: 401 Client Error: Unauthorized
MESSAGE: Invalid or missing credentials for url: http://127.0.0.1:9103/tokens

[guzzijones]

Check the journal to logs for the service. For some reason it is not starting.

…

On Fri, Feb 2, 2024 at 05:45 Shyrene Ching ***@***.***> wrote: I have change the permission of the folder, which seems to be the problem as to why it can't find the cert. I missed this and just modified the permission of the actual cert. However, even after extending the timeout to 260 secs, I still receive the worker timeout error. [2024-02-02 10:31:53 +0000] [27498] [INFO] Starting gunicorn 20.1.0 [2024-02-02 10:31:53 +0000] [27498] [INFO] Listening at: http://127.0.0.1:9103 (27498) [2024-02-02 10:31:53 +0000] [27498] [INFO] Using worker: eventlet [2024-02-02 10:31:54 +0000] [27663] [INFO] Booting worker with pid: 27663 [2024-02-02 10:32:24 +0000] [27498] [CRITICAL] WORKER TIMEOUT (pid:27663) [2024-02-02 10:32:24 +0000] [27663] [INFO] Worker exiting (pid: 27663) [2024-02-02 10:32:25 +0000] [27498] [WARNING] Worker with pid 27663 was terminated due to signal 9 [2024-02-02 10:32:25 +0000] [27985] [INFO] Booting worker with pid: 27985 [2024-02-02 10:33:41 +0000] [27498] [INFO] Handling signal: term [2024-02-02 10:33:42 +0000] [27985] [INFO] Worker exiting (pid: 27985) [2024-02-02 10:33:42 +0000] [27498] [INFO] Shutting down: Master [2024-02-02 10:33:56 +0000] [28635] [INFO] Starting gunicorn 20.1.0 [2024-02-02 10:33:56 +0000] [28635] [INFO] Listening at: http://127.0.0.1:9103 (28635) [2024-02-02 10:33:56 +0000] [28635] [INFO] Using worker: eventlet [2024-02-02 10:33:56 +0000] [28808] [INFO] Booting worker with pid: 28808 [2024-02-02 10:34:26 +0000] [28635] [CRITICAL] WORKER TIMEOUT (pid:28808) [2024-02-02 10:34:26 +0000] [28808] [INFO] Worker exiting (pid: 28808) [2024-02-02 10:34:27 +0000] [28635] [WARNING] Worker with pid 28808 was terminated due to signal 9 [2024-02-02 10:34:27 +0000] [29102] [INFO] Booting worker with pid: 29102 — Reply to this email directly, view it on GitHub <#6117 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACZ5TIMLY3JRIDOB5GC2FETYRS7TDAVCNFSM6AAAAABCL3VRFGVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DGNBVGIYDM> . You are receiving this because you commented.Message ID: ***@***.***>