Can't pam_mount a cifs volume.

[Timost]

Hi,
First of all thank you for your work on jupyterhub ! 🙏

I'm working in an environement where I think pamela prevents me from mounting a volume using pam_mount.
This is similara/related to jupyterhub/jupyterhub#810

Here are the details:

• Ubuntu 20.04, python 3.8
• Users are authenticated against an Active Directory using https://arthurdejong.org/nss-pam-ldapd.
• Their home folder is created if it doesn't exist using pam_mkhomedir.
• The Active Directory credentials are then used to automatically mount a volume from a networked file system using cifs.

The flow works properly with ssh but when connecting through jupyterhub the mounting part of the flow fails with a permission denied or pam_mount asks me to reenter the password (depending on whether the disable_interactive flag is passed to pam_mount).

After some digging, my understanding is that the authenticate method of pamela does not start a pam session and expects the pam sessions to be started later on after authentication succeeds.

If that is the case then the credentials to pam_mount are lost befire the pam_session is created and therefore the pam_mount cannot proceed accordingly.

The current dirty fix that I use is opening a pam session directly at the end of authenticate:

    if retval == 0 and resetcred:
        PAM_SETCRED(handle, resetcred)
    
    # CHANGE BEGIN
    # open_session before the handle gets destroyed with pam_end
    # (ex: pam_mount needs state info stored previously in the auth facility)
    if retval == 0:
        retval = PAM_OPEN_SESSION(handle, 0)
    # CHANGE END

    if close:
        return pam_end(handle, retval)

I've tried to pass close=False to the function but it did not work.

I'm not sure that's the proper way of doing things. What do you think ? Could that change be activated using a flag just like resetcred ?

[Timost]

I have been digging a bit more in the issue, reading pam_mount documentation and the source code (debian version) as well as JupyterHub auth module.

It seems pam_mount stores the authtoken in a store that is bound to the PAM transaction which is both opened and closed (and cleaned) in authenticate , therefore the credentials are already lost when the PAM session is opened by the spawner.

Which sadely seems to confirms that the only way to forward the auth_token to pam_mount is by opening a session while the first authenticate pam transaction is opened.

[minrk]

Our PAM session lifecycle in JupyterHub is definitely wrong for nontrivial setups where other operations are tied to the PAM session. The right thing to do is make all PAM calls in a subprocess which should be the parent of the Spawner process, as described here, passing the handle around properly, though that's pretty complicated to fit into JupyterHub's design where login and spawn are often significantly separated by hours or days or processes (Authenticator.refresh_user can be leveraged for this). If you're interested in putting in some time to figuring that out (and/or producing an isolated test case that others can run easily), that would be wonderful, and I can help with some guidance on the relevant pieces of JupyterHub.

[Timost]

Thank you for your answer,

I've gone through the linked issue. Just to make sure I understood your point. You think that the fact that PAM sessions are created in a separate process is the root cause of my issue ? Reading the issue, it does seem so but I haven't tried the fix proposed in the issue.

I can put some time in, maybe first by trying to give you a reproducible setup. Although that will most likely be quite complicated in itself. I'm thinking of maybe a docker compose file with several services (jupyterhub, a dummy ldap setup, and somehow a network share ❔ and an initialization script). Would that be okay or would you prefer something else ?

[Timost]

Ok so I've built a minial setup to reproduce the issue. You can find it here https://github.com/Timost/jupyter-pam-bug-demo.
I dropped the LDAP part of the thing because it was too hard to reproduce.
Instead it's a simple sshfs setup with user/password authentication.

The issue is the same with this setup than with my "real" use case though so I think it's a good sandbox env.

[minrk]

You think that the fact that PAM sessions are created in a separate process is the root cause of my issue ?

That's a big part of it, especially in certain SELINUX cases where the calling process is modifed (subsequent calls to open_session fail with different users because the jupyterhub process has lost the necessary permission!) - it also just happens at the wrong times except in the simplest cases.

That test sandbox will be very helpful, thanks!

[Timost]

No problem, thank you for your time ❤️

How can I help moving this forward ?
I can spend some time to work on the issue but I'm not entirely sure how to proceed.
I'm thinking of maybe creating a custom Authenticator that would provide an enhanced PAM transaction(s).

Do you think that's the right way of doing it ?

[minrk]

Yes, prototyping a custom PAM Authenticator subclass sounds right. When you have something that works, we can see if it makes sense to upstream it to the default implementation or keep it standalone.

[jeflem]

Hi,
digging in the same area as @Timost I want to share my findings here. There are several related problems with PAM and JupyterHub if c.PAMAuthenticator.open_sessions = True in JupyterHub's config:

• pam_loginuid.so fails with 'permission denied',
• pam_mount.so does not obtain the password used during PAM authentification,
• jupyterhub segfaults when the PAM session is closed (hit 'Stop my server' button in hub control panel), see Segfault in jupyterhub jupyterhub#1658.

As far as I understand these problems at the moment they all are caused by faulty PAM usage in pamela (and to some extent also by JupyterHub).

Before I go on: Sorry for this lengthy post, but the problem is not trivial and requires some more explanation.

What JupyterHub does if c.PAMAuthenticator.open_sessions = True:

1. pamela.authenticate(..., close=True) (at login)
2. pamela.open_session(...) (when single-user server is spawned)
3. pamela.close_session(...) (when single-user server is stopped)

What pamela does:

1. handle = pam_start(...)
PAM_AUTHENTICATE(handle)
pam_end(handle)
2. handle = pam_start(...)
PAM_OPEN_SESSION(handle)
pam_end(handle)
3. handle = pam_start(...)
PAM_CLOSE_SESSION(handle)
pam_end(handle)

What pamela (or JupyterHub) SHOULD do:

1. handle = pam_start(...)
2. PAM_AUTHENTICATE(handle)
3. PAM_OPEN_SESSION(handle)
4. PAM_CLOSE_SESSION(handle)
5. pam_end(handle)

Calling pamela.authenticate with close=False in 1. suppresses pam_end and makes authenticate return the PAM handle. But pamela.open_session does not use this handle.

As far as I understand, the PAM handle returned by pam_start is important. It's the reference to what sometimes is called the PAM context. Only PAM actions belonging to the same context (same handle) can interact. So pam_mount.so only gets the password from the authentification stage if corresponding session is opened in the same PAM context.

Above mentioned segfault is a result of closing a PAM session in a PAM context in which we never opened a session. Of course, this should not result in a segfault, but in an error message. I did not dig deeper in this direction. It segfaults somewhere in libc.

From the pamela side of view it would be nice, if pamela.open_session would allow for passing a PAM handle. The JupyterHub side of view is much more difficult. The (now) obvious attempt to get pam_mount.so (as well as pam_loginuid.so and some others) working correctly is as follows:

• In PAMAuthenticator.authenticate call pamela.authenticate with close=False and save the returned PAM handle in, let's say, self.pam_handle (here self is the PAMAuthenticator object).
• In PAMAuthenticator.pre_spawn_start call pamela.PAM_OPEN_SESSION(self.pam_handle).
• In PAMAuthenticator.post_spawn_stop call pamela.PAM_CLOSE_SESSION(self.pam_handle) and pamela.pam_end(self.pam_handle).

I've tested this and it works. BUT if the hub user stops and restarts its (single-user) server or if the user spawns multiple servers, then it fails (segfault when stopping the REstarted server). The reason is that calling pam_end(self.handle) invalidates the PAM handle. The next spawn tries to open a PAM session with the now invalid handle. Opening the PAM session fails, but the server spawns (without PAM session). Stopping the server tries to close a never opened PAM session... segfault.

How to properly handle PAM sessions and multiple servers in JupyterHub? I don't know because I do not have an overview of how authentication/login to the hub and spawning servers interact. Both processes are more or less independent from each other. There are at least two approaches:

• One PAM session for all servers: Open the PAM session as soon as the first server is spawned. Close the session if the last server stoppend. This requires some counting of spawns and stops. Also there might be some security issues, because servers may run several months without reauthentication of the user. Don't know the details here.
• Each server opens and closes its own PAM session. But this requires to transfer user name and password to the single-user server. And the server then has to redo the PAM authentication process to allow for opening a PAM session. In my setup this shouldn't be a problem. But what about setups where PAM authentication requires some user interaction. Then the user has to authenticate multiple times. Don't know whether such setups exist...

I'm willing to help solving the PAM/JupyterHub issues. Primarily because I need this setup, secondarily to contribute my 50 cents to the wonderful open source universe. But neither I'm a software developer nor an admin. Just a guy you wants to use JupyterHub for some data science related teaching and has to setup the server in lack of an up-to-date IT department.

Many thanks in advance for each improvement of JupyterHub and Co.!

Best regards,

jeflem

[Timost]

Hi @jeflem,
Thank you for the detailed analysis ❤️ , I kind of reached the same conclusions as you but I wouldn't have been able to put them in words as clearly as you did.

Regarding the two approaches you suggest, I think the second one with a re-authentication mechanism is favoured, but @minrk should have much more insight on that than me.
Maybe a combination of PAM handle expiry and authentication state could provide a reasonable user experience.

This is all theoretical, I haven't started on a new authenticator yet, but I've been reading as much doc/code as I can. The authentication state could help securely storing the handle . I have no idea if the PAM/pamela handle could be stored there tough.

[jeflem]

Thanks for the hint to auth_state @Timost !

Now I had a closer look at JupyterHub's source, in particular auth.py: There is only one common PAMAuthenticator object for all users of the hub and for handling all (repeated) logins. So its a bad idea to store the PAM handle in the authenticator object (we cannot use one PAM handle for all users).

But each single-user server has its own Spawner object. So storing user name and password in the Spawner objects could be an approach. Then in Spawner.start one could call PAM authentication, open the PAM session, store the handle. In Spawner.stop the PAM session gets closed. So we do not have to touch the single-user server. Username/password stay inside JupyterHub.

As far as I understand at the moment, transfer of username/password from PAMAuthenticator.authenticate to PAMAuthenticator.pre_spawn_start should be done via auth_state. And in pre_spawn_start we have access to the concrete spawner object. Will try this with LocalProcessSpawner next days...

[jeflem]

Update: auth_state works quite well for transfering username and password to the spawners. Also pamela is okay. The PAM handle object returned by pamela.authenticate(..., close=False) has two methods pam_handle.open_session() and pam_handle.close_session() for opening and closing PAM sessions in the PAM context used for authentication.

BUT if one opens multiple PAM sessions in parallel with independent PAM contexts (spawn multiple servers), then only one of the session can be closed. Closing a second session leads to segfault in pam_mount.so :-( With pam_mount.so disabled everything works fine. But the reason for digging into this JupyterHub/PAM problem was that I want to use pam_mount.so with JupyterHub.

Here is a short Python script for testing:

import pamela as p

user = 'someuser'
service = 'pamtest' # use 'login' or create custom service /etc/pam.d/pamtest
pw = 'somepassword'

h1 = p.authenticate(user, pw, service, close=False)
h1.open_session()
h1.close_session()

h2 = p.authenticate(user, pw, service, close=False)
h2.open_session()
h2.close_session()

p.pam_end(h2, 0)

p.pam_end(h1, 0)

Calling pam_end(h1, 0) before second authentication works, but closing after closing second PAM context segfaults.

To confirm that this is not a pamela issue I tested this with an equivalent C program directly calling libpam. Segfaults, too.

Maybe a bug in pam_mount.so?

[Timost]

Creating the two pam sessions from two different python processes doesn't cause the segfault I think.

# python process 1
import pamela as p
user = "jupyterhub"
pw = "admin"
service = "jupyterhub"
h1 = p.authenticate(user, pw, service, close=False)
h1.open_session()
# h1.close_session() can also be called here but I'm not sure that mimics the actual behaviour

# python process 2
import pamela as p

user = "jupyterhub"
pw = "admin"
service = "jupyterhub"
h2 = p.authenticate(user, pw, service, close=False)
h2.open_session()
h2.close_session()
p.pam_end(h2, 0)

# python process 1
h1.close_session()
p.pam_end(h1, 0)

My current mental model is that the spawners run in separate processes so I'm not sure if this is an actual issue.

[jeflem]

There are (at least) two variants for opening/closing PAM sessions:

• Open/close in JupyterHub's spawner.py, thas is, in LocalProcessSpawner.start and LocalProcessSpawner.stop. Then all PAM transactions belong to the same process jupyterhub. Tested this with os.getpid in LocalProcessSpawner.start. Each call to LocalProcessSpawner.start takes place in the same jupyterhub process.
• Open/close by the single-user servers. Assuming that each server runs as separate process (did not check this), then there is only one PAM session per process and the segfault is not relevant.

From my very limited understanding of JupyterHub's structure the first variant is preferable, because all the authentication stuff is done by JupyterHub. For the second variant each single-user server (I think, there exist different ones?) would have to implement PAM session handling.

[jeflem]

Becomes a bit off-topic, because its not really a pamela issue (see previous comment). In my opinion there are two problems:

• JupyterHub does PAM session handling completely wrong. This can be repaired by doing PAM session handling in LocalProcessSpawner on a per spawner basis (see above).
• With the per spawner approach pam_mount.so segfaults if multiple PAM transactions or started in parallel (a PAM transaction is everything between pam_start and pam_end).

Further digging revealed the cause of the segfault: pam_mount uses a global Config structure holding the module's state. There is no PAM handle dependent data management. I've checked this by writing some C code which for each PAM handle gets the memory locations of data stored by pam_mount.so. Its always the same location (independent of PAM handle).

So several parallel PAM transactions share some memory with pam_mount related data and libpam does not recognize this. Ending one transaction cleans the memory and the remaining PAM transactions still try to read and write at this location... segfault. It's clearly a problem with pam_mount, not libpam (note that pam_mount is a separate project). Fixing pam_mount would require major changes to the source. It's not a two-liner.

I'm not sure whether this memory sharing may break other functionality, too. As far as I understand from the source code, only username and auth token are managed on a per PAM handle basis. Others (list of mounted volumes for instance), are global. There are lots of bug reports out there about umounts not taking place at logout...

Since I do not have any experience with software development, bug reporting and so on, I'm not sure what to do now?

• Open an issue for JupyterHub to get PAM session handling fixed? But without parallel PAM transactions because pam_mount does not support such.
• Communicate the bug to the pam_mount developer? But seems there is low to no activity in the project (hosted on sourceforge).

Any suggestions?

[jeflem]

Update:

• reported pam_mount bug: bug report at sourceforge
• There has been a pull request [WIP] Moved PAM session creations to a preexec_fn jupyterhub#2321 for implementing server-wise PAM session handling. But it died due to unexpected complexity of the approach. Related issue is Server process being moved to first user's systemd cgroup jupyterhub#2120
• related (old) issue: PAM Auth using SSSD not creating home directories upon login for new users jupyterhub#3106 (solved without fixing JupyterHub's PAM session handling)
• another (old) related issue: jupyterhub login issue with second user jupyterhub#2406 (solved without fixing JupyterHub's PAM session handling)

[jeflem]

Reading and thinking about the JupyterHub/PAM issue brought me to the conclusion that fixing JupyterHub's PAM session handling in a way which makes pam_mount.so work correctly without fixing the above mentioned pam_mount bug is impossible. Thus, I stop my research here and try to find a solution for my JupyterHub setup that works without pam_mount.

For the record here comes a list of approaches investigated by others or me together with reasons of failure:

Two-level approach of jupyterhub/jupyterhub#2321 in 2018

The pull request tries to open and close a PAM session in an intermediate process, which is started by jupyterhub and itself starts the single-user server. The pull request died due to unexpected complexity and side effects.

Even if one would implement this approach and solve all problems discussed in the pull request, one would have to pass the user's password somehow to the intermediate process. As far is I understand the mechanics of process management, this means either storing the clear text password in an environment variable or passing it as a (command line) argument to the process or maybe via some file on disk. This sounds very dubious regarding password security. Maybe some encryption could be used similar to JupyterHub's auth_state handling, but this would add further complexity.

Note, that the discussion in the pull request does not take into account this password issue. The pull request is not aware of the fact that opening a PAM session (containing pam_mount.so) requires prior authentication within the same PAM transaction, that is, within the intermediate process.

Approach implemented by @gatoniel in 2020

Reading the source code (not testing it) I think that this implementation does not work if pam_mount.so is part of the PAM stack (which is the reason for wanting PAM sessions to work with JupyterHub). Opening and closing PAM sessions is done in the jupyterhub process. So pam_mount.so will segfault if there are more than one users and/or servers.

Also opening the PAM session is done in PAM transaction without authentication. So pam_mount will prompt the user for a password, but the user won't recognize this.

Parallel server and PAM handling processes

For each single-user server start a parallel PAM handling process. It's similar to the pull request approach above, but less complex. Passing the user password to the PAM handling process is not trivial but required.

Although for pam_mount such a parallel process would suffice, I'm not sure whether other PAM modules fail if the session is started by a different process and not by the server itself. What about pam_selinux.so for instance? Lack of knowledge...

PAM handling by single-user server

We could tell the single-user server to open a PAM session. But how to pass the password? Do we really want to do PAM/authentication stuff in the server? The server may also run in non-PAM environment, so shouldn't have to care about PAM session handling.

JupyterHub manages PAM sessions in spawners

At each spawn a PAM session is opend by JupyterHub and closed after the server stopped. Sadly, this fails due to the bug in pam_mount.so (does not support multiple PAM transactions within one process). But would solve several other problems:

• password does not have to be passed to other processes,
• simple implementation,
• no additional processes to start and manage,
• authentication and PAM session handling are well separated from the single-user server.

[jeflem]

Sorting and cleaning my notes taken the past days I found further old issues related to the JupyterHub/PAM problem, just for the record:

• Login -> Logout -> Login fails jupyterhub#486
• PAM authentication errors jupyterhub#323

[minrk]

Thanks for detailing all that!

Open an issue for JupyterHub to get PAM session handling fixed?

There already is an issue on JupyterHub to track this: jupyterhub/jupyterhub#2973

Spawner/authenticate lifetime is also a wrinkle, because login != spawn and stop != logout:

• Restarting the JupyterHub process doesn't stop servers and shouldn't generally invalidate auth. That means that a persistent handle should either live outside the JupyterHub process or be able to be serialized/deserialized.
• Authenticate and Spawner.start are often not called in immediate sequence (separated by hours/days), and not against the same JupyterHub process
• Authenticate may be called many times without spawning (e.g. once per browser session)
• Spawner start/stop may be called many times per authenticate (e.g. restarting servers)
• Spawn may occur via API without a chance to input authenticate info
• Multiple servers may be running at once for the same user (authenticate, start A, start B, stop A, stop B)
• Spawn can be triggered by other users (e.g. admins), so Authenticate can technically be called zero times prior to spawn for a given user (it's okay to have restrictions where that's not okay, but they shouldn't be default)
• Technically, the Spawner might not be LocalProcessSpawner, it could be PAMAuthenticator + DockerSpawner, so putting actions on the Spawner (as opposed to pre_spawn hooks on the Authenticator) is a possible source of problems

This does all point to turning off open_sessions by default (I've been meaning to for a long time, and should have done it in 2.0) because it's only a subset of cases where it does work: jupyterhub/jupyterhub#3787

I think it is clear that PAM session calls should not be made from the JupyterHub process. And session should receive an authenticate handle, which cannot be serialized, so authenticate must also be called from another process (at least when using sessions), and due to possible rlimit calls and friends, the session calls must be made in the parent of the server (it must not be the server process itself).

So I think it should work something like this:

• if using sessions:
  • authenticate spawns a persistent process with a lifetime tied to the hub process, possibly with a configurable idle-timeout of minutes/hours.
  • set refresh_pre_spawn=True
  • refresh_user checks for the PAM subprocess for the user, returns False if not, forcing a new login prior to spawn
  • subprocess calls authenticate and holds onto the handle
  • Provide a hook in LocalProcessSpawner.start so that it gets the args for Popen and can be passed to the appropriate PAM child instead of calling Popen directly
  • Once a PAM subprocess has been used for a session, detach it from the hub so that it exits when the child does rather than the parent
  • Admins cannot launch spawners on behalf of users when sessions are in use; that, or when they launch servers, do so without launching a session
• else: just call authenticate, like we are already doing

So the PAM child is roughly:

def pam_child(username, password):
    pam_h = pamela.authenticate(username, password, close=False)
    # propagate error if fails
    wait_for(parent exit _or_ popen input)
    if parent_exited:
        return
    pam_h.open_session()
    with Popen(popen_args) as child:
        send_to_parent(child.pid)
    pam_h.close_session()
    pamela.pam_end(pam_h) # there should be a pam_h.end(), but there isn't yet

This doesn't satisfy the first condition - that restarting the hub shouldn't force re-login, but it greatly simplifies the lifecycle because we don't have to re-establish connections to the PAM children on Hub restart. It also means that Spawning via the API in general won't work if sessions are in use. But the only way I see that possibly working is by storing passwords persistently, which I think we really shouldn't do.

Running this test script:

test script

import getpass
import os
import time
import pamela

def parent(level="session"):
    username = getpass.getuser()
    password = getpass.getpass()

    # once: reuse pam handle across subprocesses
    if level == "once":
        pam_h = pamela.authenticate(username, password, close=False)

    children = []
    for i in range(3):
        print("starting", i)
        session_pid = os.fork()
        if session_pid == 0:
            if level == "session":
                pam_h = pamela.authenticate(username, password, close=False)
            # child for each session
            for j in range(3):
                print("opning", i, j, os.getpid())
                pam_h.open_session()
                print("opened", i, j, os.getpid())
                child_pid = os.fork()
                if child_pid == 0:
                    print("worker", i, j, os.getpid())
                    time.sleep(5)
                    print("worked", i, j, os.getpid())
                    return
                else:
                    os.waitpid(child_pid, 0)
                print("closing", i, j, child_pid)
                pam_h.close_session()
                print("closed", i, j, child_pid)
            if level == "session":
                pamela.pam_end(pam_h, 0)
            return
        else:
            print("started", session_pid)
            children.append(session_pid)
            time.sleep(1)

    for session_pid in children:
        print(session_pid, os.getpid())
        os.waitpid(session_pid, 0)

    if level == "once":
        pamela.pam_end(pam_h, 0)

if __name__ == "__main__":
    parent()

suggests the following information: that pam authenticate and session open/close must be called in the same process, which in turn suggests that a new authenticate call must be made for each session/spawn to produce a new handle (reopening a session after closing it may work, as it does in my test, but I don't know if some implementations may disallow that).

[jeflem]

Thank you @minrk for developing a possible path to fixing this. Sounds good, although I do not understand all the details concerning processes and their communication (always writing single process apps with less than 100 lines of code...).

Would be great to have PAM sessions in JupyterHub some day. For the time being I stick to telling my users "login via ssh if you want to see your Windows shares in Jupyter". Worked for two years and will work another year.

Many thanks for all the effort put into Jupyter and friends!

[steffen-moser]

I've just run into the pam_mount problem with JupyterHub. Do I get it right that the best workaround to make user-individual authenticated DAVFS or CIFS mounts work with JupyterHub is the pampylho authenticator mentioned by @dsoares in this comment? jupyterhub/jupyterhub#810 (comment)

[jeflem]

Not using pam_mount for my JupyterHubs anymore. I don't think that jupyterhub/jupyterhub#810 (comment) really works in a stable way due to a bug in pam_mount, which does not allow for multiple parallel PAM sessions within one process (JHub).

Have a look at jupyter-fs, a JLab extension for connecting to DAV and many other local or remote file systems. Have this running on my hubs and users configure their remote file systems individually (mostly Nextclouds).