Internal Server Error 500 for user without experiment permissions instead of hiding the experiment

[mshodge]

Hi there,

Great package, thanks! Please can someone explain why the expected behaviour is not happening below.

Situation

A new user signs up, and hasn't been given access to any experiments yet. They then log into the Mlflow UI.

Expected behaviour

They only see the Default experiment and no others.

Actual behaviour

They see other experiments but get a 500 Internal Server Error when clicking them (see screenshot below).

In the original MLflow basic auth setup the user would not see an experiment they have no permissions to.

[kharkevich]

Hello @mshodge,
Could you please provide network logs from the browser (network tab in debugger) and mlflow + plugin logs as well.
Unfortunately, is really hard to understand what's wrong by single picture

[mshodge]

Hi there.

Below is a screen capture of the error. I have implemented the new code now with Entra ID (I had a few other issues running that but resolved them) but the same thing happens once a user is logged in.

Screen.Recording.2024-12-11.at.13.16.50.mov

Here is the trackback from Python:

Traceback (most recent call last):
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/sqlalchemy_store.py", line 137, in _get_experiment_permission
    session.query(SqlExperimentPermission)
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2778, in one
    return self._iter().one()  # type: ignore
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1810, in one
    return self._only_one_row(
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 752, in _only_one_row
    raise exc.NoResultFound(
sqlalchemy.exc.NoResultFound: No row was found when one was required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/_permissions.py", line 18, in get_permission_from_store_or_default
    perm = store_permission_user_func()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/experiment.py", line 20, in <lambda>
    lambda: store.get_experiment_permission(experiment_id, username).permission,
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/sqlalchemy_store.py", line 176, in get_experiment_permission
    return self._get_experiment_permission(session, experiment_id, username).to_mlflow_entity()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/sqlalchemy_store.py", line 145, in _get_experiment_permission
    raise MlflowException(
mlflow.exceptions.MlflowException: Experiment permission with experiment_id=642901579502662379 and [email protected] not found

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/mlflow/store/db/utils.py", line 147, in make_managed_session
    yield session
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/sqlalchemy_store.py", line 190, in get_user_groups_experiment_permission
    return user_perms.to_mlflow_entity()
UnboundLocalError: local variable 'user_perms' referenced before assignment

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/flask/app.py", line 915, in full_dispatch_request
    rv = self.preprocess_request()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/flask/app.py", line 1291, in preprocess_request
    rv = self.ensure_sync(before_func)()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/hooks/before_request.py", line 208, in before_request_hook
    if not validator():
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/experiment.py", line 62, in validate_can_read_experiment
    return _get_permission_from_experiment_id().can_read
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/experiment.py", line 19, in _get_permission_from_experiment_id
    return get_permission_from_store_or_default(
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/_permissions.py", line 23, in get_permission_from_store_or_default
    perm = store_permission_group_func()
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/validators/experiment.py", line 21, in <lambda>
    lambda: store.get_user_groups_experiment_permission(experiment_id, username).permission,
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/mlflow_oidc_auth/sqlalchemy_store.py", line 192, in get_user_groups_experiment_permission
    raise MlflowException(
  File "/Users/michael.hodge/.pyenv/versions/3.9.16/lib/python3.9/contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/Users/michael.hodge/GitProjects/mlflow-oidc-auth-github/venv/lib/python3.9/site-packages/mlflow/store/db/utils.py", line 164, in make_managed_session
    raise MlflowException(message=e, error_code=INTERNAL_ERROR)
mlflow.exceptions.MlflowException: local variable 'user_perms' referenced before assignment

I have a .har file of the network logs but would prefer to not upload them here.

[mshodge]

So I think it is to do with this line:

mlflow-oidc-auth/mlflow_oidc_auth/hooks/after_request.py

Line 76 in 777de16

if not can_read.get(e.experiment_id, default_can_read):

I'm not completely understanding the logic, but it does not seem to be working as expected.

[mshodge]

For me if not can_read.get(e.experiment_id): did what I expected it to, that is, it filtered all experiments the user was not explicitly given permissions for