Open OnDemand UI Issue: Newly Mounted AWS S3 Bucket Not Visible in Files Section

[mharini]

Hi,

We are currently encountering an issue with OnDemand when attempting to view a mounted S3 bucket in the UI. A new bucket was created with the appropriate permissions and added to the user groups. We run a cron job to mount/unmount the S3 buckets in periodic basis, based on user group and permissions. The bucket appears in the mounted path, and we can list the files from the cluster shell. However, they are not accessible from the OnDemand UI. While restarting the OnDemand service resolved the issue, we are seeking a solution that doesn't require a restart.

Any inputs would be great to resolve this issue.

[johrstrom]

While restarting the OnDemand service resolved the issue, we are seeking a solution that doesn't require a restart.

I'd guess it's an issue of when and how it's mounted. If it's mounted as a filesystem just like any other NFS drive that's been mounted I can only think there's a race condition of some kind. Like we're trying to access it when it's not available yet.

How and when is it mounted?

[mharini]

The S3 buckets are mounted using s3fs using a cron job running on a periodic basis, to identify if any new buckets or permissions or groups are added/modified.

We also see that "Restart Web Server" in the help option fixes the issue without restarting the all of the ondemand.

[mharini]

We have modified the cron job to kill the nginx process for a specific user using pid whenever there is a change in group or user.

We also identified comments in the existing code base that the web server should be restarted automatically when there is a change in the group or user. Any additional configurations to be done for this to happen?

[johrstrom]

Any additional configurations to be done for this to happen?

I don't think so. IIRC the app checks to see if any groups have changed then provides a button for the user to restart? But that's just from memory.