Design Doc: Visualize private S3 assets with Neuroglancer #111
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
✅ Deploy Preview for lincbrain-org ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
aaronkanzer
changed the title
|
Thank you, Aaron. |
kabilar
reviewed
Feb 29, 2024
kabilar
reviewed
Feb 29, 2024
kabilar
reviewed
Feb 29, 2024
kabilar
reviewed
Feb 29, 2024
kabilar
reviewed
Mar 6, 2024
kabilar
reviewed
Mar 9, 2024
2 tasks
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
doc/design/linc_permissions.md
Outdated
Comment on lines
7
to
10
| • As a CONNECTS developer, I shall be able to view multiple datasets/layers (e.g. dMRI, tractography, HiP-CT) in the same neuroglancer | ||
| • As a CONNECTS developer, I shall be able to view data locally or on neuroglancer | ||
| • As a CONNECTS developer, I shall be able to have direct access to S3 asset URI | ||
| • As a LINC Archive user, I shall be able to render neuroglancer layers view similar to DANDI |
There was a problem hiding this comment.
Suggested change
| • As a CONNECTS developer, I shall be able to view multiple datasets/layers (e.g. dMRI, tractography, HiP-CT) in the same neuroglancer | |
| • As a CONNECTS developer, I shall be able to view data locally or on neuroglancer | |
| • As a CONNECTS developer, I shall be able to have direct access to S3 asset URI | |
| • As a LINC Archive user, I shall be able to render neuroglancer layers view similar to DANDI | |
| ### Use case 2 | |
| Set a single Neuroglancer viewer programmatically to visualize multiple assets (e.g. dMRI, HiP-CT) as layers. | |
| ### Use case 3 | |
| Use case 2 + Load the tractography-generated streamlines (i.e. `trk` file) into memory, convert the streamlines into the Neuroglancer skeleton format, and visualize in the Neuroglancer viewer as an additional layer. |
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
doc/design/linc_permissions.md
Outdated
|
|
||
| Rendering via Neuroglancer | ||
|
|
||
| - (Might need to modify fork of neuroglancer to handle CloudFront asset path in same format that it handles S3 asset path) |
There was a problem hiding this comment.
Suggested change
| - (Might need to modify fork of neuroglancer to handle CloudFront asset path in same format that it handles S3 asset path) | |
| - Modify the Neuroglancer source code to handle a CloudFront asset path (i.e. `neuroglancer.lincbrain.org` since the domains of the presigned cookie are limited to `*.lincbrain.org` domains due to same-site requirements for cookies) that is similar to any S3 asset path (i.e. `s3://`). |
There was a problem hiding this comment.
Just for clarification, there is no "traditional" backend yet, as the code used is just static JS, HTML, CSS. If we use something like AWS Lambda and/or API Gateway, then we'd technically have a "backend"
Perhaps this could read:
Modify Neuroglancer source code to handle CloudFront asset path (i.e. `neuroglancer.lincbrain.org` sincethe domain of the presigned cookie are limited to `*.lincbrain.org` domains due to same-site requirements for cookies) in same format that it handles the asset similar to any S3 asset path (i.e. `s3://`).
There was a problem hiding this comment.
Thanks Aaron. I have updated my suggestion based on your feedback.
kabilar
reviewed
Mar 14, 2024
doc/design/linc_permissions.md
Outdated
| ###### CloudFront distribution with Origin Access Identity control for relevant S3 buckets | ||
|
|
||
| - S3 Asset Bucket is served via a CloudFront distribution | ||
| - User hits a get_presigned_cookie/ API endpoint in LINC Archive. In the endpoint response is a valid cookie that allows the user's browser to be able to access the asset via their browser |
There was a problem hiding this comment.
Suggested change
| - User hits a get_presigned_cookie/ API endpoint in LINC Archive. In the endpoint response is a valid cookie that allows the user's browser to be able to access the asset via their browser | |
| - User navigates to LINC Archive (i.e. `lincbrain.org`) to authenticate. Thereby the user hits a get_presigned_cookie/ API endpoint. In the endpoint response is a valid cookie that allows the user's browser to be able to access the asset via their browser. |
kabilar
reviewed
Mar 14, 2024
kabilar
reviewed
Mar 14, 2024
doc/design/linc_permissions.md
Outdated
Comment on lines
32
to
34
|
|
||
| Rendering via Neuroglancer | ||
|
|
There was a problem hiding this comment.
Suggested change
| Rendering via Neuroglancer |
kabilar
reviewed
Mar 14, 2024
doc/design/linc_permissions.md
Outdated
Comment on lines
37
to
40
| Accessing via LINC Archive API | ||
|
|
||
| - Cookies would get passed in request to get asset | ||
| - Provide LINC CLI option for retrieving asset. Instead of just referencing the asset directly, we would need to provide a helper function that wraps the request with the cookie |
There was a problem hiding this comment.
Suggested change
| Accessing via LINC Archive API | |
| - Cookies would get passed in request to get asset | |
| - Provide LINC CLI option for retrieving asset. Instead of just referencing the asset directly, we would need to provide a helper function that wraps the request with the cookie | |
| Render with local Neuroglancer | |
| - Access assets with LINC CLI | |
| - Cookies would get passed in request to get asset | |
| - Provide LINC CLI option for retrieving asset. Instead of just referencing the asset directly, we would need to provide a helper function that wraps the request with the cookie. | |
| - Install modified Neuroglancer described above, with additional updates to the Python API. |
aaronkanzer
changed the title
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
kabilar
reviewed
Apr 10, 2024
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
Co-authored-by: Kabilar Gunalan <[email protected]>
|
@kabilar thanks for the updates here -- I committed each of the newer ones, as they all seemed to make sense for me. |
kabilar
reviewed
Apr 10, 2024
doc/design/linc_permissions.md
Outdated
Comment on lines
54
to
62
| ```mermaid | ||
| %%{init: {"flowchart": {"curve": "linear"}}}%% | ||
| flowchart LR | ||
| E --> B | ||
| A(User) -->| If client has CloudFront cookies from prior session, <br/> then proceed. | B(Static Webpage i.e. Neuroglancer) | ||
| A --> | If client does not have CloudFront cookies, <br/> then GET upon /api/permissions/s3 in LINC Archive API. | E(LINC Archive API) | ||
| B --> | Upon user activity, <br/> sends presigned cookies | C(AWS CloudFront) | ||
| C -->| Allows data to be fetched | D(Private AWS S3 Bucket) | ||
| D -->| 1. Neuroglancer able to access S3 data <br/> 2. Data rendered on screen | B |
There was a problem hiding this comment.
Please add a closing to the code block (i.e. ```). GitHub suggestions does not know how to handle this suggestion.
kabilar
reviewed
Apr 10, 2024
doc/design/linc_permissions.md
Outdated
| • As a CONNECTS developer, I shall be able to have direct access to S3 asset URI | ||
| • As a LINC Archive user, I shall be able to render neuroglancer layers view similar to DANDI | ||
| 1. Zarr archives are stored in a private AWS S3 bucket | ||
| 2. Provide direct access to the LINC users of the assets on the private S3 bucket using a URI. | ||
|
|
||
| #### Relevant Links |
There was a problem hiding this comment.
Suggested change
| #### Relevant Links |
kabilar
reviewed
Apr 10, 2024
doc/design/linc_permissions.md
Outdated
| @@ -17,27 +32,48 @@ https://github.com/neuroscales/ngtools/blob/main/notebooks/show_tract.ipynb | |||
|
|
|||
| https://github.com/google/neuroglancer/issues/507 | |||
There was a problem hiding this comment.
Suggested change
| https://github.com/google/neuroglancer/issues/507 |
kabilar
reviewed
Apr 26, 2024
Co-authored-by: Kabilar Gunalan <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Relates to https://github.com/orgs/lincbrain/projects/2?pane=issue&itemId=54650571
Cc @balbasty @ayendiki -- I will keep you in-the-loop here, but this work should encapsulate the ability to use the private S3 assets as you wish for the LINC project in your analyses
Cc @MikeSchutzman -- this work might be relevant to some of our annotation/rendering tools such as
neuroglancer-- more to come here (you can just use DANDI public S3 assets for now)Cc @kabilar -- tagging for visibility