Upload ODK submission user photos to S3 for easy access #1744
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sujanadh
added
enhancement
for more information, see https://pre-commit.ci
for more information, see https://pre-commit.ci
… feat/upload-image-to-s3
spwoodcock
changed the title
spwoodcock
requested changes
Aug 13, 2024
| for idx, filename in enumerate(attachments): | ||
| s3_key = f"{s3_base_path}/{instance_id}/{idx+1}.jpeg" | ||
|
|
||
| if object_exists(s3_bucket, s3_key): |
There was a problem hiding this comment.
This will make a lot of calls to S3 potentially!
Not a problem with Minio, but slow if external S3.
Could we do a comparison for what the submission says should exist against the database table instead?
We would need the stat approach if we didn't gave the db table 👍
|
|
||
| # Insert the record into submission_photos table | ||
| sql = text(""" | ||
| INSERT INTO submission_photos ( |
There was a problem hiding this comment.
Also this could be batched for efficiency.
It's generally good to not make db queries in a for loop - instead gather the data (images uploaded) there in a dict or something, then do a batch db table insert in one go 😄
| -- Start a transaction | ||
| BEGIN; | ||
|
|
||
| CREATE TABLE submission_photos ( |
There was a problem hiding this comment.
Check this SQL doesn't error if you run it twice 👍
Maybe create table if not exists will fix
|
for more information, see https://pre-commit.ci
spwoodcock
approved these changes
Aug 14, 2024
|
Looks good! Merging 💫 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What type of PR is this? (check all applicable)
Related Issue
- Extract user photos from submissions and make available via S3 #1701
- Add a database record of photos taken in the field #1729
Describe this PR
This PR creates a function to upload submission images to s3
submission_photostable with fieldsproject_id,task_id,submission_id, ands3_pathScreenshots
N/A
Alternative Approaches Considered
Did you attempt any other approaches that are not documented in code?
Review Guide
Checklist before requesting a review
[optional] What gif best describes this PR or how it makes you feel?