-
-
Notifications
You must be signed in to change notification settings - Fork 409
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use Case: fetching many files from s3 and generating zip #291
Comments
UPDATE: I was able to make the download and zip of many files (say: 3000 100kb files) in parallel not using await for each fetch. initially, it caused the ERR::INSUFFICIENT..., and then I made it so each promise returned by the fetch was put in an array of promises. So, for every 1 MB of data downloaded, I awaited (await Promise.all(allPromises)). It made the trick. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
I'm building a solution that requires broad compatibility.
Got a few questions.
@jimmywarting
I was able to test downloading many files and getting them as a zip. This saves a lot of processing power/server costs, since there's nothing to do with the files, just give to the user as a zip.
I was also able to plug this into a react app.
I have these questions, base on comments seen from @jimmywarting :
native-file-system-adapter
that pertain to fetching remote files.I know that at some point this will not the way of getting to this solution, but I must aim for maximum compatibility
Thanks!
The text was updated successfully, but these errors were encountered: