POST modules api with Content not supported

[vincentgna]

Just to clarify if it's currently possible to send Content binary stream to v1/api/modules/:namespace/:name/:provider/:version/upload:

Seems not, because:

StorageInput supports Content:

https://github.com/valentindeaconu/terralist/blob/661f9ab0830c48f73cf650c275ff323387b03716/pkg/storage/resolver.go#L4-L12

but Controller does not support providing module binary data:

https://github.com/valentindeaconu/terralist/blob/661f9ab0830c48f73cf650c275ff323387b03716/internal/server/controllers/module.go#L100-L123

because the Module repository only supports Uploads based on location
https://github.com/valentindeaconu/terralist/blob/661f9ab0830c48f73cf650c275ff323387b03716/internal/server/repositories/module.go#L143-L153

[vincentgna]

seems in my PoC using this dev version, I have 2 options given download_url is parsed by go-getter, so the url protocols can be:

1. git (with ref/deployKey) or
2. s3 (assuming go-getter s3 implementation will use sdk default credential chain, ec2 iam profile will be used to fetch these files)

currently the storage call is hardcoded to archive the temp dir once the files are downloaded, so I guess I shouldn't archive the files?

I'm able to use the API Key after manually inserting it into the Database, so my CI will curl POST the git url with a readonly deploy ssh key for terralist to get the version of the module and package it up into it's own s3 bucket

[valentindeaconu]

An endpoint in which a user can upload modules using the content option will definitely be an option in the future. Thanks for opening this tracking issue.

currently the storage call is hardcoded to archive the temp dir once the files are downloaded, so I guess I shouldn't archive the files?

go-getter package is set to Any mode, which tries to bring downloaded files to a uniform state (a directory containing the source files), which will be later archived and uploaded to S3. In my testings, I was using archived modules, but in theory, you should be able to upload directories and they will be archived by the zipper package.

[canidam]

If I understand correctly, this ticket is about adding a POST endpoint for modules such as :namespace/:name/:system/:version/upload that supports binary data, and the ModuleService will save that data to the storage using a ModuleRepository and serve it. Am I getting this right?

[valentindeaconu]

If I understand correctly, this ticket is about adding a POST endpoint for modules such as :namespace/:name/:system/:version/upload that supports binary data, and the ModuleService will save that data to the storage using a ModuleRepository and serve it. Am I getting this right?

That's somewhat true. The current implementation accepts from the endpoint an URL and then the service will download the files from that particular URL and then uploads them to the storage. The ModuleRepository only saves the storage key into the database and that can be used later to serve the module archive (pass the key to the S3 resolver to generate a pre-signed URL). The difference between the current implementation and the one discussed in this topic is that this one should skip the download part and continue from uploading it to the storage.

I had a draft of this implementation some time ago, but I dropped it because I was facing some problems with the signatures and checksums in the case of providers, but I think those problems should be fixed now since I changed the go-getter implementation in #40.