Use-Case: s3proxy as fullpage cache with query-string support

[christoph-kluge]

I am implementing this handler right now as full-page-cache to delvier static content. (I'm running a website-as-a-service product)

While this is fast and nice for any URL w/o params. This does not work with querystring-parameters at all.
This is because S3 does not really support querystring parameters. If you want this, you need to convert the querystring to be urlencoded.

In my application I've simply used this pattern for reading and writing s3 files.

Example Host: customer1.website.local

• / will create a s3://my-bucket/customer1.website.local/index.html file
• /new-page will create a s3://bucket/customer1.website.local/new-page file
• /new-page?a=b&c=d will create a s3://bucket/customer1.website.local/new-page/sha1('a=b&c=d') file, where sha1 is the sha1 of the query-string

Since this is not an regular use-case I would like to know if this makes sense to be implemented properly or if I should keep my hardcoded-example just in my fork. As of now I'm thinking of multiple configurable scenarios:

Enable with sane default settings

s3 {
    allow_query_string
}

Example: /new-page
Key-Result: s3://my-bucket/new-page

Example#2: /new-page?a=b
Key-Result: s3://my-bucket/new-page%3Fa%3Db

Enable with query-string as directory

s3 {
    allow_query_string {
        as_directory
    }
}

Example: /new-page
Result: s3://my-bucket/new-page

Example#2: /new-page?a=b
Result: s3://my-bucket/new-page/a%3Db

Enable with query-string as directory and convert it to sha1

s3 {
    allow_query_string {
        as_directory
        as_sha1
    }
}

Example: /new-page
Result: s3://my-bucket/new-page

Example#2: /new-page?a=b
Result: s3://my-bucket/new-page/ccff2fee4b15e0b46f79f86ce5d1de59163bb483

[rayjlinden]

In caddy manipulating the path is more the domain of directives that just do that. There is the rewrite and uri directive in particular. With the rewrite directive you can "almost" do what you want with the use of place holders:

rewrite * /{host/{path}/{query}

But in the above the query would not be in sha1. (In fact the above would be a broken url because the ? would be missing.))

In any case, I don't think this kind of thing should be in the s3proxy. Worse case, maybe you create a http module that does your hack for you. It would be cleaner than maintaining a fork at least...

It would be cool if Caddy let you register functions that could be used as place holders. (Kind of like go template functions.). You might bring this up on the Caddy forums. I'd think Matt or Francislovole might have some good ideas.