Proposed API changes

[emanuele6]

Remove -s 'url=...'

It is just an alternate version of --redirect ... that does not respect --accept-space and --no-guess-url, I propose removing it.

Rework --json output

It is a bit annoying to work with --json now that there are meta properties like params, raw_port, and others that may be added in the future at the same level of the components.

E.g. if, in jq, I want to check if two URLs are semantically equivalent except for one component (e.g. host), I have to do something like:

[ $url1, $url2 | del(.host, .raw_port, .url) ] | .[0] == .[1]
# del(..., .raw_port, .url), and I have to rememeber to add new meta
#  props to the list if they are added after an update
# I always have to delete .url and .raw_port, and if I want to ignore
#  .query I also have to delete .params.

Or:

[ $url1, $url2 | { scheme, user, password, options, port, path, query, fragment, zoneid } ] | .[0] == .[1]
# I have to list all the components except the one I want to ignore

I think --json would be much more usable and easier to handle if the parts were in a separate object like so (note, "url" is NOT a part):

{
    "url": "...",
    "parts": {
        ...
    },
    "params": [
        ...
    ],
    ... raw_port and other meta stuff
}

I mostly want to use trurl --json to extract URL parts, and get the normalised URL in "url".

If it was structured that way, I could just use:

[ $url1, $url2 | .parts | del(.host) ] | .[0] == .[1]

If I wanted to filter out URLs that are not http:// or https:// and remove URLs that are equivalent ignoring the scheme, and keep https:// over http:// (#200 (comment)), I can use:

trurl --json -f - |
jq -r '
    group_by(.parts | del(.scheme))[] |
    first(
        ("https", "http") as $s |
        .[] | select(.parts.scheme == $s).url
    )
'

And, if I wanted to write a jq script that modifies one of the parts, and outputs a shell command that runs trurl with some --set options, and outputs the modified URL, I could simply iterate the parts object without listing all the properties, checking if the key is present, etc:

trurl --json -f - |
jq -r '
    .[].parts |
    # example modification: upcase "user" and "password" if they are present
    (.user, .password | values) |= ascii_upcase |
    [ "trurl", (to_entries[] | "-s", "\(.key)=\(.value)") ] |
    @sh
'

Or to process URLs as they come in a pipeline:

get-urls |
trurl --json -f - |
jq --unbuffered --stream -nr '
    fromstream(1 | truncate_stream(inputs)).parts |
    (.user, .password | values) |= ascii_upcase |
    [ "trurl", (to_entries[] | "-s", "\(.key)=\(.value)") ] |
    @sh
' |
sh

URL in default output, and --get and --json

When using trurl, I often want it to behave as if I passed -g '{url}' or -g '{puny:url}' so i get the default port for known schemes printed, and punyfied host names.

But, if I use --json, I can't access the punyfied version of the URL or the URL with explicit default port.

Not an extremely big deal, at least in a shell script that uses jq, since I can just pipe the URLs outputted by jq into another trurl process like so, but it would surely be annoying in other languages:

trurl -f - --json | jq ... | trurl -f - -g '{puny:url}'

Also, there is no way to get the punyfied URL without the default port. Except using hacks like this one:

trurl -s 'scheme=x' -g '{puny:url}' ... |
sed -n 's@^x://@http://@p'
# or, if working with multiple schemes
trurl -g '{scheme}:{url}:' ... |
while IFS=: read -r scheme url; do
    pu=$(trurl --verify -s 'scheme=x' -g '{puny:url}' -- "$url") &&
    case $pu in x://*) printf %s\\n "$scheme://${pu#x://}"; esac
done

And there is no way to get the URL without default port at all in -g.

I propose to add two new options, one that makes the default output print the hostname in punycode (--punycode), and one that makes the default output include default ports --default-port. You may specify both of them to make trurl output punyfied URLs with explict default ports.

The "url" field in --json will reflect the default output.

Remove {raw:}, and replace it with {default:}.

{puny:} and {default:} may be specified for any component in any order (e.g., {puny:default:host}, {default:puny:query}, {:puny:default:url}) as generic modfiers like {:component}, naturally they will not change anything for some components.

{url} expands to the default output, unless any modifier is specified. In which case, curl_url_get() will be called with the bitwise of: CURLU_PUNYCODE if {puny: is specified, CURLU_URLENCODE if {: is specified, CURLU_DEFAULT_PORT if {default: is specified.

This will incidentally also fix the behaviour of {raw:port} that is currently different from the raw_port present in the --json output.
{raw:port} and {port} currently make trurl call curl_get_url() with a combination of flags that yields the following results:

$ trurl 'http://localhost/hello'    -g '{port}'
80
$ trurl 'http://localhost:80/hello' -g '{port}'

$ trurl 'http://localhost/hello'    -g '{raw:port}'

$ trurl 'http://localhost:80/hello' -g '{raw:port}'

vs

$ trurl --json 'http://localhost/hello'    | jq -c '.[]|{port,raw_port}'
{"port":"80","raw_port":""}
$ trurl --json 'http://localhost:80/hello' | jq -c '.[]|{port,raw_port}'
{"port":"80","raw_port":"80"}

---

Thoughts?

[jacobmealey]

This all looks good, I think you point out a lot of operations that don't have good ergonomics so they require some strange tricks within a calling script. I think splitting the JSON stuff across multiple objects makes sense and would allow for easier-to-read JSON (as well as making scripting easier).

The only part I'm a bit confused about is:

{puny:} and {default:} may be specified for any component in any order (e.g., {puny:default:host}, {default:puny:query}, {:puny:default:url}) as generic modifiers like {:component}, naturally they will not change anything for some components.

Can you provide an example output of what these would be like?

[emanuele6]

{puny:} and {default:} may be specified for any component in any order (e.g., {puny:default:host}, {default:puny:query}, {:puny:default:url}) as generic modifiers like {:component}, naturally they will not change anything for some components.

Can you provide an example output of what these would be like?

They just specify the flags to be used by -g when calling curl_url_get() to fetch that component:

$ trurl -g '{url}\n{puny:url}\n{default:url}\n{puny:default:url}' -a 'path=a=b' pè.example.org
http://pè.example.org/a%3db
http://xn--p-8fa.example.org/a%3db
http://pè.example.org:80/a%3db
http://xn--p-8fa.example.org:80/a%3db

I omitted {:url}, {:puny:url}, {:default:url}, and {:puny:default:url} since {: has no effect on a URL.

Naturally, some combinations of modifiers and components will be useless, e.g. {puny:port}, {default:fragment}, just like {: for some componets is useless now {:url}, {:port}, etc. trurl will still pass the specified flags to curl_url_get() so the user has control.

If --punycode is specified, {url} is equivalent to {puny:url}; If --default-port is specified, {url} is equivalent to {default:url}; If both are specified it is equivalent to {puny:default:url}; otherwise if neither are specified, it is just the plain URL without explicit default port, and with the hostname not punyfied.

{url} is currently equivalent to the proposed {default:url} (note: currently -g '{url}' is not the same as default output).
{puny:url} is currently equivalent to the proposed {puny:default:url}.

[jacobmealey]

Okay, that's what I was thinking you meant I just wanted to make sure I was on the same page. I think this all makes good sense.

[bagder]

(You should probably have submitted these as separate ideas to better facilitate discussions on each separate idea)

Remove -s 'url=...'

Let's get rid of that. It was never a good idea and basically just happened because the set code uses the same list of components.

Rework --json output

Very sensible reasoning. The existing output format was just organically created and mostly by me, and I do not have a lot of experience with working with JSON data in scripts etc so likely to do newbie mistakes like this.

URL in default output

I think I'm with you on this one as well. It seems to lead to better consistency and that's something I like.