trurl.1

.\" You can view this file with:
.\" man -l trurl.1
.\" Written by Daniel Stenberg
.\"
.TH trurl 1 "April 27, 2023" "trurl" "trurl Manual"
.SH NAME
trurl \- transpose URLs
.SH SYNOPSIS
.B trurl [options]
.SH DESCRIPTION
.B trurl
parses, manipulates and outputs URLs and parts of URLs.

It uses the RFC 3986 definition of URLs and it uses libcurl's URL parser to do
so, which includes a few "extensions". The URL support is limited to
"hierarchical" URLs, the ones that use "://" separators after the scheme.

Typically you pass in one or more URLs and decide what of that you want
output. Posssibly modifying the URL as well.

trurl knows URLs and every URL consists of up to ten separate and independent
"components". These components can be extracted, removed and updated with
trurl and they are referred to by their respective names: scheme, user,
password, options, host, port, path, query, fragment and zoneid.
.SH "OPTIONS"
.IP "-a, --append [component]=[data]"
Append data to a component. This can only append data to the path and the
query components.

For path, this URL encodes and appends the new segment to the path, separated
with a slash.

For query, this URL encodes and appends the new segment to the query,
separated with an ampersand (&). If the appended segment contains an equal
sign ('=') that one will be kept verbatim and both sides of the first
occurrence will be URL encoded separately.
.IP "--accept-space"
When set, trurl will try to accept spaces as part of the URL and instead URL
encode such occurrences accordingly.

According to RFC 3986, a space cannot legally be part of a URL. This option
provides a best-effort to convert the provided string into a valid URL.
.IP "-f, --url-file [file name]"
Read URLs to work on from the given file. Use the file name "-" (a single
minus) to tell trurl to read the URLs from stdin.

Each line needs to be a single valid URL - but trurl will trim off any
trailing spaces and tabs from the end of the line. The maximum URL length
supported in a file like this is 4095 bytes.
.IP "-g, --get [format]"
Output text and URL data according to the provided format string. Components
from the URL can be output when specified as \fB{component}\fP or
\fB[component]\fP, with the name of the part show within curly braces or
brackets. You can not mix braces and brackets for this purpose in the same
command line.

The following component names are available (case sensitive): url, scheme,
user, password, options, host, port, path, query, fragment and zoneid.

Components are shown URL decoded by default. If you instead write the
component prefixed with a colon like "{:path}", it gets output URL encoded.

Hosts provided as IPv6 numerical addresses will be provided within square
brackets. Like "[fe80::20c:29ff:fe9c:409b]".

Hosts provided as IPv4 numerical addresses will be "normalized" and provided
as four dot-separated decimal numbers when output.

You can access specific keys in the query string using the format
\fB{query:key}\fP. Then the value of the first matching key will be output
using a case sensitive match. When extracting a URL decoded query key that
contains %00, such octet will be replaced with a single period '.' in the
output.

You can access specific keys in the query string and out all values using the
format \fB{query-all:key}\fP. This looks for 'key' case sensitively and will
output all values for that key space-separated.

You can access the url and host components in their "punycoded" version, which
is how International Domain Names are converted into plain ascii, by using the
form \fB{puny:url}\fP and \fB{puny:host}\fP. If the host name is not using
IDN, this option provides the regular ascii name.

You can determine if a port is explicitly defined in a url by using the "raw" 
keyword in your format string, which would look like \fB{raw:port}\fP. If the 
port is explicitly defined trurl will return the port number, if it is not
explicitly defined then trurl will return an empty string.

The "format" string supports the following backslash sequences:

\&\\\\ - backslash

\&\\t - tab

\&\\n - newline

\&\\r - carriage return

\&\\{ - an open curly brace that does not start a variable

\&\\[ - an open bracket that does not start a variable

All other text in the format string will be shown as-is.
.IP "-h, --help"
Show the help output.
.IP "--iterate [component]=[item1 item2 ...]"
Set the component to multiple values and output the result once for each
iteration. Several combined iterations are allowed to generate combinations,
but only one \fI--iterate\fP option per component. The listed items to iterate
over should be separated by single spaces.
.IP "--json"
Outputs all set components of the URLs as JSON objects. All components of the
URL that has data will get populated in the object using their component
names. See below for details on the format.
.IP "--query-separator [what]"
Specify the single letter used for separating query pairs. The default is "&"
but at least in the past sometimes semicolons ";" or even colons ":" have been
used for this purpose. If your URL uses something other than the default
letter, setting the right one makes sure trurl can do its query operations
properly.
.IP "--redirect [URL]"
Redirect the URL to this new location. It requires that you set the base url
with \fB--url\fP
.IP "-s, --set [component][:]=[data]"
Set this URL component. Setting blank string ("") will clear the component
from the URL.

The following components can be set: url, scheme, user, password,
options, host, port, path, query, fragment and zoneid.

If a simple "="-assignment is used, the data is URL encoded when applied. If
":=" is used, the data is assumed to already be URL encoded and will be stored
as-is.
.IP "--sort-query"
The "variable=content" tuplets in the query component are sorted in a case
insensitive alphabetical order. This helps making URLs identical that
otherwise only had their query pairs in different orders.
.IP "--url [URL]"
Set the input URL to work with. The URL may be provided without a scheme,
which then typically is not actually a legal URL but trurl will try to figure
out what is meant and guess what scheme to use.

Providing multiple URLs will make trurl act on all URLs in a serial fashion.

If the URL cannot be parsed for whatever reason, trurl will simply move on to
the next provided URL - unless \fI--verify\fP is used.
.IP "--trim [component]=[what]"
Trims data off a component. Currently this can only trim a query component.

"what" is specified as a full word or as a word prefix (using a single
trailing asterisk ('*')) which makes trurl remove the tuples from the query
string that match the instruction.
.IP "-v, --version"
Show version information and exit.
.IP "--verify"
When a URL is provided, return error immediately if it does not parse as a
valid URL. In normal cases, trurl can forgive a bad URL input.
.SH "JSON output format"
The \fI--json\fP option outputs a JSON array with one or more objects. One for
each URL.

Each URL JSON object contains a number of properties, a series of key/value
pairs. The exact set depends on the given URL.
.IP "url"
This key exists in every object. It is the complete URL, in a URL encoded
format.
.IP "scheme"
The URL scheme.
.IP "user"
The URL decoded user name.
.IP "password"
The URL decoded password.
.IP "options"
The URL decoded options. Note that only a few URL schemes support the
"options" component.
.IP "host"
The URL decoded and normalized host name. It might be a UTF-8 name if an IDN
name was used. It can also be a normalized IPv4 or IPv6 address. An IPv6
address always starts with a bracket (\fB[\fP) - and no other host names can
contain such a symbol.
.IP "port"
The provided port number. If the port number was not provided in the URL, but
the scheme is a known one, the default port for that scheme will be provided
here.
.IP "raw_port"
The port number exactly as provided in the URL. This will be a zero length
string if there was no explicit port number provided.
.IP "path"
The URL decoded path. Including the leading slash.
.IP "query"
The URL decoded full query, excluding the question mark separator.
.IP "fragment"
The URL decoded fragment, excluding the pound sign separator.
.IP "zoneid"
The zone id, which can only be present in an IPv6 address. When this key is
present, then \fBhost\fP is an IPv6 numerical address.
.IP "params"
This key contains an array of query key/value objects. Each such pair is
listed with "key" and "value" and their respective contents in the output.

The key/values are extracted from the query where they are separated by
amperands (\fB&\fP) - or the user sets with \fB--query-separator\fP.

The query pairs are listed in the order of appearance in a left-to-right
order, but can be made alpha-sorted with \fB--sort-query\fP
.SH EXAMPLES
.IP "Replace the host name of a URL"
.nf
$ trurl --url https://curl.se --set host=example.com
https://example.com/
.fi
.IP "Create a URL by setting components"
.nf
 $ trurl --set host=example.com --set scheme=ftp
 ftp://example.com/
.fi
.IP "Redirect a URL"
.nf
$ trurl --url https://curl.se/we/are.html --redirect here.html
https://curl.se/we/here.html
.fi
.IP "Change port number"
This also shows how trurl will remove dot-dot sequences

.nf
$ trurl --url https://curl.se/we/../are.html --set port=8080
https://curl.se:8080/are.html
.fi
.IP "Extract the path from a URL"
.nf
$ trurl --url https://curl.se/we/are.html --get '{path}'
/we/are.html
.fi
.IP "Extract the port from a URL"
This gets the default port based on the scheme if the port is not set in the
URL.

.nf
$ trurl --url https://curl.se/we/are.html --get '{port}'
443
.fi
.IP "Append a path segment to a URL"
.nf
$ trurl --url https://curl.se/hello --append path=you
https://curl.se/hello/you
.fi
.IP "Append a query segment to a URL"
.nf
$ trurl --url "https://curl.se?name=hello" --append query=search=string
 https://curl.se/?name=hello&search=string
.fi
.IP "Read URLs from stdin"
.nf
$ cat urllist.txt | trurl --url-file -
\&...
.fi
.IP "Output JSON"
.nf
$ trurl "https://fake.host/search?q=answers&user=me#frag" --json
[
  {
    "url": "https://fake.host/search?q=answers&user=me#frag",
    "scheme": "https",
    "host": "fake.host",
    "port": "443",
    "raw_port": "",
    "path": "/search",
    "query": "q=answers&user=me",
    "fragment": "frag",
    "params": [
      {
        "key": "q",
        "value": "answers"
      },
      {
        "key": "user",
        "value": "me"
      }
    ]
  }
]
.fi
.IP "Remove tracking tuples from query"
.nf
$ trurl "https://curl.se?search=hey&utm_source=tracker" --trim query="utm_*"
https://curl.se/?search=hey
.fi
.IP "Show a specific query key value"
.nf
$ trurl "https://example.com?a=home&here=now&thisthen" -g '{query:a}'
home
.fi
.IP "Sort the key/value pairs in the query component"
.nf
$ trurl "https://example.com?b=a&c=b&a=c" --sort-query
https://example.com?a=c&b=a&c=b
.fi
.IP "Work with a query that uses a semicolon separator"
.nf
$ trurl "https://curl.se?search=fool;page=5" --trim query="search" --query-separator ";"
https://curl.se?page=5
.fi
.IP "Accept spaces in the URL path"
.nf
$ trurl "https://curl.se/this has space/index.html" --accept-space
https://curl.se/this%20has%20space/index.html
.fi
.IP "Create multiple variations of a URL with different schemes"
.nf
$ trurl "https://curl.se/path/index.html" --iterate "scheme=http ftp sftp"
http://curl.se/path/index.html
ftp://curl.se/path/index.html
sftp://curl.se/path/index.html
.fi
.SH WWW
https://curl.se/trurl