command substitution vs read benchmarks

[IlanCosman]

fish, version 3.3.1-701-gceade1629

set -g string (string repeat -n 1 h)
set -q list || set -U list (seq 500)

function command_substitution
    set -l foo
    for i in $list
        set foo (echo $string$i)
    end
end

function builtin_read_command
    set -l foo
    for i in $list
        echo $string$i | read foo
    end
end

ilan@arch ~> hyperfine --shell=fish --warmup=100 --runs=100 command_substitution builtin_read_command
Benchmark 1: command_substitution
  Time (mean ± σ):      53.1 ms ±   7.4 ms    [User: 44.2 ms, System: 18.0 ms]
  Range (min … max):    26.7 ms …  61.9 ms    100 runs
 
Benchmark 2: builtin_read_command
  Time (mean ± σ):      43.1 ms ±   2.4 ms    [User: 21.8 ms, System: 12.4 ms]
  Range (min … max):    27.5 ms …  46.6 ms    100 runs
 
Summary
  'builtin_read_command' ran
    1.23 ± 0.18 times faster than 'command_substitution'

read performance is highly dependant on string length however. If we set string like so:

set -g string (string repeat -n 100 h)

The following happens:

ilan@arch ~> hyperfine --shell=fish --warmup=100 --runs=100 command_substitution builtin_read_command
Benchmark 1: command_substitution
  Time (mean ± σ):      61.0 ms ±   8.7 ms    [User: 54.4 ms, System: 15.1 ms]
  Range (min … max):    30.3 ms …  67.9 ms    100 runs
 
Benchmark 2: builtin_read_command
  Time (mean ± σ):      86.3 ms ±   2.1 ms    [User: 34.7 ms, System: 42.0 ms]
  Range (min … max):    70.9 ms …  91.6 ms    100 runs
 
Summary
  'command_substitution' ran
    1.42 ± 0.20 times faster than 'builtin_read_command'

The equivalence point is around 32 on my machine.

[faho]

The read performance is very dependent on length because read has to, in the general case, read byte-for-byte. This is so:

echo foo\nbar | begin
    read -l foo # consumes the "foo" and nothing more
    read -l bar
    echo $bar
end

prints bar. Unfortunately we can't put bytes back into the pipe (or read from it without consuming) so the only way that read -l foo doesn't consume more than a single line is to read one byte at a time. This means one syscall per byte. So the time needed scales linearly with the number of bytes in the line (or up to a NULL with -z), with a large constant factor because syscalls are expensive. See also #4972 (which I don't believe to be fixable in general).

It's probably possible to change this if the read is directly redirected like your echo | read here - in that case the pipe is closed after the read is done anyway so all the data is lost, so we can probably get away with reading a chunk of bytes at a time. This way we would only need one syscall per 128 bytes or so, so we would slash the linear factor by 128.

The command substitution performance here is caused by fish's parser being overly simplistic. It doesn't recurse into command substitutions, so it then re-parses them again and again, every time through the loop. This means there's a high setup cost for command substitutions, but once they do read they read quickly, in chunks. I've messed with a command substitution "cache" as a bandaid but have not had great results, so the proper solution is to make the parser recurse. This is a bigger task.

[faho]

Also: If you're benchmarking this, you definitely want longer benchmarks, to remove system noise from the equation. I've seen that randomly add 200ms (although this was on WSL, where you'd expect it - what with virus scanners and other background jobs).

I would shoot for a per-run time of a second or more, and reduce the number of runs accordingly. Especially warmup runs really aren't worth all that much so you only want 2 or 3 of those. Once things are in cache, they are in cache. Can't be more in-cachier than that.

This also removes other influences like startup costs.

[faho]

Also, to put this in context: This is 500 runs, and the difference is 10ms. That suggests the overhead of the command substitution is about 0.02ms, compared to piping to read.

As an alternative benchmark:

for f in (seq 10000); true (true); end

runs about 1s slower than

for f in (seq 10000); true; true; end

This seems like a big deal until you realize 1s for 10000 runs is 0.1ms per run. That matches up with other attempts with more command substitutions (true (true) (true) (true) (true) vs true; true; true; true; true and various other permutations). The cost is constant at about 0.1ms per run per command substitution. Adding more arguments (true (true a b c d e f g)) or making the command substitution more complicated ((true; true; true; true)) doesn't seem to impact it much.

[IlanCosman]

It's true that this isn't a big deal. Ultimately external commands and globbing will take the most time. But Tide, even after lots of optimization, still probably has like 30 command substitutions/reads per prompt render. And this could theoretically go a lot higher with tight loops like so:

while set -l truncation_length (math $truncation_length + 1) &&
        set -l truncated (string sub --length $truncation_length -- $dir_section) &&
        test $truncated != $dir_section -a (count $parent_dir/$truncated*/) -gt 1
end

which could theoretically add 3 command substitutions per character in $PWD. Again, the globbing no doubt takes the most time, and this while loop has since been optimized down to 2 command substitutions. Still, more performance is never unappreciated. 😄

[floam]

This thread is actually an example of something that would probably be better as a discussion. It's not really a bug report or an enhancement request: nothing was asked for, nothing was complained about, but it is interesting. I'm converting it.

[IlanCosman]

Also of note is that if you are already piping, read performs even better, 60% faster instead of 20% on the small strings.
i.e set foo (echo blah | string replace foo bar) vs echo blah | string replace foo bar | read foo