Rootless network performance (pasta vs slirp4netns)

[mlegenovic]

I did some network performance tests to see how good pasta is compared to slirp4netns. I start on one physical machine nginx (as podman container), and send requests from other phyiscal mashine with "Apache HTTP server benchmarking tool" (ab). Here the results:

Default rootless network (pasta)

ab -c 1 -n 10000

Time taken for tests:   5.885 seconds
Requests per second:    1699.26 [#/sec] (mean)
Time per request:       0.588 [ms] (mean)
Time per request:       0.588 [ms] (mean, across all concurrent requests)

ab -c 4 -n 40000

Time taken for tests:   4.498 seconds
Requests per second:    8893.57 [#/sec] (mean)
Time per request:       0.450 [ms] (mean)
Time per request:       0.112 [ms] (mean, across all concurrent requests)

ab -c 8 -n 80000

Time taken for tests:   5.775 seconds
Requests per second:    13851.79 [#/sec] (mean)
Time per request:       0.578 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)

ab -c 16 -n 160000

Time taken for tests:   11.197 seconds
Requests per second:    14289.42 [#/sec] (mean)
Time per request:       1.120 [ms] (mean)
Time per request:       0.070 [ms] (mean, across all concurrent requests)

ab -c 32 -n 320000

Time taken for tests:   21.100 seconds
Requests per second:    15165.95 [#/sec] (mean)
Time per request:       2.110 [ms] (mean)
Time per request:       0.066 [ms] (mean, across all concurrent requests)

Rootless network with slirp4netns

ab -c 1 -n 10000

Time taken for tests:   8.713 seconds
Requests per second:    1147.76 [#/sec] (mean)
Time per request:       0.871 [ms] (mean)
Time per request:       0.871 [ms] (mean, across all concurrent requests)

ab -c 4 -n 40000

Time taken for tests:   7.143 seconds
Requests per second:    5600.11 [#/sec] (mean)
Time per request:       0.714 [ms] (mean)
Time per request:       0.179 [ms] (mean, across all concurrent requests)

ab -c 8 -n 80000

Time taken for tests:   5.745 seconds
Requests per second:    13924.88 [#/sec] (mean)
Time per request:       0.575 [ms] (mean)
Time per request:       0.072 [ms] (mean, across all concurrent requests)

ab -c 16 -n 160000

Time taken for tests:   6.438 seconds
Requests per second:    24852.06 [#/sec] (mean)
Time per request:       0.644 [ms] (mean)
Time per request:       0.040 [ms] (mean, across all concurrent requests)

ab -c 32 -n 320000

Time taken for tests:   11.061 seconds
Requests per second:    28930.86 [#/sec] (mean)
Time per request:       1.106 [ms] (mean)
Time per request:       0.035 [ms] (mean, across all concurrent requests)

Default rootfull network

ab -c 1 -n 10000

Time taken for tests:   5.074 seconds
Requests per second:    1970.86 [#/sec] (mean)
Time per request:       0.507 [ms] (mean)
Time per request:       0.507 [ms] (mean, across all concurrent requests)

ab -c 4 -n 40000

Time taken for tests:   4.779 seconds
Requests per second:    8369.89 [#/sec] (mean)
Time per request:       0.478 [ms] (mean)
Time per request:       0.119 [ms] (mean, across all concurrent requests)

ab -c 8 -n 80000

Time taken for tests:   4.174 seconds
Requests per second:    19165.58 [#/sec] (mean)
Time per request:       0.417 [ms] (mean)
Time per request:       0.052 [ms] (mean, across all concurrent requests)

ab -c 16 -n 160000

Time taken for tests:   5.182 seconds
Requests per second:    30878.20 [#/sec] (mean)
Time per request:       0.518 [ms] (mean)
Time per request:       0.032 [ms] (mean, across all concurrent requests)

ab -c 32 -n 320000

Time taken for tests:   8.508 seconds
Requests per second:    37610.08 [#/sec] (mean)
Time per request:       0.851 [ms] (mean)
Time per request:       0.027 [ms] (mean, across all concurrent requests)

Conclusion
As you can see, pasta performs better than slirp4netns only up to a certain parallelism (specifically up to 8). Then slirp4netns starts to perform significantly better. Rootfull network scales best, which is to be expected.

Question
Are there any options to improve pasta's scalability?

[sbrivio-rh]

Are there any options to improve pasta's scalability?

Not really options, at the moment. We check latency using neper in the upstream CI (perf/pasta_tcp set), but it's all single-threaded.

There are plans to introduce multithreading, eventually, and much more concrete work in progress to support vhost-user for virtual machines (passt(1)), first.

Once vhost-user support is merged, we can start playing with multithreading for virtual machines (vhost-user supports multiple queues). That would be kind of symmetric with VDUSE for containers (see "VDUSE support for containers" in this article): for pasta, it could look like this.

I'm not excluding you could get much better results by tweaking some #defines in tcp.c and tap.c, though. I would start with TAP_SEQS and TCP_HASH_TABLE_LOAD -- or perhaps "simply" profiling with perf(1) in the ab -c 32 case.