Use http/2 and keepalives #1378

technillogue · 2023-07-18T23:03:56Z

Details

We noticed connections were being dropped frequently. This adds rustls, so that http/2 ALPN works. To benefit from this more, it also sets tcp_keepalive and a connection pool. I'm not attached to the specific values for those.

It also adds a little bit of logging on throughput, which I'm happy to discard if it doesn't fit.

Types of changes

What types of changes does your PullRequest introduce? Put an x in all the boxes that apply:

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have updated the documentation accordingly.
I have added tests to cover my changes.

anolis-bot · 2023-07-18T23:04:04Z

@technillogue , a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/85448

anolis-bot · 2023-07-18T23:05:49Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

anolis-bot · 2023-07-18T23:06:04Z

@technillogue , the title has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/85449

anolis-bot · 2023-07-18T23:07:34Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

adamqqqplay

Please fix broken CI first, thanks.

imeoer · 2023-07-19T02:18:16Z

storage/src/backend/connection.rs

@@ -626,14 +626,19 @@ impl Connection {
 } else {
 None
 };
+ // get pool size from envvar
+ let pool_max_idle_per_host = match env::var("REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST") {


Do these idle connections take pressure on the registry server? Especially when the container is started and prefetch work is done. Maybe we can make the value as the one option of ConnectionConfig.

I don't think it would make a significant impact for reasonable values. Most servers should close idle connections if it becomes a problem.

How about making the value as the one option of ConnectionConfig?

It's better to fetch REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST from configuration.

imeoer · 2023-07-19T02:24:55Z

@technillogue Thanks for the PR! It seems to have some compilation errors that still need to be solved.
Does this change improve your bandwidth utilization?

anolis-bot · 2023-07-19T02:57:55Z

@technillogue , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/85469

anolis-bot · 2023-07-19T02:59:40Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

technillogue · 2023-07-19T03:31:39Z

Does this change improve your bandwidth utilization?

It was a small improvement. It made less of a difference than changing merging_size, but still helped.

anolis-bot · 2023-07-19T03:33:23Z

@technillogue , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/85490

anolis-bot · 2023-07-19T03:36:23Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

codecov · 2023-07-19T05:49:32Z

Codecov Report

Merging #1378 (dfa8491) into master (9ab1ec1) will increase coverage by 0.01%.
The diff coverage is 64.28%.

@@            Coverage Diff             @@
##           master    #1378      +/-   ##
==========================================
+ Coverage   46.28%   46.29%   +0.01%     
==========================================
  Files         123      123              
  Lines       38872    38883      +11     
  Branches    38872    38883      +11     
==========================================
+ Hits        17990    18000      +10     
- Misses      19908    19909       +1     
  Partials      974      974

Files	Coverage Δ
storage/src/backend/connection.rs	`43.47% <85.71%> (+0.47%)`	⬆️
storage/src/cache/mod.rs	`50.67% <42.85%> (+0.09%)`	⬆️

... and 3 files with indirect coverage changes

jiangliu · 2023-07-19T07:52:48Z

storage/src/backend/connection.rs


 let mut cb = Client::builder()
 .timeout(timeout)
 .connect_timeout(connect_timeout)
- .redirect(Policy::none());
+ .redirect(Policy::none())
+ .use_rustls_tls()


why use_rustls_tls() here? It introduces heavy dependencies.

How about using native-tls instead? We already have vendored openssl-sys crate.

openssl = { version = "0.10.55", features = ["vendored"] }

it's required for http/2, as far as I know native-tls does not provide for the ALPN required for http/2

would this be acceptable?

It should be acceptable I think.

Should I make any other changes to this then?

How about making the env REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST value as the one option of ConnectionConfig?

anolis-bot · 2023-07-25T16:32:40Z

@technillogue , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/86935

adamqqqplay

The current commit log looks a bit messy, it would be nice if you could reorganize them.

anolis-bot · 2023-07-29T20:46:29Z

@technillogue , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/87687

technillogue · 2023-07-29T20:47:50Z

@adamqqqplay, I've reorganized the commit log

anolis-bot · 2023-07-29T20:48:28Z

@technillogue , the code has been updated, so a new test job has been submitted. Please wait in patience. The test job url: https://tone.openanolis.cn/ws/nrh4nnio/test_result/87688

anolis-bot · 2023-07-29T20:51:44Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		❌ FAIL

Sorry, your test job failed. Please get the details in the link.

anolis-bot · 2023-07-29T21:03:14Z

@technillogue , The CI test is completed, please check result:

Test Case		Test Result
build rust golang image		✅ SUCCESS
compile nydusd		✅ SUCCESS
compile ctr remote		✅ SUCCESS
compile nydus snapshotter		✅ SUCCESS
run container with rafs		✅ SUCCESS
run container with zran		✅ SUCCESS
run container with rafs and compile linux		✅ SUCCESS

Congratulations, your test job passed!

technillogue · 2023-08-08T12:40:13Z

Any update on this?

imeoer · 2023-08-09T02:02:48Z

Any update on this?

How about making the env REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST value as the one option of ConnectionConfig?

Signed-off-by: technillogue <[email protected]>

ccx1024cc · 2023-10-30T03:11:47Z

storage/src/backend/connection.rs

@@ -626,14 +626,19 @@ impl Connection {
 } else {
 None
 };
+ // get pool size from envvar
+ let pool_max_idle_per_host = match env::var("REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST") {


It's better to fetch REGISTRY_CLIENT_POOL_MAX_IDLE_PER_HOST from configuration.

ccx1024cc · 2023-10-30T03:12:33Z

storage/src/backend/connection.rs

- .redirect(Policy::none());
+ .redirect(Policy::none())
+ .use_rustls_tls()
+ .tcp_keepalive(Some(Duration::from_secs(5 * 60)))


Should be configurable.

technillogue requested a review from a team as a code owner July 18, 2023 23:03

technillogue requested review from liubin, luodw and adamqqqplay and removed request for a team July 18, 2023 23:03

anolis-bot added the anolis_testing label Jul 18, 2023

anolis-bot added anolis_test_fail and removed anolis_testing labels Jul 18, 2023

anolis-bot added anolis_testing and removed anolis_test_fail labels Jul 18, 2023

anolis-bot added anolis_test_fail and removed anolis_testing labels Jul 18, 2023

adamqqqplay reviewed Jul 19, 2023

View reviewed changes

imeoer reviewed Jul 19, 2023

View reviewed changes

technillogue force-pushed the master branch from 3c84c00 to 0c15c0d Compare July 19, 2023 02:57

anolis-bot added anolis_testing and removed anolis_test_fail labels Jul 19, 2023

anolis-bot added anolis_test_fail and removed anolis_testing labels Jul 19, 2023

anolis-bot added anolis_testing and removed anolis_test_fail labels Jul 19, 2023

anolis-bot added anolis_test_fail and removed anolis_testing labels Jul 19, 2023

jiangliu reviewed Jul 19, 2023

View reviewed changes

anolis-bot added anolis_testing and removed anolis_test_fail labels Jul 25, 2023

adamqqqplay reviewed Jul 27, 2023

View reviewed changes

technillogue force-pushed the master branch from e017b18 to 59caeec Compare July 29, 2023 20:46

technillogue force-pushed the master branch from 59caeec to 0d5d211 Compare July 29, 2023 20:48

anolis-bot added anolis_test_fail and removed anolis_testing labels Jul 29, 2023

anolis-bot added anolis_test_pass and removed anolis_test_fail labels Jul 29, 2023

technillogue added 2 commits October 10, 2023 21:34

use rustls feature in addition, 5min keepalive, connection pool 20

b50c614

Signed-off-by: technillogue <[email protected]>

make connection pool size configurable and add throughput logging

dfa8491

Signed-off-by: technillogue <[email protected]>

jiangliu force-pushed the master branch from 0d5d211 to dfa8491 Compare October 10, 2023 13:34

ccx1024cc requested changes Oct 30, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use http/2 and keepalives #1378

Use http/2 and keepalives #1378

technillogue commented Jul 18, 2023 •

edited

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

adamqqqplay left a comment

imeoer Jul 19, 2023

technillogue Jul 19, 2023

imeoer Jul 20, 2023

ccx1024cc Oct 30, 2023

imeoer commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

technillogue commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

codecov bot commented Jul 19, 2023 •

edited

jiangliu Jul 19, 2023

jiangliu Jul 19, 2023

technillogue Jul 19, 2023

technillogue Jul 20, 2023

imeoer Jul 24, 2023

technillogue Jul 26, 2023

imeoer Jul 31, 2023

anolis-bot commented Jul 25, 2023

adamqqqplay left a comment

anolis-bot commented Jul 29, 2023

technillogue commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

technillogue commented Aug 8, 2023

imeoer commented Aug 9, 2023

ccx1024cc Oct 30, 2023

ccx1024cc Oct 30, 2023

Use http/2 and keepalives #1378

Are you sure you want to change the base?

Use http/2 and keepalives #1378

Conversation

technillogue commented Jul 18, 2023 • edited

Details

Types of changes

Checklist

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

anolis-bot commented Jul 18, 2023

adamqqqplay left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

imeoer commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

technillogue commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

anolis-bot commented Jul 19, 2023

codecov bot commented Jul 19, 2023 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

anolis-bot commented Jul 25, 2023

adamqqqplay left a comment

Choose a reason for hiding this comment

anolis-bot commented Jul 29, 2023

technillogue commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

anolis-bot commented Jul 29, 2023

technillogue commented Aug 8, 2023

imeoer commented Aug 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

technillogue commented Jul 18, 2023 •

edited

codecov bot commented Jul 19, 2023 •

edited