Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CURLOPT_FORBID_REUSE=1 + Graceful TCP shutdown with FIN-ACK FIN-ACK and SSL #13153

Open
dorrogeray opened this issue Mar 20, 2024 · 0 comments
Open

CURLOPT_FORBID_REUSE=1 + Graceful TCP shutdown with FIN-ACK FIN-ACK and SSL #13153

dorrogeray opened this issue Mar 20, 2024 · 0 comments
Labels
connecting & proxies

Comments

@dorrogeray
Copy link

dorrogeray commented Mar 20, 2024 •
edited by bagder

I did this

Hi, we have started to use CURLOPT_FORBID_REUSE in effort to get rid of "idle connections" in our AWS Nat Gateway, which seem to be happening when TCP connections are not closed with a graceful shutdown, with FIN-ACK from both sides. I observe this behavior also with cli curl, it seems that curl for the sake of convenience does not wait for the graceful shutdown to finish. So unfortunately, using CURLOPT_FORBID_REUSE did not fix our problem. Is there any way this can be addressed, perhaps behind a new option, like CURL_WAIT_FOR_GRACEFUL_SHUTDOWN_MS?

Or maybe there is something else wrong here, I am not sure.

Repository where the issue can be reproduced locally: https://github.com/dorrogeray/curl-kinesis-rst-issue
Script used to generate following pcap: https://github.com/dorrogeray/curl-kinesis-rst-issue/blob/master/curl-push-kinesis.sh
Pcap: https://github.com/dorrogeray/curl-kinesis-rst-issue/blob/master/kinesis_tcp_curl.pcap
image

curl logs 
09:04:00.669742 [0-0] * Host kinesalite:4567 was resolved.
09:04:00.669783 [0-0] * IPv6: (none)
09:04:00.669793 [0-0] * IPv4: 172.27.0.2
09:04:00.669807 [0-0] * [HTTPS-CONNECT] added
09:04:00.669818 [0-0] * [HTTPS-CONNECT] connect, init
09:04:00.669830 [0-0] * [HTTPS-CONNECT] connect, check h21
09:04:00.669847 [0-0] * [HAPPY-EYEBALLS] created ipv4 (timeout 300000ms)
09:04:00.669856 [0-0] * [HAPPY-EYEBALLS] ipv4 starting (timeout=300000ms)
09:04:00.669873 [0-0] *   Trying 172.27.0.2:4567...
09:04:00.669890 [0-0] * [TCP] cf_socket_open() -> 0, fd=5
09:04:00.670011 [0-0] * [TCP] local address 172.27.0.3 port 54646...
09:04:00.670027 [0-0] * [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=0
09:04:00.670043 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
09:04:00.670062 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
09:04:00.670077 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
09:04:00.670089 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:04:00.670105 [0-0] * [TCP] adjust_pollset, !connected, POLLOUT fd=5
09:04:00.670118 [0-0] * [HAPPY-EYEBALLS] adjust_pollset -> 1 socks
09:04:00.670134 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:04:00.670150 [0-0] * [HTTPS-CONNECT] connect, check h21
09:04:00.670166 [0-0] * [TCP] connected
09:04:00.670181 [0-0] * [HAPPY-EYEBALLS] ipv4 connect -> 0, connected=1
09:04:00.670195 [0-0] * Connected to kinesalite (172.27.0.2) port 4567
09:04:00.670210 [0-0] * [SSL] cf_connect()
09:04:00.670343 [0-0] * ALPN: curl offers http/1.1
09:04:00.670427 [0-0] * TLSv1.3 (OUT), TLS handshake, Client hello (1):
09:04:00.670461 [0-0] * [TCP] send(len=517) -> 517, err=0
09:04:00.670471 [0-0] * [SSL] ossl_bio_cf_out_write(len=517) -> 517, err=0
09:04:00.670481 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> -1, err=81
09:04:00.670491 [0-0] * [TCP] recv(len=5) -> -1, err=81
09:04:00.670500 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
09:04:00.670511 [0-0] * [SSL] populate_x509_store, path=./combined-ca-bundle.pem, blob=0
09:04:00.672978 [0-0] *  CAfile: ./combined-ca-bundle.pem
09:04:00.672988 [0-0] *  CApath: /etc/ssl/certs
09:04:00.672996 [0-0] * [SSL] cf_connect() -> 0, done=0
09:04:00.673006 [0-0] * [HTTPS-CONNECT] connect -> 0, done=0
09:04:00.673016 [0-0] * [SSL] adjust_pollset, POLLIN fd=5
09:04:00.673026 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:04:00.673037 [0-0] * [SSL] adjust_pollset, POLLIN fd=5
09:04:00.673047 [0-0] * [HTTPS-CONNECT] adjust_pollset -> 1 socks
09:04:00.673741 [0-0] * [HTTPS-CONNECT] connect, check h21
09:04:00.673777 [0-0] * [SSL] cf_connect()
09:04:00.673835 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.673861 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.673892 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.673921 [0-0] * [TCP] nw_in_read(len=122, fd=5) -> 122, err=0
09:04:00.673943 [0-0] * [TCP] recv(len=122) -> 122, err=0
09:04:00.673969 [0-0] * [SSL] ossl_bio_cf_in_read(len=122) -> 122, err=0
09:04:00.673998 [0-0] * TLSv1.3 (IN), TLS handshake, Server hello (2):
09:04:00.674105 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.674128 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.674150 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.674169 [0-0] * [TCP] nw_in_read(len=1, fd=5) -> 1, err=0
09:04:00.674188 [0-0] * [TCP] recv(len=1) -> 1, err=0
09:04:00.674212 [0-0] * [SSL] ossl_bio_cf_in_read(len=1) -> 1, err=0
09:04:00.674234 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.674252 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.674276 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.674305 [0-0] * [TCP] nw_in_read(len=38, fd=5) -> 38, err=0
09:04:00.674332 [0-0] * [TCP] recv(len=38) -> 38, err=0
09:04:00.674360 [0-0] * [SSL] ossl_bio_cf_in_read(len=38) -> 38, err=0
09:04:00.674392 [0-0] * TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
09:04:00.674420 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.674446 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.674473 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.674502 [0-0] * [TCP] nw_in_read(len=2921, fd=5) -> 2921, err=0
09:04:00.674523 [0-0] * [TCP] recv(len=2921) -> 2921, err=0
09:04:00.674554 [0-0] * [SSL] ossl_bio_cf_in_read(len=2921) -> 2921, err=0
09:04:00.674592 [0-0] * TLSv1.3 (IN), TLS handshake, Certificate (11):
09:04:00.674787 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.674801 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.674811 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.674832 [0-0] * [TCP] nw_in_read(len=537, fd=5) -> 537, err=0
09:04:00.674847 [0-0] * [TCP] recv(len=537) -> 537, err=0
09:04:00.674869 [0-0] * [SSL] ossl_bio_cf_in_read(len=537) -> 537, err=0
09:04:00.674889 [0-0] * TLSv1.3 (IN), TLS handshake, CERT verify (15):
09:04:00.674965 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.674977 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.674991 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.675014 [0-0] * [TCP] nw_in_read(len=69, fd=5) -> 69, err=0
09:04:00.675035 [0-0] * [TCP] recv(len=69) -> 69, err=0
09:04:00.675051 [0-0] * [SSL] ossl_bio_cf_in_read(len=69) -> 69, err=0
09:04:00.675075 [0-0] * TLSv1.3 (IN), TLS handshake, Finished (20):
09:04:00.675102 [0-0] * TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
09:04:00.675128 [0-0] * TLSv1.3 (OUT), TLS handshake, Finished (20):
09:04:00.675161 [0-0] * [TCP] send(len=80) -> 80, err=0
09:04:00.675177 [0-0] * [SSL] ossl_bio_cf_out_write(len=80) -> 80, err=0
09:04:00.675205 [0-0] * SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / [blank] / UNDEF
09:04:00.675221 [0-0] * ALPN: server accepted http/1.1
09:04:00.675236 [0-0] * Server certificate:
09:04:00.675255 [0-0] *  subject: C=US; ST=NY; L=New York; O=Kinesalite; CN=kinesalite
09:04:00.675270 [0-0] *  start date: Mar  8 09:57:09 2024 GMT
09:04:00.675283 [0-0] *  expire date: Mar  4 09:57:09 2040 GMT
09:04:00.675303 [0-0] *  subjectAltName: host "kinesalite" matched cert's "kinesalite"
09:04:00.675322 [0-0] *  issuer: C=US; ST=NY; L=New York; O=Kinesalite; CN=Kinesalite CA
09:04:00.675334 [0-0] *  SSL certificate verify ok.
09:04:00.675351 [0-0] *   Certificate level 0: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
09:04:00.675369 [0-0] *   Certificate level 1: Public key type ? (4096/128 Bits/secBits), signed using sha256WithRSAEncryption
09:04:00.675387 [0-0] * [SSL] cf_connect() -> 0, done=1
09:04:00.675409 [0-0] * [HTTPS-CONNECT] connect+handshake h21: 5ms, 1st data: 4ms
09:04:00.675428 [0-0] * using HTTP/1.x
09:04:00.675454 [0-0] * [HTTPS-CONNECT] connect -> 0, done=1
09:04:00.675501 [0-0] * [TCP] send(len=542) -> 542, err=0
09:04:00.675520 [0-0] * [SSL] ossl_bio_cf_out_write(len=542) -> 542, err=0
09:04:00.675546 [0-0] > POST / HTTP/1.1
09:04:00.675546 [0-0] > Host: kinesalite:4567
09:04:00.675546 [0-0] > User-Agent: curl/8.7.0-DEV
09:04:00.675546 [0-0] > Accept: */*
09:04:00.675546 [0-0] > Content-Type: application/x-amz-json-1.1
09:04:00.675546 [0-0] > X-Amz-Target: Kinesis_20131202.PutRecord
09:04:00.675546 [0-0] > Authorization: AWS4-HMAC-SHA256 Credential=root/20240320/us-east-1/kinesis/aws4_request, SignedHeaders=content-type;host;x-amz-target, Signature=3890472fc2557bd8c71dccdadc7ea86c0903f438bc29934d2b5e9191f19b47c1
09:04:00.675546 [0-0] > X-Amz-Date: 20240320T090400Z
09:04:00.675546 [0-0] > Content-Length: 92
09:04:00.675546 [0-0] > 
09:04:00.675703 [0-0] * upload completely sent off: 92 bytes
09:04:00.675729 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.675745 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.675768 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.675783 [0-0] * [TCP] nw_in_read(len=282, fd=5) -> 282, err=0
09:04:00.675795 [0-0] * [TCP] recv(len=282) -> 282, err=0
09:04:00.675803 [0-0] * [SSL] ossl_bio_cf_in_read(len=282) -> 282, err=0
09:04:00.675818 [0-0] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
09:04:00.675845 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.675865 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.675882 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.675897 [0-0] * [TCP] nw_in_read(len=282, fd=5) -> 282, err=0
09:04:00.675912 [0-0] * [TCP] recv(len=282) -> 282, err=0
09:04:00.675932 [0-0] * [SSL] ossl_bio_cf_in_read(len=282) -> 282, err=0
09:04:00.675960 [0-0] * TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
09:04:00.675987 [0-0] * old SSL session ID is stale, removing
09:04:00.676006 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> -1, err=81
09:04:00.676025 [0-0] * [TCP] recv(len=5) -> -1, err=81
09:04:00.676047 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
09:04:00.676063 [0-0] * [SSL] cf_recv(len=102400) -> -1, 81
09:04:00.676186 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> 5, err=0
09:04:00.676202 [0-0] * [TCP] recv(len=5) -> 5, err=0
09:04:00.676217 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> 5, err=0
09:04:00.676232 [0-0] * [TCP] nw_in_read(len=459, fd=5) -> 459, err=0
09:04:00.676247 [0-0] * [TCP] recv(len=459) -> 459, err=0
09:04:00.676262 [0-0] * [SSL] ossl_bio_cf_in_read(len=459) -> 459, err=0
09:04:00.676285 [0-0] * [SSL] cf_recv(len=102400) -> 442, 0
09:04:00.676302 [0-0] < HTTP/1.1 200 OK
09:04:00.676318 [0-0] < x-amzn-RequestId: cab0dcb0-e698-11ee-9862-69a049b10c61
09:04:00.676338 [0-0] < x-amz-id-2: PQJWyMPAgYGwk117gFICLfTlmfF2snsSQyV0ca/rTatmOFWpAbZAjGykrLRgiz+j/DTRjJr9UYbGMiF/7D3LNh13awp4e2i/
09:04:00.676358 [0-0] < Content-Type: application/x-amz-json-1.1
09:04:00.676377 [0-0] < Content-Length: 110
09:04:00.676392 [0-0] < Date: Wed, 20 Mar 2024 09:04:00 GMT
09:04:00.676405 [0-0] < Connection: keep-alive
09:04:00.676419 [0-0] < Keep-Alive: timeout=5
09:04:00.676434 [0-0] < 
09:04:00.676462 [0-0] * Connection #0 to host kinesalite left intact
{"ShardId":"shardId-000000000000","SequenceNumber":"49649967789711958815854551123516276095458061256270807042"}

Section potentially of interest:

09:04:00.675987 [0-0] * old SSL session ID is stale, removing
09:04:00.676006 [0-0] * [TCP] nw_in_read(len=5, fd=5) -> -1, err=81
09:04:00.676025 [0-0] * [TCP] recv(len=5) -> -1, err=81
09:04:00.676047 [0-0] * [SSL] ossl_bio_cf_in_read(len=5) -> -1, err=81
09:04:00.676063 [0-0] * [SSL] cf_recv(len=102400) -> -1, 81

Not sure if related to #10290 or #13087

I expected the following

I expect a graceful shutdown when using the TCP connection, with no RST packet present in the pcap. This should apply to both running curl from cli, or when using https://curl.se/libcurl/c/CURLOPT_FORBID_REUSE.html with value set to 1. I except that this could be a opt-in feature, which is used only in scale intensive scenarios where graceful TCP shutdown is important.

curl/libcurl version

curl 8.7.0-DEV

operating system

Debian
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
connecting & proxies
Milestone
No milestone
Development

No branches or pull requests
2 participants
@bagder @dorrogeray