503 UC not being recorded in traces

[Stono]

Is this the right place to submit this?

• This is not a security vulnerability or a crashing bug
• This is not a question about how to use Istio

Bug Description

It appears that traces aren't being recorded for certain scenarios that result in a 503. In this case; connection resets.

As you can see here, we've got a reasonable stream of 503UC's from storage.googleapis.com.

I've come to investigate this and there are no traces being recorded for those errors. Only the 200/201's etc.

I enabled istio proxy debug logs and could see the error:

{"level":"debug","time":"2024-05-06T20:12:45.128689Z","scope":"envoy pool","msg":"[Tags: \"ConnectionId\":\"468579\"] creating stream","caller":"external/envoy/source/common/conn_pool/conn_pool_base.cc:182","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128697Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"468655\",\"StreamId\":\"7492250774811389721\"] pool ready","caller":"external/envoy/source/common/router/upstream_request.cc:572","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128718Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"468579\"] encode complete","caller":"external/envoy/source/common/http/codec_client.cc:141","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128969Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"468579\"] remote address:142.250.145.207:443,TLS_error:|33554536:system library:OPENSSL_internal:Connection reset by peer:TLS_error_end","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:241","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128981Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"468579\"] remote close","caller":"external/envoy/source/common/network/connection_impl.cc:714","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128985Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"468579\"] closing socket: 0","caller":"external/envoy/source/common/network/connection_impl.cc:278","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.128996Z","scope":"envoy misc","msg":"Unknown error code 32 details Broken pipe","caller":"external/envoy/source/common/network/io_socket_error_impl.cc:64","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129000Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"468579\"] SSL shutdown: rc=-1","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:329","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129004Z","scope":"envoy connection","msg":"[Tags: \"ConnectionId\":\"468579\"] remote address:142.250.145.207:443,TLS_error:|33554536:system library:OPENSSL_internal:Connection reset by peer:TLS_error_end|33554464:system library:OPENSSL_internal:Broken pipe:TLS_error_end","caller":"external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:241","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129037Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"468579\"] disconnect. resetting 1 pending requests","caller":"external/envoy/source/common/http/codec_client.cc:107","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129047Z","scope":"envoy client","msg":"[Tags: \"ConnectionId\":\"468579\"] request reset","caller":"external/envoy/source/common/http/codec_client.cc:158","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129056Z","scope":"envoy router","msg":"[Tags: \"ConnectionId\":\"468655\",\"StreamId\":\"7492250774811389721\"] upstream reset: reset reason: connection termination, transport failure reason: ","caller":"external/envoy/source/common/router/router.cc:1324","thread":49}
{"level":"debug","time":"2024-05-06T20:12:45.129105Z","scope":"envoy http","msg":"[Tags: \"ConnectionId\":\"468655\",\"StreamId\":\"7492250774811389721\"] Sending local reply with details upstream_reset_before_response_started{connection_termination}","caller":"external/envoy/source/common/http/filter_manager.cc:1035","thread":49}

I'd be expecting to see traces for these failed requests in Jaeger? I do see other traces which result in a 503, so it appears to be 503UC specifically.

Version

`1.20.4`

Additional Information

These aren't being captured by any retries or anything like that, the app ends up seeing them (this is from app logs):

Caused by: com.google.cloud.storage.StorageException: 503 Service Unavailable
POST http://storage.googleapis.com/upload/storage/v1/b/.......
upstream connect error or disconnect/reset before headers. reset reason: connection termination

[zirain]

this should be report to envoy.

[Stono]

Whilst on paper I agree, from a consumer experience perspective - I don't. Users of Istio will report issues to Istio. Some Istio users wouldn't even know there's Envoy under the hood as it's abstracted away.

It's like my car, if my brakes aren't working - I'd report it to Tesla (the manufacturer of my car), not Bosch (the manufacturer of my brakes).