UI and MW dying because of nightly apache reload

[tpurschke]

this happens at large customer site f1 only on prod server but has not been seen anywhere else
The next reload usually (24h later) fixes the problem and everyhting works after that.

Logs

(logs from 2024-06-16 before midnight look normal)

2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:  ---


-----------------------------------


2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-16T00:00:30+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: --- End of stack trace from previous location ---
2024-03-16T00:00:30.408+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-16T00:00:32+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  The server returned status code '503' when status code '101' was expected.
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ConnectAsync(CancellationToken token) in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 432
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLWSProtocolHandler.<>c__DisplayClass7_0`1.<<CreateSubscriptionObservable>b__0>d.MoveNext() in /_/src/GraphQL.Client/Websocket/GraphQLWSProtocolHandler.cs:line 134
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-18T00:00:30+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: --- End of stack trace from previous location ---
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-03-18T00:00:32.788+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-18T00:00:32+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-18T00:00:32.788+01:00 orcprocust1201 fworch-ui[2804]:  ---

[tpurschke]

as the config subscription is also involved, might there be a connection to other issues like #2404