Nodes stop receiving connections to the public API

[avacom]

Andriy Apostol @avacom 12:39
Hello! We have faced some strange issue. A test environment with 4 nodes is running 24/7, the transactions are constantly being committed. However, after ~12 hrs, the nodes simply stop receiving the connections to the public API. The restart of the nodes does not help - I cannot connect to them neither with our software, nor with Postman. Only wiping the DB and configs help. Do you have any ideas why this can happen? Thanks in advance

Ilya Bogdanov @vitvakatu 13:12
@avacom hello. Thanks for your report, can't say anything concrete at the moment. I'll contact our QA team. What is your Exonum version? Do you have 4 validators? How much transactions do you have?

Andriy Apostol @avacom 13:19
@vitvakatu The Exonum version is 0.9.1. All 4 nodes are validators. It happened several times: 2-3 times with the low transaction rate (~30 / 12 hrs), and one time with ~8600 xactions / 12 hrs. For now, I cannot give you steps to reproduce, but if you have seen anything like this during your tests, I will very appreciate if you share the workaround

[ivan-ochc]

@avacom, hi. So far, i didn't manage to reproduce your issue. It would be great if you provide additional details, for example node's logs.
Just in case: did you have enough disk space?

[avacom]

So, the current situation is the following: the node abnormally stopped. We have restarted it without the tracing. The API did not work for 10-15 minutes, and then it began to work. But now, the transactions posted by this node are not accepted by the blockchain. If I post the transaction from another node, it is committed, and the blockchain state is changed. But the 1st problemmatic node shows the old state. We restarted the problemmatic node, posted the transaction with it, and collected the following trace:

Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node::consensus Broadcast prevote: Prevote { validator: ValidatorId(3), height: Height(605996), round: Round(171), propose_hash: Hash(18babeb5...), locked_round: Round(4) } Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6334 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6332 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6333 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node::consensus Broadcast prevote: Prevote { validator: ValidatorId(3), height: Height(605996), round: Round(172), propose_hash: Hash(18babeb5...), locked_round: Round(4) } Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6334 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6332 Fri, 31 Aug 2018 13:21:03.250 +0000 TRACE exonum::node Send to address: 127.0.0.1:6333 Fri, 31 Aug 2018 13:22:42.390 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:22:45.210 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:33:02.033 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node::consensus Broadcast prevote: Prevote { validator: ValidatorId(3), height: Height(605996), round: Round(171), propose_hash: Hash(18babeb5...), locked_round: Round(4) } Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6334 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6332 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6333 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node::consensus Broadcast prevote: Prevote { validator: ValidatorId(3), height: Height(605996), round: Round(172), propose_hash: Hash(18babeb5...), locked_round: Round(4) } Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6334 Fri, 31 Aug 2018 13:21:03.249 +0000 TRACE exonum::node Send to address: 127.0.0.1:6332 Fri, 31 Aug 2018 13:21:03.250 +0000 TRACE exonum::node Send to address: 127.0.0.1:6333 Fri, 31 Aug 2018 13:22:42.390 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:22:45.210 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:33:02.033 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} Fri, 31 Aug 2018 13:45:13.270 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"}
This Fri, 31 Aug 2018 13:45:13.270 +0000 DEBUG exonum::api::node::public::explorer {"type":"unknown"} was printed right after the unsuccessful transaction posting. Do you see any issues here?

[aleksuss]

@avacom just to clarify. Is a free space is present on a hard disk ?

[avacom]

@aleksuss hello. The disk space was enough. But the RAM was almost out

[ivan-ochc]

@avacom, do you still face the problem with receiving connections to the public API?