[BUG] Cannot process the vote or it's invalid

[platfowner]

node-8080-error-16-Mar-22.log:

{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensus
Message] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/con
sensus/number/319627/0x2988e548c0efb0b755b07c8993f6d536134093e78d6f1eecfdfa60d0aae29a34/vote/0x007C36bf5D0F77836eE13
8EEAc8df7051b43209b\",\"value\":{\"block_hash\":\"0x2988e548c0efb0b755b07c8993f6d536134093e78d6f1eecfdfa60d0aae29a34
\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647396840146}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1
647396840146},\"signature\":\"0xa7c28ff16d77f1d74b3f682f45ade293ad027150f90fb2d8f1cb7fca34a56a9c7819858adbae804b6b4b
ed9d0f476491a8ba864caa0b8d0487ea938feba4981d2c3f3ba1fc971999ea08a076cfe3fbb181caddf558fecbb6d670d10d2373a12426\",\"h
ash\":\"0xa7c28ff16d77f1d74b3f682f45ade293ad027150f90fb2d8f1cb7fca34a56a9c\",\"address\":\"0x007C36bf5D0F77836eE138E
EAc8df7051b43209b\"}","time":"2022-03-16T02:14:00.217Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensus
Message] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/con
sensus/number/319627/0x2988e548c0efb0b755b07c8993f6d536134093e78d6f1eecfdfa60d0aae29a34/vote/0x002C76f0aeA9Ba615428d
9dF7fedEC6f8ed5369f\",\"value\":{\"block_hash\":\"0x2988e548c0efb0b755b07c8993f6d536134093e78d6f1eecfdfa60d0aae29a34
\",\"stake\":10000000,\"is_against\":false,\"vote_nonce\":1647396840107}},\"nonce\":-1,\"gas_price\":0,\"timestamp\"
:1647396840107},\"signature\":\"0x82e95548ad10088a91d0b27ad517d1bb0741b9f8a8f1bf1dc37cbdef39670ed24e6be6d35430ed3bbd
6cd06b61666598a65c4121d543851b8b23de3cafd653a852c4b4c39d5f5022240e083866e5c56a453f52d2a0d328f44395512d5384f33126\",\
"hash\":\"0x82e95548ad10088a91d0b27ad517d1bb0741b9f8a8f1bf1dc37cbdef39670ed2\",\"address\":\"0x002C76f0aeA9Ba615428d
9dF7fedEC6f8ed5369f\"}","time":"2022-03-16T02:14:00.221Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensus
Message] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/con
sensus/number/319679/0xe8bbf2f5a7f979a2a0ced6f3bccc39f2c8583e9f396f67896e8f249ada8a273a/vote/0x007C36bf5D0F77836eE13
8EEAc8df7051b43209b\",\"value\":{\"block_hash\":\"0xe8bbf2f5a7f979a2a0ced6f3bccc39f2c8583e9f396f67896e8f249ada8a273a
\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647397940194}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1
647397940194},\"signature\":\"0x2f2384cf35320aa38fce26bbdb69bc87769df833810aae394c5d2bbbe98528fe59d957c8e29751a656cc
3cd805541a5b29fe56667e7f29a036f0b2a99afb89950c010621d9888e47f9388163c7cb22f0b3f8f3f0803584617361492978d61d7825\",\"h
ash\":\"0x2f2384cf35320aa38fce26bbdb69bc87769df833810aae394c5d2bbbe98528fe\",\"address\":\"0x007C36bf5D0F77836eE138E
EAc8df7051b43209b\"}","time":"2022-03-16T02:32:20.375Z","v":0}
...

[platfowner]

Spin-off'ed from:

• [BUG] Blockchain alert: ~ 2022-03-16 12:50:00 #973

[platfowner]

Not sure whether it's directly related or not but this issue was seen around the time we observed the blockchain alert.
@liayoo could you take a look?

[liayoo]

As suspected, the errors were caused by not receiving blocks to vote on time. I think it could be due to the network topology issues or slowing down of the process. Once we have the topology recording feature implemented, the source of the problem may become clearer.

{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.293Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x006C672861e9DBb09232307c17Be6554BC90687c\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000152}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000152},\"signature\":\"0xd1b50eee68659545d0c59f043bef27103e4e5ba959f0d4539f09fbf469e2c4afbcb71f138963fc62c8b5c177dc4e2d03ed8379579ff338c1a5ad862fa8fb0dd92e8508b88ca5213d8aea9aa07ec898d5a2e4e20b240ecb516edb89f56b5113e425\",\"hash\":\"0xd1b50eee68659545d0c59f043bef27103e4e5ba959f0d4539f09fbf469e2c4af\",\"address\":\"0x006C672861e9DBb09232307c17Be6554BC90687c\"}","time":"2022-03-16T03:56:40.293Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.298Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x001C3C9C4a5669eCD8b78946f6fa5549b33362F8\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":10000000,\"is_against\":false,\"vote_nonce\":1647403000090}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000090},\"signature\":\"0xdbf1bfe9b974689fd1e39f9633ae8239dac0a1c868c5a8acbfe87a7eb43d9bcffd3730dd7fa05bdad964b6bf969601a3add397632e68e40aabf017f6dbab9ec11eca5f381414642fd909f683dde029893e7de898211f8e57c6443c0af8dea3e925\",\"hash\":\"0xdbf1bfe9b974689fd1e39f9633ae8239dac0a1c868c5a8acbfe87a7eb43d9bcf\",\"address\":\"0x001C3C9C4a5669eCD8b78946f6fa5549b33362F8\"}","time":"2022-03-16T03:56:40.298Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.357Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x005C99Db64845e5BF24cd152b22c932989479907\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000187}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000187},\"signature\":\"0xeef546a42f42d533cff53549ed5c11a151fa205a0a4549632ab1f122056d43f05002627ba45c59abc0475c6e6c69211dd116aab0d45fc93fe43b4c7a62f07c3c4c12d476b850d1a93d34bd3f9585fad24af9f937371c2d09c1b44df5340b8a8326\",\"hash\":\"0xeef546a42f42d533cff53549ed5c11a151fa205a0a4549632ab1f122056d43f0\",\"address\":\"0x005C99Db64845e5BF24cd152b22c932989479907\"}","time":"2022-03-16T03:56:40.357Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.429Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x009C66DBce144003f8C4B859fFFce78F80fDD639\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000245}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000245},\"signature\":\"0xe7b8a8edd9f6a09bab52e2dc56cc42042c54902922700964e18c263228e358da0af22484d294ec774f84801a1c6bec5b7180040196ff89266bc624bb452d576e308d0709b89c2c985c52a9a5212a49207c3b734f208bd7fcbc83fe5877e66db025\",\"hash\":\"0xe7b8a8edd9f6a09bab52e2dc56cc42042c54902922700964e18c263228e358da\",\"address\":\"0x009C66DBce144003f8C4B859fFFce78F80fDD639\"}","time":"2022-03-16T03:56:40.429Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.510Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x001C3C9C4a5669eCD8b78946f6fa5549b33362F8\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":10000000,\"is_against\":false,\"vote_nonce\":1647403000090}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000090},\"signature\":\"0xdbf1bfe9b974689fd1e39f9633ae8239dac0a1c868c5a8acbfe87a7eb43d9bcffd3730dd7fa05bdad964b6bf969601a3add397632e68e40aabf017f6dbab9ec11eca5f381414642fd909f683dde029893e7de898211f8e57c6443c0af8dea3e925\",\"hash\":\"0xdbf1bfe9b974689fd1e39f9633ae8239dac0a1c868c5a8acbfe87a7eb43d9bcf\",\"address\":\"0x001C3C9C4a5669eCD8b78946f6fa5549b33362F8\"}","time":"2022-03-16T03:56:40.510Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.838Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x006C672861e9DBb09232307c17Be6554BC90687c\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000152}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000152},\"signature\":\"0xd1b50eee68659545d0c59f043bef27103e4e5ba959f0d4539f09fbf469e2c4afbcb71f138963fc62c8b5c177dc4e2d03ed8379579ff338c1a5ad862fa8fb0dd92e8508b88ca5213d8aea9aa07ec898d5a2e4e20b240ecb516edb89f56b5113e425\",\"hash\":\"0xd1b50eee68659545d0c59f043bef27103e4e5ba959f0d4539f09fbf469e2c4af\",\"address\":\"0x006C672861e9DBb09232307c17Be6554BC90687c\"}","time":"2022-03-16T03:56:40.838Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.848Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x005C99Db64845e5BF24cd152b22c932989479907\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000187}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000187},\"signature\":\"0xeef546a42f42d533cff53549ed5c11a151fa205a0a4549632ab1f122056d43f05002627ba45c59abc0475c6e6c69211dd116aab0d45fc93fe43b4c7a62f07c3c4c12d476b850d1a93d34bd3f9585fad24af9f937371c2d09c1b44df5340b8a8326\",\"hash\":\"0xeef546a42f42d533cff53549ed5c11a151fa205a0a4549632ab1f122056d43f0\",\"address\":\"0x005C99Db64845e5BF24cd152b22c932989479907\"}","time":"2022-03-16T03:56:40.849Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":30,"msg":"[CONSENSUS] [checkVoteTx] Cannot verify the vote without the block it's voting for: 0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c / undefined","time":"2022-03-16T03:56:40.849Z","v":0}
{"name":"mainnet-prod","hostname":"mainnet-node-0-taiwan","pid":16797,"level":50,"msg":"[CONSENSUS] [handleConsensusMessage] Cannot process the vote or it's invalid: {\"tx_body\":{\"operation\":{\"type\":\"SET_VALUE\",\"ref\":\"/consensus/number/319905/0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c/vote/0x009C66DBce144003f8C4B859fFFce78F80fDD639\",\"value\":{\"block_hash\":\"0x7b4530081d791cab19d028d4a0fc7bd6b26f9ed9d76ad51c474ee4bb58f83a5c\",\"stake\":100000,\"is_against\":false,\"vote_nonce\":1647403000245}},\"nonce\":-1,\"gas_price\":0,\"timestamp\":1647403000245},\"signature\":\"0xe7b8a8edd9f6a09bab52e2dc56cc42042c54902922700964e18c263228e358da0af22484d294ec774f84801a1c6bec5b7180040196ff89266bc624bb452d576e308d0709b89c2c985c52a9a5212a49207c3b734f208bd7fcbc83fe5877e66db025\",\"hash\":\"0xe7b8a8edd9f6a09bab52e2dc56cc42042c54902922700964e18c263228e358da\",\"address\":\"0x009C66DBce144003f8C4B859fFFce78F80fDD639\"}","time":"2022-03-16T03:56:40.849Z","v":0}

[platfowner]

@liayoo thanks for the update.
@minsulee2 do we have any progress on the topology logging?
#985