Investigate errors from logs - relayer Kusama to BHP finality

[bkontur]

https://grafana.teleport.parity.io/explore?schemaVersion=1&panes=%7B%227ff%22:%7B%22datasource%22:%22P8781AF333E0457B4%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D~%5C%22bridges-common-relay%2Fbridges-common-relay-headers-a-76f49567fc-m5r8w%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P8781AF333E0457B4%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-2d%22,%22to%22:%22now%22%7D%7D%7D&orgId=1

	2024-10-17 23:20:54.377	
2024-10-17 21:20:54 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.    
2024-10-17 23:20:45.049	
2024-10-17 21:20:45 +00  INFO bridge: Synced 25385581 of 25385586 headers    
	
2024-10
	2024-10-17 23:20:20.609	
2024-10-17 21:20:20 +00  INFO bridge: Synced 25385581 of 25385582 headers    
2024-10-17 23:20:07.897	
2024-10-17 21:20:07 +00  INFO bridge: Synced 25385569 of 25385581 headers    
2024-10-17 23:19:54.868	
2024-10-17 21:19:54 +00 ERROR bridge: Finality sync loop iteration has failed with error: Target(FailedToSubmitTransaction { chain: "BridgeHubPolkadot", error: RpcError(Call(ErrorObject { code: ServerError(1010), message: "Invalid Transaction", data: Some(RawValue("Transaction has a bad signature")) })) })    
	
2024-10-17 23:19:54.868	
2024-10-17 21:19:54 +00 ERROR bridge: Failed to send transaction to BridgeHubPolkadot node: Call(ErrorObject { code: ServerError(1010), message: "Invalid Transaction", data: Some(RawValue("Transaction has a bad signature")) })    
	
2024-10-17 23:19:53.828	
2024-10-17 21:19:53 +00  INFO bridge: Synced 25385569 of 25385579 headers    

and after few minutes later, there are no errors, but syncing stopped on 25385726:

	2024-10-17 23:36:36.368	
2024-10-17 21:36:36 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:24.262	
2024-10-17 21:36:24 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:12.157	
2024-10-17 21:36:12 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:36:00.005	
2024-10-17 21:36:00 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:48.731	
2024-10-17 21:35:48 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:36.624	
2024-10-17 21:35:36 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:24.384	
2024-10-17 21:35:24 +00  INFO bridge: Synced 25385726 of 25385727 headers    
2024-10-17 23:35:10.203	
2024-10-17 21:35:10 +00  INFO bridge: Synced 25385721 of 25385726 headers    
2024-10-17 23:34:57.629	
2024-10-17 21:34:57 +00  INFO bridge: Synced 25385721 of 25385723 headers    

[bkontur]

hmm, "of header" 25385727 is not changing Synced 25385726 of 25385727 headers

[bkontur]

when I ran that locally now, the "of header" is advancing 25412898

2024-10-19 19:50:11 +00  INFO bridge: Synced 25385726 of 25412898 headers

[bkontur]

• investigate why "of header" stopped advancing

[bkontur]

• also wondering what those ProofSubmissionTxLost are:

2024-10-20 19:29:00 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:29:13 +00  INFO bridge: Synced 25426767 of 25426769 headers
2024-10-20 19:29:26 +00  INFO bridge: Synced 25426767 of 25426771 headers
2024-10-20 19:29:39 +00  INFO bridge: Synced 25426767 of 25426771 headers
2024-10-20 19:29:51 +00  INFO bridge: Synced 25426767 of 25426775 headers
2024-10-20 19:30:04 +00  INFO bridge: Synced 25426775 of 25426777 headers
2024-10-20 19:30:16 +00  INFO bridge: Synced 25426775 of 25426779 headers
2024-10-20 19:30:29 +00  INFO bridge: Synced 25426775 of 25426782 headers
2024-10-20 19:30:42 +00  INFO bridge: Synced 25426782 of 25426783 headers
2024-10-20 19:30:54 +00  INFO bridge: Synced 25426782 of 25426787 headers
2024-10-20 19:31:07 +00  INFO bridge: Synced 25426787 of 25426789 headers
2024-10-20 19:31:20 +00  INFO bridge: Synced 25426787 of 25426791 headers
2024-10-20 19:31:32 +00  INFO bridge: Synced 25426787 of 25426792 headers
2024-10-20 19:31:37 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:31:49 +00  INFO bridge: Synced 25426787 of 25426796 headers
2024-10-20 19:32:02 +00  INFO bridge: Synced 25426787 of 25426796 headers
2024-10-20 19:32:14 +00  INFO bridge: Synced 25426796 of 25426800 headers
2024-10-20 19:32:26 +00  INFO bridge: Synced 25426796 of 25426802 headers
2024-10-20 19:32:40 +00  INFO bridge: Synced 25426802 of 25426805 headers
2024-10-20 19:32:52 +00  INFO bridge: Synced 25426802 of 25426806 headers
2024-10-20 19:33:13 +00  INFO bridge: Synced 25426807 of 25426810 headers
2024-10-20 19:33:26 +00  INFO bridge: Synced 25426807 of 25426812 headers
2024-10-20 19:33:39 +00  INFO bridge: Synced 25426812 of 25426814 headers
2024-10-20 19:33:51 +00  INFO bridge: Synced 25426812 of 25426816 headers
2024-10-20 19:34:10 +00  INFO bridge: Synced 25426817 of 25426819 headers
2024-10-20 19:34:22 +00  INFO bridge: Synced 25426817 of 25426821 headers
2024-10-20 19:34:34 +00  INFO bridge: Synced 25426817 of 25426823 headers
2024-10-20 19:34:47 +00  INFO bridge: Synced 25426823 of 25426825 headers
2024-10-20 19:34:59 +00  INFO bridge: Synced 25426823 of 25426827 headers
2024-10-20 19:35:12 +00  INFO bridge: Synced 25426823 of 25426829 headers
2024-10-20 19:35:18 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:35:31 +00  INFO bridge: Synced 25426831 of 25426833 headers
2024-10-20 19:35:44 +00  INFO bridge: Synced 25426831 of 25426834 headers
2024-10-20 19:35:56 +00  INFO bridge: Synced 25426831 of 25426836 headers
2024-10-20 19:36:06 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:36:19 +00  INFO bridge: Synced 25426831 of 25426841 headers
2024-10-20 19:36:31 +00  INFO bridge: Synced 25426838 of 25426843 headers
2024-10-20 19:36:36 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.
2024-10-20 19:36:48 +00 ERROR bridge: Finality sync proof submission tx to BridgeHubPolkadot has failed with error: ProofSubmissionTxLost.