Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate parachain finality relayer for Rococo/Westend #3014

Open
bkontur opened this issue Jun 25, 2024 · 2 comments
Open

Investigate parachain finality relayer for Rococo/Westend #3014

bkontur opened this issue Jun 25, 2024 · 2 comments

Comments

@bkontur
Copy link
Contributor

bkontur commented Jun 25, 2024 •
edited
Loading

Looks like parachain finality relayer from BridgeHubWestend to BridgeHubRococo has stopped advancing.
No bridgeWestendParachains.UpdatedParachainHead on Rococo BH, I had to run locally and then delivery starts working.

RUST_LOG=bridge=trace ./target/debug/substrate-relay relay-parachains westend-to-bridge-hub-rococo \
  --source-uri wss://westend-rpc.dwellir.com \
  --target-uri wss://bridge-hub-rococo-rpc.dwellir.com \
  --target-signer "derive milk cigar crime monkey lounge dwarf inner proud lava reform empower" \
  --source-version-mode Auto \
  --target-version-mode Auto

https://grafana.teleport.parity.io/explore?schemaVersion=1&panes=%7B%22yph%22:%7B%22datasource%22:%22P03E52D76DFE188C3%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bpod%3D~%5C%22bridges-common-relay%2F.%2A%5C%22,%20container%3D%5C%22parachains-b%5C%22%7D%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22P03E52D76DFE188C3%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221718909446000%22,%22to%22:%221718911246000%22%7D%7D%7D&orgId=1

Explore-logs-2024-06-25 14 25 59.txt

2024-06-20 21:20:38.771	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:20:28.771	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:20:18.770	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:20:08.770	TRACE bridge-metrics Refreshed process metrics: CPU=0.10090817511081696, memory=28810674176
2024-06-20 21:19:58.770	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:19:48.770	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:19:38.769	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:19:28.769	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:19:18.769	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:19:08.769	TRACE bridge-metrics Refreshed process metrics: CPU=0.10078106075525284, memory=28810674176
2024-06-20 21:18:58.768	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:18:48.768	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:18:38.768	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:18:28.768	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:18:18.767	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:18:08.767	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:17:58.767	TRACE bridge-metrics Refreshed process metrics: CPU=0.10101010650396347, memory=28810674176
2024-06-20 21:17:48.767	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:17:38.766	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:17:28.766	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:17:18.766	TRACE bridge-metrics Refreshed process metrics: CPU=0.20156212151050568, memory=28810674176
2024-06-20 21:17:08.765	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:58.765	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:48.765	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:38.765	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:28.764	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:18.764	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:16:08.764	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:15:58.763	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:15:48.763	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:15:38.763	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:15:28.762	TRACE bridge-metrics Refreshed process metrics: CPU=0.10095911473035812, memory=28810674176
2024-06-20 21:15:18.762	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:15:08.762	TRACE bridge-metrics Refreshed process metrics: CPU=0.1013684794306755, memory=28810674176
2024-06-20 21:14:58.762	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:14:48.761	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:14:38.761	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:14:28.761	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:14:18.760	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:14:08.760	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:13:58.760	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:13:48.760	TRACE bridge-metrics Refreshed process metrics: CPU=0.10093363374471664, memory=28810674176
2024-06-20 21:13:38.759	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:13:28.759	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:13:18.759	TRACE bridge-metrics Refreshed process metrics: CPU=0.10118896514177322, memory=28810674176
2024-06-20 21:13:08.758	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:12:58.758	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:12:48.758	TRACE bridge-metrics Refreshed process metrics: CPU=0.10090817511081696, memory=28810674176
2024-06-20 21:12:38.757	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:12:28.757	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:12:18.757	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:12:08.757	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:11:58.756	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:11:48.756	TRACE bridge-metrics Refreshed process metrics: CPU=0.10101010650396347, memory=28810674176
2024-06-20 21:11:38.755	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:11:28.755	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:11:18.755	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:11:08.755	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:10:58.754	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:10:48.754	TRACE bridge-metrics Refreshed process metrics: CPU=0.10118896514177322, memory=28810674176
2024-06-20 21:10:38.754	TRACE bridge-metrics Refreshed process metrics: CPU=0.10108668357133865, memory=28810674176
2024-06-20 21:10:28.753	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:10:18.753	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:10:08.753	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:09:58.752	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:09:48.752	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:09:38.752	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:09:28.761	TRACE bridge-metrics Refreshed process metrics: CPU=0.10118896514177322, memory=28810674176
2024-06-20 21:09:18.751	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:09:08.751	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:08:58.751	TRACE bridge-metrics Refreshed process metrics: CPU=0.10080644488334656, memory=28810674176
2024-06-20 21:08:48.750	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:08:38.750	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:08:28.750	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:08:18.750	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:08:08.749	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:07:58.749	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:07:48.749	TRACE bridge-metrics Refreshed process metrics: CPU=0.10101010650396347, memory=28810674176
2024-06-20 21:07:38.749	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:07:28.748	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:07:18.748	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:07:08.748	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:06:58.748	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:06:48.747	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:06:38.747	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:06:28.747	TRACE bridge-metrics Refreshed process metrics: CPU=0.20202021300792694, memory=28810674176
2024-06-20 21:06:18.746	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:06:08.746	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:58.746	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:48.746	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:38.745	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:28.745	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:18.745	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:05:08.744	TRACE bridge-metrics Refreshed process metrics: CPU=0.10126582533121109, memory=28810674176
2024-06-20 21:04:58.744	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:04:48.744	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:04:38.744	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:04:28.743	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:04:18.743	TRACE bridge-metrics Refreshed process metrics: CPU=0.10095911473035812, memory=28810674176
2024-06-20 21:04:08.743	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:03:58.742	TRACE bridge-metrics Refreshed process metrics: CPU=0.1007302924990654, memory=28810674176
2024-06-20 21:03:48.742	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:03:38.742	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:03:28.742	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:03:18.741	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:03:08.741	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:02:58.741	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:02:48.740	TRACE bridge-metrics Refreshed process metrics: CPU=0.10083185881376266, memory=28810674176
2024-06-20 21:02:38.740	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:02:28.740	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:02:18.739	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:02:08.739	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:01:58.739	TRACE bridge-metrics Refreshed process metrics: CPU=0.10101010650396347, memory=28810674176
2024-06-20 21:01:48.738	TRACE bridge-metrics Refreshed process metrics: CPU=0.2018163502216339, memory=28810674176
2024-06-20 21:01:47.395	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to Westend node at wss://westend-rpc.polkadot.io
2024-06-20 21:01:47.348	DEBUG bridge best headers headers stream of BridgeHubRococo has returned `None`. It may need to be restarted
2024-06-20 21:01:45.077	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to BridgeHubRococo node at wss://rococo-bridge-hub-rpc.polkadot.io
2024-06-20 21:01:45.077	DEBUG bridge best finalized headers headers stream of Westend has returned `None`. It may need to be restarted
2024-06-20 21:01:45.077	DEBUG bridge best finalized headers headers stream of Westend has returned `None`. It may need to be restarted
2024-06-20 21:01:43.638	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to Westend node at wss://westend-rpc.polkadot.io
2024-06-20 21:01:43.638	[Westend_to_BridgeHubRococo_Parachains_1002] DEBUG bridge Restarting relay loop
2024-06-20 21:01:43.638	[Westend_to_BridgeHubRococo_Parachains_1002] WARN bridge Parachains synchronization from Westend to BridgeHubRococo has stalled. Going to restart
2024-06-20 21:01:43.637	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge BridgeHubRococo transaction 0x03710674d027d73d5337cde56eeb2ac5ef3dbbf97c5304fa7bfb58f163e07f3c has been invalidated
2024-06-20 21:01:43.637	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:38.738	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:01:37.609	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:31.597	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:28.738	TRACE bridge-metrics Refreshed process metrics: CPU=0.1011122390627861, memory=28810674176
2024-06-20 21:01:25.587	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:19.576	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:18.738	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:01:13.566	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:08.737	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:01:07.556	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:01:01.539	TRACE bridge Sent transaction to BridgeHubRococo node: 0x03710674d027d73d5337cde56eeb2ac5ef3dbbf97c5304fa7bfb58f163e07f3c
2024-06-20 21:01:01.473	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Submitting Westend parachain ParaId(1002) head update transaction to BridgeHubRococo. Para hash at source relay HeaderId(21349294, 0x86cc6b42a5f79c0f66aac238ddb8941663b0311cb211babe7f7b8ef4c9138708): 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7
2024-06-20 21:01:01.463	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Westend parachain ParaId(1002) needs update at BridgeHubRococo: Available(HeaderId(4181335, 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7)) vs Some(HeaderId(4181270, 0x89fd524927c0a76e785dceae89c7497c798e6e38c6cc9fbced7cec8829862186))
2024-06-20 21:01:01.463		At BridgeHubRococo (HeaderId(3859144, 0xc8354abd31d32bb32d6df711719a7aee8d16c4b9ad2719730eb9ad5ed33cbdd3)): Some(HeaderId(4181270, 0x89fd524927c0a76e785dceae89c7497c798e6e38c6cc9fbced7cec8829862186))
2024-06-20 21:01:01.463		At Westend (HeaderId(21349294, 0x86cc6b42a5f79c0f66aac238ddb8941663b0311cb211babe7f7b8ef4c9138708)): Available(HeaderId(4181335, 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7))
2024-06-20 21:01:01.463	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Checking if Westend parachain ParaId(1002) needs update at BridgeHubRococo:
2024-06-20 21:01:01.463	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_source[ParaId(1002)]': 4181335
2024-06-20 21:01:01.463	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:00:58.737	TRACE bridge-metrics Refreshed process metrics: CPU=0.30349013209342957, memory=28810674176
2024-06-20 21:00:55.449	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Free Westend headers interval at BridgeHubRococo: 5
2024-06-20 21:00:55.378	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to Westend node at wss://westend-rpc.polkadot.io
2024-06-20 21:00:54.906	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to BridgeHubRococo node at wss://rococo-bridge-hub-rpc.polkadot.io
2024-06-20 21:00:54.906	[Westend_to_BridgeHubRococo_Parachains_1002] DEBUG bridge Restarting relay loop
2024-06-20 21:00:54.906	[Westend_to_BridgeHubRococo_Parachains_1002] WARN bridge Failed to read free Westend headers interval at BridgeHubRococo: ChannelError("Background task of BridgeHubRococo client has exited with result: Err(ChannelError(\"Finalized headers subscription for BridgeHubRococo has finished\"))")
2024-06-20 21:00:52.177	DEBUG bridge best finalized headers headers stream of BridgeHubRococo has returned `None`. It may need to be restarted
2024-06-20 21:00:52.172	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Connecting to Westend node at wss://westend-rpc.polkadot.io
2024-06-20 21:00:52.172	[Westend_to_BridgeHubRococo_Parachains_1002] DEBUG bridge Restarting relay loop
2024-06-20 21:00:52.172	  17: clone))) }
2024-06-20 21:00:52.172	  16: start_thread
2024-06-20 21:00:52.172	  15: <unknown>
2024-06-20 21:00:52.172	  14: <unknown>
2024-06-20 21:00:52.172	  13: <unknown>
2024-06-20 21:00:52.172	  12: <unknown>
2024-06-20 21:00:52.172	  11: <unknown>
2024-06-20 21:00:52.172	  10: <unknown>
2024-06-20 21:00:52.172	   9: <unknown>
2024-06-20 21:00:52.172	   8: <unknown>
2024-06-20 21:00:52.172	   7: <unknown>
2024-06-20 21:00:52.172	   6: <unknown>
2024-06-20 21:00:52.172	   5: <unknown>
2024-06-20 21:00:52.172	   4: <unknown>
2024-06-20 21:00:52.172	   3: <unknown>
2024-06-20 21:00:52.172	   2: <unknown>
2024-06-20 21:00:52.172	   1: <unknown>
2024-06-20 21:00:52.172	   0: <unknown>
2024-06-20 21:00:52.172	Stack backtrace:
2024-06-20 21:00:52.172	
2024-06-20 21:00:52.172	    1: Connection reset by peer (os error 104)
2024-06-20 21:00:52.172	    0: i/o error: Connection reset by peer (os error 104)
2024-06-20 21:00:52.172	Caused by:
2024-06-20 21:00:52.172	
2024-06-20 21:00:52.172	[Westend_to_BridgeHubRococo_Parachains_1002] WARN bridge Westend client has failed to return its sync status: FailedToGetSystemHealth { chain: "Westend", error: RpcError(RestartNeeded(Transport(i/o error: Connection reset by peer (os error 104)
2024-06-20 21:00:49.194	DEBUG bridge best headers headers stream of Westend has returned `None`. It may need to be restarted
2024-06-20 21:00:49.194	DEBUG bridge best headers headers stream of Westend has returned `None`. It may need to be restarted
2024-06-20 21:00:48.737	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:00:46.171	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:00:38.915	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:00:38.736	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 21:00:31.619	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:00:28.736	TRACE bridge-metrics Refreshed process metrics: CPU=0.10108668357133865, memory=28810674176
2024-06-20 21:00:24.614	TRACE bridge Sent transaction to BridgeHubRococo node: 0xd0ccce6d29ae7c988a2d2a4aae2387bda7289ea137f43c14452d76ca866d129a
2024-06-20 21:00:21.080	[Westend_to_BridgeHubRococo_Parachains_1002] INFO bridge Submitting Westend parachain ParaId(1002) head update transaction to BridgeHubRococo. Para hash at source relay HeaderId(21349294, 0x86cc6b42a5f79c0f66aac238ddb8941663b0311cb211babe7f7b8ef4c9138708): 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7
2024-06-20 21:00:19.929	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Westend parachain ParaId(1002) needs update at BridgeHubRococo: Available(HeaderId(4181335, 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7)) vs Some(HeaderId(4181270, 0x89fd524927c0a76e785dceae89c7497c798e6e38c6cc9fbced7cec8829862186))
2024-06-20 21:00:19.929		At BridgeHubRococo (HeaderId(3859144, 0xc8354abd31d32bb32d6df711719a7aee8d16c4b9ad2719730eb9ad5ed33cbdd3)): Some(HeaderId(4181270, 0x89fd524927c0a76e785dceae89c7497c798e6e38c6cc9fbced7cec8829862186))
2024-06-20 21:00:19.929		At Westend (HeaderId(21349294, 0x86cc6b42a5f79c0f66aac238ddb8941663b0311cb211babe7f7b8ef4c9138708)): Available(HeaderId(4181335, 0x94a44b8376cc30cba7d6cfa51a5aab1f91bbe776464097faa8907ec6721877c7))
2024-06-20 21:00:19.929	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Checking if Westend parachain ParaId(1002) needs update at BridgeHubRococo:
2024-06-20 21:00:19.929	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_source[ParaId(1002)]': 4181335
2024-06-20 21:00:18.736	TRACE bridge-metrics Refreshed process metrics: CPU=0.10101010650396347, memory=28810674176
2024-06-20 21:00:18.217	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge-metrics Updated value of metric 'best_parachain_block_number_at_target[ParaId(1002)]': 4181270
2024-06-20 21:00:09.883	[Westend_to_BridgeHubRococo_Parachains_1002] TRACE bridge Free Westend headers interval at BridgeHubRococo: 5
2024-06-20 21:00:08.736	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 20:59:58.735	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
2024-06-20 20:59:48.735	TRACE bridge-metrics Refreshed process metrics: CPU=0, memory=28810674176
@bkontur
Copy link
Contributor Author

bkontur commented Jun 25, 2024

after I ran relayer locally, status command started to fail (maybe not related):

bparity@bkontur-ThinkPad-P14s-Gen-2i:~/parity/bridge-troubleshooting$ node main status rococo-westend
2024-06-25 14:43:24        REGISTRY: Unknown signed extensions BridgeRejectObsoleteHeadersAndMessages, OnBridgeHubRococoRefundBridgeHubWestendMessages, OnBridgeHubRococoRefundRococoBulletinMessages, StorageWeightReclaim found, treating them as no-effect
2024-06-25 14:43:24        API/INIT: RPC methods not decorated: chainHead_v1_body, chainHead_v1_call, chainHead_v1_continue, chainHead_v1_follow, chainHead_v1_header, chainHead_v1_stopOperation, chainHead_v1_storage, chainHead_v1_unfollow, chainHead_v1_unpin, transactionWatch_v1_submitAndWatch, transactionWatch_v1_unwatch, transaction_v1_broadcast, transaction_v1_stop
2024-06-25 14:43:24        API/INIT: bridge-hub-rococo/1013000: Not decorating unknown runtime apis: 0xd7bdd8a272ca0d65/1, 0x6ff52ee858e6c5bd/1, 0x91b1c8b16328eb92/1, 0x3ed67502c56fbb7c/1, 0x0ae972bc9baccc81/1, 0x15566834f0a8b169/1, 0xec5372fdb1645189/1, 0xee271bd2dfe3f753/1, 0x17b1e7eadee78096/1, 0x0c7dfd595d10ee14/1, 0x76ee5cebb2fa8fff/1
2024-06-25 14:43:25        REGISTRY: Unknown signed extensions CheckMetadataHash found, treating them as no-effect
2024-06-25 14:43:25        API/INIT: RPC methods not decorated: chainHead_v1_body, chainHead_v1_call, chainHead_v1_continue, chainHead_v1_follow, chainHead_v1_header, chainHead_v1_stopOperation, chainHead_v1_storage, chainHead_v1_unfollow, chainHead_v1_unpin, chainSpec_v1_chainName, chainSpec_v1_genesisHash, chainSpec_v1_properties, transactionWatch_v1_submitAndWatch, transactionWatch_v1_unwatch, transaction_v1_broadcast, transaction_v1_stop
2024-06-25 14:43:25        API/INIT: rococo/1013000: Not decorating runtime apis without matching versions: ParachainHost/11 (1/10/2/3/4/5/6/7/8/9 known)
2024-06-25 14:43:25        API/INIT: rococo/1013000: Not decorating unknown runtime apis: 0x6ff52ee858e6c5bd/1, 0x91b1c8b16328eb92/1
2024-06-25 14:43:25        REGISTRY: Unknown signed extensions BridgeRejectObsoleteHeadersAndMessages, OnBridgeHubWestendRefundBridgeHubRococoMessages, StorageWeightReclaim found, treating them as no-effect
2024-06-25 14:43:25        API/INIT: RPC methods not decorated: chainHead_v1_body, chainHead_v1_call, chainHead_v1_continue, chainHead_v1_follow, chainHead_v1_header, chainHead_v1_stopOperation, chainHead_v1_storage, chainHead_v1_unfollow, chainHead_v1_unpin, transactionWatch_v1_submitAndWatch, transactionWatch_v1_unwatch, transaction_v1_broadcast, transaction_v1_stop
2024-06-25 14:43:25        API/INIT: bridge-hub-westend/1013000: Not decorating unknown runtime apis: 0xd7bdd8a272ca0d65/1, 0x6ff52ee858e6c5bd/1, 0x91b1c8b16328eb92/1, 0x23f8d0d46b9198e3/1, 0xc319038f6660974e/1, 0xcac69fecf89cb17d/1
2024-06-25 14:43:26        REGISTRY: Unknown signed extensions CheckMetadataHash found, treating them as no-effect
2024-06-25 14:43:26        API/INIT: RPC methods not decorated: chainHead_v1_body, chainHead_v1_call, chainHead_v1_continue, chainHead_v1_follow, chainHead_v1_header, chainHead_v1_stopOperation, chainHead_v1_storage, chainHead_v1_unfollow, chainHead_v1_unpin, chainSpec_v1_chainName, chainSpec_v1_genesisHash, chainSpec_v1_properties, transactionWatch_v1_submitAndWatch, transactionWatch_v1_unwatch, transaction_v1_broadcast, transaction_v1_stop
2024-06-25 14:43:26        API/INIT: westend/1013000: Not decorating runtime apis without matching versions: ParachainHost/11 (1/10/2/3/4/5/6/7/8/9 known)
2024-06-25 14:43:26        API/INIT: westend/1013000: Not decorating unknown runtime apis: 0x6ff52ee858e6c5bd/1, 0x91b1c8b16328eb92/1
GRANDPA client of rococo at bridge-hub-westend: 
  Best header number: 11,044,405 of 11,044,412 (missing 7 headers or ~0 sessions)
  Best header hash: 0x44d1ef58bfd4ff3f89d7fef1183050ac6d38fb51fd6c6a76f2718773b487469e (MATCH)
  GRANDPA authorities set id: 18,811 of 18811 (MATCH)
  Recently imported rococo headers:
    11,044,405: 0x44d1ef58bfd4ff3f89d7fef1183050ac6d38fb51fd6c6a76f2718773b487469e
    11,044,399: 0xd360d0c2e8cdea984901641c8689605a30c31043f927baab98183326bee6e305
    11,044,393: 0x820f0ed4ac5ab626c4be8a5f13f2220efac73c6184b27483d64dc45423651d62
    11,044,387: 0xf25589601ba7d37877aa9fbd175b917c3bbdb9b4fb0a23a60efa0e85cb996347
    11,044,381: 0xaec7e910a7e74632f72389ca28f74b30b04fe39323ed1401a26c581ef324a4b4
    11,044,375: 0x73687769889b4a12297d8d7a31270fb7631c2436cee4965f5fdf8977a2d135ce
    11,044,369: 0x62b54f8d34fcdda24218c2c9ba633588e0f61800087a30c9e6dd2cb7b44ea25a
    11,044,362: 0x9fd54b5f70c4d207ff494a7aea5f424352e78b7af41b7aa4776ecae6df513ab2
    11,044,356: 0xaf34774a4132594cd9162038c70dd0aa08bdbba78dea8bbe2732310c541bf021
    11,044,350: 0x3c5fe6327087d3b3ac2422b4dd12c0b848b2699a617db9d4074d933306af024a
GRANDPA client of westend at bridge-hub-rococo: 
  Best header number: 21,417,062 of 21,417,066 (missing 4 headers or ~0 sessions)
  Best header hash: 0x8876a4d49bd387fdce06fcb53a5a8f3aca118d0d63aa46bce2877c153151cdcc (MATCH)
  GRANDPA authorities set id: 7,997 of 7997 (MATCH)
  Recently imported westend headers:
    21,417,062: 0x8876a4d49bd387fdce06fcb53a5a8f3aca118d0d63aa46bce2877c153151cdcc
    21,417,055: 0xa3d2525a2b10f16e901cf3f8233334f4eaad88bd2bd1e79ddd251cd49d3fdafe
    21,417,049: 0x315b32e12cc20a09dad9bafa95cae88bec4ad67d9f715fc8c63ca97a3fbf243d
    21,417,043: 0x8e55c857d6562dae1cd15b7ee7622ea312eb19e5f614c4a4d4b0e874d7fc0467
    21,417,037: 0x3f4a5d0196c431638275defb5a25ced526b4ffaba65ed518294afcff9b604ead
    21,417,031: 0xd3a34044aa40ec9bbe3ba2ede6970bddffb2edc4ce0f48b89c3f86819c1ca9cd
    21,417,025: 0xe62c9c3968ca7eff45f39fd71b27cd572294c96cd98b78482dee96098123f7ad
    21,417,019: 0x99939fd6da23f5492d1040680c42c0ac956db79a459386dfaab06cb69c8b003f
    21,417,010: 0x9f0d1048b428eb7f66c28ee7431a51f80507a086baa40c6f4c263934536070bb
    21,417,004: 0x73b8e4ae5a80d349b37fc3cbe114ecfc087deae7cc29e94cf4ed012247fd13d7
Parachain client of bridge-hub-rococo at bridge-hub-westend: 
  Best header number: 3,899,566 of 3,899,576
  Best header hash: 0xd288391c4b0e17e76ca022f0a25d6d32d2e2982dba4a0ef85fb26e8e255c94cb (MATCH)
  Recently imported bridge-hub-rococo headers:
    3,899,566: 0xd288391c4b0e17e76ca022f0a25d6d32d2e2982dba4a0ef85fb26e8e255c94cb
    3,899,560: 0xeb1b299703e952a0788defd5ad17eb820f5c48df78d2ded923c36ba3134566d1
    3,899,554: 0xa231a60656918508eaebfaf7ad1f7be4cbee0ab1e7f80f82cb755a6332575f86
    3,899,548: 0x28871d1e9653007fd33b322ff4683730c4b0c30f46d651f532b0c8402bdd298a
    3,899,542: 0x0776f5ac2e060248590e02e137dd59d55bb4ffebaff7ca1b298d5376539fa467
    3,899,536: 0x5046253a30c2ffb95710bbb73826e6d1187cbfaa16c588c105e598803ccf67dd
    3,899,530: 0xecc0dba54e8802a6c7443a40e666fd6935d54c9895785d8c50a884fae03206cf
    3,899,527: 0x261dd3f5e520e23cbd7b56b526cd97f0c7400cfa2da9f298e7ccabeb202da080
    3,899,521: 0xaae67d4d56fb52bf11242e82a63aa87e621bd6a8135fd61dbb0601bdc7cc4e5d
    3,899,515: 0xeac7728f26b4c0c59357001afd02569e983a2da0120405b55a9134ad6db50577
Parachain client of bridge-hub-westend at bridge-hub-rococo: 
  Best header number: 4,241,993 of 4,242,001
  Best header hash: 0xb89942565bc8efd2c163882a26329f86aab8f29c912b67243e0c2892a217dea7 (MATCH)
  Recently imported bridge-hub-westend headers:
    4,241,993: 0xb89942565bc8efd2c163882a26329f86aab8f29c912b67243e0c2892a217dea7
    4,241,986: 0x574f958f462ccc33600ec71dffc0a01507e23cae362df259dc86808372189db3
    4,241,980: 0x99b901664588f659b1a589664b8a5af6e682534c3e21b6448c3e954ebf01737e
    4,241,974: 0x06313759a72bf2e395c8d75f0281a0b20f19726acb5c38630de55554151b431a
2024-06-25 14:43:32             VEC: Unable to decode on index 0 createType(Lookup4):: Number needs to be an integer <= Number.MAX_SAFE_INTEGER, i.e. 2 ^ 53 - 1
2024-06-25 14:43:32        RPC-CORE: queryStorageAt(keys: Vec<StorageKey>, at?: BlockHash): Vec<StorageChangeSet>:: createType(Vec<StorageKey>):: createType(Lookup4):: Number needs to be an integer <= Number.MAX_SAFE_INTEGER, i.e. 2 ^ 53 - 1
Error occured: {} Error: createType(Vec<StorageKey>):: createType(Lookup4):: Number needs to be an integer <= Number.MAX_SAFE_INTEGER, i.e. 2 ^ 53 - 1
    at createTypeUnsafe (/home/bparity/parity/bridge-troubleshooting/node_modules/@polkadot/types-create/cjs/create/type.js:54:22)
    at TypeRegistry.createTypeUnsafe (/home/bparity/parity/bridge-troubleshooting/node_modules/@polkadot/types/cjs/create/registry.js:230:52)
    at RpcCore._formatParams (/home/bparity/parity/bridge-troubleshooting/node_modules/@polkadot/rpc-core/cjs/bundle.js:300:34)
    at callWithRegistry (/home/bparity/parity/bridge-troubleshooting/node_modules/@polkadot/rpc-core/cjs/bundle.js:187:33)
    at Observable._subscribe (/home/bparity/parity/bridge-troubleshooting/node_modules/@polkadot/rpc-core/cjs/bundle.js:195:17)
    at Observable._trySubscribe (/home/bparity/parity/bridge-troubleshooting/node_modules/rxjs/dist/cjs/internal/Observable.js:41:25)
    at /home/bparity/parity/bridge-troubleshooting/node_modules/rxjs/dist/cjs/internal/Observable.js:35:31
    at Object.errorContext (/home/bparity/parity/bridge-troubleshooting/node_modules/rxjs/dist/cjs/internal/util/errorContext.js:22:9)
    at Observable.subscribe (/home/bparity/parity/bridge-troubleshooting/node_modules/rxjs/dist/cjs/internal/Observable.js:26:24)
    at ConnectableObservable.connect (/home/bparity/parity/bridge-troubleshooting/node_modules/rxjs/dist/cjs/internal/observable/ConnectableObservable.js:60:40)

@bkontur
Copy link
Contributor Author

bkontur commented Jun 25, 2024

also we had to restart parachains-b relayer manually

@acatangiu acatangiu moved this to Todo in Bridges + XCM Jun 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
Bridges + XCM
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bkontur