Delay Mechanism for Retrying L1 Requests

[wojciechos]

Currently, Juno retries failed L1 requests immediately, causing potential spamming of l1 node. We need to introduce a delay between retries to manage load and improve system stability.

Sample logs - ~40 request in on sec

21:24:35.993 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.968 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.942 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.916 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.891 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.866 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.840 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.811 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.783 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.756 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.731 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.706 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.681 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.655 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.628 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.603 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.577 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.551 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.526 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.501 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.476 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.451 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.426 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.397 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.372 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.346 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.318 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.291 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.266 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.240 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.215 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.190 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.164 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.139 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.111 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.086 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.060 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.035 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}
21:24:35.009 21/04/2024 +00:00	DEBUG	l1/l1.go:180	Failed to retrieve L1 finalised height, retrying...	{"error": "get finalised Ethereum block: daily request count exceeded, request rate limited"}