re-evaluate effectiveness of broadcast retries triggered by replay/propagated-stats

[behzadnouri]

Problem

Based on PropagatedStats, replay code may trigger broadcast retries for its leader slots:
https://github.com/solana-labs/solana/blob/ae7803a55/core/src/progress_map.rs#L192
https://github.com/solana-labs/solana/blob/ae7803a55/core/src/replay_stage.rs#L927-L931
https://github.com/solana-labs/solana/blob/ae7803a55/core/src/replay_stage.rs#L1747-L1754

The downside is that this can exacerbate network traffic because of duplicate shreds and the fact that there will be two nodes simultaneously broadcasting shreds from their respective leader slots, which then might even further hurt shreds propagation for the following slots.

The alternative is to just let the cluster skip the slot.
Similar issue with repair: #28637

Proposed Solution

• Evaluate if these broadcast retries are effective/helpful.
  • e.g. one metric could be that how often a slot which was retried ultimately got rooted; or was it skipped anyways. Retried slots are tracked by: https://github.com/solana-labs/solana/blob/b77e12b3a/core/src/replay_stage.rs#L1068-L1073
  • or, how often retrying a slot causes more instability for the following slots; e.g. in terms of repair metrics for those.
  • or, we can run an a/b testing experiment with these retries disabled and then compare.
• If there is not enough evidence of retries helping the cluster overall then remove this code.

cc @carllin @jbiseda @bw-solana

[behzadnouri]

From discord:
https://discord.com/channels/428295358100013066/439194979856809985/1071195371402964992

I think this broadcast retry was introduced to handle long partitions where you have multiple forks, say 10 forks each with 10% stake. Then per the design, this retry would:

1. Prevent the individual partitions from adding more blocks before the network connections healed (b/c they would stop making new blocks in favor of retransmitting their old ones)
2. After the network healed, leaders would rebroadcast which would allow for faster resolution of the partitions than relying only on repair

@carllin
I think we can temporarily pause block production regardless of retrying old shreds or not (i.e that sounds orthogonal). That being said

• If my node pauses block production during its leader slots, what prevents the next leader from generating ticks and skipping my leader slots anyways? iow wouldn't this require even more coordination between nodes and make things even more unstable if they are not in sync?

• What is the best way to measure if retrying broadcast is effective in achieving this?

  faster resolution of the partitions than relying only on repair

  One issue is that what if half the cluster is doing one thing and the other half another thing? For example, my node is retrying its old shreds while the current leader is also generating shreds, wouldn't that 2x shreds traffic and make things worse?

Also should note that current turbine is less subject to network partitions in the sense that staked nodes placement on the randomly shuffled broadcast tree is the same across nodes, even if some are not visible in gossip. Also with 32:32 erasure batches theoretically at least there should be more resilience to network partitions.

[carllin]

If my node pauses block production during its leader slots, what prevents the next leader from generating ticks and >skipping my leader slots anyways? iow wouldn't this require even more coordination between nodes and make things even >more unstable if they are not in sync?

Nothing :), this was designed assuming everybody would halt block production during a major partition

What is the best way to measure if retrying broadcast is effective in achieving this?

I think measuring if a retried slot gets confirmed is probably the best way

One issue is that what if half the cluster is doing one thing and the other half another thing? For example, my node is retrying its old shreds while the current leader is also generating shreds, wouldn't that 2x shreds traffic and make things worse?

The idea was that because the leader is skipping its new slot in favor of rebroadcasting the old one, the bandwidth usage shouldn't be too much worse, but I can see how the traffic could be an issue.

Also should note that current turbine is less subject to network partitions in the sense that staked nodes placement on the randomly shuffled broadcast tree is the same across nodes, even if some are not visible in gossip. Also with 32:32 erasure batches theoretically at least there should be more resilience to network partitions.

Yeah this is a fair point, but what would happen during a long running 66-33 partition? I"m guessing each side of the partition would still see their own blocks because repairing from the leader makes up for the holes in turbine?

Then when the partition resolves, the smaller side would repair the blocks made by the heavier side? Should we have an alternative fast/large repair protocol for this? Regular repair might take a while if there are a lot of blocks on the heavy side that didn't stop block production. I think if such a protocol existed we could get rid of the rebroadcast logic.

[behzadnouri]

The idea was that because the leader is skipping its new slot in favor of rebroadcasting the old one, the bandwidth usage shouldn't be too much worse

But there are 2 different leaders. no?
I mean while my node is retrying its old shreds, another node is the current leader and they might be broadcasting shreds for their leader slot. So 2x shreds traffic.

what would happen during a long running 66-33 partition? I"m guessing each side of the partition would still see their own blocks because repairing from the leader makes up for the holes in turbine?

I think this is where repair can be improved. For one thing, nodes shouldn't be repairing from the slot leader because that would make it easy for a malicious leader to create partitions in the cluster. Related issue:

• Debug slots repaired ~100% by majority of the cluster #28637

Should we have an alternative fast/large repair protocol for this?

Might be a bit tricky security-wise if the slot is not rooted yet. I mean if a node gets all shreds of a certain slot from few nodes then again might make it easier to create partitions.

[bw-solana]

The idea was that because the leader is skipping its new slot in favor of rebroadcasting the old one, the bandwidth usage shouldn't be too much worse

But there are 2 different leaders. no? I mean while my node is retrying its old shreds, another node is the current leader and they might be broadcasting shreds for their leader slot. So 2x shreds traffic.

There are two places where we will signal to retransmit shreds:

1. Near the end of replay stage
2. As part of checking if we should start leader

I suspect 1 is where the bulk of the retransmit signal occurrences would be happening. In this case, I believe @behzadnouri is right that we would see shreds for multiple blocks from different leaders transmitting over turbine concurrently. I'm guessing 2 is the case that @carllin is referring to, which wouldn't exacerbate network traffic because this node would be "stealing" its own leader slot

[bw-solana]

Created draft PR #30155 for adding retransmit slot metrics. It's pretty ugly, but I can clean it up if this is somewhat what we're going for

[bw-solana]

So far, looking back 24 hours, I'm not seeing any evidence that retransmitting is helping.

Looking at metrics on testnet:

• No reporting of replay_stage-retransmit metrics by any node
• replay_stage-retransmit-timing-based entries for node td3n5NGhP7JKWrL638gzau3NY7mF4K3ztZww3GkpywJ for slots 173,756,697 & 173,756,698. There are >5k retry iterations spanning multiple days. Slot 173,756,695 was confirmed back in mid-Jan.

I mean... time to give up my guy

Looking at metrics on MNB:

• No reporting of replay_stage-retransmit metrics by any node
• BYHuaEKgW3KsvGLeydrs9cuEkfDjUGmLZxwjC1FWdQtR 176097752-5. >260 retry iterations spanning multiple hours. None of these slots were confirmed.
• 6NmKxzGAdJ9ttewC8hwJEFaJdn4GScjpqimaL5BnbWgx 176309096-9. 18 retries. None of these slots were confirmed.
• 4RLcStbSkt5S1Xm4mStup6N13PGyAfAWy7Vs5d7yJRnY 176354880-3. 9 retries. None of these slots were confirmed.
• 5LGN129TL3TsMU7NkUsAZJvxyNHwFAXhNYy3Z7WUsbr1 176361692-5 36 retries. None of these slots were confirmed.

[carllin]

There are two places where we will signal to retransmit shreds:
Near the end of replay stage

Ah yeah, thanks for clarifying

think this is where repair can be improved. For one thing, nodes shouldn't be repairing from the slot leader because that would make it easy for a malicious leader to create partitions in the cluster. Related issue:

@behzadnouri this makes sense

I think we can temporarily pause block production regardless of retrying old shreds or not (i.e that sounds orthogonal).

Ok, I think we can remove the rebroadcast logic but keep the block production pausing?

I think we can have a separate design discussion for efficient repair after a long partition. Even with block production paused PoH is still running so there will be a lot of ticks, we'll probably need tick compression @bw-solana 😃

[behzadnouri]

From the metrics on mainnet right now, effectively every slot being retried only a few milliseconds after the initial broadcast which might indicate there is a bug somewhere.

[behzadnouri]

Seems like this .unwrap_or(true) is the bug:
https://github.com/solana-labs/solana/blob/94ef881de/core/src/progress_map.rs#L178
It always returns true the first time around because retry_time is None.
So every slot is broadcasted twice.

#30681 to patch this.

[github-actions]

This repository is no longer in use. Please re-open this issue in the agave repo: https://github.com/anza-xyz/agave