Reducing missed mempool tx's between ecosystem entities

[CharlieC3]

Context

When a transaction (tx) is submitted to a stacks-node, it can take some time before the tx propagates to a neighboring node which broadcasts the event to a public stacks-blockchain-api (API). Depending on the topography of the blockchain, this event can make its way to an API in anywhere from a couple seconds to over an hour in the worst cases. This can cause mempool tx's to not appear in any service or client which ingests mempool data from an API.

There's currently a couple different solutions available to the public that Hiro utilizes to help reduce the number of missing or delayed mempool tx's that appear in the stacks-blockchain-api's (API) database.

1. With the latest version of the API (0.68.0), the API can now multicast a tx it received to other internal stacks-nodes which they then re-broadcast the event to other API instances. This helps keep mempool data in-sync across internal API databases when an entity is load-balancing multiple API instances to serve traffic.
2. The mempool-pusher script can be ran alongside a stacks-node to push all tx's it receives to another stacks-node or set of stacks-nodes. However its usefulness is limited, as it's susceptible to the same issue of missing or delayed mempool tx's.

Problem

The above solutions would help a single entity propagate mempool tx's within their own infrastructure, but if one entity hosts an API instance and receives a tx, other entities running their own API may not receive the mempool tx in a timely manner and would be susceptible to the issue described in the context above.

As decentralization continues to expand, this problem where an entity may be missing mempool tx's from other entities will likely get worse.

Possible Solution

One idea I've had to mitigate this problem was for an entity to broadcast tx's received to other known entities that opt-in to receiving tx's from external sources. For example, if an exchange or partner submitted a tx to an API they host, it would then broadcast the same tx to a list of public endpoints that Hiro and others entities own. Entities receiving the broadcast would then be able to distribute the tx as needed to their internal infrastructure.

The list of entities to receive this broadcast would be a publicly-curated list, perhaps in Github, and would be opt-in. So entities will not receive the broadcast unless they add themselves to the list. The list could be a json or yaml file which simply lists entity names and a public endpoint they own.

This feature could likely be done by the API with some code changes to the current multicast logic. It could also likely be modified to accept tx's as well which would then be forwarded to backend stacks-nodes internally.

It's possible this is something that can be solved with another solution, or perhaps with changes to the stacks-node, but I'm not sure of their feasibility or what those would look like.

[CharlieC3]

Thoughts? @kantai @zone117x @rafaelcr @lgalabru @wileyj @diwakergupta @agraebe

[CharlieC3]

Is this the issue you're referring to? If the problem described above can be fixed by resolving an issue in the stacks-node, I agree that I'd much rather have the issue resolved than tack on something additional to how the API works.

[kantai]

Yep, that's the one.

[CharlieC3]

@kantai Are there any plans to work on that issue in Q4? If not, can we?

[kantai]

It was discussed in the last blockchain meeting, I believe that the Stacks Foundation is planning to begin working on this problem immediately after a prototype of app chains.

[CharlieC3]

Thanks, that's good to know. Since this problem will exacerbate as decentralization expands, I assume this would also be in Hiro's interest to contribute to resolving 2193 if that meant it would be resolved faster. Though I'm not sure what it's priority would be nor if it would qualify for Q4.

I'll defer that to you and others to determine.

cc @diwakergupta @agraebe

[rafaelcr]

@CharlieC3 Could there be a scenario in which a lot of overhead (and unnecessary) traffic is created by either the API multicast or the mempool pusher script?

If a node that receives a tx can then rebroadcast that same transaction to other nodes, how can we control that we really need to rebroadcast that tx? So for example if the last node received a tx from an API multicast, it would then rebroadcast that same tx to the rest of the nodes which have already received that tx.

[CharlieC3]

Assuming you're referring to an API instance and not a stacks-node. Not sure what the best option would be, but I think a simple header could help make the distinction.

So when an API multicasts a tx to external entities, it could add a header to the HTTP POST. The API on the receiving end would have some logic to understand how the incoming tx should be multicasted. If the header is included, the tx would only be multicasted to internal stacks-nodes (like 0.68.0 does today). If there's no header, it'll be multicasted to internal stacks-nodes and external entities.