Telemetry for local payments (incoming to outgoing within the same Rafiki instance)

[beniaminmunteanu]

reference to an older discussion that was on the telemetry PR:

Telemetry has 2 environments, named TESTNET and LIVENET. Rafiki.money is currently sending metrics to the TESTNET infra.
However, transactions that represent actual amounts/values that are flowing through the network are disregarded, as the implementation does not consider local payments as a means to observe the network's growth (which is the telemetry goal)

As such, rafiki.money and any other deployment that would have local payments only track transactions count at the moment.

Looking to clarify and continue the discussion if local payments is something we want to track, and also get some ideas on the changes required.

[mkurapov]

Thinking about this, if the goal of the project is to only track the activity of the network we should not count "local" payments at all, which includes changing how we transactions_total to ignore local payments.

This means the actual telemetry tracking should probably live primarily within the ILP connector, if we want to track ILP network traffic. This way, we could track the number of individual packets being sent as well, if we want to.

[beniaminmunteanu]

Yes, you are right. There shouldn't be a difference between counts and amounts, the question is if we track local payments or not.
I'm guessing since TESTNET IS a requirement, we should track all of it.
The "insight" will be how much do developers interact with the TESTNET through playground+rafiki.money.

However, i'd still look for a way to only collect at sending/receiving (sending preffered) time, and removing that PEER conditional would also imply not being able to make that distinction, would need another solution for it, unless we go for de-duplicating requests which would be last resort.

A solution we previously had for collection was the ILP connector, where we could do that distinction and only react on sending side.
However, that still needs to be discussed in terms of multiple payment methods, see my comment below

Another simple solution would be just leaving the amount metrics as duplicates, but halving the actual value that they hold.
It would then be doubled back in the average transaction graph, which would be affected

[mkurapov]

and removing that PEER conditional would also imply not being able to make that distinction

What do you mean by this exactly?

We can also just have xxx_metric_sent and xxx_metric_received if we want, to track from both ends without worrying about picking one side or another

[beniaminmunteanu]

oh i meant if we keep the collection in the same spot, and just remove the PEER property check.
Where would xxx_metric_sent and xxx_metric_received be captured? both in accouting?

[mkurapov]

I think we should also be pretty liberal with adding metrics, and we can always differentiate between local & network payments, ie:
network transactions:
total_amount_peer_to_peer_ilp_transactions_received
total_value_peer_to_peer_transactions_received

local transactions:
total_amount_peer_to_peer_ilp_transactions_received
total_amount_local_ilp_transactions_recevied

Likewise, we can later (if we want) think about tracking the amount of Open Payments transactions, ie how many incoming payments were created, how many outgoing payments etc. Those should be tracked in a separate place, away from the ILP connector.

[mkurapov]

@beniaminmunteanu can you remind me about the issues of adding telemetry inside the ILP connector middleware? For example, if we add a middleware for telemetry operations after createBalanceMiddleware and check that the packet was fulfilled, we can differentiate between local and network payments by checking the source/incoming account. If it is a peer -> it is a network payment, if it is an incoming payment -> it is a local payment.

[beniaminmunteanu]

In the same discussion that I linked from the telemetry PR :
#2299 (comment)

@sabineschaller raised the issue of collecting regardless of payment method, and as such going back to the accounting service instead of the ILP connector, which was the collection point then.

[mkurapov]

Right, should have added that. I think @sabineschaller 's comment made a lot of sense when it came to multiple payment methods, but (after multiple discussions) we won't be adding multiple payment methods in the near future, and likely never in this repo specifically.

[beniaminmunteanu]

Oh, well in that case, if you agree with it, I would move the collection in ILP connector again 🤡

[sabineschaller]

Let's add it to the connector. I'm not even sure my comment made sense back then, just because we want to measure how the ILP network is growing, not the adoption of Rafiki.

In terms of local vs. network, I agree with @mkurapov, let's just try to capture both. But let's be consistent regarding amounts and counts.

[beniaminmunteanu]

Thanks!

Created #2531, and added it to my plate. Feel free to re-assign if you consider so.