Add metrics for RPC

[xemul]

RPC metrics are added in 5.4 with the scylladb/scylladb@0c69a31 seastar update and are enhanced (see below) with the scylladb/scylladb#15785 merge. In ent. it's going to be 2024.1

The metrics include

• scylla_rpc_client_count -- gauge showing total number of connections
• scylla_rpc_client_sent_messages -- counter with total number of messages sent
• scylla_rpc_client_replied -- counter with total number of responses received. This is less-or-equal than the above, because some messages-sent can be one-way calls not asking for the response or can result in exception or timeout (there's metrics for that too)
• scylla_rpc_client_exception_received -- counter with total number of exceptional replies
• scylla_rpc_client_timeout -- counter with total number of request timeouts
• scylla_rpc_client_pending -- gauge with the number of requests queued for sending, but not yet sent
• scylla_rpc_client_wait_reply -- gauge with the number of requests waiting for the reply

The metrics are labeled with "domain" and "shard" values, each domain should have its own set of plots on the dashboard. Domains are dynamic.

[amnonh]

@xemul I tried to test it with 5.4-rc1 didn't see those metrics

[xemul]

If you're testing it with a single scylla instance, there will be no RPC connections and no such metrics

[amnonh]

@xemul I've tested it with a 3 nodes cluster

[xemul]

Hm... I tested with two scylla processes launched by hand. Can you give me the IPs of those nodes so I could check?

[xemul]

Ah, wait. scylla-5.4.0-rc1 doesn't have this seastar update (yet?)

[xemul]

I guess 5.4's seastar is not going to be just merged from master, so it's going to be 5.5/6.0 then

[amnonh]

postponding until we'll have a version with those metrics

[amnonh]

@xemul is this part of 2024.1?

[xemul]

Nope :(

[amnonh]

@xemul, did it make it to 6.0?

[xemul]

Yes

[amnonh]

@xemul, @mykaul who is going to use those metrics and how?
Those are 7 additional panels, I collected data from the compression test and can use that to add the panels, but the question, should we?

[amnonh]

@xemul I'm assigning it to you to get your input, you don't need to add any code

[xemul]

From my perspective it's for Advanced dashboard that includes IO-queue metrics and CPU-scheduler ones

[amnonh]

I'm worried that we just add more and more metrics, the dashboard will work slower and slower and in the end, it's not useful

[mykaul]

Is the content of scylladb/seastar#2293 included here, or do we need a separate issue for it? (rpc DELAY metrics) ?

[amnonh]

Is the content of scylladb/seastar#2293 included here, or do we need a separate issue for it? (rpc DELAY metrics) ?

It's not, it will require a new issue with what version it's part of and some additional information and samples