check_solr_metrics.pl does not return correct list of keys for category QUERY in Solr 7.6

[azandar-eur]

The command does not return all the list of categories, for example using "-A QUERY -K /select" you can't have the typical "15minRateRequestsPerSecond" and 5minRateRequestsPerSecond", while those data are available via an api call.

Example with Solr 7.6.0:
/etc/nagios-plugins/check_solr_metrics.pl -H solr-prod.eanadev.org -P 9292 -C search-prod -A QUERY -K /select

OK: Solr query /select GET_FACETS=1032, GET_FIELDS=1768, GET_HIGHLIGHTS=3, GET_TOP_IDS=4993, count=0, handlerStart=1603107446416, meanRate=0.0478526278593185, requests=8400, total=6759, totalTime=290902431152, query time 69ms, QTime 0ms | '/select GET_FACETS'=1032 '/select GET_FIELDS'=1768 '/select GET_HIGHLIGHTS'=3 '/select GET_TOP_IDS'=4993 '/select count'=0 '/select handlerStart'=1603107446416 '/select meanRate'=0.0478526278593185 '/select requests'=8400 '/select total'=6759 '/select totalTime'=290902431152 query_time=69ms query_QTime=0ms

Example with Solr 6.6.5:
/etc/nagios-plugins/check_solr_metrics.pl -H solr-prod.eanadev.org -P 9191 -C search-prod -A QUERY -K /select
OK: Solr query /select 15minRateRequestsPerSecond=0.659448245268916, 5minRateRequestsPerSecond=0.0956139830903445, 75thPcRequestTime=0.155083, 95thPcRequestTime=0.401197, 999thPcRequestTime=13.208008, 99thPcRequestTime=0.401197, avgRequestsPerSecond=0.158742181164243, avgTimePerRequest=0.243951540508014, clientErrors=132, errors=132, handlerStart=1603005203013, medianRequestTime=0.155083, requests=44024, serverErrors=0, timeouts=0, totalTime=2880087.795976, query time 71ms, QTime 0ms | '/select 15minRateRequestsPerSecond'=0.659448245268916 '/select 5minRateRequestsPerSecond'=0.0956139830903445 '/select 75thPcRequestTime'=0.155083 '/select 95thPcRequestTime'=0.401197 '/select 999thPcRequestTime'=13.208008 '/select 99thPcRequestTime'=0.401197 '/select avgRequestsPerSecond'=0.158742181164243 '/select avgTimePerRequest'=0.243951540508014 '/select clientErrors'=132 '/select errors'=132 '/select handlerStart'=1603005203013 '/select medianRequestTime'=0.155083 '/select requests'=44024 '/select serverErrors'=0 '/select timeouts'=0 '/select totalTime'=2880087.795976 query_time=71ms query_QTime=0ms

cURL command contains the metrics:
Example with Solr 7.6.0:
curl 'http://localhost:9292/solr/admin/metrics?group=core&prefix=QUERY./select.requestTimes'
{
"responseHeader":{
"status":0,
"QTime":1},
"metrics":{
"solr.core.search-prod.shard1.replica_n1":{
"QUERY./select.requestTimes":{
"count":49,
"meanRate":4.8681480619888593E-4,
"1minRate":2.940655617036234E-12,
"5minRate":3.7340293216305956E-5,
"15minRate":2.486123505929501E-4,
"min_ms":87.782058,
"max_ms":87.782058,
"mean_ms":87.782058,
"median_ms":87.782058,
"stddev_ms":0.0,
"p75_ms":87.782058,
"p95_ms":87.782058,
"p99_ms":87.782058,
"p999_ms":87.782058}},
"solr.core.search-prod.shard2.replica_n2":{
"QUERY./select.requestTimes":{
"count":50,
"meanRate":4.967498360984714E-4,
"1minRate":5.881311234072468E-12,
"5minRate":7.468058643261191E-5,
"15minRate":4.972247011859002E-4,
"min_ms":3.940434,
"max_ms":117.006585,
"mean_ms":60.4735095,
"median_ms":117.006585,
"stddev_ms":56.5330755,
"p75_ms":117.006585,
"p95_ms":117.006585,
"p99_ms":117.006585,
"p999_ms":117.006585}},
"solr.core.search-prod.shard3.replica_n4":{
"QUERY./select.requestTimes":{
"count":50,
"meanRate":4.967497791390166E-4,
"1minRate":2.940655617036234E-12,
"5minRate":3.7340293216305956E-5,
"15minRate":2.486123505929501E-4,
"min_ms":3.954119,
"max_ms":3.954119,
"mean_ms":3.954119,
"median_ms":3.954119,
"stddev_ms":0.0,
"p75_ms":3.954119,
"p95_ms":3.954119,
"p99_ms":3.954119,
"p999_ms":3.954119}}}}

[HariSekhon]

Could you please post the full -v -v -v debug level output?

[azandar-eur]

/etc/nagios-plugins/check_solr_metrics.pl -H solr-prod.eanadev.org -P 9292 -C search-prod -A QUERY -K /select -vvv
verbose mode on

check_solr_metrics.pl version 0.6.0 => Hari Sekhon Utils version 1.19.6

host: solr-prod.eanadev.org
port: 9292
collection: search-prod
category: QUERY
http context: /solr

setting timeout to 10 secs

querying Solr
HTTP GET http://solr-prod.eanadev.org:9292/solr/search-prod/admin/mbeans?stats=true&cat=QUERY&key=%2Fselect&omitHeader=off&wt=json&start=0&rows=3&indent=true
returned HTML:

{
"responseHeader":{
"status":0,
"QTime":0},
"solr-mbeans":[
"QUERY",{
"/select":{
"class":"org.apache.solr.handler.component.AliasingSearchHandler",
"description":"Expands keyword arguments and pseudofields into Solr-parseable queries",
"stats":{
"QUERY./select.handlerStart":1603107446416,
"QUERY./select.clientErrors.count":0,
"QUERY./select.serverErrors.count":0,
"QUERY./select.shardRequests.GET_FACETS":1359,
"QUERY./select.shardRequests.GET_TOP_IDS":6769,
"QUERY./select.shardRequests.GET_FIELDS":2429,
"QUERY./select.totalTime":422606769496,
"QUERY./select.errors.count":0,
"QUERY./select.shardRequests.GET_HIGHLIGHTS":3,
"QUERY./select.requestTimes.meanRate":0.047231588224149475,
"QUERY./select.timeouts.count":0,
"QUERY./select.shardRequests.total":9196,
"QUERY./select.requests":11427}}}]}

http status code: 200
http status message: OK

parsing output from Solr

/select => count = 0
/select => count = 0
/select => handlerStart = 1603107446416
/select => meanRate = 0.0472315882241495
/select => requests = 11427
/select => count = 0
/select => GET_FACETS = 1359
/select => GET_FIELDS = 2429
/select => GET_HIGHLIGHTS = 3
/select => GET_TOP_IDS = 6769
/select => total = 9196
/select => count = 0
/select => totalTime = 422606769496
10 stats collected

OK: Solr query /select GET_FACETS=1359, GET_FIELDS=2429, GET_HIGHLIGHTS=3, GET_TOP_IDS=6769, count=0, handlerStart=1603107446416, meanRate=0.0472315882241495, requests=11427, total=9196, totalTime=422606769496, query time 71ms, QTime 0ms | '/select GET_FACETS'=1359 '/select GET_FIELDS'=2429 '/select GET_HIGHLIGHTS'=3 '/select GET_TOP_IDS'=6769 '/select count'=0 '/select handlerStart'=1603107446416 '/select meanRate'=0.0472315882241495 '/select requests'=11427 '/select total'=9196 '/select totalTime'=422606769496 query_time=71ms query_QTime=0ms

[HariSekhon]

If you look at the response from Solr that is what it is returning - there is no "15minRateRequestsPerSecond" result in that query returned by Solr:

{
"responseHeader":{
"status":0,
"QTime":0},
"solr-mbeans":[
"QUERY",{
"/select":{
"class":"org.apache.solr.handler.component.AliasingSearchHandler",
"description":"Expands keyword arguments and pseudofields into Solr-parseable queries",
"stats":{
"QUERY./select.handlerStart":1603107446416,
"QUERY./select.clientErrors.count":0,
"QUERY./select.serverErrors.count":0,
"QUERY./select.shardRequests.GET_FACETS":1359,
"QUERY./select.shardRequests.GET_TOP_IDS":6769,
"QUERY./select.shardRequests.GET_FIELDS":2429,
"QUERY./select.totalTime":422606769496,
"QUERY./select.errors.count":0,
"QUERY./select.shardRequests.GET_HIGHLIGHTS":3,
"QUERY./select.requestTimes.meanRate":0.047231588224149475,
"QUERY./select.timeouts.count":0,
"QUERY./select.shardRequests.total":9196,
"QUERY./select.requests":11427}}}]}

Your curl query and the query being sent by the plugin are different, try changing your query parameters.

Your curl:

curl 'http://localhost:9292/solr/admin/metrics?group=core&prefix=QUERY./select.requestTimes'

what the plugin is sending:

http://localhost:9292/solr/search-prod/admin/mbeans?stats=true&cat=QUERY&key=%2Fselect&omitHeader=off&wt=json&start=0&rows=3&indent=true

The plugin returns a specific key, whereas your curl is doing a prefix search, maybe they've changed the mbeans between solr versions and you need to specify the new key name in this version.

[azandar-eur]

Hi, the problem is that in solr 7.x the key is different and is not listed in within the category.

In Solr 6.X for category "QUERY" the key was "/select"
https://lucene.apache.org/solr/guide/6_6/performance-statistics-reference.html

In solr 7.X for category "QUERY" the key is "/select.requestTimes" and this key is not listed within the available ones
https://lucene.apache.org/solr/guide/7_0/performance-statistics-reference.html

[HariSekhon]

That's what I thought. I think this isn't a matter of the code, but just tuning your CLI options to query the new key instead if you're using Solr 7?

[azandar-eur]

The new key is not available, this is the issue.
For category QUERY the old key was "/select" the new key is "/select.requestTimes" and this key is not displayed when doing --list-keys.

The point is that the metric "15minRateRequestsPerSecond" is officially available on both Solr 6 and Solr 7 version but the plugin can't get it in Solr 7