[Question] carbonapi /metrics/find result is different from Graphite webapp

[cynici]

Describe the bug

For exactly the same query, the "id" value in /metrics/find results are different between carbonapi and Graphite webapp, e.g.

"id": "agents" vs "id": "carbon.agents"

Shall be most grateful for any pointers.

This is my setup:

• go-carbon on 2 servers carbonserver-1, carbonserver-2
• Both carbonapi and webapp run on the same server
• carbonapi uses broadcast method to query both carbonservers
• Graphite webapp CLUSTER_SERVERS = carbonapi:8081

carbonapi's version
0.15.4 from stable repo

Does this happened before
This is my first time trying out carbonapi as a possible replacement for Graphite webapp

carbonapi's config

listen: "127.0.0.1:8081"
prefix: ""
useCachingDNSResolver: false
cachingDNSRefreshTime: "1m"
expvar:
  enabled: true
  pprofEnabled: false
  listen: ""
headersToPass:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
headersToLog:
  - "X-Dashboard-Id"
  - "X-Grafana-Org-Id"
  - "X-Panel-Id"
define:
  -
    name: "perMinute"
    template: "perSecond({{.argString}})|scale(60)"
notFoundStatusCode: 200
concurency: 1000
cache:
  type: "mem"
  size_mb: 0
  defaultTimeoutSec: 60
cpus: 0
tz: ""
functionsConfig:
maxBatchSize: 100
graphite:
    host: ""
    interval: "60s"
    prefix: "carbon.api"
    pattern: "{prefix}.{fqdn}"
idleConnections:
pidFile: ""
upstreams:
    tldCacheDisabled: false
    buckets: 10
    slowLogThreshold: "1s"
    timeouts:
        find: "2s"
        render: "10s"
        connect: "200ms"
    concurrencyLimitPerServer: 0
    keepAliveInterval: "30s"
    maxIdleConnsPerHost: 100
    doMultipleRequestsIfSplit: false
    backendsv2:
        backends:
          -
            groupName: "group1"
            protocol: "carbonapi_v3_pb"
            lbMethod: "broadcast"
            maxTries: 3
            maxBatchSize: 100
            keepAliveInterval: "10s"
            concurrencyLimit: 0
            maxIdleConnsPerHost: 1000
            forceAttemptHTTP2: false
            doMultipleRequestsIfSplit: false
            timeouts:
                find: "2s"
                render: "50s"
                connect: "200ms"
            servers:
              - "http://carbonserver-1:8070"
              - "http://carbonserver-2:8070"
    graphite09compat: false
graphTemplates: "/etc/carbonapi/graphTemplates.example.yaml"
expireDelaySec: 600

backend software and config
go-carbon 0.15.4 carbonserver, also from stable repo

Query that causes problems

/metrics/find/?query=carbon.*&format=treejson&path=carbon&node=carbon

Compare the "id" value below

Result from carbonapi

[
  {
    "allowChildren": 1,
    "expandable": 1,
    "leaf": 0,
    "id": "agents",
    "text": "agents",
    "context": {}
  },
  {
    "allowChildren": 1,
    "expandable": 1,
    "leaf": 0,
    "id": "api",
    "text": "api",
    "context": {}
  },
  {
    "allowChildren": 1,
    "expandable": 1,
    "leaf": 0,
    "id": "relays",
    "text": "relays",
    "context": {}
  }
]

Result from Graphite webapp

[
  {
    "text": "agents",
    "expandable": 1,
    "leaf": 0,
    "id": "carbon.agents",
    "allowChildren": 1
  },
  {
    "text": "api",
    "expandable": 1,
    "leaf": 0,
    "id": "carbon.api",
    "allowChildren": 1
  },
  {
    "text": "relays",
    "expandable": 1,
    "leaf": 0,
    "id": "carbon.relays",
    "allowChildren": 1
  }
]

[deniszh]

I would say carbon is a bit special metric. Can you reproduce on different path, not carbon ?

[cynici]

Another different query path
/metrics/find/?query=application.*&format=treejson&path=application&node=application

carbonapi

[
  {
    "allowChildren": 1,
    "expandable": 1,
    "leaf": 0,
    "id": "app-1",
    "text": "app-1",
    "context": {}
  },
  {
    "allowChildren": 1,
    "expandable": 1,
    "leaf": 0,
    "id": "app-2",
    "text": "app-2",
    "context": {}
  }
]

Graphite webapp

[
  {
    "text": "app-1",
    "expandable": 1,
    "leaf": 0,
    "id": "application.app-1",
    "allowChildren": 1
  },
  {
    "text": "app-2",
    "expandable": 1,
    "leaf": 0,
    "id": "application.app-2",
    "allowChildren": 1
  }
]

[deniszh]

Not sure if it's carbonapi fault. Coiuld you please also share graphite-web and go-carbon configs?

[cynici]

/opt/graphite/webapp/graphite/local_settings.py

TIME_ZONE = 'UTC'
DEBUG = True
DEFAULT_XFILES_FACTOR = 0
WHISPER_DIR = '/opt/graphite/storage/whisper'
DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
       ...
        'PORT': '3306'
    }
}
CLUSTER_SERVERS = ["127.0.0.1:8081"]  # carbonapi on the same host
USE_WORKER_POOL = True
POOL_MAX_WORKERS = 10
REMOTE_STORE_USE_POST = True
FUNCTION_PLUGINS = []

go-carbon.conf

[common]
user = "graphite"
graph-prefix = "carbon.agents.{host}"
metric-endpoint = "local"
metric-interval = "1m0s"
max-cpu = 2
[whisper]
data-dir = "/opt/graphite/storage/whisper"
schemas-file = "/opt/graphite/conf/storage-schemas.conf"
aggregation-file = "/opt/graphite/conf/storage-aggregation.conf"
workers = 2
max-updates-per-second = 4000
max-creates-per-second = 1000
hard-max-creates-per-second = false
sparse-create = true
flock = false
enabled = true
hash-filenames = false
compressed = false
remove-empty-file = true
[cache]
max-size = 1000000
write-strategy = "sorted"
[udp]
listen = ":2003"
enabled = false
buffer-size = 0
[tcp]
listen = ":2003"
enabled = true
buffer-size = 0
[pickle]
listen = ":2004"
max-message-size = 67108864
enabled = true
buffer-size = 0
[carbonlink]
listen = ":7002"
enabled = true
read-timeout = "10s"
[carbonserver]
listen = ":8070"
enabled = true
buckets = 10
metrics-as-counters = false
read-timeout = "60s"
write-timeout = "60s"
query-cache-enabled = true
query-cache-size-mb = 0
find-cache-enabled = true
trigram-index = true
scan-frequency = "10m0s"
trie-index = false
file-list-cache = ""
concurrent-index = false
realtime-index = 0
cache-scan = false
max-globs = 5000
fail-on-max-globs = true
max-metrics-globbed  = 10000000
max-metrics-rendered = 1000000
graphite-web-10-strict-mode = true
internal-stats-dir = ""
stats-percentiles = []
[dump]
enabled = false
path = "/var/lib/graphite/dump/"
restore-per-second = 0
[pprof]
listen = "localhost:7007"
enabled = false
[[logging]]
logger = ""
file = "/var/log/go-carbon/go-carbon.log"
level = "fatal"
encoding = "mixed"
encoding-time = "iso8601"
encoding-duration = "seconds"

[Civil]

I think carbonapi doesn't support node parameter as it's relatively new. I'll need to have a look at that though.

[cynici]

I think carbonapi doesn't support node parameter as it's relatively new. I'll need to have a look at that though.

Thanks, thought I was losing my mind. I am using grapite-web 1.1.8

[Civil]

It shouldn't matter much for grafana and other software that uses graphite (at least I'm not aware about those).