Jaeger query shows error in log

[mmshaikh88]

I am running jaeger with elasticsearch and elasticsearch is running on another VM.

But the jaeger query pod on kubernetes is showing following error in logs:

{"level":"error","ts":1548655854.701871,"caller":"zap/logger.go:33","msg":"error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap/logger.go:33\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:257\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:267"}

[mmshaikh88]

ubuntu@kube-master-1:~/jaeger-kubernetes$ kubectl logs jaeger-query-676d6658d6-cdqs9 -f  
{"level":"info","ts":1548655820.6596537,"caller":"healthcheck/handler.go:99","msg":"Health Check server started","http-port":16687,"status":"unavailable"}
{"level":"info","ts":1548655820.7135236,"caller":"query/main.go:184","msg":"Archive storage not created","reason":"Archive storage not supported"}
{"level":"info","ts":1548655821.1784363,"caller":"query/main.go:132","msg":"Registering metrics handler with HTTP server","route":"/metrics"}
{"level":"info","ts":1548655821.1785743,"caller":"healthcheck/handler.go:133","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1548655821.1786165,"caller":"query/main.go:141","msg":"Starting HTTP server","port":16686}
{"level":"error","ts":1548655854.701871,"caller":"zap/logger.go:33","msg":"error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused","stacktrace":"github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap.(*Logger).Error\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/log/zap/logger.go:33\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue.func1\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:257\ngithub.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go.(*remoteReporter).processQueue\n\t/home/travis/gopath/src/github.com/jaegertracing/jaeger/vendor/github.com/uber/jaeger-client-go/reporter.go:267"}

[jpkrohling]

This is the relevant error message:

error when flushing the buffer: write udp 127.0.0.1:43658->127.0.0.1:6831: write: connection refused

This happens because there's no agent at localhost available to the query component. This is usually done by adding a agent container to the query pod. Without it, you won't see spans that are created by the query component itself. If that's not a problem for you, you can ignore the error messages.

If you still need assistance, feel free to reopen this issue.

[yurishkuro]

@jpkrohling do you have that sidecar setup for the agent in our k8s templates?

[jpkrohling]

We should, but looks like we don't:

jaeger-kubernetes/jaeger-production-template.yml

Lines 129 to 139 in da22250

	containers:
	- image: jaegertracing/jaeger-query:1.7.0
	name: jaeger-query
	args: ["--config-file=/conf/query.yaml"]
	ports:
	- containerPort: 16686
	protocol: TCP
	readinessProbe:
	httpGet:
	path: "/"
	port: 16687

I think we only have it in the operator. I'll leave this open, to add the sidecar to this template.

[mmshaikh88]

I don't have agent running along with query. Are you suggesting to setup agent along with query.

I do have jaeger-agent setup along with application sending trace to agent which further send trace to collector.

apiVersion: v1
kind: List
items:
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: collector-pod
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "14268"
      spec:
        containers:
        - image: jaegertracing/jaeger-collector:1.7.0
          name: jaeger-collector
          args: ["--config-file=/conf/collector.yaml"]
          ports:
          - containerPort: 14267
            protocol: TCP
          - containerPort: 14268
            protocol: TCP
          - containerPort: 9411
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 14269
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: collector
                  path: collector.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-collector
    labels:
      app: jaeger
      jaeger-infra: collector-service
  spec:
    ports:
    - name: jaeger-collector-tchannel
      port: 14267
      protocol: TCP
      targetPort: 14267
    - name: jaeger-collector-http
      port: 14268
      protocol: TCP
      targetPort: 14268
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: ClusterIP
- apiVersion: v1
  kind: Service
  metadata:
    name: zipkin
    labels:
      app: jaeger
      jaeger-infra: zipkin-service
  spec:
    ports:
    - name: jaeger-collector-zipkin
      port: 9411
      protocol: TCP
      targetPort: 9411
    selector:
      jaeger-infra: collector-pod
    type: ClusterIP
- apiVersion: extensions/v1beta1
  kind: Deployment
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-deployment
  spec:
    replicas: 1
    strategy:
      type: Recreate
    template:
      metadata:
        labels:
          app: jaeger
          jaeger-infra: query-pod
        annotations:
          prometheus.io/scrape: "true"
          prometheus.io/port: "16686"
      spec:
        containers:
        - image: jaegertracing/jaeger-query:1.7.0
          name: jaeger-query
          args: ["--config-file=/conf/query.yaml"]
          ports:
          - containerPort: 16686
            protocol: TCP
          readinessProbe:
            httpGet:
              path: "/"
              port: 16687
          volumeMounts:
          - name: jaeger-configuration-volume
            mountPath: /conf
          env:
          - name: SPAN_STORAGE_TYPE
            valueFrom:
              configMapKeyRef:
                name: jaeger-configuration
                key: span-storage-type
        volumes:
          - configMap:
              name: jaeger-configuration
              items:
                - key: query
                  path: query.yaml
            name: jaeger-configuration-volume
- apiVersion: v1
  kind: Service
  metadata:
    name: jaeger-query
    labels:
      app: jaeger
      jaeger-infra: query-service
  spec:
    ports:
    - name: jaeger-query
      port: 80
      protocol: TCP
      targetPort: 16686
    selector:
      jaeger-infra: query-pod
    type: LoadBalancer

[mmshaikh88]

Configmap for jaeger to connect to elasticsearch running on separate VM as:

apiVersion: v1
kind: ConfigMap
metadata:
  name: jaeger-configuration
  labels:
    app: jaeger
    jaeger-infra: configuration
data:
  span-storage-type: elasticsearch
  collector: |
    es:
      server-urls: http://10.145.70.97:9200
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://10.145.70.97:9200
  agent: |
    collector:
      host-port: "jaeger-collector:14267"

[mmshaikh88]

Deployed hotrod with sidecar as jaeger-agent

apiVersion: v1
kind: Service
metadata:
  name: example-hotrod
spec:
  ports:
    - name: ui 
      port: 8080
      targetPort: 8080 
    - name: interface1 
      port: 8081
      targetPort: 8081
    - name: interface2 
      port: 8082
      targetPort: 8082
    - name: interface3 
      port: 8083
      targetPort: 8083
  selector:
    app: example-hotrod 
  #clusterIP: None
  type: LoadBalancer
---
apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: example-hotrod 
spec:
  selector:
    matchLabels:
      app: example-hotrod 
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: example-hotrod 
    spec:
      containers:
      **- image: jaegertracing/example-hotrod:latest**
        name: example-hotrod 
        ports:
          - containerPort: 8080 
          - containerPort: 8081 
          - containerPort: 8082
          - containerPort: 8083 
        args:
          - "all"
      **- image: jaegertracing/jaeger-agent**
        name: jaeger-agent
        ports:
          - containerPort: 5775
            protocol: UDP
          - containerPort: 6831
            protocol: UDP
          - containerPort: 6832
            protocol: UDP
          - containerPort: 5778
            protocol: TCP
        args: ["--collector.host-port=jaeger-collector:14267"]

[jpkrohling]

Are you suggesting to setup agent along with query

Yes, if you want to get the Query traces reported to the collector, or if the error messages are bothering you. If you don't care about the spans generated by the query component, you can just ignore the error messages.

[yurishkuro]

You can also configure the sampling strategy in the collector for the query service to use 0 probability.

[naseemkullah]

Is it possible to configure query to talk to an agent elsewhere than localhost?

e.g. running agent as a daemonset with useHostPort: true and have query talk to the host directly...

[mmshaikh88]

As long as the agent with specific IP/Port is reachable from the Pod then yes you can configure it.

[naseemkullah]

As long as the agent with specific IP/Port is reachable from the Pod then yes you can configure it.

Thanks, do you know what the ENV_VAR would be?

[mmshaikh88]

Which ENV_VAR? Can you please clarify and elaborate a bit more?

[naseemkullah]

You mentioned we could configure it so I presumed there would be an environment variable for query to get query to talk to an agent that is not on local host.

[jpkrohling]

I don't think there's a way to customize the agent location for query. See jaegertracing/jaeger#1044.

[naseemkullah]

Hi @jpkrohling can you please confirm if we stil’ Need to add a agent sidecar to query in version 1.12 of jaeger? If so I will add it to the helm chart for jaeger via helm/charts#13693 (comment)

[jpkrohling]

@naseemkullah: yes, in order to have spans generated at the query component to reach the collector, it needs the agent as sidecar.

[naseemkullah]

Thank you for confirming @jpkrohling , an optional agent sidecar for query has been added to the jaeger helm chart.