Missing traces for some services in Skywalking + Istio

[azibhassan]

Hi,

This could possibly be related to my configuration, but I am hoping if someone could direct me to the right direction.
Currently, I am running Skywalking v9.7.0 in Kubernetes cluster (Azure) along with Service Mesh (Istio v1.19).

Most of the services (GRPC and HTTP) are reporting traces in Skywalking but services which sends/receive data to Kafka are not showing up in Skywalking. I don't see any traces related to those services under SERVICE MESH > SERVICES dashboard. These services are only showing up under SERVICE MESH > DATA PLANE. I don`t see their traces in Zipkin instance as well.

Following is the configuration I am using for Skywalking and Istio:

SW_HEALTH_CHECKER_INTERVAL_SECONDS: 30
SW_RECEIVER_ZIPKIN: default
SW_QUERY_ZIPKIN: default
SW_ZIPKIN_HTTP_COLLECTOR_ENABLED: true
SW_RECEIVER_ZIPKIN_REST_HOST: 0.0.0.0
SW_RECEIVER_ZIPKIN_REST_PORT: 9411
SW_ENVOY_METRIC: default
SW_ENVOY_METRIC_SERVICE: true
SW_ENVOY_METRIC_ALS_HTTP_ANALYSIS: mx-mesh,k8s-mesh,persistence
SW_ENVOY_METRIC_ALS_TCP_ANALYSIS: mx-mesh,k8s-mesh,persistence
SW_LOG_ANALYZER: default
SW_LOG_LAL_FILES: mesh-dp,pgsql-slowsql,redis-slowsql,k8s-service
SW_OTEL_RECEIVER_ENABLED_HANDLERS: otlp,otlp-metrics,otlp-logs,otlp-traces
K8S_SERVICE_NAME_RULE: "app::${service.metadata.name}.${service.metadata.namespace}"
SW_ENABLE_ON_DEMAND_POD_LOG: true
SW_TELEMETRY: prometheus
SW_TELEMETRY_PROMETHEUS_HOST: 0.0.0.0
SW_TELEMETRY_PROMETHEUS_PORT: 1234
SW_ENABLE_UPDATE_UI_TEMPLATE: true
SW_METER_ANALYZER_ACTIVE_FILES: satellite,network-profiling,continuous-profiling
SW_SLOW_DB_THRESHOLD: default:10,mongodb:100
SW_RECEIVER_SHARING_REST_HOST: 0.0.0.0
SW_RECEIVER_SHARING_REST_PORT: 13800
SW_RECEIVER_SHARING_REST_CONTEXT_PATH: /
SW_SEARCHABLE_LOGS_TAG_KEYS: level,http.status_code
SW_SEARCHABLE_TAG_KEYS: http.method,http.status_code,rpc.status_code,db.type,db.instance
SW_QUERY_GRAPHQL_ENABLE_LOG_TEST_TOOL: true
SW_ZIPKIN_KAFKA_COLLECTOR_ENABLED: true
SW_ZIPKIN_KAFKA_SERVERS: kafka-controller-0.kafka-controller-headless.namespace.svc.cluster.local:9092

Istio Configuration:

  mesh: |-
    defaultConfig:
      holdApplicationUntilProxyStarts: true
      tracing:
        zipkin:
          address: skywalking-oap.aks-istio-system.svc:9411
      envoyAccessLogService:
        address: skywalking-satellite.aks-istio-system.svc:11800
      envoyMetricsService:
        address: skywalking-satellite.aks-istio-system.svc:11800
      proxyStatsMatcher:
        inclusionRegexps:
           - ".*outlier_detection.*"
           - ".*membership_healthy.*"
           - ".*upstream_cx_active.*"
           - ".*upstream_cx_total.*"
           - ".*upstream_rq_active.*"
           - ".*upstream_rq_total.*"
           - ".*upstream_rq_pending_active.*"
           - ".*lb_healthy_panic.*"
           - ".*upstream_cx_none_healthy.*"
        inclusionSuffixes:
           - "upstream_rq_timeout"
    enableEnvoyAccessLogService: true
    enablePrometheusMerge: true
    enableTracing: true
    extensionProviders:
      - skywalking:
          service: skywalking-satellite.aks-istio-system.svc.cluster.local
          port: 11800
        name: skywalking
      - zipkin:
          service: skywalking-oap.aks-istio-system.svc.cluster.local
          port: 9411
        name: zipkin
    defaultProviders:
      tracing: 
      - zipkin
      accessLogging:
      - skywalking

Any suggestions which I could try to fix missing traces between services
Is there any misconfiguration in above code or any configuration I have not included

[wu-sheng]

SkyWalking mesh services are discovered by ALS(Access Log Service) of envoy, so, they are not related to Zipkin trace.

And Zipkin traces are different from SkyWalking native tracer, which services show up in the general service dashboard.

[azibhassan]

I am talking about zipkin tracer. According to what I have understood, all the data is coming through Istio right.

Adding some screenshots to give you clear idea of what I am talking about. On following pages, as I said before, some of the services are missing.

[wu-sheng]

That may be related to the sampling policy, the default should be 1%, I assume.
Zipkin traces are from envoy zipkin tracer directly, rather than Istio. Istio just applied the tracer configurations to envoy. I would suggest you to check trace rate of envoys.

From you described, zipkin tracer may only work for HTTP traffic, Kafka connected services in Istio are TCP based only.

You could reach Istio community to confirm this.

[azibhassan]

Okay great. I will definitely check. Thanks for your time and help

[azibhassan]

Hey Wu-sheng, just to test, I switched from zipkin-tracer to skywalking-tracer and I am seeing the same results with kafka-based applications. Seems like only http connection traces are showing up.

[wu-sheng]

I don't think there is nothing related Zipkin or SkyWalking tracer. I think this is more about these tracers support status on HTTP traffic or TCP.