docs: show example of tracing over http->grpc boundary

[tmc]

We should show how to propagate an incoming traced http request over the grpc boundary.

Further, perhaps this just be supported out of the box.

[maxkondr]

Is here any news? Any examples,docs will be appreciated

[uschen]

I believe the stackdriver trace works with grpc-gateway.

[maxkondr]

@uschen If I understood correctly, stackdriver works in google cloud and AWS only. I want to use tracing in own solution.

[tmc]

@maxkondr you can use stackdriver trace if your workloads aren't in google's cloud.

[smthpickboy]

Any examples or docs for supporting opentracing? Thanks!

[theRealWardo]

similar to how I solved #7 - I ended up wrapping the mux and writing a little bit of code to propagate the tracing context correctly. here's how I did it:

import (
   ...
   "github.com/opentracing/opentracing-go"
   "github.com/opentracing/opentracing-go/ext"
)

var grpcGatewayTag = opentracing.Tag{Key: string(ext.Component), Value: "grpc-gateway"}

func tracingWrapper(h http.Handler) http.Handler {
  return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    parentSpanContext, err := opentracing.GlobalTracer().Extract(
      opentracing.HTTPHeaders,
      opentracing.HTTPHeadersCarrier(r.Header))
    if err == nil || err == opentracing.ErrSpanContextNotFound {
      serverSpan := opentracing.GlobalTracer().StartSpan(
        "ServeHTTP",
        // this is magical, it attaches the new span to the parent parentSpanContext, and creates an unparented one if empty.
        ext.RPCServerOption(parentSpanContext),
        grpcGatewayTag,
      )
      r = r.WithContext(opentracing.ContextWithSpan(r.Context(), serverSpan))
      defer serverSpan.Finish()
    }
    h.ServeHTTP(w, r)
  })
}

// Then just wrap your mux...
if err := http.ListenAndServe(":8080", tracingWrapper(mux)); err != nil {
  log.Fatalf("failed to start gateway server on 8080: %v", err)
}

[maxkondr]

there is a ready for use solution from github.com/opentracing-contrib/go-stdlib/nethttp lib:

import "github.com/opentracing-contrib/go-stdlib/nethttp"

...
return http.ListenAndServe(fmt.Sprintf(":%d", myPort),
		nethttp.Middleware(
			opentracing.GlobalTracer(),
			mux,
			nethttp.MWComponentName(os.Getenv("MY_POD_NAME")),
		),
	)

[ZachEddy]

Not sure if this is the right place to ask questions, but I didn't know where else to post.

Does the gRPC Gateway client need to be initiated with a tracing interceptor? I struggled to get @theRealWardo's example to work until I added a client-side interceptor. If so, I think it would be helpful to bring this up in the documentation (happy to open a PR for this).

There's some relevant information here, but I wanted to double check.

[johanbrandhorst]

Hi @ZachEddy, we'd happily welcome any documentation improvements you could contribute. I haven't got any experience with this myself so your knowledge would be very good to capture.

[ZachEddy]

Sounds good! Some additional documentation would be useful, but I like the original suggestion to add tracing support out of the box.

Maybe this can be accomplished by adding another ServeMuxOption to the runtime package?

Something like runtime.WithTracer(tracer opentracing.Tracer) perhaps? Not really sure, but I'll tinker around with it.

[kristiandrucker]

Hi. @johanbrandhorst asked me to write how I have forwarded span from gateway -> grpc server. Well after a few hours of debugging this is my final code. For tracing in gateway I have used nethttp.Middleware() and for actually sending the span via metadata I wrote my own function that returns metadata.MD map.

Implementation of nethttp:

func NewTracedRuntimeMuxServer(tracer opentracing.Tracer, mux *runtime.ServeMux) *http.Server {
	return &http.Server{
		Handler: nethttp.Middleware(
			tracer,
			mux,
			nethttp.OperationNameFunc(func(r *http.Request) string {
				return fmt.Sprintf("HTTP-gRPC %s %s", r.Method, r.URL.String())
			}),
		),
	}
}

Custom metadata from span function:

func MetadataFromSpan(span opentracing.Span) metadata.MD {
	ctx := span.Context()
	carrier := make(map[string]string)

	span.Tracer().Inject(
		ctx,
		opentracing.TextMap,
		opentracing.TextMapCarrier(carrier),
	)

	return metadata.New(carrier)
}

Implementation of the custom metadata from span function:

mux := runtime.NewServeMux(
	runtime.WithMetadata(
		func(ctx context.Context, r *http.Request) metadata.MD {
			span := opentracing.SpanFromContext(ctx)
			tracing.MetadataFromSpan(span)

			return tracing.MetadataFromSpan(span)
		},
	),
)

Getting the values in gRPC is done via this package github.com/grpc-ecosystem/go-grpc-middleware/tracing/opentracing. There I have created a UnaryInterceptor for grpc, but you can also create a StreamInterceptor and pass the UnaryServerInterceptor or the StreamServerInterceptor along with a tracer to create a trace for every call to the gRPC service and also it will enable you to create the span from context.