What are the best practices to dynamically control whether to enable/disable OpenTelemetry tracing

[luoxiao352]

Hello,

I would like to have a flight in my service to dynamically control whether to enable OpenTelemetry tracing, and change the sample rate when the service is on. But from https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/src/OpenTelemetry/README.md#getting-started-with-tracing I noticed that Once TracerProvider is built, changes to its configuration are not allowed.

Therefore, I wonder what the best practices are if I want to enable/disable tracing or make change to the sampler (or the inner config of the sampler) when the service is running. Is it okay to dispose the TracerProvider when I want to disable tracing and rebuild a new one when I want to reenable it?

Thank you for your time.

[p10tyr]

You are correct once it is set up there is no way to change it. I am not sure if this is by W3C specification or just an implementation detail.

The only way you can change it, is the change the values and restart your service, unless you have it load balanced (web sites maybe) or doesn't matter for a short down time (worker services or docker workers)

One could argue that you shouldn't be changing it because Tracing is not meant to work like logging and you shouldn't need to export 100% just a fraction large enough for you to find problems. for example you could have 1 million requests a day.. even a 25% will find outliers and performance issues.

The way I have it setup all development machines have 100%, roll out to QA machines has 75%, our Staging has 50% and Prod has 25%

Logs are different, all logs and metrics will be exported with OTLP

[luoxiao352]

Thank you, Piotr. I did a local testing and found out that disposing the TracerProvider could disable tracing, and rebuild one would re-enable tracing with the new configs during the service running. I am wondering whether this is a good practice or an acceptable practice?

[CodeBlanch]

@luoxiao352 It should generally work if you want to do that. Some stuff like the .NET Framework HttpWebRequest instrumentation hooks in reflectively when started, that won't be removed with disposal. In that case HttpWebRequest would continue to attempt to create telemetry, but there would be no listener so it would just no-op. If you just want to dynamically change sampling rate you could make your own sampler to do that.