OpenTelemetry tracing samples GET / when /q/metrics is called

[brunobat]

Describe the bug

This is a follow up to #45300

It turns out that if we call the prometheus metrics endpoint /q/metrics a trace for / is registered:

Please mind that the /q/metrics/ endpoint doesn't exist in the provided reproducer.
The response status is 404.

Expected behavior

The default behaviour samples out all /q/* endpoints

Actual behavior

A trace for / is sent out by OTel

How to Reproduce?

Build last Quarkus main

Checkout https://github.com/brunobat/quarkus-lab/tree/main/otel-tracing

Start in dev mode: mvn quarkus:dev

Look at the Grafana traces.
Current main has the Grafana LGTM devservice scraper pooling /q/metrics (There is a PR to fix that).
You will see many requests.

Output of uname -a or ver

24.2.0 Darwin Kernel Version 24.2.0: Fri Dec 6 19:01:59 PST 2024; root:xnu-11215.61.5~2/RELEASE_ARM64_T6000 arm64

Output of java -version

OpenJDK 64-Bit Server VM Temurin-22.0.2+9

Quarkus version or git rev

No response

Build tool (ie. output of mvnw --version or gradlew --version)

No response

Additional information

No response

[quarkus-bot]

/cc @ebullient (metrics), @jmartisk (metrics), @radcortez (opentelemetry)

[gsmet]

@brunobat ah yes, so this is something I saw: all the 404 are normalized to either / or /* (no idea why they are not all normalized to the same thing). It might be a dev mode thing, though.

[brunobat]

Will confirm.

[brunobat]

It's also happening with java -jar target/quarkus-app/quarkus-run.jar
And not only related with /q/*, /something will also trigger a span.

[gsmet]

Yeah so the route is set when the request is routed. And basically any route that will match in the end will trigger it.

So if we have a route on / that will somehow be the default, you end up getting a span.

Now I don't know how we want to handle a 404 but that's how it is.

Note that we somehow want to normalize 404 as it could trigger a huge amount of spans if we keep the whole URL.

[brunobat]

I think we had a bug on the way we handled 404 and didn't realised it.
We should be generating spans if the route was not matched, yes.
In the past, I guess we were sampling out these /q requests somehow.

If a lot of spans are a problem, we could evaluate a property to sample out 4** errors.
Closing this.