Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Webhook Panic on Unexpected Parser Expression #1181

Closed
squat opened this issue May 22, 2024 · 2 comments · Fixed by #1182
Closed

Webhook Panic on Unexpected Parser Expression #1181

squat opened this issue May 22, 2024 · 2 comments · Fixed by #1182

Comments

@squat
Copy link
Contributor

squat commented May 22, 2024

If you're like me and tried mistakenly wrote a weird SLO like

spec:
  description: ""
  indicator:
    latency:
      success:
        metric: foo{le="1.0"} or vector(0)
                            # ^^^^^^^^^^^^
      total:
        metric: foo{le="1.0"}

The Kubernetes webhook API handler will panic with:

2024/05/22 11:25:56 http: panic serving 10.52.5.26:37498: interface conversion: parser.Expr is *parser.BinaryExpr, not *parser.VectorSelector
goroutine 576365 [running]:
net/http.(*conn).serve.func1()
	net/http/server.go:1854 +0xbf
panic({0x19859a0, 0xc0009bde00})
	runtime/panic.go:890 +0x263
github.com/pyrra-dev/pyrra/kubernetes/api/v1alpha1.(*ServiceLevelObjective).validate(0xc000896b60)
	github.com/pyrra-dev/pyrra/kubernetes/api/v1alpha1/servicelevelobjective_types.go:283 +0xfab
github.com/pyrra-dev/pyrra/kubernetes/api/v1alpha1.(*ServiceLevelObjective).ValidateUpdate(0xc00036af40?, {0xc0004e2000?, 0x781?})
	github.com/pyrra-dev/pyrra/kubernetes/api/v1alpha1/servicelevelobjective_types.go:191 +0x19
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*validatingHandler).Handle(_, {_, _}, {{{0xc000c6a210, 0x24}, {{0xc000951150, 0x9}, {0xc000951128, 0x8}, {0xc00012c168, ...}}, ...}})
	sigs.k8s.io/[email protected]/pkg/webhook/admission/validator.go:103 +0x6a9
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_, {_, _}, {{{0xc000c6a210, 0x24}, {{0xc000951150, 0x9}, {0xc000951128, 0x8}, {0xc00012c168, ...}}, ...}})
	sigs.k8s.io/[email protected]/pkg/webhook/admission/webhook.go:169 +0x20a
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0xc00003ceb0, {0x7fd32b84c0a0?, 0xc000466550}, 0xc0005c8300)
	sigs.k8s.io/[email protected]/pkg/webhook/admission/http.go:98 +0xc94
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7fd32b84c0a0, 0xc000466550}, 0x222c900?)
	github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:60 +0xd4
net/http.HandlerFunc.ServeHTTP(0x222c900?, {0x7fd32b84c0a0?, 0xc000466550?}, 0xc000355828?)
	net/http/server.go:2122 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x222c900?, 0xc00032c2a0?}, 0xc0005c8300)
	github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:147 +0xc5
net/http.HandlerFunc.ServeHTTP(0x6fbce5?, {0x222c900?, 0xc00032c2a0?}, 0x40dc2a?)
	net/http/server.go:2122 +0x2f
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x222c900, 0xc00032c2a0}, 0xc0005c8300)
	github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:109 +0xc7
net/http.HandlerFunc.ServeHTTP(0xc00032c2a0?, {0x222c900?, 0xc00032c2a0?}, 0x1bbea7e?)
	net/http/server.go:2122 +0x2f
net/http.(*ServeMux).ServeHTTP(0xc0008937a8?, {0x222c900, 0xc00032c2a0}, 0xc0005c8300)
	net/http/server.go:2500 +0x149
net/http.serverHandler.ServeHTTP({0x221d810?}, {0x222c900, 0xc00032c2a0}, 0xc0005c8300)
	net/http/server.go:2936 +0x316
net/http.(*conn).serve(0xc000622240, {0x222d7b0, 0xc000663bf0})
	net/http/server.go:1995 +0x612
created by net/http.(*Server).Serve
	net/http/server.go:3089 +0x5ed
squat added a commit to squat/pyrra that referenced this issue May 22, 2024
Even if a vector expression in an SLO returns a vector, the webhook API
handler will panic if the underlying expression is not a vector selctor.

Fixes: pyrra-dev#1181

Signed-off-by: squat <[email protected]>
squat added a commit to squat/pyrra that referenced this issue May 22, 2024
Even if a vector expression in an SLO returns a vector, the webhook API
handler will panic if the underlying expression is not a vector selctor.

Fixes: pyrra-dev#1181

Signed-off-by: squat <[email protected]>
@metalmatze
Copy link
Member

Opening again until we have a patch release.

@metalmatze metalmatze reopened this May 22, 2024
squat added a commit to squat/pyrra that referenced this issue May 23, 2024
Even if a vector expression in an SLO returns a vector, the webhook API
handler will panic if the underlying expression is not a vector selctor.

Fixes: pyrra-dev#1181

Signed-off-by: squat <[email protected]>
squat added a commit to squat/pyrra that referenced this issue May 23, 2024
Even if a vector expression in an SLO returns a vector, the webhook API
handler will panic if the underlying expression is not a vector selctor.

Fixes: pyrra-dev#1181

Signed-off-by: squat <[email protected]>
@metalmatze
Copy link
Member

By the way, Pyrra already adds the or vector(0) for the error matches in some rules and alert expressions.
Therefore there shouldn't be a need to do this manually.

sum(errorMetric{matchers="errors"} or vector(0))

availability, err := parser.ParseExpr(`1 - sum(errorMetric{matchers="errors"} or vector(0)) / sum(metric{matchers="total"})`)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants