runtime error: index out of range [43] with length 30 in HandlerForTransactional

[huanghy6268]

Background

It panics suddenly and does not happen again.

Envs

Go version: go1.18

github.com/prometheus/client_golang: v1.14.0

github.com/prometheus/common: v0.39.0

Stack:

2024-10-25 18:59:34.483444|ERROR|-|-:-:-|-|-:-|-:-|-|adminhttp: panic serving <ip:port>: runtime error: index out of range [43] with length 30|-:-|
2024-10-25 18:59:34.483468|ERROR|-|-:-:-|-|-:-|-:-|-|goroutine 2062665 [running]:|-:-|
2024-10-25 18:59:34.483473|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*conn).serve.func1()|-:-|
2024-10-25 18:59:34.483478|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:1825 +0xbf|-:-|
2024-10-25 18:59:34.483482|ERROR|-|-:-:-|-|-:-|-:-|-|panic({0x4510960, 0xc0094bd530})|-:-|
2024-10-25 18:59:34.483487|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/runtime/panic.go:844 +0x258|-:-|
2024-10-25 18:59:34.483491|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*huffmanBitWriter).indexTokens(0xc01976a5a0, {0xc01563e000, 0x1641, 0xc012ac6d48?})|-:-|
2024-10-25 18:59:34.483496|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/huffman_bit_writer.go:551 +0x2a5|-:-|
2024-10-25 18:59:34.483501|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*huffmanBitWriter).writeBlock(0xc01976a5a0, {0xc01563e000?, 0x39?, 0x1ef8?}, 0x0, {0x0, 0x0, 0x0})|-:-|
2024-10-25 18:59:34.483505|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/huffman_bit_writer.go:440 +0xcf|-:-|
2024-10-25 18:59:34.483509|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).writeBlock(0xc01f7a6000, {0xc01563e000?, 0xc012ac6e68?, 0x59146a?}, 0xc01f7a6000?)|-:-|
2024-10-25 18:59:34.483513|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:170 +0x9c|-:-|
2024-10-25 18:59:34.483517|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).deflate(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483522|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:415 +0x6d9|-:-|
2024-10-25 18:59:34.483526|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).close(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483530|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:646 +0x43|-:-|
2024-10-25 18:59:34.483534|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*Writer).Close(...)|-:-|
2024-10-25 18:59:34.483538|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:732|-:-|
2024-10-25 18:59:34.483542|ERROR|-|-:-:-|-|-:-|-:-|-|compress/gzip.(*Writer).Close(0xc0114b5970)|-:-|
2024-10-25 18:59:34.483547|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/gzip/gzip.go:242 +0x87|-:-|
2024-10-25 18:59:34.483551|ERROR|-|-:-:-|-|-:-|-:-|-|panic({0x4510960, 0xc0094bd518})|-:-|
2024-10-25 18:59:34.483555|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/runtime/panic.go:844 +0x258|-:-|
2024-10-25 18:59:34.483560|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).findMatch(0xc01ecbaa50?, 0x4857785?, 0xc012ac70a8?, 0x4a72d4?, 0x1?)|-:-|
2024-10-25 18:59:34.483564|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:254 +0x2bf|-:-|
2024-10-25 18:59:34.483568|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).deflate(0xc01f7a6000)|-:-|
2024-10-25 18:59:34.483572|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:443 +0x215|-:-|
2024-10-25 18:59:34.483576|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*compressor).write(0xc01f7a6000, {0xc0154b6000?, 0x1000, 0xc00a612a50?})|-:-|
2024-10-25 18:59:34.483580|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:554 +0x82|-:-|
2024-10-25 18:59:34.483584|ERROR|-|-:-:-|-|-:-|-:-|-|compress/flate.(*Writer).Write(...)|-:-|
2024-10-25 18:59:34.483590|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/flate/deflate.go:712|-:-|
2024-10-25 18:59:34.483595|ERROR|-|-:-:-|-|-:-|-:-|-|compress/gzip.(*Writer).Write(0xc0114b5970, {0xc0154b6000, 0x1000, 0x1000})|-:-|
2024-10-25 18:59:34.483599|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/compress/gzip/gzip.go:196 +0x34a|-:-|
2024-10-25 18:59:34.483603|ERROR|-|-:-:-|-|-:-|-:-|-|bufio.(*Writer).Flush(0xc0158a51c0)|-:-|
2024-10-25 18:59:34.483613|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/bufio/bufio.go:628 +0x62|-:-|
2024-10-25 18:59:34.483617|ERROR|-|-:-:-|-|-:-|-:-|-|bufio.(*Writer).WriteString(0xc0158a51c0, {0x4830c59?, 0xc0158a51c0?})|-:-|
2024-10-25 18:59:34.483621|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/bufio/bufio.go:740 +0x30|-:-|
2024-10-25 18:59:34.483626|ERROR|-|-:-:-|-|-:-|-:-|-|strings.(*byteStringReplacer).WriteString(0xc0000ec600, {0x4f3f280, 0xc0158a51c0}, {0x4830c59, 0x1})|-:-|
2024-10-25 18:59:34.483631|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/strings/replace.go:565 +0x1ce|-:-|
2024-10-25 18:59:34.483635|ERROR|-|-:-:-|-|-:-|-:-|-|strings.(*Replacer).WriteString(0xc012ac7287?, {0x4f3f280?, 0xc0158a51c0?}, {0x4830c59?, 0x0?})|-:-|
2024-10-25 18:59:34.483639|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/strings/replace.go:103 +0x90|-:-|
2024-10-25 18:59:34.483643|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeEscapedString({0x4f92218?, 0xc0158a51c0}, {0x4830c59, 0x1}, 0x58?)|-:-|
2024-10-25 18:59:34.483648|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:424 +0x66|-:-|
2024-10-25 18:59:34.483653|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeLabelPairs({0x4f92218, 0xc0158a51c0}, {0xc01ea2fe20, 0x4, 0x0?}, {0x0, 0x0}, 0xc012ac73d0?)|-:-|
2024-10-25 18:59:34.483657|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:368 +0x1cd|-:-|
2024-10-25 18:59:34.483662|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.writeSample({0x4f92218, 0xc0158a51c0}, {0x486b5e2?, 0x0?}, {0x0, 0x0}, 0xc01ed29260, {0x0, 0x0}, 0x24?, ...)|-:-|
2024-10-25 18:59:34.483668|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:296 +0x105|-:-|
2024-10-25 18:59:34.483673|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.MetricFamilyToText({0x4f3f3a0?, 0xc0114b5970}, 0xc01ee061e0)|-:-|
2024-10-25 18:59:34.483678|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/text_create.go:161 +0x985|-:-|
2024-10-25 18:59:34.483682|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.NewEncoder.func7(0x4a454c0?)|-:-|
2024-10-25 18:59:34.483686|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:144 +0x27|-:-|
2024-10-25 18:59:34.483690|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/common/expfmt.encoderCloser.Encode(...)|-:-|
2024-10-25 18:59:34.483695|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/expfmt/encode.go:50|-:-|
2024-10-25 18:59:34.483700|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.HandlerForTransactional.func1({0x7f73371cf7e0, 0xc01e189cc0}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483704|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/http.go:205 +0x7dc|-:-|
2024-10-25 18:59:34.483708|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x4f478e0?, {0x7f73371cf7e0?, 0xc01e189cc0?}, 0xc01e292ae0?)|-:-|
2024-10-25 18:59:34.483712|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483717|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x7f73371cf7e0, 0xc01e189cc0}, 0x4f90900?)|-:-|
2024-10-25 18:59:34.483721|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:60 +0xd4|-:-|
2024-10-25 18:59:34.483727|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x4f90900?, {0x7f73371cf7e0?, 0xc01e189cc0?}, 0x0?)|-:-|
2024-10-25 18:59:34.483731|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483735|ERROR|-|-:-:-|-|-:-|-:-|-|github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x4f90900?, 0xc00056f180?}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483742|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/github.com/prometheus/[email protected]/prometheus/promhttp/instrument_server.go:146 +0xb8|-:-|
2024-10-25 18:59:34.483747|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0xc002a94c30?, {0x4f90900?, 0xc00056f180?}, 0x0?)|-:-|
2024-10-25 18:59:34.483751|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483756|ERROR|-|-:-:-|-|-:-|-:-|-|<mask>/internal.(*adminService).metricsToggleable.func1({0x4f90900, 0xc00056f180}, 0xeb5ac1f9490c?)|-:-|
2024-10-25 18:59:34.483760|ERROR|-|-:-:-|-|-:-|-:-|-|	/root/go/pkg/mod/<mask>/internal/admin_mod.go:305 +0x89|-:-|
2024-10-25 18:59:34.483764|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.HandlerFunc.ServeHTTP(0x7f7370415908?, {0x4f90900?, 0xc00056f180?}, 0x40f3e5?)|-:-|
2024-10-25 18:59:34.483768|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2084 +0x2f|-:-|
2024-10-25 18:59:34.483773|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*ServeMux).ServeHTTP(0x0?, {0x4f90900, 0xc00056f180}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483777|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2462 +0x149|-:-|
2024-10-25 18:59:34.483781|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.serverHandler.ServeHTTP({0xc01e2925a0?}, {0x4f90900, 0xc00056f180}, 0xc01c905000)|-:-|
2024-10-25 18:59:34.483785|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:2916 +0x43b|-:-|
2024-10-25 18:59:34.483789|ERROR|-|-:-:-|-|-:-|-:-|-|net/http.(*conn).serve(0xc008076640, {0x4f922f8, 0xc00dcd6690})|-:-|
2024-10-25 18:59:34.483794|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:1966 +0x5d7|-:-|
2024-10-25 18:59:34.483798|ERROR|-|-:-:-|-|-:-|-:-|-|created by net/http.(*Server).Serve|-:-|
2024-10-25 18:59:34.483803|ERROR|-|-:-:-|-|-:-|-:-|-|	/go/go1.18/src/net/http/server.go:3071 +0x4db|-:-|

[bwplotka]

👋🏽 Thanks!

First of all, it would be nice if you could upgrade to the latest client_golang. Then we might need to ask for some reproduction tests.

@vesari actually found a very similar (if not same) issue related to gzip golang/go#40008 (comment) where the root cause was that gzip writer was used concurrently. Sincer our code is not using response writer concurrently maybe your middleware or server setup causes concurrent response writer use? 🤔 Do you mind giving us more context on how promhttp is used (ideally in the small isolate go test).