Fix head stats and hooks when replaying a corrupted snapshot #14079

alanprot · 2024-05-10T21:57:04Z

When loading a snapshot and encountering a corrupted chunk, we discard previously loaded series from the snapshot and resort to replaying the wall. In such cases, we were not resetting the number of series in the head, leading to double counting them.

Additionally, we did not invoke the PostDeletion hook when resetting the memory - this needs to be called as the PostCreation was called for the series which we were able to replay from the snapshot but were subsequently discarded.

yeya24

Thanks make sense to me.

In Cortex, we rely on series lifecycle callback to keep track of active series. We hit this series double counting bug when getting a corrupted chunk and active series reached limit because post delete hook was not called.

bboreham

Thanks for this; looks fairly good but some comments from the bug-scrub meeting.

bboreham · 2024-05-14T11:45:36Z

tsdb/head.go

+ totalSeries += len(deletedForCallback)
+ deletedFromPrevStripe = len(deletedForCallback)
+ }
+ s.series = make([]map[chunks.HeadSeriesRef]*memSeries, s.size)


This seems a no-op because it is overwritten on line 319?

Yeah.. it is.. i just wanted to do that for correctness of the "reset" method - reset seems that we are resetting the struct to the initial state (empty) and could be reused. WDYT?

I can no do that and rename the method to something else (maybe flush? clean?)

Updated the code:

Renamed the method from reset to flush

Removed the extra logic to clean the series

I can still see s.series = make([]map[chunks.HeadSeriesRef]*memSeries, s.size), is it still needed?
I added some comments below regarding iter, below.
As its invocation would become simpler, we can just move it directly to resetInMemoryState() with a comment.
(And entrust the task of naming to the person who will extract this code in the future :))

My suggestion: get rid of flush() and inline it in resetInMemoryState() without this allocation. Or remove this line of s.series = ... and probably rename this function to callPostDeletionForAll().

ok.. make sense!

Ok.. i just did that! its better indeed!

tsdb/head.go

bboreham · 2024-05-14T11:48:48Z

tsdb/head_test.go

@@ -4007,24 +4008,44 @@ func TestSnapshotError(t *testing.T) {
 require.NoError(t, err)
 f, err := os.OpenFile(path.Join(snapDir, files[0].Name()), os.O_RDWR, 0)
 require.NoError(t, err)
- _, err = f.WriteAt([]byte{0b11111111}, 18)
+ // lets corrupt middle of the snapshot, so we can replay some entries
+ _, err = f.WriteAt([]byte{0b11111111}, 300)


Is the change from 18 to 300 significant?

Yeah.. it is..

If we corrupt the byte 18, we will not be able to restore any series (and so we cannot see the problem).

Corrupting the byte 300, we are able to restore 2 timeseries before reaching the corrupted position, and so, highlight the problem.

So we don’t make it appear as if we’ve deleted a test (we'd want to continue running the other checks when no series has been restored) Let’s add that as another scenario. You can simply recreate a head at the end and run the new checks related to callbacks, or even better, run all checks for the two cases.

I will need to create a new snapshot as the old one is already corrupted in the beginning. Will do

I backed up the snapshot and restored to create the other test case! PTAL?

alanprot · 2024-05-16T20:43:43Z

@bboreham PTAL?

jeromeinsf · 2024-05-22T18:32:29Z

@codesome is this something you could help review?

machine424

Nice catch!

machine424 · 2024-05-24T08:08:44Z

tsdb/head.go

+ return deleted, rmChunks, actualMint, minOOOTime, minMmapFile
+}
+
+func (s *stripeSeries) iter(f func(int, uint64, *memSeries, map[chunks.HeadSeriesRef]labels.Labels), endShard func(map[chunks.HeadSeriesRef]labels.Labels)) {


I think we can make it more specific (by renaming it deleteFunc or iterForDeletion or sth else + some comments), call PostDeletion inside it and make it return the number of deleted series.

we can also note in the comments what f should do with the map.

Let's not have "deletion" in the name, because this function in itself has nothing to do with deletion. The users of this just happen to use it for deletion.

If we dont have deletion on the name we cannot call the post deletion hook inside the function! =/ What u guys think is better here? TBH i like the iterForDeletion as it makes the code cleaner!

tsdb/head.go

machine424 · 2024-05-24T08:29:00Z

tsdb/head.go

+ totalSeries += len(deletedForCallback)
+ deletedFromPrevStripe = len(deletedForCallback)
+ }
+ s.series = make([]map[chunks.HeadSeriesRef]*memSeries, s.size)


I can still see s.series = make([]map[chunks.HeadSeriesRef]*memSeries, s.size), is it still needed?
I added some comments below regarding iter, below.
As its invocation would become simpler, we can just move it directly to resetInMemoryState() with a comment.
(And entrust the task of naming to the person who will extract this code in the future :))

tsdb/head_test.go

machine424 · 2024-05-24T08:57:36Z

tsdb/head_test.go

@@ -4007,24 +4008,44 @@ func TestSnapshotError(t *testing.T) {
 require.NoError(t, err)
 f, err := os.OpenFile(path.Join(snapDir, files[0].Name()), os.O_RDWR, 0)
 require.NoError(t, err)
- _, err = f.WriteAt([]byte{0b11111111}, 18)
+ // lets corrupt middle of the snapshot, so we can replay some entries
+ _, err = f.WriteAt([]byte{0b11111111}, 300)


So we don’t make it appear as if we’ve deleted a test (we'd want to continue running the other checks when no series has been restored) Let’s add that as another scenario. You can simply recreate a head at the end and run the new checks related to callbacks, or even better, run all checks for the two cases.

Signed-off-by: alanprot <[email protected]>

Co-authored-by: Ayoub Mrini <[email protected]> Signed-off-by: Alan Protasio <[email protected]>

codesome

Checking the unit tests after this

codesome · 2024-05-24T19:17:32Z

tsdb/head.go

+ totalSeries += len(deletedForCallback)
+ deletedFromPrevStripe = len(deletedForCallback)
+ }
+ s.series = make([]map[chunks.HeadSeriesRef]*memSeries, s.size)


My suggestion: get rid of flush() and inline it in resetInMemoryState() without this allocation. Or remove this line of s.series = ... and probably rename this function to callPostDeletionForAll().

tsdb/head.go

codesome

Just one comment on tests and the above comments, looks good otherwise. Thanks!

tsdb/head_test.go

Signed-off-by: alanprot <[email protected]>

codesome

LGTM. Small nits.

tsdb/head_test.go

Co-authored-by: Ganesh Vernekar <[email protected]> Signed-off-by: Alan Protasio <[email protected]>

machine424 · 2024-05-25T12:52:59Z

tsdb/head.go

+ // and increment the series removed metrics
+ fs := h.series.iterForDeletion(func(_ int, _ uint64, s *memSeries, flushedForCallback map[chunks.HeadSeriesRef]labels.Labels) {
+ // All series should be flushed
+ flushedForCallback[s.ref] = s.lset


I forgot to submit this comment that was more of a question:
I don't know if we should lock s here, I don't know if we could have any races.

iterForDeletion is already locking here, so should be good?

I was talking about this lock

prometheus/tsdb/head.go

Line 1883 in e6f1f7e

series.Lock()

actually.

Hum..

Do you think is needed? I can add those locks but right now i dont see a case where they are racing on the reset method.

On the check function we are reading and modifying the series at the same time we are possibly appending more samples, but maybe its not the case on the reset? I can still add just to be safe though.

machine424 · 2024-05-25T12:53:38Z

Thanks @alanprot, this lgtm ;)

…r-prometheus-37b408c * upstream/main: (34 commits) Feature: Allow configuration of a rule evaluation delay (prometheus#14061) optimize regex matching for empty label values in posting match (prometheus#14075) Enable additional Go metrics Bump golangci-lint action (prometheus#14154) Upgrade to golangci-lint v1.59.0 docs/configuration: clarify OpenStack metadata labels (prometheus#14149) Join errors Fix head stats and hooks when replaying a corrupted snapshot (prometheus#14079) .gitpod.Dockerfile: Auto-fetch Go and goyacc vers fix: correct the typo in azuread sdk auth (prometheus#14106) fix(api): Send warnings only if the limit is really exceeded (prometheus#14116) doc: Clarify the limits of dumping/backfilling via OpenMetrics Check context every 128 labels instead of 100 (prometheus#14118) web/api: export defaultStatsRenderer (prometheus#14121) added line When set, query.max-concurrency may need to be adjusted accordingly. Signed-off-by: kushagra Shukla <[email protected]> tsdb/index: Refactor Reader tests (prometheus#14071) Fix flaky test Enable perfsprint linter and fix up code [TEST] Rules: Sleep 15ms to fit Windows behaviour better removes the added tests from engine_test.go ...

* makefile: let golangci-lint also run on arm64 Signed-off-by: Mickael Carl <[email protected]> * docs: add a quick note on linting in contributing guidelines Signed-off-by: Mickael Carl <[email protected]> * chore:fix typo Signed-off-by: tyltr <[email protected]> * Code optimization: The relabel operation is used very frequently, and strconv.FormatInt() with better performance should be used. Signed-off-by: roger.wang <[email protected]> * Clarify batch_send_deadline docs This is the time period covered by a batch of samples, when the number of waiting samples is lower than max_samples_per_send. It does not affect timeouts or retries. Co-authored-by: Bartlomiej Plotka <[email protected]> Signed-off-by: Federico Leva <[email protected]> * chore(tsdb): add a sandboxDir to DBReadOnly, the directory can be used for transient file writes. use it in loadDataAsQueryable to make sure the RO Head doesn't truncate or cut new chunks in data/chunks_head/. add a -sandbox-dir-root flag to "promtool tsdb dump/dump-openmetrics" to control the root of that sandbox dirrectory. Signed-off-by: machine424 <[email protected]> * refactor: add max func to maxTimestamp Signed-off-by: komisan19 <[email protected]> * fix Signed-off-by: komisan19 <[email protected]> * Rule Manager: Add `rule_group_last_restore_duration_seconds` to measure restore time per rule group When a rule group changes or prometheus is restarted we need to ensure we restore the active alerts that were firing for a corresponding rule, for that Prometheus uses the `ALERTS_FOR_STATE` series to query the previous state and restore it. If a given rule has high cardinality (think 100s of 1000s for series) this proccess can take a bit of time - this is the first of a series of PRs to improve this problem and I'd like to start with exposing the time it takes to restore a rule group as a gauge. Signed-off-by: gotjosh <[email protected]> * docs: storage.md: clarify storage.tsdb.retention.time description Signed-off-by: tesla59 <[email protected]> * Change variable name to `restoreStartTime` from `now` and introduce a log line to record total time Signed-off-by: gotjosh <[email protected]> * Add a changelog entry Signed-off-by: gotjosh <[email protected]> * Improve the metric description Signed-off-by: gotjosh <[email protected]> * Avoid creating new slices for labels values on postings for matchers (#13958) * Avoid creating new slices for labels values on postings for matchers Signed-off-by: alanprot <[email protected]> * refactor Signed-off-by: alanprot <[email protected]> --------- Signed-off-by: alanprot <[email protected]> * Rule Manager: Only query once per alert rule when restoring alert state Prometheus restores alert state between restarts and updates. For each rule, it looks at the alerts that are meant to be active and then queries the `ALERTS_FOR_STATE` series for _each_ alert within the rules. If the alert rule has 120 instances (or series) it'll execute the same query with slightly different labels. This PR changes the approach so that we only query once per alert rule and then match the corresponding alert that we're about to restore against the series-set. While the approach might use a bit more memory at start-up (if even?) the restore proccess is only ran once per restart so I'd consider this a big win. This builds on top of #13974 Signed-off-by: gotjosh <[email protected]> * bug: nil check against the series set not errors Signed-off-by: gotjosh <[email protected]> * Fix tests and a bug with the series lookup logic. Signed-off-by: gotjosh <[email protected]> * Use the string representation of the labels instead of the hash Signed-off-by: gotjosh <[email protected]> * - Add a changelog entry - Improve variable name of the map produced by the series set Signed-off-by: gotjosh <[email protected]> * Allow the result map for the series set before hand with a hint. Signed-off-by: gotjosh <[email protected]> * fix typo Signed-off-by: gotjosh <[email protected]> * bugfix: Decouple native histogram ingestions and protobuf parsing Up until this point, if a scrape was done with the protobuf format Prometheus would always try to ingest native histograms even with the feature flag disabled. This causes problems with other feature-flags that depend on the protobuf format, like 'created-timestamp-zero-ingestion'. This commit decouples native histogram parsing from ingestion, making sure ingestion only happens when the 'native-histogram' feature-flag is enabled. Signed-off-by: Arthur Silva Sens <[email protected]> * Improve comments around resending resolved alerts (#13990) Signed-off-by: George Robinson <[email protected]> * discovery(k8s): Only register client-go metrics adapters when needed Previously the metrics adapters for client-go were registered in an init function. This resulted in clobbering default metrics providers when these packages are imported into an application that leverages the default client-go metrics registry. Instead, let's only register these adapters when requested. Signed-off-by: Stephen Heckler <[email protected]> * Update docs/storage.md Co-authored-by: Ayoub Mrini <[email protected]> Signed-off-by: Nishant Singh <[email protected]> * Update docs/storage.md Co-authored-by: Ayoub Mrini <[email protected]> Signed-off-by: Nishant Singh <[email protected]> * fix(scaleway-sd): use public IPs if no private IP present (#13941) * fix(scaleway-sd): use public IPs if no private IP present * tests(scaleway-sd): add instance with routed public ip and no private ip --------- Signed-off-by: Heyoxe <[email protected]> * UTF-8: updates UI parser to support UTF-8 characters (#13590) Signed-off-by: Neeraj Gartia <[email protected]> * fix Signed-off-by: komisan19 <[email protected]> * fix(promql/query_logger): close file in error handling (#13948) Signed-off-by: guoguangwu <[email protected]> * otlp: Prometheus to own its own copy of the otlptranslator package (#13991) After a lot of productive discussion between the Prometheus and OpenTelemetry community we decided that it made sense for Prometheus to own its own copy of the code in charge for handling OTLP ingestion traffic. This commit is removing the README and update-copy.sh files that had the previous steps to update the code. Also it is updating the licensing of all the files to make sure the OpenTelemetry provenance is explicit and to state the new ownership. Signed-off-by: Jesus Vazquez <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> * docs: [ovh sd] Added missing label for OVH dedicated server in service discovery doc Signed-off-by: Jiekun <[email protected]> * OTLP: Use PrometheusConverter directly Signed-off-by: Arve Knudsen <[email protected]> * prometheusremotewrite: Add PrometheusConverter.FromMetrics benchmark Signed-off-by: Arve Knudsen <[email protected]> * Add an assertion on the count of alerts before adding an active alert Signed-off-by: gotjosh <[email protected]> * Rename QueryforStateSeries to QueryForStateSeries Signed-off-by: gotjosh <[email protected]> * Use labels.Len() instead of manually counting the labels Signed-off-by: gotjosh <[email protected]> * Remove duplicated sorted and assignment of expected alerts. Signed-off-by: gotjosh <[email protected]> * Rename `alerts` to `expectedAlerts` in the test case input Signed-off-by: gotjosh <[email protected]> * querier.Select cannot return a nil series set. Signed-off-by: gotjosh <[email protected]> * ci: check generated parser code before running unit tests Check that the generated parser code is consistent with the input definition. Remove the file before re-generating to make sure that missing goyacc is not effecting the check. Fixes: #7488 Signed-off-by: György Krajcsovits <[email protected]> * Add missing OTLP fixes to changelog (#14014) Signed-off-by: Arve Knudsen <[email protected]> * prometheusremotewrite: Move TimeSeries method to timeseries.go To facilitate generating OTel translation code for other Prometheus compatible backends, modify the prometheusremotewrite sources slightly so that the PrometheusConverter.TimeSeries method is in a file called timeseries.go. The rationale is to allow other backends to define their own implementation of this method. Signed-off-by: Arve Knudsen <[email protected]> * Update promu Update promu to the latest release. Signed-off-by: SuperQ <[email protected]> * build(deps): bump golangci/golangci-lint-action in /scripts Bumps [golangci/golangci-lint-action](https://github.com/golangci/golangci-lint-action) from 4.0.0 to 5.1.0. - [Release notes](https://github.com/golangci/golangci-lint-action/releases) - [Commits](https://github.com/golangci/golangci-lint-action/compare/3cfe3a4abbb849e10058ce4af15d205b6da42804...9d1e0624a798bb64f6c3cea93db47765312263dc) --- updated-dependencies: - dependency-name: golangci/golangci-lint-action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/checkout from 4.1.2 to 4.1.4 Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.2 to 4.1.4. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/9bb56186c3b09b4f86b1c65136769dd318469633...0ad4b8fadaa221de15dcec353f45205ec38ea70b) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/upload-artifact from 4.3.1 to 4.3.3 Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.3.1 to 4.3.3. - [Release notes](https://github.com/actions/upload-artifact/releases) - [Commits](https://github.com/actions/upload-artifact/compare/5d5d22a31266ced268874388b861e4b58bb5c2f3...65462800fd760344b1a7b4382951275a0abb4808) --- updated-dependencies: - dependency-name: actions/upload-artifact dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.51.1 to 0.51.2. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.51.1...v0.51.2) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/linode/linodego from 1.32.0 to 1.33.0 Bumps [github.com/linode/linodego](https://github.com/linode/linodego) from 1.32.0 to 1.33.0. - [Release notes](https://github.com/linode/linodego/releases) - [Commits](https://github.com/linode/linodego/compare/v1.32.0...v1.33.0) --- updated-dependencies: - dependency-name: github.com/linode/linodego dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump google.golang.org/api from 0.174.0 to 0.177.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.174.0 to 0.177.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.174.0...v0.177.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * removed formateoverview section (#13994) docs: Remove outdated information about remote-read API --------- Signed-off-by: kushagra Shukla <[email protected]> Signed-off-by: Kushal shukla <[email protected]> Signed-off-by: Arthur Silva Sens <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> * BUGFIX: Mark the rule's restoration process as completed always (#14048) * BUGFIX: Mark the rule's restoration process as completed always In https://github.com/prometheus/prometheus/pull/13980 I introduced a change to reduce the number of queries executed when we restore alert statuses. With this, the querying semantics changed as we now need to go through all series before we enter the alert restoration loop and I missed the fact that exiting early when there are no rules to restore would lead to an incomplete restoration. An alert being restored is used as a proxy for "we're now ready to write `ALERTS/ALERTS_FOR_SERIES` metrics" so as a result we weren't writing the series if we didn't restore anything the first time around. --------- Signed-off-by: gotjosh <[email protected]> * Fix `parser.VectorSelector.String()` with empty name matcher (#14015) The check fell into "this matcher equals vector selector's name" case when vector selector doesn't have a name and the matcher is an explicit matcher for an empty __name__ label. To provide some context about why this is important: some downstream projects use the promql.Parse(expr.String()) to clone an expression's AST, and with this bug that matcher disappears in the cloning. Signed-off-by: Oleg Zaytsev <[email protected]> * Fix FastRegexMatcher matching multibyte runes with . (#14059) When `zeroOrOneCharacterStringMatcher` wach checking the input string, it assumed that if there are more than one bytes, then there are more than one runes, but that's not necessarily true. Signed-off-by: Oleg Zaytsev <[email protected]> * bugfix: allow opting-out of multi-cluster setups Allow users to opt-out of the multi-cluster setup for Prometheus dashboard, in environments where it isn't applicable. Refer: https://github.com/prometheus/prometheus/pull/13180. Signed-off-by: Pranshu Srivastava <[email protected]> * [REFACTOR] PromQL: simplify rangeEvalTimestampFunctionOverVectorSelector (#14021) The function `rangeEvalTimestampFunctionOverVectorSelector` appeared to be checking histogram size, however the value it used was always 0 due to subtle variable shadowing. However we don't need to pass sample values to the `timestamp` function, since the latter only cares about timestamps. This also affects peak sample count in statistics, since we are no longer copying histogram samples. Signed-off-by: Arve Knudsen <[email protected]> * tsdb/chunkenc.Pool: Refactor Get and Put Signed-off-by: Arve Knudsen <[email protected]> * refactor: extract almost.Equal() to new package To avoid a circular reference between promql and promqltest. Signed-off-by: Bryan Boreham <[email protected]> * test: turn TestKahanSum into scripted test This saves having a function solely to call kahanSumInc. Signed-off-by: Bryan Boreham <[email protected]> * test: check for @-modifier without using engine internals Signed-off-by: Bryan Boreham <[email protected]> * test: move test files into new promqltest package So that promql package does not bring in test-only dependencies. Signed-off-by: Bryan Boreham <[email protected]> * test: move promqltest tests together with the implementation Signed-off-by: Bryan Boreham <[email protected]> * test: make field initializers explicit Lint started complaining after I moved the file. Signed-off-by: Bryan Boreham <[email protected]> * test: clean up promqltest package references So it nearly compiles. Signed-off-by: Bryan Boreham <[email protected]> * Quote label name in matchers when needed When the label name of a matcher contains non-standard characters, like a dot, or starts with a digit, it should be quoted. If it's not quoted, then `VectorSelector.String()` isn't a valid PromQL. Signed-off-by: Oleg Zaytsev <[email protected]> * Optimize Matcher.String() Signed-off-by: Oleg Zaytsev <[email protected]> * refactor: Move NewTestEngine into promqltest And export `DefaultMaxSamplesPerQuery` so callers can replicate previous behaviour. Signed-off-by: Bryan Boreham <[email protected]> * promql: export NewTestQuery So that tests can call it from another package. Signed-off-by: Bryan Boreham <[email protected]> * test: add promqltest package references To packages outside of promql. Signed-off-by: Bryan Boreham <[email protected]> * refactor: extract some PromQL Engine tests which use unexported structs Signed-off-by: Bryan Boreham <[email protected]> * test: move most PromQL tests into separate test package So that they can import promqltest which imports promql. Signed-off-by: Bryan Boreham <[email protected]> * Fix language in docs and comments (#14041) Fix language in docs and comments --------- Signed-off-by: Arve Knudsen <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * test: PromQL: stop using internal fields of engine * set enablePerStepStats and lookback duration via `NewTestEngine` parameters. * check maxSamples by recreating query engine * check lookback without modifying internals Signed-off-by: Bryan Boreham <[email protected]> * Use bytes.Buffer from stack buf in Matcher.String() Also removed the growing until there's a benchmark for that. Signed-off-by: Oleg Zaytsev <[email protected]> * [ENHANCEMENT] TSDB: Optimize querying with regexp matchers Add method `PostingsForLabelMatching` to `tsdb.IndexReader`, to obtain postings for labels with a certain name and values accepted by a provided callback, and use it from `tsdb.PostingsForMatchers`. The intention is to optimize regexp matcher paths, especially not having to load all label values before matching on them. Plus tests, and refactor some `tsdb/index.Reader` methods. Benchmarking shows memory reduction up to ~100%, and speedup of up to ~50%. Signed-off-by: Arve Knudsen <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> * promql.ActiveQueryTracker: Unmap mmapped file when done Signed-off-by: Arve Knudsen <[email protected]> * otlp: Remove OTel feature gate registration from copied translation package (#13932) Signed-off-by: Anthony J Mirabella <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> * Add failing test case Signed-off-by: Charles Korn <[email protected]> * Ensure series in matrix values returned for instant queries are always sorted Signed-off-by: Charles Korn <[email protected]> * Capture timing information while sorting Signed-off-by: Charles Korn <[email protected]> * Benchmark zeroOrOneCharacterStringMatcher.Matches This adds some more test cases for unicode values, and also a benchmark for zeroOrOneCharacterStringMatcher.Matches() Signed-off-by: Oleg Zaytsev <[email protected]> * Use utf8.DecodeRuneInString(s) This replaces the custom `moreThanOneRune` function with the standard `utf8.DecodeRuneInString(s)` that can be used to figure out the size of the first rune. Signed-off-by: Oleg Zaytsev <[email protected]> * Add invalid utf8 test cases to regexp Signed-off-by: Oleg Zaytsev <[email protected]> * Check utf8.RuneError result Signed-off-by: Oleg Zaytsev <[email protected]> * adds TestNativeHistogramRate func to promql test framework Signed-off-by: Neeraj Gartia <[email protected]> * adds test for native histogram rate func in promql testing framework Signed-off-by: Neeraj Gartia <[email protected]> * adds test for sum, count, stddev, stdvar, quantile and fraction func to promql testing framework Signed-off-by: Neeraj Gartia <[email protected]> * some nits Signed-off-by: Neeraj Gartia <[email protected]> * removes the added tests from engine_test.go Signed-off-by: Neeraj Gartia <[email protected]> * [TEST] Rules: Sleep 15ms to fit Windows behaviour better On Windows, Go will sleep 15ms if you ask for less. TestAsyncRuleEvaluation compares actual delay to the nominal time, so using 15ms should work better on Windows, and be hardly noticeable elsewhere. Signed-off-by: Bryan Boreham <[email protected]> * tsdb: check for context cancel before regex matching postings (#14096) * tsdb: check for context cancel before regex matching postings Regex matching can be heavy if the regex takes a lot of cycles to evaluate and we can get stuck evaluating postings for a long time without this fix. The constant checkContextEveryNIterations=100 may be changed later. Signed-off-by: György Krajcsovits <[email protected]> * tsdb/index/postings: fix missing lock unlock Followup to #14096 Unfortunately the previous PR introduced this bug by not releasing the lock before returning. Signed-off-by: György Krajcsovits <[email protected]> * Enable perfsprint linter and fix up code Signed-off-by: Oleksandr Redko <[email protected]> * Fix flaky test Signed-off-by: Arve Knudsen <[email protected]> * tsdb/index: Refactor Reader tests (#14071) tsdb/index: Refactor Reader tests Co-authored-by: Björn Rabenstein <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> --------- Signed-off-by: Arve Knudsen <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> * Document sorting behaviour Signed-off-by: Charles Korn <[email protected]> * added line When set, query.max-concurrency may need to be adjusted accordingly. Signed-off-by: kushagra Shukla <[email protected]> Signed-off-by: kushagra Shukla <[email protected]> * web/api: export defaultStatsRenderer (#14121) defaultStatsRenderer->DefaultStatsRenderer Add short docstrings. I'd like to use the stats renderer to peek at the statistics in https://github.com/grafana/mimir/pull/7966 However I cannot call the original function without this export afterwards. Signed-off-by: György Krajcsovits <[email protected]> * Check context every 128 labels instead of 100 (#14118) Follow up on https://github.com/prometheus/prometheus/pull/14096 As promised, I bring a benchmark, which shows a very small improvement if context is checked every 128 iterations of label instead of every 100. It's much easier for a computer to check modulo 128 than modulo 100. This is a very small 0-2% improvement but I'd say this is one of the hottest paths of the app so this is still relevant. Signed-off-by: Oleg Zaytsev <[email protected]> * doc: Clarify the limits of dumping/backfilling via OpenMetrics This is about native histograms (not yet supported) and staleness markers (for which OpenMetrics support isn't even planned). Signed-off-by: beorn7 <[email protected]> * fix(api): Send warnings only if the limit is really exceeded (#14116) for the the series, label names and label values APIs Add warnings count check to TestEndpoints The limit param was added in https://github.com/prometheus/prometheus/pull/13396 Signed-off-by: machine424 <[email protected]> * fix: correct the typo in azuread sdk auth (#14106) Signed-off-by: Jayapriya Pai <[email protected]> * .gitpod.Dockerfile: Auto-fetch Go and goyacc vers In this commit we auto-fetch Go version from go.mod and goyacc version from Makefile in the Prometheus repo. Signed-off-by: Mohamed Awnallah <[email protected]> * Fix head stats and hooks when replaying a corrupted snapshot (#14079) * Fixing head stats and hooks when replaying a corrupted snapshot Signed-off-by: alanprot <[email protected]> * Fixing create/removed series metrics Signed-off-by: alanprot <[email protected]> * Refactoring to have common code between gc and flush method Signed-off-by: alanprot <[email protected]> * Update tsdb/head.go Co-authored-by: Ayoub Mrini <[email protected]> Signed-off-by: Alan Protasio <[email protected]> * refactor Signed-off-by: alanprot <[email protected]> * Update tsdb/head_test.go Co-authored-by: Ganesh Vernekar <[email protected]> Signed-off-by: Alan Protasio <[email protected]> * Update tsdb/head_test.go Co-authored-by: Ganesh Vernekar <[email protected]> Signed-off-by: Alan Protasio <[email protected]> --------- Signed-off-by: alanprot <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Co-authored-by: Ayoub Mrini <[email protected]> Co-authored-by: Ganesh Vernekar <[email protected]> * Join errors Signed-off-by: Arve Knudsen <[email protected]> * docs/configuration: clarify OpenStack metadata labels (#14149) On several occasions, users assumed that the `__meta_openstack_tag_<key>` labels were about tags [1] instead of metadata [2]. While we can't really change the Prometheus label name, we can at least clarify in the documentation what's the information carried in the label. [1] https://specs.openstack.org/openstack/api-wg/guidelines/tags.html [2] https://docs.openstack.org/api-ref/compute/#server-metadata-servers-metadata Signed-off-by: Simon Pasquier <[email protected]> * BUGFIX: Need seperate listOptions structs since linodego writes into them for pagination Signed-off-by: David Andruczyk <[email protected]> * Upgrade to golangci-lint v1.59.0 Signed-off-by: Arve Knudsen <[email protected]> * Bump golangci-lint action (#14154) * Bump golangci-lint action to 6.0.1 * Synchronize script/golangci-lint.yml and workflows/ci.yml Signed-off-by: Matthieu MOREL <[email protected]> * Enable additional Go metrics Enable some additioal Go runtime metrics in order to observe additional performance data. Enables a number of new metrics: ``` HELP go_gc_cycles_automatic_gc_cycles_total Count of completed GC cycles generated by the Go runtime. HELP go_gc_cycles_forced_gc_cycles_total Count of completed GC cycles forced by the application. HELP go_gc_cycles_total_gc_cycles_total Count of all completed GC cycles. HELP go_gc_gogc_percent Heap size target percentage configured by the user, otherwise 100. This value is set by the GOGC environment variable, and the runtime/debug.SetGCPercent function. HELP go_gc_gomemlimit_bytes Go runtime memory limit configured by the user, otherwise math.MaxInt64. This value is set by the GOMEMLIMIT environment variable, and the runtime/debug.SetMemoryLimit function. HELP go_gc_heap_allocs_by_size_bytes Distribution of heap allocations by approximate size. Bucket counts increase monotonically. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. HELP go_gc_heap_allocs_bytes_total Cumulative sum of memory allocated to the heap by the application. HELP go_gc_heap_allocs_objects_total Cumulative count of heap allocations triggered by the application. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. HELP go_gc_heap_frees_by_size_bytes Distribution of freed heap allocations by approximate size. Bucket counts increase monotonically. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. HELP go_gc_heap_frees_bytes_total Cumulative sum of heap memory freed by the garbage collector. HELP go_gc_heap_frees_objects_total Cumulative count of heap allocations whose storage was freed by the garbage collector. Note that this does not include tiny objects as defined by /gc/heap/tiny/allocs:objects, only tiny blocks. HELP go_gc_heap_goal_bytes Heap size target for the end of the GC cycle. HELP go_gc_heap_live_bytes Heap memory occupied by live objects that were marked by the previous GC. HELP go_gc_heap_objects_objects Number of objects, live or unswept, occupying heap memory. HELP go_gc_heap_tiny_allocs_objects_total Count of small allocations that are packed together into blocks. These allocations are counted separately from other allocations because each individual allocation is not tracked by the runtime, only their block. Each block is already accounted for in allocs-by-size and frees-by-size. HELP go_gc_limiter_last_enabled_gc_cycle GC cycle the last time the GC CPU limiter was enabled. This metric is useful for diagnosing the root cause of an out-of-memory error, because the limiter trades memory for CPU time when the GC's CPU time gets too high. This is most likely to occur with use of SetMemoryLimit. The first GC cycle is cycle 1, so a value of 0 indicates that it was never enabled. HELP go_gc_pauses_seconds Deprecated. Prefer the identical /sched/pauses/total/gc:seconds. HELP go_gc_scan_globals_bytes The total amount of global variable space that is scannable. HELP go_gc_scan_heap_bytes The total amount of heap space that is scannable. HELP go_gc_scan_stack_bytes The number of bytes of stack that were scanned last GC cycle. HELP go_gc_scan_total_bytes The total amount space that is scannable. Sum of all metrics in /gc/scan. HELP go_gc_stack_starting_size_bytes The stack size of new goroutines. HELP go_sched_gomaxprocs_threads The current runtime.GOMAXPROCS setting, or the number of operating system threads that can execute user-level Go code simultaneously. HELP go_sched_goroutines_goroutines Count of live goroutines. HELP go_sched_latencies_seconds Distribution of the time goroutines have spent in the scheduler in a runnable state before actually running. Bucket counts increase monotonically. HELP go_sched_pauses_stopping_gc_seconds Distribution of individual GC-related stop-the-world stopping latencies. This is the time it takes from deciding to stop the world until all Ps are stopped. This is a subset of the total GC-related stop-the-world time (/sched/pauses/total/gc:seconds). During this time, some threads may be executing. Bucket counts increase monotonically. HELP go_sched_pauses_stopping_other_seconds Distribution of individual non-GC-related stop-the-world stopping latencies. This is the time it takes from deciding to stop the world until all Ps are stopped. This is a subset of the total non-GC-related stop-the-world time (/sched/pauses/total/other:seconds). During this time, some threads may be executing. Bucket counts increase monotonically. HELP go_sched_pauses_total_gc_seconds Distribution of individual GC-related stop-the-world pause latencies. This is the time from deciding to stop the world until the world is started again. Some of this time is spent getting all threads to stop (this is measured directly in /sched/pauses/stopping/gc:seconds), during which some threads may still be running. Bucket counts increase monotonically. HELP go_sched_pauses_total_other_seconds Distribution of individual non-GC-related stop-the-world pause latencies. This is the time from deciding to stop the world until the world is started again. Some of this time is spent getting all threads to stop (measured directly in /sched/pauses/stopping/other:seconds). Bucket counts increase monotonically. ``` Signed-off-by: SuperQ <[email protected]> * Prepare v2.52.1 release Signed-off-by: Arthur Silva Sens <[email protected]> * optimize regex matching for empty label values in posting match (#14075) Also update tests. Signed-off-by: Ben Ye <[email protected]> * [Test] TSDB: let BenchmarkAddExemplar reuse slots Test with different amounts of capacity and exemplars, so that sometimes new exemplars are evicting older exemplars. Signed-off-by: Bryan Boreham <[email protected]> * [Test] TSDB: BenchmarkResizeExemplar multiple per series One exemplar per series is not a typical workload. Make it the same as `BenchmarkAddExemplar`. Signed-off-by: Bryan Boreham <[email protected]> * [ENHANCEMENT] TSDB: Reduce map lookups on exemplar index In many cases we already have a pointer to the entry. Signed-off-by: Bryan Boreham <[email protected]> * [ENHANCEMENT] TSDB: Eliminate pointer when storing exemplars Saves memory and effort. Signed-off-by: Bryan Boreham <[email protected]> * [ENHANCEMENT] TSDB: let Resize re-use buffer This saves having to zero the buffer every time. Signed-off-by: Bryan Boreham <[email protected]> * [ENHANCEMENT] TSDB: Save map lookup on validation Goes faster. Signed-off-by: Bryan Boreham <[email protected]> * Feature: Allow configuration of a rule evaluation delay (#14061) * [PATCH] Allow having evaluation delay for rule groups Signed-off-by: Ganesh Vernekar <[email protected]> * [PATCH] Fix lint Signed-off-by: Ganesh Vernekar <[email protected]> * [PATCH] Move the option to ManagerOptions Signed-off-by: Ganesh Vernekar <[email protected]> * [PATCH] Include evaluation_delay in the group config Signed-off-by: Ganesh Vernekar <[email protected]> * Fix comments Signed-off-by: gotjosh <[email protected]> * Add a server configuration option. Signed-off-by: gotjosh <[email protected]> * Appease the linter #1 Signed-off-by: gotjosh <[email protected]> * Add the new server flag documentation Signed-off-by: gotjosh <[email protected]> * Improve documentation of the new flag and configuration Signed-off-by: gotjosh <[email protected]> * Use named parameters for clarity on the `Rule` interface Signed-off-by: gotjosh <[email protected]> * Add `initial` to the flag help Signed-off-by: gotjosh <[email protected]> * Change the CHANGELOG area from `ruler` to `rules` Signed-off-by: gotjosh <[email protected]> * Rename evaluation_delay to `rule_query_offset`/`query_offset` and make it a global configuration option. Signed-off-by: gotjosh <[email protected]> E Your branch is up to date with 'origin/gotjosh/evaluation-delay'. * more docs Signed-off-by: gotjosh <[email protected]> * Improve wording on CHANGELOG Signed-off-by: gotjosh <[email protected]> * Add `RuleQueryOffset` to the default config in tests in case it changes Signed-off-by: gotjosh <[email protected]> * Update docs/configuration/recording_rules.md Co-authored-by: Julius Volz <[email protected]> Signed-off-by: gotjosh <[email protected]> * Rename `RuleQueryOffset` to `QueryOffset` when in the group context. Signed-off-by: gotjosh <[email protected]> * Improve docstring and documentation on the `rule_query_offset` Signed-off-by: gotjosh <[email protected]> --------- Signed-off-by: Ganesh Vernekar <[email protected]> Signed-off-by: gotjosh <[email protected]> Co-authored-by: Ganesh Vernekar <[email protected]> Co-authored-by: Julius Volz <[email protected]> * build(deps): bump github.com/prometheus/client_golang Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.19.0 to 1.19.1. - [Release notes](https://github.com/prometheus/client_golang/releases) - [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/client_golang/compare/v1.19.0...v1.19.1) --- updated-dependencies: - dependency-name: github.com/prometheus/client_golang dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/checkout from 4.1.4 to 4.1.6 Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.4 to 4.1.6. - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v4.1.4...a5ac7e51b41094c92402da3b24376905380afc29) --- updated-dependencies: - dependency-name: actions/checkout dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump ossf/scorecard-action from 2.3.1 to 2.3.3 Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.3.1 to 2.3.3. - [Release notes](https://github.com/ossf/scorecard-action/releases) - [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md) - [Commits](https://github.com/ossf/scorecard-action/compare/0864cf19026789058feabb7e87baa5f140aac736...dc50aa9510b46c811795eb24b2f1ba02a914e534) --- updated-dependencies: - dependency-name: ossf/scorecard-action dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/hetznercloud/hcloud-go/v2 Bumps [github.com/hetznercloud/hcloud-go/v2](https://github.com/hetznercloud/hcloud-go) from 2.7.2 to 2.9.0. - [Release notes](https://github.com/hetznercloud/hcloud-go/releases) - [Changelog](https://github.com/hetznercloud/hcloud-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/hetznercloud/hcloud-go/compare/v2.7.2...v2.9.0) --- updated-dependencies: - dependency-name: github.com/hetznercloud/hcloud-go/v2 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 Bumps [github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5](https://github.com/Azure/azure-sdk-for-go) from 5.6.0 to 5.7.0. - [Release notes](https://github.com/Azure/azure-sdk-for-go/releases) - [Changelog](https://github.com/Azure/azure-sdk-for-go/blob/main/documentation/release.md) - [Commits](https://github.com/Azure/azure-sdk-for-go/compare/sdk/resourcemanager/compute/armcompute/v5.6.0...sdk/resourcemanager/compute/armcompute/v5.7.0) --- updated-dependencies: - dependency-name: github.com/Azure/azure-sdk-for-go/sdk/resourcemanager/compute/armcompute/v5 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/setup-go from 5.0.0 to 5.0.1 in /scripts Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.0.0 to 5.0.1. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/0c52d547c9bc32b1aa3301fd7a9cb496313a4491...cdcb36043654635271a94b9a6d1392de5bb323a7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump actions/setup-go from 5.0.0 to 5.0.1 Bumps [actions/setup-go](https://github.com/actions/setup-go) from 5.0.0 to 5.0.1. - [Release notes](https://github.com/actions/setup-go/releases) - [Commits](https://github.com/actions/setup-go/compare/0c52d547c9bc32b1aa3301fd7a9cb496313a4491...cdcb36043654635271a94b9a6d1392de5bb323a7) --- updated-dependencies: - dependency-name: actions/setup-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump bufbuild/buf-setup-action from 1.30.0 to 1.32.2 Bumps [bufbuild/buf-setup-action](https://github.com/bufbuild/buf-setup-action) from 1.30.0 to 1.32.2. - [Release notes](https://github.com/bufbuild/buf-setup-action/releases) - [Commits](https://github.com/bufbuild/buf-setup-action/compare/517ee23296d5caf38df31c21945e6a54bbc8a89f...dde0b9351db90fbf78e345f41a57de8514bf1091) --- updated-dependencies: - dependency-name: bufbuild/buf-setup-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * tsdb/index: Fix TestReader_PostingsForLabelMatchingHonorsContextCancel Fix number of series in TestReader_PostingsForLabelMatchingHonorsContextCancel (off by one). Signed-off-by: Arve Knudsen <[email protected]> * tsdb: add details to duplicate sample error (#13277) Now the error will include the timestamp and the existing and new values. When you are trying to track down the source of this error, it can be useful to see that the values are close, or alternating, or something else. Signed-off-by: Bryan Boreham <[email protected]> * build(deps): bump github.com/prometheus/prometheus Bumps [github.com/prometheus/prometheus](https://github.com/prometheus/prometheus) from 0.51.2 to 0.52.1. - [Release notes](https://github.com/prometheus/prometheus/releases) - [Changelog](https://github.com/prometheus/prometheus/blob/main/CHANGELOG.md) - [Commits](https://github.com/prometheus/prometheus/compare/v0.51.2...v0.52.1) --- updated-dependencies: - dependency-name: github.com/prometheus/prometheus dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * CI: Use default behavior of update-container-description-action Previously, we always used README.md as the readme to push as container description. By not explicitly specifying this file name, we use the default behavior of the action, which is to push README-containers.md if it exist, and push README.md otherwise. In short, nothing will directly change with this commit, but now repositories can provide a README-containers.md if they want to push a different README as the container description. Signed-off-by: beorn7 <[email protected]> * build(deps): bump the go-opentelemetry-io group across 1 directory with 9 updates Bumps the go-opentelemetry-io group with 6 updates in the / directory: | Package | From | To | | --- | --- | --- | | [go.opentelemetry.io/collector/pdata](https://github.com/open-telemetry/opentelemetry-collector) | `1.5.0` | `1.8.0` | | [go.opentelemetry.io/collector/semconv](https://github.com/open-telemetry/opentelemetry-collector) | `0.98.0` | `0.101.0` | | [go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp](https://github.com/open-telemetry/opentelemetry-go-contrib) | `0.50.0` | `0.52.0` | | [go.opentelemetry.io/otel/exporters/otlp/otlptrace](https://github.com/open-telemetry/opentelemetry-go) | `1.25.0` | `1.27.0` | | [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc](https://github.com/open-telemetry/opentelemetry-go) | `1.25.0` | `1.27.0` | | [go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp](https://github.com/open-telemetry/opentelemetry-go) | `1.25.0` | `1.27.0` | Updates `go.opentelemetry.io/collector/pdata` from 1.5.0 to 1.8.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](https://github.com/open-telemetry/opentelemetry-collector/compare/pdata/v1.5.0...pdata/v1.8.0) Updates `go.opentelemetry.io/collector/semconv` from 0.98.0 to 0.101.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-collector/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-collector/blob/main/CHANGELOG-API.md) - [Commits](https://github.com/open-telemetry/opentelemetry-collector/compare/v0.98.0...v0.101.0) Updates `go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp` from 0.50.0 to 0.52.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go-contrib/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go-contrib/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go-contrib/compare/zpages/v0.50.0...zpages/v0.52.0) Updates `go.opentelemetry.io/otel` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) Updates `go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) Updates `go.opentelemetry.io/otel/sdk` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) Updates `go.opentelemetry.io/otel/trace` from 1.25.0 to 1.27.0 - [Release notes](https://github.com/open-telemetry/opentelemetry-go/releases) - [Changelog](https://github.com/open-telemetry/opentelemetry-go/blob/main/CHANGELOG.md) - [Commits](https://github.com/open-telemetry/opentelemetry-go/compare/v1.25.0...v1.27.0) --- updated-dependencies: - dependency-name: go.opentelemetry.io/collector/pdata dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/collector/semconv dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel/sdk dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io - dependency-name: go.opentelemetry.io/otel/trace dependency-type: direct:production update-type: version-update:semver-minor dependency-group: go-opentelemetry-io ... Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> * build(deps): bump github/codeql-action from 3.22.12 to 3.25.7 Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.22.12 to 3.25.7. - [Release notes](https://github.com/github/codeql-action/releases) - [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md) - [Commits](https://github.com/github/codeql-action/compare/012739e5082ff0c22ca6d6ab32e07c36df03c4a4...f079b8493333aace61c81488f8bd40919487bd9f) --- updated-dependencies: - dependency-name: github/codeql-action dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/docker/docker Bumps [github.com/docker/docker](https://github.com/docker/docker) from 26.0.1+incompatible to 26.1.3+incompatible. - [Release notes](https://github.com/docker/docker/releases) - [Commits](https://github.com/docker/docker/compare/v26.0.1...v26.1.3) --- updated-dependencies: - dependency-name: github.com/docker/docker dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Update Go dependencies for 2.53 Ran "make update-all-go-deps" , but do not bump go version and remove toolchain clause from go.mod Signed-off-by: György Krajcsovits <[email protected]> * tsdb: Allow passing a custom compactor to override the default one (#14113) * expose hook in tsdb to allow customizing compactor Signed-off-by: Ben Ye <[email protected]> * address comment Signed-off-by: Ben Ye <[email protected]> --------- Signed-off-by: Ben Ye <[email protected]> * Revert update of opentelemetry collector components These bring in a hard dependecy on toolchain we don't want via https://github.com/open-telemetry/opentelemetry-collector/pull/10165 The dependency should be going away https://github.com/open-telemetry/opentelemetry-collector/pull/10165#pullrequestreview-2060814562 Signed-off-by: György Krajcsovits <[email protected]> * Revert change to scaleway-sdk-go The change brings in non trivial deprecations. Multiple public IPs can be returned by the API now instead of one. Need to decide what to do in that case as we used a single meta label for that __meta_scaleway_instance_public_ipv4. Signed-off-by: György Krajcsovits <[email protected]> * Add configuration option for GOGC Add the ability to adjust the `GOGC` variable from the Prometheus configuration file. * Create a new top-level `runtime` section in the config. * Adjust from the Go default of 100 to 50 to reduce wasted memory. * Use the `GOGC` env value if no configuraiton is used. Signed-off-by: SuperQ <[email protected]> * Increase the depndabot open PR limit The default limit of 5 is a bit small given the number of dependencies we have for Go and JS. Increase to 20 to allow more updates to be pushed. Signed-off-by: SuperQ <[email protected]> * Update scaleway-sdk-go and silence deprecation warnings This reverts commit 777daea86e0ee2cee1e00f3cd75293bde9aa7f51. * Format linter pragma correctly Signed-off-by: György Krajcsovits <[email protected]> * build(deps): bump golang.org/x/net from 0.25.0 to 0.26.0 Bumps [golang.org/x/net](https://github.com/golang/net) from 0.25.0 to 0.26.0. - [Commits](https://github.com/golang/net/compare/v0.25.0...v0.26.0) --- updated-dependencies: - dependency-name: golang.org/x/net dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/digitalocean/godo from 1.116.0 to 1.117.0 Bumps [github.com/digitalocean/godo](https://github.com/digitalocean/godo) from 1.116.0 to 1.117.0. - [Release notes](https://github.com/digitalocean/godo/releases) - [Changelog](https://github.com/digitalocean/godo/blob/main/CHANGELOG.md) - [Commits](https://github.com/digitalocean/godo/compare/v1.116.0...v1.117.0) --- updated-dependencies: - dependency-name: github.com/digitalocean/godo dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * build(deps): bump github.com/aws/aws-sdk-go from 1.53.15 to 1.53.16 Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.53.15 to 1.53.16. - [Release notes](https://github.com/aws/aws-sdk-go/releases) - [Commits](https://github.com/aws/aws-sdk-go/compare/v1.53.15...v1.53.16) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> * RuleQueryOffset: Add omitempty for the global configuration (#14216) A small oversight of when I introduced https://github.com/prometheus/prometheus/pull/14061, I could add a test to cover it but it seems like an overkill given other similar attributes don't have it either. Let me know if you think it's worth it. Signed-off-by: gotjosh <[email protected]> * build(deps): bump golang.org/x/tools from 0.21.0 to 0.22.0 Bumps [golang.org/x/tools](https://github.com/golang/tools) from 0.21.0 to 0.22.0. - [Release notes](https://github.com/golang/tools/releases) - [Commits](https://github.com/golang/tools/compare/v0.21.0...v0.22.0) --- updated-dependencies: - dependency-name: golang.org/x/tools dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * CI: Actually enable the default fallback behavior of docker-pushrm The Github action explicitly sets `README.md` as the default file to push, see https://github.com/christian-korneck/update-container-description-action/blob/master/action.yml#L17 This disables the fallback to `README-containers.md`, as implemented in the actual tool that the Github action uses, i.e. https://github.com/christian-korneck/docker-pushrm However, by setting the file name explicitly to an empty string, we can trigger the default fallback behavior of dockre-pushrm after all. Signed-off-by: beorn7 <[email protected]> * build(deps): bump google.golang.org/api from 0.182.0 to 0.183.0 Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.182.0 to 0.183.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.182.0...v0.183.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> * Version bump to v2.53.0-rc.0 Signed-off-by: György Krajcsovits <[email protected]> * Ammend changelog with missing user impact entries Signed-off-by: György Krajcsovits <[email protected]> * Clarify action to take with regards to the changelog Signed-off-by: György Krajcsovits <[email protected]> * Fix typo in changelog Signed-off-by: György Krajcsovits <[email protected]> * Update changelog from review comments Signed-off-by: György Krajcsovits <[email protected]> * Fix Group.Equals() to take in account the new queryOffset too (#14273) Signed-off-by: Marco Pracucci <[email protected]> * Update changelog due to pr 14273 Signed-off-by: György Krajcsovits <[email protected]> * Tune default GOGC Adjust the default GOGC value to 75. This is less of a memory savings, but has less impact on CPU use. Signed-off-by: SuperQ <[email protected]> * Update changelog for GOGC tuning Include #14285 in changelog. Signed-off-by: SuperQ <[email protected]> * Revert "Update changelog due to pr 14273" This reverts commit dd4400146521c996239da57d2a225a608e3915cb. Signed-off-by: György Krajcsovits <[email protected]> * Prepare 2.53.0-rc.1 release Signed-off-by: György Krajcsovits <[email protected]> * Prepare release 2.53.0 Signed-off-by: György Krajcsovits <[email protected]> * Update CHANGELOG.md Co-authored-by: Julien <[email protected]> Signed-off-by: George Krajcsovits <[email protected]> * [release 2.53] Revert 13583 to stop dropping samples in remote-write catch-up (#14446) * Revert "fix bug that would cause us to endlessly fall behind (#13583)" This reverts commit 0c71230784368da829f1f02d412d181d7a06aee6. (leaving the new test in place) * TSDB: enhance TestRun_AvoidNotifyWhenBehind With code suggested by @cstyan in #14439. * WAL watcher: add back log line showing current segment --------- Signed-off-by: Bryan Boreham <[email protected]> * Prepare release 2.53.1 (#14452) Co-authored-by: George Krajcsovits <[email protected]> Signed-off-by: Bryan Boreham <[email protected]> * [bot] assets: generate Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> * [create-pull-request] automated change Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> --------- Signed-off-by: Mickael Carl <[email protected]> Signed-off-by: tyltr <[email protected]> Signed-off-by: roger.wang <[email protected]> Signed-off-by: Federico Leva <[email protected]> Signed-off-by: machine424 <[email protected]> Signed-off-by: komisan19 <[email protected]> Signed-off-by: gotjosh <[email protected]> Signed-off-by: tesla59 <[email protected]> Signed-off-by: alanprot <[email protected]> Signed-off-by: Arthur Silva Sens <[email protected]> Signed-off-by: George Robinson <[email protected]> Signed-off-by: Stephen Heckler <[email protected]> Signed-off-by: Nishant Singh <[email protected]> Signed-off-by: Heyoxe <[email protected]> Signed-off-by: Neeraj Gartia <[email protected]> Signed-off-by: guoguangwu <[email protected]> Signed-off-by: Jesus Vazquez <[email protected]> Signed-off-by: Jiekun <[email protected]> Signed-off-by: Arve Knudsen <[email protected]> Signed-off-by: György Krajcsovits <[email protected]> Signed-off-by: SuperQ <[email protected]> Signed-off-by: dependabot[bot] <[email protected]> Signed-off-by: kushagra Shukla <[email protected]> Signed-off-by: Kushal shukla <[email protected]> Signed-off-by: Oleg Zaytsev <[email protected]> Signed-off-by: Pranshu Srivastava <[email protected]> Signed-off-by: Bryan Boreham <[email protected]> Signed-off-by: Anthony J Mirabella <[email protected]> Signed-off-by: Charles Korn <[email protected]> Signed-off-by: Oleksandr Redko <[email protected]> Signed-off-by: beorn7 <[email protected]> Signed-off-by: Jayapriya Pai <[email protected]> Signed-off-by: Mohamed Awnallah <[email protected]> Signed-off-by: Julien <[email protected]> Signed-off-by: Alan Protasio <[email protected]> Signed-off-by: Simon Pasquier <[email protected]> Signed-off-by: David Andruczyk <[email protected]> Signed-off-by: Matthieu MOREL <[email protected]> Signed-off-by: Arthur Silva Sens <[email protected]> Signed-off-by: Ben Ye <[email protected]> Signed-off-by: Ganesh Vernekar <[email protected]> Signed-off-by: Ben Kochie <[email protected]> Signed-off-by: Marco Pracucci <[email protected]> Signed-off-by: George Krajcsovits <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Mickael Carl <[email protected]> Co-authored-by: tyltr <[email protected]> Co-authored-by: roger.wang <[email protected]> Co-authored-by: Federico Leva <[email protected]> Co-authored-by: Bartlomiej Plotka <[email protected]> Co-authored-by: machine424 <[email protected]> Co-authored-by: komisan19 <[email protected]> Co-authored-by: gotjosh <[email protected]> Co-authored-by: tesla59 <[email protected]> Co-authored-by: Alan Protasio <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> Co-authored-by: George Robinson <[email protected]> Co-authored-by: Arthur Silva Sens <[email protected]> Co-authored-by: Stephen Heckler <[email protected]> Co-authored-by: Heyoxe <[email protected]> Co-authored-by: Neeraj Gartia <[email protected]> Co-authored-by: Julien <[email protected]> Co-authored-by: guangwu <[email protected]> Co-authored-by: Jesus Vazquez <[email protected]> Co-authored-by: Arve Knudsen <[email protected]> Co-authored-by: Jiekun <[email protected]> Co-authored-by: György Krajcsovits <[email protected]> Co-authored-by: George Krajcsovits <[email protected]> Co-authored-by: SuperQ <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Augustin Husson <[email protected]> Co-authored-by: Kushal shukla <[email protected]> Co-authored-by: Oleg Zaytsev <[email protected]> Co-authored-by: Pranshu Srivastava <[email protected]> Co-authored-by: Bryan Boreham <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: Björn Rabenstein <[email protected]> Co-authored-by: Anthony Mirabella <[email protected]> Co-authored-by: Charles Korn <[email protected]> Co-authored-by: Neeraj Gartia <[email protected]> Co-authored-by: Oleksandr Redko <[email protected]> Co-authored-by: Matthias Loibl <[email protected]> Co-authored-by: kushagra Shukla <[email protected]> Co-authored-by: Jayapriya Pai <[email protected]> Co-authored-by: Mohamed Awnallah <[email protected]> Co-authored-by: Ganesh Vernekar <[email protected]> Co-authored-by: Simon Pasquier <[email protected]> Co-authored-by: David Andruczyk <[email protected]> Co-authored-by: Matthieu MOREL <[email protected]> Co-authored-by: Ben Ye <[email protected]> Co-authored-by: Julius Volz <[email protected]> Co-authored-by: Marco Pracucci <[email protected]> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

alanprot requested a review from jesusvazquez as a code owner May 10, 2024 21:57

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch 4 times, most recently from f8d34ad to 6ee4190 Compare May 10, 2024 23:23

yeya24 approved these changes May 11, 2024

View reviewed changes

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch 3 times, most recently from cb8c835 to 61bf6b6 Compare May 13, 2024 18:19

bboreham reviewed May 14, 2024

View reviewed changes

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch from 0b2ec4b to 70a26fd Compare May 14, 2024 16:24

alanprot requested a review from bboreham May 21, 2024 17:11

machine424 reviewed May 24, 2024

View reviewed changes

alanprot and others added 4 commits May 24, 2024 12:32

Fixing head stats and hooks when replaying a corrupted snapshot

3ae7efe

Signed-off-by: alanprot <[email protected]>

Fixing create/removed series metrics

97f4263

Signed-off-by: alanprot <[email protected]>

Refactoring to have common code between gc and flush method

250271a

Signed-off-by: alanprot <[email protected]>

Update tsdb/head.go

37aae47

Co-authored-by: Ayoub Mrini <[email protected]> Signed-off-by: Alan Protasio <[email protected]>

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch from a14e433 to 37aae47 Compare May 24, 2024 19:32

codesome reviewed May 24, 2024

View reviewed changes

tsdb/head_test.go Outdated Show resolved Hide resolved

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch from cbc345d to c19fb3c Compare May 24, 2024 21:15

refactor

ad98c84

Signed-off-by: alanprot <[email protected]>

alanprot force-pushed the fix-corrupted-snapshot-callbacks branch from c19fb3c to ad98c84 Compare May 24, 2024 21:17

codesome approved these changes May 24, 2024

View reviewed changes

tsdb/head_test.go Outdated Show resolved Hide resolved

tsdb/head_test.go Outdated Show resolved Hide resolved

alanprot and others added 2 commits May 24, 2024 14:41

Update tsdb/head_test.go

3d6b15d

Co-authored-by: Ganesh Vernekar <[email protected]> Signed-off-by: Alan Protasio <[email protected]>

Update tsdb/head_test.go

f5fb207

Co-authored-by: Ganesh Vernekar <[email protected]> Signed-off-by: Alan Protasio <[email protected]>

codesome merged commit 8894d65 into prometheus:main May 25, 2024
25 checks passed

machine424 reviewed May 25, 2024

View reviewed changes

gotjosh mentioned this pull request May 30, 2024

Merge upstream prometheus/prometheus at 37b408c grafana/mimir-prometheus#639

Merged

pracucci mentioned this pull request Jun 6, 2024

Update vendored prometheus grafana/mimir#8295

Merged

4 tasks

alanprot mentioned this pull request Jun 10, 2024

Update Prometheus and Thanos cortexproject/cortex#6002

Merged

1 task

yeya24 mentioned this pull request Jul 31, 2024

prometheus_tsdb_head_series increases to double previous value during WAL replay then drops #13385

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix head stats and hooks when replaying a corrupted snapshot #14079

Fix head stats and hooks when replaying a corrupted snapshot #14079

alanprot commented May 10, 2024 •

edited

Loading

yeya24 left a comment

bboreham left a comment

bboreham May 14, 2024

alanprot May 14, 2024 •

edited

Loading

alanprot May 14, 2024

machine424 May 24, 2024

codesome May 24, 2024

alanprot May 24, 2024

alanprot May 24, 2024

bboreham May 14, 2024

alanprot May 14, 2024

machine424 May 24, 2024

alanprot May 24, 2024

alanprot May 24, 2024

alanprot commented May 16, 2024

jeromeinsf commented May 22, 2024

machine424 left a comment

machine424 May 24, 2024

machine424 May 24, 2024

codesome May 24, 2024

alanprot May 24, 2024 •

edited

Loading

machine424 May 24, 2024

machine424 May 24, 2024

codesome left a comment

codesome May 24, 2024

codesome left a comment •

edited

Loading

codesome left a comment

machine424 May 25, 2024

alanprot May 27, 2024

machine424 May 27, 2024

alanprot May 27, 2024

machine424 commented May 25, 2024

Fix head stats and hooks when replaying a corrupted snapshot #14079

Fix head stats and hooks when replaying a corrupted snapshot #14079

Conversation

alanprot commented May 10, 2024 • edited Loading

yeya24 left a comment

Choose a reason for hiding this comment

bboreham left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanprot May 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanprot commented May 16, 2024

jeromeinsf commented May 22, 2024

machine424 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alanprot May 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codesome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codesome left a comment • edited Loading

Choose a reason for hiding this comment

codesome left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

machine424 commented May 25, 2024

alanprot commented May 10, 2024 •

edited

Loading

alanprot May 14, 2024 •

edited

Loading

alanprot May 24, 2024 •

edited

Loading

codesome left a comment •

edited

Loading