31 Aug 14:13

7f5c066

v0.32.2

This patch release again brings a few fixes for the Store Gateway. In the previous release, certain users reported receiving a nil postings error on some of their queries, alongside certain race conditions, and empty query stats on debug messages. These have now been addressed.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

#6675 Store: Fix race when iterating blocks
#6679 Store: Record stats even on ExpandPostings error
#6681 Store: Fix forgotten field in store stats merge
#6684 Store: Fix postings reader short reads to address nil postings bug

Assets 9

0 Join discussion

28 Aug 11:45

saswatamcode

v0.32.1

5bf3a9e

v0.32.1

This patch release contains a few fixes for the Store Gateway. In the previous release, a memory issue was detected in the label names set implementation, alongside certain ignored or unchecked errors in store postings. These have now been addressed.
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Fixed

#6650 Store: fix error handling in decodePostings
#6654 Store: fix ignored error in postings
#6655 Store: fix bufio pool handling
#6669 Store: Fix mutable stringset memory usage

Assets 9

0 Join discussion

23 Aug 13:55

saswatamcode

v0.32.0

041eb41

v0.32.0

v0.32.0 is out after a long wait, as we were busy fixing a rather challenging issue!
Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.
Some of the highlights include support for PromQL query explanations in the UI when using the thanos PromQL engine, AZ-aware replication for Receive and other new flags, tools bucket replicate improvements, and lots of optimizations and bug/race fixes!
Do take note of some of the breaking metric name changes and the change in container image user.
You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems! Enjoy! 🎉

Changes

Added

#6437 Receive: make tenant stats limit configurable
#6369 Receive: add az-aware replication support for Ketama algorithm
#6185 Tracing: tracing in OTLP support configuring service_name.
#6192 Store: add flag bucket-web-label to select the label to use as timeline title in web UI
#6195 Receive: add flag tsdb.too-far-in-future.time-window to prevent clock skewed samples to pollute TSDB head and block all valid incoming samples.
#6273 Mixin: Allow specifying an instance name filter in dashboards
#6163 Receiver: Add hidden flag --receive-forward-max-backoff to configure the max backoff for forwarding requests.
#5777 Receive: Allow specifying tenant-specific external labels in Router Ingestor.
#6352 Store: Expose store gateway query stats in series response hints.
#6420 Index Cache: Cache expanded postings.
#6441 Compact: Compactor will set index_stats in meta.json file with max series and chunk size information.
#6466 Mixin (Receive): add limits alerting for configuration reload and meta-monitoring.
#6467 Mixin (Receive): add alert for tenant reaching head series limit.
#6528 Index Cache: Add histogram metric thanos_store_index_cache_stored_data_size_bytes for item size.
#6560 Thanos ruler: add flag to optionally disable adding Thanos params when querying metrics
#6574 Tools: Add min and max compactions range flags to bucket replicate command.
#6593 Store: Add thanos_bucket_store_chunk_refetches_total metric to track number of chunk refetches.
#6264 Query: Add Thanos logo in navbar
#6234 Query: Add ability to switch between thanos and prometheus engines dynamically via UI and API.
#6346 Query: Add ability to generate SQL-like query explanations when thanos engine is used.

Fixed

#6503 *: Change the engine behind ContentPathReloader to be completely independent of any filesystem concept. This effectively fixes this configuration reload when used with Kubernetes ConfigMaps, Secrets, or other volume mounts.
#6456 Store: fix crash when computing set matches from regex pattern
#6427 Receive: increased log level for failed uploads to error
#6172 query-frontend: return JSON formatted errors for invalid PromQL expression in the split by interval middleware.
#6171 Store: fix error handling on limits.
#6183 Receiver: fix off by one in multitsdb flush that will result in empty blocks if the head only contains one sample
#6197 Exemplar OTel: Fix exemplar for otel to use traceId instead of spanId and sample only if trace is sampled
#6207 Receive: Remove the shipper once a tenant has been pruned.
#6216 Receiver: removed hard-coded value of EnableExemplarStorage flag and set it according to max-exemplar value.
#6222 mixin(Receive): Fix tenant series received dashboard widget.
#6218 mixin(Store): handle ResourceExhausted as a non-server error. As a consequence, this error won't contribute to Store's grpc errors alerts.
#6271 Receive: Fix segfault in LabelValues during head compaction.
#6306 Tracing: tracing in OTLP utilize the OTEL_TRACES_SAMPLER env variable
#6330 Store: Fix inconsistent error for series limits.
#6342 Cache/Redis: Upgrade rueidis to v1.0.2 to to improve error handling while shrinking a redis cluster.
#6325 Store: return gRPC resource exhausted error for byte limiter.
#6399 *: Fix double-counting bug in http_request_duration metric
#6428 Report gRPC connnection errors in the logs.
#6519 Reloader: Use timeout for initial apply.
#6509 Store Gateway: Remove memWriter from fileWriter to reduce memory usage when sync index headers.
#6556 Thanos compact: respect block-files-concurrency setting when downsampling
#6592 Query Frontend: fix bugs in vertical sharding without and union function to allow more queries to be shardable.
#6317 *: Fix internal label deduplication bug, by resorting store response set.
#6189 Rule: Fix panic when calling API /api/v1/rules?type=alert.

Changed

#6049 Compact: breaking ⚠️ Replace group with resolution in compact metrics to avoid cardinality explosion on compact metrics for large numbers of groups.
#6168 Receiver: Make ketama hashring fail early when configured with number of nodes lower than the replication factor.
#6201 Query-Frontend: Disable absent and absent_over_time for vertical sharding.
#6212 Query-Frontend: Disable scalar for vertical sharding.
#6107 breaking ⚠️ Change default user id in container image from 0(root) to 1001
#6228 Conditionally generate debug messages in ProxyStore to avoid memory bloat.
#6231 mixins: Add code/grpc-code dimension to error widgets.
#6244 mixin(Rule): Add rule evaluation failures to the Rule dashboard.
#6303 Store: added and start using streamed snappy encoding for postings list instead of block based one. This leads to constant memory usage during decompression. This approximately halves memory usage when decompressing a postings list in index cache.
#6071 Query Frontend: breaking ⚠️ Add experimental native histogram support for which we updated and aligned with the Prometheus common model, which is used for caching so a cache reset required.
#6163 Receiver: changed default max backoff from 30s to 5s for forwarding requests. Can be configured with --receive-forward-max-backoff.
#6327 *: breaking ⚠️ Use histograms instead of summaries for instrumented handlers.
#6322 Logging: Avoid expensive log.Valuer evaluation for disallowed levels.
#6358 Query: Add +Inf bucket to query duration metrics
#6363 Store: Check context error when expanding postings.
#6405 Index Cache: Change postings cache key to include the encoding format used so that older Thanos versions would not try to decode it during the deployment of a new version.
#6479 Store: breaking ⚠️ Rename thanos_bucket_store_cached_series_fetch_duration_seconds to thanos_bucket_store_series_fetch_duration_seconds and thanos_bucket_store_cached_postings_fetch_duration_seconds to thanos_bucket_store_postings_fetch_duration_seconds.
#6474 Store/Compact: Reduce a ...

Contributors

jianwu, TomMD, and 33 other contributors

Assets 9

0 Join discussion

21 Aug 09:08

saswatamcode

v0.32.0-rc.1

0ea99ac

v0.32.0-rc.1 Pre-release

Pre-release

#6612: Store: Fix missing flush when handling pushed down queries

Assets 9

0 Join discussion

16 Aug 11:39

saswatamcode

v0.32.0-rc.0

48a30b6

v0.32.0-rc.0 Pre-release

Pre-release

v0.32.0-rc.0 is out after a long wait, as we were busy fixing a rather challenging issue!
Thank you to all contributors who have contributed to this release. It wouldn't be possible without you.

Some of the highlights include support for PromQL query explanations in the UI when using the thanos PromQL engine, AZ-aware replication for Receive and other new flags, tools bucket replicate improvements, and lots of optimizations and bug/race fixes!
Do take note of some of the breaking metric name changes and the change in container image user.

You can find the changelog with all of the details below. Let's also celebrate all our new contributors!
Please try it out and let us know if you spot any problems! Full-release/next rc will be in 3 days!

Changes

Added

#6437 Receive: make tenant stats limit configurable
#6369 Receive: add az-aware replication support for Ketama algorithm
#6185 Tracing: tracing in OTLP support configuring service_name.
#6192 Store: add flag bucket-web-label to select the label to use as timeline title in web UI
#6195 Receive: add flag tsdb.too-far-in-future.time-window to prevent clock skewed samples to pollute TSDB head and block all valid incoming samples.
#6273 Mixin: Allow specifying an instance name filter in dashboards
#6163 Receiver: Add hidden flag --receive-forward-max-backoff to configure the max backoff for forwarding requests.
#5777 Receive: Allow specifying tenant-specific external labels in Router Ingestor.
#6352 Store: Expose store gateway query stats in series response hints.
#6420 Index Cache: Cache expanded postings.
#6441 Compact: Compactor will set index_stats in meta.json file with max series and chunk size information.
#6466 Mixin (Receive): add limits alerting for configuration reload and meta-monitoring.
#6467 Mixin (Receive): add alert for tenant reaching head series limit.
#6528 Index Cache: Add histogram metric thanos_store_index_cache_stored_data_size_bytes for item size.
#6560 Thanos ruler: add flag to optionally disable adding Thanos params when querying metrics
#6574 Tools: Add min and max compactions range flags to bucket replicate command.
#6593 Store: Add thanos_bucket_store_chunk_refetches_total metric to track number of chunk refetches.
#6264 Query: Add Thanos logo in navbar
#6234 Query: Add ability to switch between thanos and prometheus engines dynamically via UI and API.
#6346 Query: Add ability to generate SQL-like query explanations when thanos engine is used.

Fixed

#6503 *: Change the engine behind ContentPathReloader to be completely independent of any filesystem concept. This effectively fixes this configuration reload when used with Kubernetes ConfigMaps, Secrets, or other volume mounts.
#6456 Store: fix crash when computing set matches from regex pattern
#6427 Receive: increased log level for failed uploads to error
#6172 query-frontend: return JSON formatted errors for invalid PromQL expression in the split by interval middleware.
#6171 Store: fix error handling on limits.
#6183 Receiver: fix off by one in multitsdb flush that will result in empty blocks if the head only contains one sample
#6197 Exemplar OTel: Fix exemplar for otel to use traceId instead of spanId and sample only if trace is sampled
#6207 Receive: Remove the shipper once a tenant has been pruned.
#6216 Receiver: removed hard-coded value of EnableExemplarStorage flag and set it according to max-exemplar value.
#6222 mixin(Receive): Fix tenant series received dashboard widget.
#6218 mixin(Store): handle ResourceExhausted as a non-server error. As a consequence, this error won't contribute to Store's grpc errors alerts.
#6271 Receive: Fix segfault in LabelValues during head compaction.
#6306 Tracing: tracing in OTLP utilize the OTEL_TRACES_SAMPLER env variable
#6330 Store: Fix inconsistent error for series limits.
#6342 Cache/Redis: Upgrade rueidis to v1.0.2 to to improve error handling while shrinking a redis cluster.
#6325 Store: return gRPC resource exhausted error for byte limiter.
#6399 *: Fix double-counting bug in http_request_duration metric
#6428 Report gRPC connnection errors in the logs.
#6519 Reloader: Use timeout for initial apply.
#6509 Store Gateway: Remove memWriter from fileWriter to reduce memory usage when sync index headers.
#6556 Thanos compact: respect block-files-concurrency setting when downsampling
#6592 Query Frontend: fix bugs in vertical sharding without and union function to allow more queries to be shardable.
#6317 *: Fix internal label deduplication bug, by resorting store response set.
#6189 Rule: Fix panic when calling API /api/v1/rules?type=alert.

Changed

#6049 Compact: breaking ⚠️ Replace group with resolution in compact metrics to avoid cardinality explosion on compact metrics for large numbers of groups.
#6168 Receiver: Make ketama hashring fail early when configured with number of nodes lower than the replication factor.
#6201 Query-Frontend: Disable absent and absent_over_time for vertical sharding.
#6212 Query-Frontend: Disable scalar for vertical sharding.
#6107 breaking ⚠️ Change default user id in container image from 0(root) to 1001
#6228 Conditionally generate debug messages in ProxyStore to avoid memory bloat.
#6231 mixins: Add code/grpc-code dimension to error widgets.
#6244 mixin(Rule): Add rule evaluation failures to the Rule dashboard.
#6303 Store: added and start using streamed snappy encoding for postings list instead of block based one. This leads to constant memory usage during decompression. This approximately halves memory usage when decompressing a postings list in index cache.
#6071 Query Frontend: breaking ⚠️ Add experimental native histogram support for which we updated and aligned with the Prometheus common model, which is used for caching so a cache reset required.
#6163 Receiver: changed default max backoff from 30s to 5s for forwarding requests. Can be configured with --receive-forward-max-backoff.
#6327 *: breaking ⚠️ Use histograms instead of summaries for instrumented handlers.
#6322 Logging: Avoid expensive log.Valuer evaluation for disallowed levels.
#6358 Query: Add +Inf bucket to query duration metrics
#6363 Store: Check context error when expanding postings.
#6405 Index Cache: Change postings cache key to include the encoding format used so that older Thanos versions would not try to decode it during the deployment of a new version.
#6479 Store: breaking ⚠️ Rename thanos_bucket_store_cached_series_fetch_duration_seconds to thanos_bucket_store_series_fetch_duration_seconds and thanos_bucket_store_cached_postings_fetch_duration_seconds to thanos_bucket_store_postings_fetch_duration_seconds.
[#6474](https://github.com/thanos-io/...

Contributors

jianwu, TomMD, and 34 other contributors

Assets 9

0 Join discussion

23 Mar 10:32

fpetkovski

v0.31.0

50c4641

v0.31.0

What's Changed

Added

#5990 Cache/Redis: add support for Redis Sentinel via new option master_name.
#6008 *: Add counter metric gate_queries_total to gate.
#5926 Receiver: Add experimental string interning in writer. Can be enabled with a hidden flag --writer.intern.
#5773 Store: Support disabling cache index header file by setting --disable-caching-index-header-file. When toggled, Stores can run without needing persistent disks.
#5653 Receive: Allow setting hashing algorithm per tenant in hashrings config.
#6074 *: Add histogram metrics thanos_store_server_series_requested and thanos_store_server_chunks_requested to all Stores.
#6074 *: Allow configuring series and sample limits per Series request for all Stores.
#6104 Store: Support S3 session token.
#5548 Query: Add experimental support for load balancing across multiple Store endpoints.
#6148 Query-frontend: Add traceID to slow query detected log line.
#6153 Query-frontend: Add remote_user (from http basic auth) and remote_addr to slow query detected log line.

Fixed

#5995 Sidecar: Loads TLS certificate during startup.
#6044 Receive: Mark out-of-window errors as conflict when out-of-window samples ingestion is used.
#6050 Store: Re-try bucket store initial sync upon failure.
#6067 Receive: Fix panic when querying uninitialized TSDBs.
#6082 Query: Don't error when no stores are matched.
#6098 Cache/Redis: Upgrade rueidis to v0.0.93 to fix potential panic when the client-side caching is disabled.
#6103 Mixins(Rule): Fix expression for long rule evaluations.
#6121 Receive: Deduplicate meta-monitoring queries for Active Series Limiting.
#6137 Downsample: Repair of non-empty XOR chunks during 1h downsampling.
#6125 Query Frontend: Fix vertical shardable instant queries do not produce sorted results for sort, sort_desc, topk and bottomk functions.
#6203 Receive: Fix panic in head compaction under high query load.

Changed

#6010 *: Upgrade Prometheus to v0.42.0.
#5999 *: Upgrade Alertmanager dependency to v0.25.0.
#5887 Tracing: Make sure rate limiting sampler is the default, as was the case in version pre-0.29.0.
#5997 Rule: switch to miekgdns DNS resolver as the default one.
#6035 Tools (replicate): Support all types of matchers to match blocks for replication. Change matcher parameter from string slice to a single string.
#6131 Store: breaking ⚠️ Use Histograms instead of Summaries for bucket metrics.

Assets 9

13 Mar 11:50

fpetkovski

v0.31.0-rc.1

f17d17c

v0.31.0-rc.1 Pre-release

Pre-release

#6203 Receive: Fix panic in head compaction under high query load.

Assets 9

24 Feb 15:26

fpetkovski

v0.31.0-rc.0

5262948

v0.31.0-rc.0 Pre-release

Pre-release

What's Changed

Added

#5990 Cache/Redis: add support for Redis Sentinel via new option master_name.
#6008 *: Add counter metric gate_queries_total to gate.
#5926 Receiver: Add experimental string interning in writer. Can be enabled with a hidden flag --writer.intern.
#5773 Store: Support disabling cache index header file by setting --disable-caching-index-header-file. When toggled, Stores can run without needing persistent disks.
#5653 Receive: Allow setting hashing algorithm per tenant in hashrings config.
#6074 *: Add histogram metrics thanos_store_server_series_requested and thanos_store_server_chunks_requested to all Stores.
#6074 *: Allow configuring series and sample limits per Series request for all Stores.
#6104 Store: Support S3 session token.
#5548 Query: Add experimental support for load balancing across multiple Store endpoints.
#6148 Query-frontend: Add traceID to slow query detected log line.
#6153 Query-frontend: Add remote_user (from http basic auth) and remote_addr to slow query detected log line.

Fixed

#5995 Sidecar: Loads TLS certificate during startup.
#6044 Receive: Mark out-of-window errors as conflict when out-of-window samples ingestion is used.
#6050 Store: Re-try bucket store initial sync upon failure.
#6067 Receive: Fix panic when querying uninitialized TSDBs.
#6082 Query: Don't error when no stores are matched.
#6098 Cache/Redis: Upgrade rueidis to v0.0.93 to fix potential panic when the client-side caching is disabled.
#6103 Mixins(Rule): Fix expression for long rule evaluations.
#6121 Receive: Deduplicate meta-monitoring queries for Active Series Limiting.
#6137 Downsample: Repair of non-empty XOR chunks during 1h downsampling.
#6125 Query Frontend: Fix vertical shardable instant queries do not produce sorted results for sort, sort_desc, topk and bottomk functions.

Changed

#6010 *: Upgrade Prometheus to v0.42.0.
#5999 *: Upgrade Alertmanager dependency to v0.25.0.
#5887 Tracing: Make sure rate limiting sampler is the default, as was the case in version pre-0.29.0.
#5997 Rule: switch to miekgdns DNS resolver as the default one.
#6035 Tools (replicate): Support all types of matchers to match blocks for replication. Change matcher parameter from string slice to a single string.
#6131 Store: breaking ⚠️ Use Histograms instead of Summaries for bucket metrics.

New Contributors

@JoaoBraveCoding made their first contribution in #5997
@rueian made their first contribution in #5998
@ashwinsrinivasmurthy made their first contribution in #6042
@farodin91 made their first contribution in #6044
@danielmellado made their first contribution in #6052
@xBazilio made their first contribution in #6066
@PradyumnaKrishna made their first contribution in #6063
@Kartik-Garg made their first contribution in #6050
@pmoncadaisla made their first contribution in #6084
@sshantel made their first contribution in #6087
@jkowall made their first contribution in #6078
@m-messiah made their first contribution in #6058
@abaguas made their first contribution in #6121
@domjaeg made their first contribution in #6136
@romanegunkov made their first contribution in #6134
@harry671003 made their first contribution in #6125

Full Changelog: v0.30.0...v0.31.0-rc.0

Contributors

harry671003, farodin91, and 14 other contributors

Assets 9

31 Jan 20:28

yeya24

v0.30.2

fe3f5d2

v0.30.2

What's Changed

Fixed panic because of nil sampler in #6066 by @xBazilio
Fix store-gateway deadlock due to not close BlockSeriesClient in #6086 by @alanprot

Full Changelog: v0.30.1...v0.30.2

Contributors

alanprot and xBazilio

Assets 9

04 Jan 19:00

GiedriusS

v0.30.1

27e4714

v0.30.1

This release contains a very small fix for the new Redis client. In the previous release, it was impossible to enable multiple caches using the new Redis client because it tries to register metrics more than once. As a result, for example, it was impossible to use Redis in Thanos Store with index cache and caching bucket enabled.

What's Changed

Fix duplicate metrics registration in Redis client in #6009 by @kama910

Full Changelog: v0.30.0...v0.30.1

Contributors

kama910

Assets 9

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes

Fixed

Changes

Fixed

Changes

Added

Fixed

Changed

Contributors

Changes

Added

Fixed

Changed

Contributors

What's Changed

Added

Fixed

Changed

What's Changed

Added

Fixed

Changed

New Contributors

Contributors

What's Changed

Contributors

What's Changed

Contributors

Releases: thanos-io/thanos

v0.32.2

Changes

Fixed

v0.32.1

Changes

Fixed

v0.32.0

Changes

Added

Fixed

Changed

Contributors

v0.32.0-rc.1

v0.32.0-rc.0

Changes

Added

Fixed

Changed

Contributors

v0.31.0

What's Changed

Added

Fixed

Changed

v0.31.0-rc.1

v0.31.0-rc.0

What's Changed

Added

Fixed

Changed

New Contributors

Contributors

v0.30.2

What's Changed

Contributors

v0.30.1

What's Changed

Contributors