v1.0.1

• [BUGFIX] Guard against negative dataLength #763 (@joe-elliott)

v1.0.0

Breaking changes

• This release contains a change to communication between distributors and ingesters which requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version.
• -auth.enabled is marked deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646 @dgzlopes

Enhancements

This release contains significant improvements for performance and stability:

• [ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681 @joe-elliott
• [ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668 @joe-elliott
• [ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679 @annanay25
• [ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 @joe-elliott
• [ENHANCEMENT] Add config option to disable write extension to the ingesters. #677 @joe-elliott
• [ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644 @annanay25
• [ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693 @annanay25
• [ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714 @kvrhdn

Bugfixes

• [BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690 @annanay25
• [BUGFIX] Zipkin Support - CombineTraces. #688 @joe-elliott
• [BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687 @annanay25
• [BUGFIX] Azure Backend - Fix an issue with the append method on the Azure backend. #736 @pedrosaraiva

v1.0.0-rc.0

Breaking changes

• This release contains a change to communication between distributors and ingesters which requires a specific rollout process to prevent dropped spans. First, rollout everything except distributors. After all ingesters have updated you can then rollout distributors to the latest version.
• -auth.enabled is marked deprecated. New flag is -multitenancy.enabled and is set to false by default. This is a breaking change if you were relying on auth/multitenancy being enabled by default. #646 @dgzlopes

Enhancements

This release contains significant improvements for performance and stability:

• [ENHANCEMENT] Performance: Improve Ingester Record Insertion. #681 @joe-elliott
• [ENHANCEMENT] Improve WAL Replay by not rebuilding the WAL. #668 @joe-elliott
• [ENHANCEMENT] Preallocate byte slices on ingester request unmarshal. #679 @annanay25
• [ENHANCEMENT] Reduce marshalling in the ingesters to improve performance. #694 @joe-elliott
• [ENHANCEMENT] Add config option to disable write extension to the ingesters. #677 @joe-elliott
• [ENHANCEMENT] Allow setting the bloom filter shard size with support dynamic shard count.#644 @annanay25
• [ENHANCEMENT] GCS SDK update v1.12.0 => v.15.0, ReadAllWithEstimate used in GCS/S3 backends. #693 @annanay25
• [ENHANCEMENT] Add a new endpoint /api/echo to test the query frontend is reachable. #714 @kvrhdn

Bugfixes

• [BUGFIX] Fix Query Frontend grpc settings to avoid noisy error log. #690 @annanay25
• [BUGFIX] Zipkin Support - CombineTraces. #688 @joe-elliott
• [BUGFIX] Zipkin support - Dedupe span IDs based on span.Kind (client/server) in Query Frontend. #687 @annanay25

v0.7.0

License Change

• v0.7.0 and future versions are licensed under AGPLv3 #660

Breaking changes

• In an effort to move to byte based limits we removed some limits options and replaced them with byte based ones:
  max_spans_per_trace => max_bytes_per_trace
ingestion_rate_limit => ingestion_rate_limit_bytes
ingestion_burst_size => ingestion_burst_size_bytes
• The Query/QueryFrontend call signature has changed so there will be a query interruption during rollout.

All Changes

• [CHANGE] Update to Go 1.16, latest OpenTelemetry proto definition and collector #546 @mdisibio
• [CHANGE] max_spans_per_trace limit override has been removed in favour of max_bytes_per_trace.
  This is a breaking change to the overrides config section. #612 @annanay25
• [CHANGE] Add new flag -ingester.lifecycler.ID to manually override the ingester ID with which to register in the ring. #625 @annanay25
• [CHANGE] ingestion_rate_limit limit override has been removed in favour of ingestion_rate_limit_bytes.
  ingestion_burst_size limit override has been removed in favour of ingestion_burst_size_bytes.
  This is a breaking change to the overrides config section. #630 @annanay25

Features

• [FEATURE] Add page based access to the index file. #557 @joe-elliott
• [FEATURE] (Experimental) WAL Compression/checksums. #638 @joe-elliott

Enhancements

• [ENHANCEMENT] Add a Shutdown handler to flush data to backend, at "/shutdown". #526 @annanay25
• [ENHANCEMENT] Queriers now query all (healthy) ingesters for a trace to mitigate 404s on ingester rollouts/scaleups.
  This is a breaking change and will likely result in query errors on rollout as the query signature b/n QueryFrontend & Querier has changed. #557 @annanay25
• [ENHANCEMENT] Add list compaction-summary command to tempo-cli #588 @mdisibio
• [ENHANCEMENT] Add list and view index commands to tempo-cli #611 @mdisibio
• [ENHANCEMENT] Add a configurable prefix for HTTP endpoints. #631 @joe-elliott
• [ENHANCEMENT] Add kafka receiver. #613 @mapno
• [ENHANCEMENT] Upgrade OTel collector to v0.21.0. #613 @mapno
• [ENHANCEMENT] Add support for Cortex Background Cache. #640 @dgzlopes

Bugfixes

• [BUGFIX] Fixes permissions errors on startup in GCS. #554 @joe-elliott
• [BUGFIX] Fixes error where Dell ECS cannot list objects. #561 @kradalby
• [BUGFIX] Fixes listing blocks in S3 when the list is truncated. #567 @jojand
• [BUGFIX] Fixes where ingester may leave file open #570 @mdisibio
• [BUGFIX] Fixes a bug where some blocks were not searched due to query sharding and randomness in blocklist poll. #583 @annanay25
• [BUGFIX] Fixes issue where wal was deleted before successful flush and adds exponential backoff for flush errors #593 @mdisibio
• [BUGFIX] Fixes issue where Tempo would not parse odd length trace ids #605 @joe-elliott
• [BUGFIX] Sort traces on flush to reduce unexpected recombination work by compactors #606 @mdisibio
• [BUGFIX] Ingester fully persists blocks locally to reduce amount of work done after restart #628 @mdisibio

v0.6.0

0.6.0 release of Tempo comes with lots of features, enhancements and bugfixes. 🎁 🚀

The highlights of this release include support for Compression and Exhaustive search, along with CPU performance gains from changing compression algorithm between distributors and ingesters.

Breaking changes

[CHANGE] Ingester cut blocks based on size instead of trace count. Replace ingester traces_per_block setting with max_block_bytes. This is a breaking change. #474 (By default, Tempo will cut blocks of size 1GB) @mdisibio
[FEATURE] Added block compression. This is a breaking change b/c some configuration fields moved. #504 (By default, Tempo will use zstd for compression.) @joe-elliott
[CHANGE] Refactor cache section in tempodb. This is a breaking change b/c the cache config section has changed. #485 @dgzlopes

Enhancements

[CHANGE/BUGFIX] Rename tempodb_compaction_objects_written and tempodb_compaction_bytes_written metrics to tempodb_compaction_objects_written_total and tempodb_compaction_bytes_written_total. #524 @gouthamve
[ENHANCEMENT] Change default ingester_client compression from gzip to snappy. #522 @mdisibio
[ENHANCEMENT] Add exhaustive search to combine traces from all blocks in the backend. #489 @annanay25
[ENHANCEMENT] Add per-tenant block retention #77 @mdisibio
[ENHANCEMENT] Change index-downsample to index-downsample-bytes. This is a breaking change #519 @joe-elliott

Bugfixes

[BUGFIX] Upgrade cortex dependency to v1.7.0-rc.0+ to address issue with forgetting ring membership #442 #512 @mdisibio
[BUGFIX] No longer raise the tempodb_blocklist_poll_errors_total metric if a block doesn't have meta or compacted meta. #481 @joe-elliott
[BUGFIX] Replay wal completely before ingesting new spans. #525 @joe-elliott

For the full list of changes see CHANGELOG

v0.5.0

0.5.0 Release of Tempo with important new features and improvements. 🚀 Note: This release contains some breaking changes.

New Features

36991f0 Add support for Azure Blob Storage backend (#340) @pedrosaraiva
bc11b55 Add Query Frontend module to allow scaling the query path (#400) @annanay25

Breaking Changes

9e0e05a The gRPC signature from distributors to ingesters has changed. This is a breaking change when running in microservices mode with separate distributors and ingesters. To prevent errors first upgrade all ingesters, which adds the new gRPC endpoint, then upgrade the distributors. (#430) @mdisibio
dd7a18e Removed disk-based caching. Please migrate to redis or memcached. (#441) @joe-elliott
a796195 The ingestion_max_batch_size setting has been renamed to ingestion_burst_size. (#445) @mdisibio

Other changes and fixes

65a0643 Prevent race conditions between querier polling and ingesters clearing complete blocks (#421) @joe-elliott
4c0ea69 Exclude blocks in last active window from compaction (#411) @mdisibio
e804efd Mixin: Ignore metrics and query-frontend route when checking for TempoRequestLatency alert. (#440) @pstibrany
40abd5d Compactor without GCS permissions fail silently (#379) @mdisibio
40abd5d Add docker-compose example for GCS along with new backend options (#397) @mdisibio @achatterjee-grafana
00d9d3a Added tempo_distributor_bytes_received_total metric (#453) @joe-elliott
9e690dd Redo tempo-cli with basic command structure and improvements (#385) @mdisibio @achatterjee-grafana

For the full list of changes see CHANGELOG

v0.4.0

0.4.0 Release of Tempo with lots of bugfixes and performance improvements! 🎉

This release saw three new contributors, welcome to the project @achatterjee-grafana, @alexissellier and @simonswine!

Changelog

cd8e6cb Add level label to tempodb_compaction_objects_combined_total metric. Update operational dashboard to match (#376) @mdisibio
82bf5fc Add metrics for bytes and objects written during compaction (#360) @mdisibio
077b806 Add new compactor_objects_combined metric and test (#339) @mdisibio
c20839c Add support for Redis caching (#354) @dgzlopes
2e7a500 Change to TOC structure and other reorg to individual topics (#365) @achatterjee-grafana
143bd10 Compact more than 2 blocks at a time (#348) @mdisibio
62af44e Fix Ingesters Occassionally Double Flushing (#364) @joe-elliott
c0cb2fa Fix value type (#383) @alexissellier
9b6edac Query Path Observability Improvements (#361) @joe-elliott
2b34ea3 [ops] Fix resources dashboard, disable tempo-query tracing by default (#353) @annanay25
aa26d61 feat: add option to S3 backend for V2 signatures (#352) @simonswine
fca9484 Remove panic in ReportFatalError (#343) @joe-elliott
78f3554 Address frequent errors logged by compactor regarding meta not found (#327) @mdisibio

v0.3.0

Changelog

eb987a8 Increase Prometheus not found metric on tempo-vulture (#301) @dgzlopes
09af806 #306 - Compactor flush to backend based on buffer size (#325) @mdisibio
493406d Add per tenant bytes counter (#331) @dgzlopes
077fd13 Add warnings for suspect configurations (#294) @dgzlopes
d91c415 Bloom sharding (#192) @annanay25
814282d Build: add support for multi arch build (#311) @morlay
5d8d2f6 Fix Tempo build-info (#295) @dgzlopes
2f0917d Prune in-memory blocks from missing tenants (#314) @dgzlopes
7156aa3 Rename maintenance cycle to blocklist poll (#315) @dgzlopes
fd114a2 Return 404 on unknown tenant (#321) @joe-elliott
4edd1fe Support multiple authentication methods for S3 (#320) @chancez

Breaking Changes

• Bloom filter sharding (#192) prevents previously written blocks from being read. Just wipe existing data and restart.
• #315 renames maintenance_cycle to blocklist_poll in configuration for clarity.

v0.2.0

🎉 First Release of Tempo 🎉
Congratulations maintainers @annanay25 @joe-elliott !