Tempo 3.0 release: a new architecture for scale and lower TCO, TraceQL metrics GA, and more

See the PR and documentation for configuration and more information.

Sampling compensation with probability sampling state 

For environments that use head-based sampling, generated metrics can undercount real traffic unless the processor knows how to scale back the metrics from the sampled spans.

Tempo has support for span_multiplier_key config to configure a span or resource attribute that has the sampling ratio and scales the metrics accordingly. For example, if spans have attribute X-SampleRatio=0.1 (10% sampling), setting span_multiplier_key: "X-SampleRatio" results in each span counting as 10 requests.

New in 3.0, we also added the enable_tracestate_span_multiplier config on span metrics and service graphs to provide an alternative approach that extracts the multiplier from the W3C tracestate header using the OpenTelemetry probability sampling threshold.

When enabled,  enable_tracestate_span_multiplier takes priority over span_multiplier_key.

For more information, check out the PR and documentation for more information.

Trace redaction to protect sensitive data 

Trace data can include sensitive data, including personally identifiable information (PII). This release includes new redaction capabilities to help operators remove trace data when needed.

The new tempo-cli redact command submits redaction jobs to the backend scheduler over gRPC. Operators provide a tenant and one or more trace IDs, and Tempo creates backend jobs to rewrite the affected blocks in object storage, removing the matching trace data.

See the PR and the documentation for more information.

Improvements to query performance

Tempo 3.0 delivers major query performance improvements across both vParquet5 and TraceQL execution. 

vParquet5

Tempo uses the Apache Parquet columnar block format to store traces, and TraceQL queries operate on traces stored in those Parquet blocks. 

vParquet5, the latest iteration of the Parquet-based columnar block format in Tempo, now supports up to 20 dedicated string columns per scope (span, resource, and event), doubled from 10, and up to 5 integer columns. It also does this dynamically, to reduce overhead to only what you actually use.

Dedicated columns can improve query performance by storing commonly used or queried attributes in their own columns, to put like-data together for best efficiency, and targeted reads when you query them. The Tempo CLI analysis and suggestion commands were also updated to help determine the best attributes to dedicate for your data.  It’s not always the case that more is better, and your optimal configuration might not need to use all available columns.

Please see the PR for more details.

TraceQL AST optimization

Tempo now automatically rewrites supported repeated conditions on the same attribute into a faster internal array operation, speeding up query execution. For example, a query like  { resource.service.name="frontend" || resource.service.name="backend" || resource.service.name="api" } can be rewritten internally before execution, with no changes required from users. 

This is a breaking change. The implicit array matching semantics of the operators != and !~ have changed: != now means NOT IN, and !~ now means MATCH NONE. Regex operands must now be strings or string arrays; values that were previously accepted because they could be converted to regex strings may now be rejected. Review the upgrade notes before moving to Tempo 3.0 if you use complex TraceQL queries.

TraceQL AST (Abstract Syntax Tree) optimizations can be disabled by setting skip_ast_transformations=all in the query frontend config. You can opt out per query with the hint skip_ast_transformations=all.

See the PR #6353 and PR #7012 for more details, including limitations and breaking changes.

A new way to enable Tempo MCP server

In Tempo 2.9, we introduced MCP server support as an experimental way for LLMs and AI agents to query and analyze Tempo data through TraceQL and other endpoints. In addition to enabling the feature via configuration, you can now use a dedicated flag to enable it as well:

See the PR and documentation for more details.

Migrating to Tempo 3.0

Because Tempo 3.0 includes major architectural changes, we created a migration guide to help you move your Tempo 2.x configurations to 3.0. Also, see Upgrade your Tempo installation for breaking changes and migration guidance before upgrading to Tempo 3.0.

We also added a new migration command to the Tempo CLI.

For monolithic deployments, we recommend using: