OpenTelemetry 10 min read

Metric cardinality

Why one innocent attribute can multiply a metrics bill, how to find the metric and the attribute responsible, and four ways to bring the series count back down.

David Granzin By David Granzin

Metric cardinality is the number of distinct time series a metric produces. It is the main driver of metrics cost, and it is easy to raise by accident: add user.id to a request counter and the metric now has one series per user.

This guide explains how series multiply, how that turns into storage and money, how to find the attribute responsible, and how to fix it at the source, in the SDK and in the Collector. The SDK and Collector examples were run against current OpenTelemetry releases.

Key takeaways

  • Every unique set of attribute values is a separate time series. Series counts multiply across attributes.
  • Histograms multiply again by their bucket count.
  • IDs, full URLs, IP addresses and error messages do not belong on metrics. Put them on spans and logs.
  • Fix it where the metric is recorded: bounded attributes and SDK views beat cleanup downstream.
  • A series with otel.metric.overflow=true means a metric hit the SDK's cardinality limit.

What cardinality is

A time series is a metric name plus one specific set of attribute values (Prometheus calls them labels). These are three different series of the same histogram:

http.server.request.duration{http.request.method="GET",  http.route="/orders/{id}", http.response.status_code="200"}
http.server.request.duration{http.request.method="GET",  http.route="/orders/{id}", http.response.status_code="404"}
http.server.request.duration{http.request.method="POST", http.route="/orders",      http.response.status_code="201"}

The metric's cardinality is how many such combinations actually occur. As an upper bound it is the product of each attribute's distinct values. Take http.server.request.duration with 5 methods, 40 routes and 10 status codes, reported by 20 instances whose service.instance.id differs:

AttributesSeries
method × route × status5 × 40 × 10 = 2,000
× 20 instances40,000
× user.id with 10,000 active users400,000,000

In practice not every combination occurs, since a route only returns some status codes. But the shape holds: adding an attribute multiplies, and an attribute with unbounded values multiplies without limit.

Histograms make it worse. A histogram series carries a count per bucket. The OpenTelemetry SDK's default explicit boundaries are 0, 5, 10, 25, 50, 75, 100, 250, 500, 750, 1000, 2500, 5000, 7500, 10000: 15 boundaries, 16 buckets. Stored the Prometheus way, that is 16 bucket series plus _sum and _count, so 18 series for every attribute combination. With those buckets, the 40,000 combinations above become 720,000 series. HTTP instrumentation usually sets its own boundaries in seconds (the semantic conventions advise 14), which changes the multiplier a little and the conclusion not at all.

How cardinality reaches your bill

Every series costs something at each stage:

  • In the application. The SDK keeps one aggregation in memory per series and exports all of them each interval. The default export interval is 60 seconds.
  • In transit. Each series becomes a data point in every export. 40,000 series at one point a minute is about 1.75 billion data points a month.
  • In storage. Time series databases index each series. Memory use and query time grow with the number of active series, and queries that aggregate across a high-cardinality attribute read every one of them.
  • On the invoice. Metrics products commonly bill per active series, per custom metric or per data point ingested. All three grow with cardinality.

Churn counts too. A series that appears for ten minutes and never again still had to be created, indexed and kept for the retention period. Attributes that change on every deploy, such as a pod name, create a new set of series per deploy, so a service deployed twenty times a day accumulates series even if its live count looks steady.

Common causes

AttributeWhy it explodesUse instead
user.id, session.id, order or request IDsOne series per entitySpan attributes
url.path, url.fullIDs inside paths: /orders/8812http.route: /orders/{id}
client.address, network.peer.addressOne series per IPNothing, or a coarse region
exception.message, error textMessages embed valueserror.type
db.query.textOne series per distinct statementdb.operation.name, db.collection.name
k8s.pod.name, container.idNew values on every deploy and restartk8s.deployment.name, or drop it from metrics
Tenant or customer IDGrows with the businessFine for a few hundred tenants; above that, a plan tier

Auto-instrumentation libraries that follow the semantic conventions already use bounded attributes on their metrics. The usual source of trouble is custom metrics, where an attribute was added for one investigation and never removed, or a framework that reports the raw path because it does not know the route template.

Find the culprit

Start with the metric, then the attribute. In a Prometheus-compatible backend, count series per metric name. This query touches every series, so run it on a recent window and not on a dashboard:

# The 10 metric names with the most series
topk(10, count by (__name__) ({__name__=~".+"}))

Then count each attribute's distinct values on the worst metric. With its default name translation, Prometheus stores url.path as url_path:

# How many distinct values one attribute has on one metric
count(count by (url_path) (http_server_request_duration_seconds_count))

Prometheus also shows the metric names and label names with the most series on its /tsdb-status page. Other backends have an equivalent: a cardinality explorer, a usage page, or a SQL table you can group by metric and attribute.

On the sending side, two signals help. A series with otel.metric.overflow=true means an SDK hit its cardinality limit, covered below. And a Collector's debug exporter with verbosity: detailed prints every data point's attributes, which shows quickly whether an unexpected attribute is present before anything reaches the backend.

Fix it

Fix cardinality as close to where the metric is recorded as you can. Each step downstream is less precise and costs more.

1. Record bounded attributes

The best fix is to not record the value. Replace the path with the route template, the error message with an error type, the user ID with nothing. If you need a per-user answer, a metric is the wrong tool; see where the detail goes.

2. Drop attributes with an SDK view

When the metric comes from a library you cannot change, a view in the metrics SDK rewrites it before aggregation. An allow-list of attribute keys keeps the ones you name and discards the rest, and the SDK aggregates the result correctly: points that only differed by a dropped attribute are summed.

Node.js

import { MeterProvider, createAllowListAttributesProcessor } from "@opentelemetry/sdk-metrics";

const meterProvider = new MeterProvider({
  readers: [reader],
  views: [
    {
      // Keep only these three attributes on this histogram.
      instrumentName: "http.server.request.duration",
      attributesProcessors: [
        createAllowListAttributesProcessor(["http.request.method", "http.route", "http.response.status_code"]),
      ],
    },
  ],
});

Python

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.view import View

provider = MeterProvider(
    metric_readers=[reader],
    views=[
        # Keep only these three attributes on this histogram.
        View(
            instrument_name="http.server.request.duration",
            attribute_keys={"http.request.method", "http.route", "http.response.status_code"},
        ),
    ],
)

We recorded 500 measurements with 500 different user.id values through each of these. Both SDKs exported a single data point with the three allowed attributes.

3. Keep the cardinality limit as a safety net

The OpenTelemetry metrics SDK caps how many attribute sets one metric can hold per collection: 2,000 by default. Past the limit, new combinations are folded into a single overflow series carrying otel.metric.overflow=true. No measurement is lost or counted twice, but the overflow series has no breakdown, so the metric stops being useful for whatever exceeded the limit. In the JavaScript SDK, a view can set a lower limit per instrument:

views: [
  { instrumentName: "orders.created", aggregationCardinalityLimit: 3 },
],

With a limit of 3 and ten different customer.id values, the export held two real series and the overflow:

orders.created points 3 [{"customer.id":"c_0"},{"customer.id":"c_1"},{"otel.metric.overflow":true}]

Alert on the overflow series. It is the earliest sign that a deploy added an unbounded attribute.

4. Re-aggregate in the Collector

For metrics you do not instrument, such as scraped Prometheus endpoints or a vendor agent, fix them in a Collector. Three processors cover most cases:

config.yaml

processors:
  transform/route:
    metric_statements:
      - context: datapoint
        statements:
          - replace_pattern(attributes["url.path"], "/orders/[0-9]+", "/orders/{id}")
  metrics_transform/drop-user:
    transforms:
      - include: http.server.request.count
        action: update
        operations:
          - action: aggregate_labels
            label_set: [http.request.method, url.path]
            aggregation_type: sum
  filter/debug-metrics:
    metrics:
      metric:
        - 'IsMatch(name, "^debug\\.")'
  • transform rewrites values: here, order IDs in paths become /orders/{id}.
  • metrics_transform with aggregate_labels keeps only the listed attributes and sums points that become identical. In our test, three points for three users collapsed into one point with their total.
  • filter drops whole metrics nobody uses.

Do not just delete the attribute

The attributes processor can delete user.id from metric points, but it does not merge the points that are now identical. In our test, three points for three users came out as three points with the same attributes and a value of 5 each, where the right answer was one point of 15. Depending on the backend, such duplicates overwrite each other or are rejected. The same applies to transform on its own, which is why aggregate_labels runs after it. List the processors in the pipeline in the order shown.

The Collector receives the full cardinality before reducing it, so applications still pay for the memory and network of the original series. Treat Collector fixes as the place for metrics you cannot change at the source.

Where the detail goes

High cardinality is a problem for metrics, not for observability. Spans and logs are stored per event, so a user.id on a span costs a few bytes, while the same attribute on a metric multiplies the series count. A good split:

  • Metrics answer "how many, how fast, how often" with bounded dimensions: route, method, status, region, version.
  • Spans carry the specifics: user, tenant, order ID, query text. When a metric shows a slow route, the spans for that route tell you which users and which queries.
  • Exemplars join the two: a metric point can carry the trace ID of one measurement, so a spike in a latency chart links to a trace from that spike.

For SLOs and alerts, keep the metric. For "which customer was affected", query the spans. Read the cardinality glossary entry for the short version, or go back to part 1 of this series to set up the Collector these fixes run in.

Maple

Metrics with the traces behind them

Send OTLP metrics, traces and logs to Maple and keep per-request detail on spans, where it costs bytes and not series.

Sources and further reading

FAQ

よくある質問

What is metric cardinality?
Metric cardinality is the number of distinct time series a metric produces. Each unique combination of attribute values (labels in Prometheus) is its own series, so the cardinality is roughly the product of the number of values each attribute takes.
Why is high cardinality expensive?
Most metrics backends store, index and bill per active series or per data point, and every series produces a data point at every export interval. An attribute with many distinct values, such as a user ID, multiplies the series count by the number of users.
Which attributes cause high cardinality?
Unbounded values: user, session, request or order IDs, full URLs and paths with IDs in them, IP addresses, error messages, timestamps, and resource attributes that change on every deploy such as pod or container IDs.
How do I reduce metric cardinality in OpenTelemetry?
Record bounded attributes, such as http.route instead of url.path. Drop attributes in the SDK with a View that allow-lists the keys you need. Use the cardinality limit as a safety net. For metrics you do not control, re-aggregate in the Collector with the metrics_transform processor's aggregate_labels action.
What is otel.metric.overflow?
When a metric reaches the SDK's cardinality limit (2,000 attribute sets per metric by default), further new attribute sets are folded into one series with the attribute otel.metric.overflow=true. Seeing it means the metric has more combinations than the limit allows.

今日、最初のトレースを。

SDKを追加し、OTLPエクスポーターをMapleに向ければトレースが届きます。

maple.dev:OpenTelemetryの上のオブザーバビリティ