OpenTelemetry 11 min read

The OpenTelemetry Collector

What the Collector does, how its pipelines fit together, a config you can run today, and how to choose between an agent and a gateway.

David Granzin By David Granzin

The OpenTelemetry Collector is a vendor-neutral process that receives telemetry, changes it, and sends it on. Your services export traces, metrics and logs to it over OTLP, the OpenTelemetry Protocol. It batches, retries, filters and enriches them, then exports them to whatever backend you use.

This guide covers the Collector's building blocks, a configuration we ran on Collector v0.165.0, how to test it with a single curl, and the two ways to deploy it. Every config on this page passes otelcol-contrib validate.

Key takeaways

  • A component does nothing until a pipeline under service.pipelines lists it.
  • Put memory_limiter first in every pipeline. Batch in the exporter's sending_queue, not with the deprecated batch processor.
  • Since v0.144.0 the OTLP exporters are otlp_grpc and otlp_http. The old names still load with a warning.
  • Inside a container, bind the OTLP receiver to 0.0.0.0. The default is localhost.
  • Run agents for local context and a gateway for credentials, sampling and routing. Tail sampling needs every span of a trace on one gateway instance.

What the Collector does

An OpenTelemetry SDK can export straight to a backend. That works for a single service. As the system grows, a Collector between the two takes over jobs you would otherwise repeat in every service:

  • Retries and queueing. The SDK hands data to a nearby Collector in milliseconds. The Collector holds it while the backend is slow or down.
  • Processing in one place. Drop a noisy attribute, redact an email address or add the Kubernetes pod name once, for every service.
  • Other sources. Receivers read host metrics, log files, Prometheus endpoints and other formats the SDKs do not produce.
  • Routing. Send the same data to two backends during a migration, or change backends without redeploying services.
  • Credentials. Only the Collector holds the backend's API key, so services never see it.

The Collector ships as several distributions. The core distribution (otelcol) has a small set of components. Contrib (otelcol-contrib) includes almost every component the project maintains and is the one most guides assume. The k8s distribution is trimmed for Kubernetes. For production you can build your own with the OpenTelemetry Collector Builder (ocb), which includes only the components you list and keeps the binary and attack surface small.

Components and pipelines

A Collector config is a set of components, plus a service block that wires them into pipelines.

ComponentWhat it doesExamples
ReceiversGet data into the Collector, by listening or by pullingotlp, prometheus, host_metrics, file_log
ProcessorsChange data in order: drop, add, rename, sample, limit memorymemory_limiter, attributes, transform, filter, tail_sampling
ExportersSend data out, with a queue and retriesotlp_grpc, otlp_http, debug
ConnectorsExporter in one pipeline and receiver in another, such as turning spans into metricsspan_metrics, count, routing
ExtensionsCapabilities that do not touch telemetryhealth_check, pprof, zpages

A pipeline has a signal type (traces, metrics or logs), at least one receiver and at least one exporter. Processors run in the order the pipeline lists them. A receiver can feed several pipelines, and a processor listed in two pipelines runs as two separate instances.

Defining a component under receivers: or exporters: does not enable it. A Collector that starts cleanly but receives nothing usually has a receiver defined and never listed in a pipeline.

Names follow type/name. attributes/drop-ids and attributes/add-env are two independent instances of the attributes processor, each with its own settings.

A working config

This config receives OTLP on both protocols, protects the process from running out of memory, adds host attributes and exports everything over OTLP/HTTP to the backend named in OTLP_ENDPOINT. Traces also go to the debug exporter, which prints a line to the Collector's own output for each batch; it is there for the first test, so take it out once data reaches the backend.

config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 20
  resource_detection:
    detectors: [env, system]
    override: false

exporters:
  otlp_http:
    endpoint: ${env:OTLP_ENDPOINT}
    headers:
      Authorization: Bearer ${env:OTLP_API_KEY}
    sending_queue:
      enabled: true
      batch:
        enabled: true
  debug:
    verbosity: basic

extensions:
  health_check:
    endpoint: 0.0.0.0:13133

service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection]
      exporters: [otlp_http, debug]   # drop debug once data arrives
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection]
      exporters: [otlp_http]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource_detection]
      exporters: [otlp_http]

Block by block:

  • otlp receiver. gRPC on 4317 and HTTP on 4318, the OTLP default ports. The Collector binds to localhost unless you say otherwise, which makes it unreachable from other containers. 0.0.0.0 listens on every interface. On a shared host, bind to a specific address instead. The receiver has no authentication here, so do not expose it outside your network.
  • memory_limiter. Checks memory every second. Above the soft limit (here 80% minus 20%, so 60% of total memory, or of the container's limit) it refuses new data with a retryable error, so senders back off. Above the hard limit (80%) it also forces garbage collection. It does not replace sizing the Collector properly; the processor's README also recommends setting the GOMEMLIMIT environment variable to about 80% of the container's memory limit.
  • resource_detection. Reads OTEL_RESOURCE_ATTRIBUTES and the host name. override: false keeps the values the SDK already set. Run it on agents, next to the app: on a central gateway, the system detector would stamp the gateway's own host name on every service that did not set one.
  • otlp_http exporter. The endpoint is a base URL. The exporter appends /v1/traces, /v1/metrics and /v1/logs itself. ${env:OTLP_ENDPOINT} and ${env:OTLP_API_KEY} are read from the environment when the Collector starts, so the file holds no secrets. Your backend's docs give both values.
  • sending_queue with batch. Queues data in memory and groups it into batches before sending. Retries on failure are on by default, backing off from 5 seconds to 30 seconds for up to 5 minutes.
  • health_check. Answers 200 on port 13133 once the pipelines are running, for container and Kubernetes probes.

Renamed components and the batch processor

Many Collector guides online were written before 2026 and use names that now log deprecation warnings. Collector v0.144.0 renamed the otlp exporter to otlp_grpc and otlphttp to otlp_http. Later releases moved other multi-word names to snake case: hostmetrics became host_metrics, spanmetrics became span_metrics, loadbalancing became load_balancing and resourcedetection became resource_detection. The old names still load as aliases. The otlp receiver kept its name.

The batch processor is deprecated. Batching moved into each exporter's sending_queue, as in the config above, with the same defaults as the old processor: 8,192 items or 200 ms, whichever comes first. The project plans to turn exporter batching on by default in v0.171.0 and to remove the processor in v0.177.0. If you remove batch from a busy pipeline, watch the queue: it now holds smaller, unbatched requests and may need a larger queue_size.

Older Collector?

The new names do not exist before the version that introduced them. On a Collector older than v0.144.0, keep otlp, otlphttp and the batch processor. Check the version with otelcol-contrib --version.

Run it and send a span

Check the file before you start anything. validate catches unknown keys, wrong types and components that are not in your distribution. The contrib image runs it without installing anything:

# Your backend's OTLP base URL and key, read by the config
export OTLP_ENDPOINT=https://otlp.example.com
export OTLP_API_KEY=your-api-key

docker run --rm \
  -e OTLP_ENDPOINT -e OTLP_API_KEY \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.165.0 \
  validate --config /etc/otelcol-contrib/config.yaml

One mistake it caught while we tested this guide: the load_balancing exporter's DNS resolver wants port as a string, so port: 4317 fails and port: "4317" passes on v0.165.0, although the exporter's README shows an integer. Then start the Collector with the config mounted and the same two variables passed in. The ports are published on localhost only, since anything that can reach them sends telemetry with your key:

docker run --rm \
  -p 127.0.0.1:4317:4317 -p 127.0.0.1:4318:4318 -p 127.0.0.1:13133:13133 \
  -e OTLP_ENDPOINT -e OTLP_API_KEY \
  -v "$PWD/config.yaml:/etc/otelcol-contrib/config.yaml" \
  otel/opentelemetry-collector-contrib:0.165.0

You do not need an instrumented app to test the pipeline. OTLP/HTTP accepts JSON, so a file with one span and curl are enough. This script writes one, stamped with the current time so the backend shows it in its default time range. Trace and span IDs are hex strings, kind: 2 is a server span, and timestamps are nanoseconds since the epoch, written as strings.

make-span.sh

START=$(date +%s)000000000
END=$((START + 250000000))
cat > span.json <<EOF
{
  "resourceSpans": [{
    "resource": {
      "attributes": [{ "key": "service.name", "value": { "stringValue": "checkout" } }]
    },
    "scopeSpans": [{
      "scope": { "name": "manual-test" },
      "spans": [{
        "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
        "spanId": "00f067aa0ba902b7",
        "name": "GET /cart",
        "kind": 2,
        "startTimeUnixNano": "$START",
        "endTimeUnixNano": "$END",
        "status": { "code": 1 }
      }]
    }]
  }]
}
EOF
curl -i http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  --data @span.json
HTTP/1.1 200 OK
Content-Type: application/json

{"partialSuccess":{}}

An empty partialSuccess means the Collector accepted the whole batch. It does not mean the backend received it: the exporter sends from its queue afterwards, so a wrong endpoint or key only shows up in the Collector's output. The debug exporter prints a line with "spans": 1 once the span has passed through the pipeline; set verbosity: detailed to print every attribute. If the backend rejects the data, the exporter logs the HTTP status and response body in the Collector's output. The OTLP guide covers what each status means.

Point your app at the Collector

Once the curl test passes, send real data. The fastest start is auto-instrumentation, which records spans for HTTP, database and queue calls without code changes. Every SDK reads the same environment variables, so the only Collector-specific setting is the endpoint. In Node.js:

npm install @opentelemetry/api @opentelemetry/auto-instrumentations-node

export OTEL_SERVICE_NAME="checkout"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
node --require @opentelemetry/auto-instrumentations-node/register app.js

In Python:

pip install opentelemetry-distro opentelemetry-exporter-otlp
opentelemetry-bootstrap -a install

export OTEL_SERVICE_NAME="checkout"
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:4318"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
opentelemetry-instrument python app.py

Make a request to the app and the debug exporter prints a batch a few seconds later. From another container, replace localhost with the Collector's service name, such as http://otel-collector:4318. If nothing arrives, the OTLP errors section lists the usual causes, and two services that appear as separate traces point at context propagation.

Processor order

Processors run in the order the pipeline lists them, and the order changes the outcome. A sequence that works for most pipelines:

  1. memory_limiter first, so it can refuse data before any other work happens.
  2. Enrichment such as resource_detection or k8s_attributes, so later steps can match on what they add.
  3. Filtering and redaction with filter, attributes, redaction or transform. Dropping early saves the work of processing data you will not keep.
  4. Sampling, for example probabilistic_sampler or tail_sampling.
  5. Batching in the exporter's sending_queue, after everything else.

Two traps. A filter processor placed after tail_sampling can drop spans from a trace the sampler already kept, which leaves gaps in the trace. And attributes you delete before tail_sampling are gone by the time its policies run, so a policy on a deleted attribute never matches.

Agent vs gateway

The same binary runs in two roles. Most production setups use both.

AgentGateway
RunsOn each host, as a Kubernetes DaemonSet, or as a sidecarAs a central Deployment behind a load balancer
Receives fromServices on the same host or podAgents and services across the cluster or region
Good atHost metrics, log files, pod and node attributes, a short hop for the SDKCredentials, tail sampling, routing, one place to change backends
Scales withThe number of nodesTraffic volume, horizontally
Main riskConfig spread over every nodeA shared bottleneck, so it needs replicas and queue alerts

Start with agents only if you have a handful of services: each agent exports straight to the backend. Add a gateway when you want one egress point, one place for the API key, or tail sampling. Then agents send OTLP to the gateway's service address and only the gateway talks to the backend.

Tail sampling adds a constraint. The tail_sampling processor decides once it has seen a whole trace, so every span of a trace must reach the same Collector instance. A plain round-robin load balancer breaks that. The fix is two gateway tiers: the first uses the load_balancing exporter to route by trace ID, the second runs the sampler.

gateway tier 1: route by trace ID

exporters:
  load_balancing:
    routing_key: traceID
    protocol:
      otlp:
        tls:
          insecure: true   # plain text inside the cluster; use TLS across networks
    resolver:
      dns:
        hostname: otel-sampler-headless.observability.svc.cluster.local
        port: "4317"

The DNS resolver watches a headless Kubernetes service, so a new sampler pod starts receiving its share of traces without a config change. The flip side: whenever the set of pods changes, part of the trace ID range moves to a different pod, and traces in flight at that moment are split. Scale the sampler tier deliberately rather than autoscaling it on every spike. The sampling guide has the second tier's tail sampling config.

Monitor the Collector

A Collector that silently drops data looks the same as a quiet system. It reports its own metrics in Prometheus format on port 8888, bound to 127.0.0.1 by default. To scrape it from another container, expose it on all interfaces:

service:
  telemetry:
    metrics:
      readers:
        - pull:
            exporter:
              prometheus:
                host: 0.0.0.0
                port: 8888

The metrics worth an alert:

MetricWhat it tells you
otelcol_exporter_send_failed_spansSpans the exporter gave up on. Any sustained rate is data loss. The same exists for metric points and log records.
otelcol_exporter_queue_size / otelcol_exporter_queue_capacityHow full the sending queue is. Near 100%, new data is dropped.
otelcol_exporter_enqueue_failed_spansSpans dropped because the sending queue was full. This is the usual way data is lost during a backend outage.
otelcol_receiver_refused_spansData turned away at the door, often by memory_limiter.
otelcol_receiver_accepted_spansThroughput. A sudden drop to zero means senders stopped reaching the Collector.

Since v0.152.0 these names have no _total suffix by default. Dashboards built for older versions may query otelcol_exporter_send_failed_spans_total and show nothing.

Send it to Maple

Maple ingests OTLP directly, so the config above works unchanged: set OTLP_ENDPOINT=https://ingest.maple.dev and OTLP_API_KEY to the private ingest key from Settings → Ingestion. For gRPC, use the otlp_grpc exporter with endpoint: ingest.maple.dev:443; the port is 443, not 4317. The ingest reference lists the limits and status codes, and OpenTelemetry with Maple covers the SDK side.

Next step

Point a Collector at Maple

Traces, logs and metrics from the same Collector land in one place, linked by trace ID.

Sources and further reading

FAQ

자주 묻는 질문

What is the OpenTelemetry Collector?
The OpenTelemetry Collector is a standalone process that receives telemetry, processes it and exports it to one or more backends. It accepts OTLP from OpenTelemetry SDKs and many other formats, and it is configured with one YAML file of receivers, processors, exporters and pipelines.
Do I need an OpenTelemetry Collector?
Not always. An SDK can export OTLP straight to a backend. Add a Collector when you need to scrub or enrich data in one place, collect host metrics or log files, tail sample, fan out to several backends, or keep retries and batching out of your application processes.
What ports does the OpenTelemetry Collector use?
The OTLP receiver listens on 4317 for gRPC and 4318 for HTTP. The health_check extension uses 13133 and the Collector's own metrics are served on 8888. Recent versions bind to localhost by default, so set the endpoint to 0.0.0.0 inside a container.
What is the difference between an agent and a gateway Collector?
An agent runs next to the application, on the same host or as a sidecar or DaemonSet, and adds local context such as host and pod attributes. A gateway is a central pool of Collectors behind a load balancer that receives from many agents or services and owns credentials, sampling and routing.
Is the batch processor deprecated?
Yes. The OpenTelemetry project deprecated the batch processor in favour of batching inside each exporter's sending_queue. Exporter batching is planned to become the default in v0.171.0 and the processor is planned for removal in v0.177.0.

오늘 첫 트레이스를 확인하세요.

SDK를 추가하고 OTLP 익스포터를 Maple로 향하게 하면 트레이스가 도착합니다.

maple.dev: OpenTelemetry 위의 옵저버빌리티