OpenTelemetry 10 min read

OTLP explained

The protocol every OpenTelemetry exporter speaks: which transport to pick, how endpoints and headers are configured, and what each export error means.

David Granzin By David Granzin

OTLP, the OpenTelemetry Protocol, is how telemetry moves between an OpenTelemetry SDK, a Collector and a backend. If you have seen 4317, 4318 or OTEL_EXPORTER_OTLP_ENDPOINT, you have configured OTLP.

Most OTLP problems are configuration problems: the wrong port for the transport, a path appended twice, a header in the wrong format. This guide explains the parts and then works through the errors in the order you are likely to hit them.

Key takeaways

  • gRPC listens on 4317 and HTTP on 4318 by default. Sending one transport to the other's port fails with confusing errors.
  • http/protobuf is the specified default and the safest choice when in doubt.
  • OTEL_EXPORTER_OTLP_ENDPOINT is a base URL. The per-signal variables are used as written, path included.
  • Retry on 429, 502, 503 and 504. Do not retry 400, 401, 403 or 413 unchanged.
  • A 200 response can still carry partialSuccess with rejected items. Read it.

What OTLP is

OTLP is a specification with two parts. The first is a set of protobuf messages: ExportTraceServiceRequest, ExportMetricsServiceRequest, ExportLogsServiceRequest and, still in development, one for profiles. Each request groups data by resource (the service and host that produced it), then by instrumentation scope (the library that recorded it).

The second part is how those messages travel: over gRPC, or as HTTP POST requests. The traces, metrics and logs parts of the protocol are stable, so an exporter written today talks to a Collector from two years ago.

A request is a batch. An SDK collects spans for a few seconds (5 seconds by default for traces) or until it has 512 of them, and sends them in one call. That is why a span shows up in the backend a few seconds after the request finishes, and why a process that exits without flushing loses its last batch.

gRPC vs HTTP

Both transports carry the same messages. They differ in where they work well.

OTLP/gRPCOTLP/HTTP
Default port43174318
Protocol valuegrpchttp/protobuf or http/json
Where data goesThe Export method of TraceService, MetricsService or LogsServicePOST /v1/traces, /v1/metrics, /v1/logs
EncodingProtobuf onlyProtobuf or JSON
BrowsersNoYes
Proxies and load balancersNeeds HTTP/2 end to end; long-lived connections can pin to one backendWorks through anything that forwards HTTP
ErrorsgRPC status codes such as UNAVAILABLEHTTP status codes

The specification says SDKs should default to http/protobuf, though some SDKs kept grpc as their default for compatibility with earlier releases. Check your SDK's docs, or set OTEL_EXPORTER_OTLP_PROTOCOL explicitly so nobody has to guess.

Choose HTTP when traffic crosses a proxy, an API gateway or the public internet, or when it comes from a browser. Choose gRPC between services and Collectors you run yourself, where its single multiplexed connection saves a little overhead. The difference in throughput rarely matters next to batch size and compression.

One gRPC caveat: a layer-4 load balancer balances connections, not requests. A gRPC exporter opens one connection and keeps it, so all its data lands on whichever Collector it first reached. Use a load balancer that understands gRPC, or OTLP/HTTP.

Protobuf vs JSON

Over HTTP you choose the encoding with Content-Type: application/x-protobuf for binary protobuf, application/json for JSON. The server answers in the same encoding it received.

Protobuf is smaller and cheaper to encode, and it is what you want in production. JSON is for places where protobuf is awkward, such as a hand-written curl test, a shell script or a small browser exporter. OTLP/JSON is not the JSON you would guess from the protobuf schema. Four rules trip people up:

{
  "traceId": "4bf92f3577b34da6a3ce929d0e0e4736",
  "spanId": "00f067aa0ba902b7",
  "kind": 2,
  "startTimeUnixNano": "1763712000000000000",
  "droppedAttributesCount": 0
}
  • IDs are hex strings. traceId is 32 hex characters and spanId is 16. The standard protobuf-to-JSON mapping would use base64, and a base64 ID is rejected with a 400.
  • Enums are integers. "kind": 2 for a server span, never "SPAN_KIND_SERVER".
  • Keys are lowerCamelCase. droppedAttributesCount, not dropped_attributes_count.
  • 64-bit integers are strings. Timestamps such as startTimeUnixNano are decimal strings. Receivers accept numbers too, but a JavaScript number loses precision past 253.

Compression is separate from encoding. Set OTEL_EXPORTER_OTLP_COMPRESSION=gzip in the SDK; the Collector's OTLP exporters already use gzip by default. Telemetry repeats the same keys and values constantly and compresses well, which matters for both egress cost and the 413 errors below.

Endpoints and environment variables

Every OpenTelemetry SDK reads the same environment variables, so this setup works in any language:

export OTEL_SERVICE_NAME="checkout"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"

Each variable has a per-signal form, such as OTEL_EXPORTER_OTLP_TRACES_ENDPOINT or OTEL_EXPORTER_OTLP_LOGS_HEADERS, which overrides the generic one for that signal. The two endpoint forms behave differently, and that difference causes most OTLP 404s:

# Generic variable: a base URL. The SDK appends /v1/traces, /v1/metrics, /v1/logs.
OTEL_EXPORTER_OTLP_ENDPOINT=http://collector:4318
#   traces  -> http://collector:4318/v1/traces

# Per-signal variable: used exactly as written. Include the path yourself.
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://collector:4318/v1/traces

With no endpoint set, SDKs send to http://localhost:4318 for HTTP or http://localhost:4317 for gRPC. That default is right when a Collector runs on the same host, and wrong almost everywhere else.

The export timeout defaults to 10 seconds per batch (OTEL_EXPORTER_OTLP_TIMEOUT, in milliseconds). Retries happen inside that budget, so a slow endpoint shows up as timeouts, then dropped batches.

Auth headers and TLS

OTLP has no authentication of its own. Backends read a header: usually Authorization: Bearer <key>, sometimes a vendor header. Over HTTP it is a request header; over gRPC it is metadata, where keys are lowercase.

OTEL_EXPORTER_OTLP_HEADERS takes a comma-separated list of key=value pairs in the W3C Baggage format. Two details matter. A comma inside a value ends it, so a key containing a comma must be percent-encoded. And quoting is part of the value, so Authorization="Bearer abc" sends the quotes. Most SDKs accept the space in Bearer abc as written; if yours rejects it, encode it as Bearer%20abc.

In a Collector, headers go on the exporter, and the key comes from the environment rather than the file:

config.yaml

exporters:
  otlp_http:
    endpoint: https://ingest.maple.dev
    headers:
      Authorization: Bearer ${env:MAPLE_INGEST_KEY}
  otlp_grpc:
    endpoint: https://ingest.maple.dev
    headers:
      authorization: Bearer ${env:MAPLE_INGEST_KEY}

The scheme in the endpoint sets transport security. https:// means TLS and http:// means plain text, for both transports. A gRPC endpoint with no scheme falls back to the insecure setting (OTEL_EXPORTER_OTLP_INSECURE), which defaults to TLS on. For a private certificate authority, point OTEL_EXPORTER_OTLP_CERTIFICATE at its PEM file; for mutual TLS, also set OTEL_EXPORTER_OTLP_CLIENT_CERTIFICATE and OTEL_EXPORTER_OTLP_CLIENT_KEY.

Common errors and fixes

OTLP defines which failures a client retries. Over HTTP, 429, 502, 503 and 504 are retryable, with exponential backoff or after the Retry-After header when the server sends one. Every other 4xx and 5xx is final: sending the same batch again gets the same answer. Over gRPC the retryable signal is UNAVAILABLE.

SymptomUsual causeFix
Connection refusedNothing listens at that host and portCheck the host, the port and the Collector's bind address
400 Bad RequestUndecodable body, such as JSON with base64 IDsFix the payload; never retry it
401 / 403Missing, wrong or revoked key, or a key from another regionCheck the header name, the Bearer prefix and the key
404 Not FoundWrong path, usually a missing or doubled /v1/tracesUse the base URL in the generic variable
413The request body is over the server's limitSmaller batches, gzip on
415Unsupported Content-Type or compressionProtobuf or JSON, gzip or none
429 / 503Rate limit or backpressureLet the exporter retry; lower volume if it persists
Protocol or parse errorsgRPC sent to the HTTP port, or the reverseMatch the protocol to the port

Connection refused

The TCP connection never opened. In order of likelihood: the Collector is not running; the exporter uses the default localhost while the Collector runs in another container, so localhost is the app's own container; or the Collector listens only on localhost inside its container, so nothing outside can reach it. The last one is the default since Collector v0.104.0. Bind the receiver to 0.0.0.0:4318 and point the app at the Collector's service name, such as http://otel-collector:4318.

401 and 403

A 401 means the backend did not accept the credential. Check that the header reached the exporter: a variable set in your shell is not set inside a container or a systemd unit. Check the format: Authorization=Bearer key in the environment variable, and the exact header name your backend documents. Some backends issue keys per region and reject a key sent to the other region's endpoint. A 403 usually means the key is valid but not allowed to send that signal.

404

Almost always the path. Setting OTEL_EXPORTER_OTLP_ENDPOINT=https://host/v1/traces sends traces to /v1/traces/v1/traces. Setting OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://host sends them to /. In a Node.js test, the HTTP exporter given a bare http://localhost:4318 as its URL got Not Found from the Collector, because a URL passed in code is also used as written.

413 Content Too Large

The specification tells servers to answer 413 when a request is over their size limit, and suggests supporting at least 64 MiB. Hosted backends often set lower limits, and proxies in front of a Collector (an nginx default is 1 MB) are lower still. Make batches smaller at the source:

# Smaller span batches from the SDK (default 512 spans)
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=256

In a Collector, cap the batch the exporter builds. max_size splits anything larger:

exporters:
  otlp_http:
    endpoint: https://ingest.maple.dev
    sending_queue:
      enabled: true
      batch:
        enabled: true
        sizer: items
        min_size: 2000   # the default minimum is 8192, above this maximum
        max_size: 4000

Turn on gzip if it is off. Check which size the limit applies to: some backends measure the compressed body, others the decompressed one.

Protocol mismatch

Sending gRPC to the HTTP port, or HTTP to the gRPC port, does not produce a clear message. In a test with the Node.js exporters against a Collector, the gRPC exporter pointed at 4318 failed with 14 UNAVAILABLE: No connection established. Last error: Protocol error, and the HTTP exporter pointed at 4317 failed with Parse Error: Expected HTTP/, RTSP/ or ICE/. Other languages word it differently, often as an HTTP/2 frame or preface error. When you see a protocol-level error, check that OTEL_EXPORTER_OTLP_PROTOCOL and the port agree before anything else.

Debug an exporter

When data does not arrive, take the application out of the picture and test the endpoint directly. An empty OTLP/JSON request is valid and checks the URL, the TLS setup and the key in one go:

curl -i https://ingest.maple.dev/v1/traces \
  -H "Authorization: Bearer YOUR_INGEST_KEY" \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'

200 means the endpoint and key work, so the problem is in the app's configuration. 401 points at the key and 404 at the path. If curl cannot connect at all, the network is the problem: DNS, a firewall, or a proxy that needs its own settings.

Then check the SDK side:

  • Print the effective config. Log OTEL_EXPORTER_OTLP_* at startup, minus the header values. A typo such as OTEL_EXPORTER_OTLP_ENDPIONT is silently ignored.
  • Turn on SDK logs. OTEL_LOG_LEVEL=debug makes the SDKs that support it log each failed export. Support varies by language.
  • Flush on exit. Short-lived processes, such as CLI tools, cron jobs and serverless functions, must call shutdown() or forceFlush() on the provider before exiting, or the last batch never leaves.
  • Read partial success. A 200 whose body has partialSuccess.rejectedSpans greater than zero means some items were dropped, with the reason in errorMessage. Clients do not retry it, so log it.
  • Put a Collector in between. A local Collector with the debug exporter shows exactly what the SDK sends. The Collector guide has a config that does this.

Data that arrives but is incomplete is usually not an OTLP problem. Missing child spans point at context propagation, and traces that appear for some requests but not others point at sampling.

Maple

Send OTLP to Maple

Set the endpoint to https://ingest.maple.dev and the header to your ingest key. Both transports work on port 443, and the ingest reference lists every status code with its fix.

Sources and further reading

FAQ

Common questions

What is OTLP?
OTLP, the OpenTelemetry Protocol, is the wire format OpenTelemetry uses to send traces, metrics, logs and profiles between SDKs, Collectors and backends. It defines protobuf messages and two transports, gRPC and HTTP.
What ports does OTLP use?
4317 for OTLP over gRPC and 4318 for OTLP over HTTP. These are defaults, so a hosted backend may serve both on 443 instead.
Should I use OTLP over gRPC or HTTP?
The specification makes http/protobuf the default, and it works through any HTTP proxy, load balancer or browser. gRPC can be slightly more efficient between services you control. If you are not sure, use http/protobuf.
Why does my OTLP exporter get a 404?
Usually the path. OTEL_EXPORTER_OTLP_ENDPOINT is a base URL and the SDK appends /v1/traces. The per-signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is used exactly as written, so it must include /v1/traces. Setting /v1/traces on the generic variable produces /v1/traces/v1/traces.
What does 413 mean from an OTLP endpoint?
The request body was larger than the server accepts. Lower the export batch size in the SDK or set max_size on the Collector exporter's batch settings, and turn on gzip compression if it is off.
Can I send OTLP as JSON?
Yes, over HTTP with Content-Type: application/json. Trace and span IDs are hex strings, enums are integers, field names are lowerCamelCase and 64-bit integers such as timestamps are written as strings. gRPC always uses protobuf.

See your first trace today.

Add the SDK, point your OTLP exporter at Maple, and traces arrive.

maple.dev: observability on OpenTelemetry