OpenTelemetry for Rust with the tracing crate

Rust services already log and time work with the tracing crate. tracing-opentelemetry turns those spans into OpenTelemetry spans, a second layer turns events into log records, and both export to Maple over OTLP.

request 5 spans · 39ms
DurationStatus
requestorders-rs
39.0msOk
load_order
11.0msOk
SELECT orders
8.0msOk
HTTP POST
20.0msOk
/chargespayments-api
16.0msOk

Example: an Axum request with #[instrument] and reqwest spans

Quick start

Set up Rust

1

Add the crates

toml
# Cargo.toml: the opentelemetry* crates share one minor version
[dependencies]
opentelemetry = "0.31"
opentelemetry_sdk = "0.31"
opentelemetry-otlp = { version = "0.31", features = ["reqwest-rustls"] }
opentelemetry-appender-tracing = "0.31"
tracing-opentelemetry = "0.32"
tracing = "0.1"
tracing-subscriber = { version = "0.3", features = ["env-filter"] }
tokio = { version = "1", features = ["full"] }
axum = "0.8"
tower-http = { version = "0.6", features = ["trace"] }
2

Install the bridge layers

rust
// src/telemetry.rs: the exporters read OTEL_EXPORTER_OTLP_* from the environment
use opentelemetry::{global, trace::TracerProvider as _};
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use opentelemetry_otlp::{LogExporter, SpanExporter};
use opentelemetry_sdk::{
    logs::SdkLoggerProvider, propagation::TraceContextPropagator, trace::SdkTracerProvider, Resource,
};
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter, Layer};

pub fn init() -> Result<(SdkTracerProvider, SdkLoggerProvider), Box<dyn std::error::Error>> {
    let resource = Resource::builder().build();

    let tracer_provider = SdkTracerProvider::builder()
        .with_batch_exporter(SpanExporter::builder().with_http().build()?)
        .with_resource(resource.clone())
        .build();
    global::set_tracer_provider(tracer_provider.clone());
    global::set_text_map_propagator(TraceContextPropagator::new());

    let logger_provider = SdkLoggerProvider::builder()
        .with_batch_exporter(LogExporter::builder().with_http().build()?)
        .with_resource(resource)
        .build();

    // Keep the exporter's own HTTP client logs out of the OpenTelemetry log pipeline.
    let otel_log_filter = EnvFilter::new("info")
        .add_directive("hyper=off".parse()?)
        .add_directive("h2=off".parse()?)
        .add_directive("reqwest=off".parse()?);

    tracing_subscriber::registry()
        .with(EnvFilter::try_from_default_env().unwrap_or_else(|_| EnvFilter::new("info")))
        .with(tracing_subscriber::fmt::layer())
        .with(tracing_opentelemetry::layer().with_tracer(tracer_provider.tracer("my-rust-app")))
        .with(OpenTelemetryTracingBridge::new(&logger_provider).with_filter(otel_log_filter))
        .init();

    Ok((tracer_provider, logger_provider))
}
3

Trace requests and flush on exit

rust
// src/main.rs
mod telemetry;

use axum::{routing::get, Router};
use tower_http::trace::TraceLayer;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let (tracer_provider, logger_provider) = telemetry::init()?;

    let app = Router::new()
        .route("/api/orders", get(handle_orders))
        // Request spans default to DEBUG, below the INFO filter, so raise them
        .layer(TraceLayer::new_for_http().make_span_with(
            tower_http::trace::DefaultMakeSpan::new().level(tracing::Level::INFO),
        ));
    let listener = tokio::net::TcpListener::bind("0.0.0.0:8080").await?;
    let served = axum::serve(listener, app)
        .with_graceful_shutdown(shutdown_signal())
        .await;

    // Reached on Ctrl-C, SIGTERM or a server error: send the last batch first
    let _ = tracer_provider.shutdown();
    let _ = logger_provider.shutdown();
    Ok(served?)
}

async fn shutdown_signal() {
    let ctrl_c = async {
        let _ = tokio::signal::ctrl_c().await;
    };
    #[cfg(unix)]
    let terminate = async {
        if let Ok(mut sigterm) =
            tokio::signal::unix::signal(tokio::signal::unix::SignalKind::terminate())
        {
            sigterm.recv().await;
        }
    };
    #[cfg(not(unix))]
    let terminate = std::future::pending::<()>();
    tokio::select! {
        _ = ctrl_c => {},
        _ = terminate => {},
    }
}
4

Point the exporters at Maple

bash
export OTEL_SERVICE_NAME="my-rust-app"
export OTEL_EXPORTER_OTLP_ENDPOINT="https://ingest.maple.dev"
export OTEL_EXPORTER_OTLP_PROTOCOL="http/protobuf"
export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer YOUR_INGEST_KEY"
export OTEL_RESOURCE_ATTRIBUTES="deployment.environment.name=production"

cargo run --release

This is the short path. The full Rust guide covers the rest.

Organization in the EU region? Send to https://ingest.eu.maple.dev instead of https://ingest.maple.dev. An ingest key only works in its own region.

What you get

What a Rust service sends to Maple

#[instrument] spans

Every function you mark with #[instrument] becomes a span, with the fields you list as attributes.

Request spans from Tower

TraceLayer opens a span per Axum request, so the functions it calls nest under it in one trace.

tracing events as logs

info! and error! inside a span become OpenTelemetry log records that carry that span's trace and span id.

reqwest calls

reqwest-tracing adds a client span and injects traceparent, so an instrumented callee joins the same trace.

Database spans

Give a span otel.kind = "client" and the db.* fields, and Maple draws the database on the service map.

Metrics

The full guide adds a meter provider. Metrics export every 60 seconds over the same endpoint.

What is different in Rust

Versions move together

The opentelemetry crates share one minor version, and tracing-opentelemetry must be the release built for it: 0.32 pairs with 0.31. A mismatch shows up as trait errors at compile time.

TLS is a feature flag

The default opentelemetry-otlp HTTP client has no TLS. Without the reqwest-rustls feature it cannot reach an https endpoint.

TraceLayer does not propagate

tower-http's TraceLayer ignores incoming traceparent headers. To join a caller's trace, use a crate built for it, such as axum-tracing-opentelemetry.

통합

다른 통합

오늘 첫 트레이스를 확인하세요.

SDK를 추가하고 OTLP 익스포터를 Maple로 향하게 하면 트레이스가 도착합니다.

maple.dev: OpenTelemetry 위의 옵저버빌리티