Distributed Tracing

Glintlog provides full distributed tracing support via OpenTelemetry. Visualize request flows, analyze latency, and understand service dependencies.

Concepts

Traces

A trace represents a complete request as it flows through your distributed system. Each trace has a unique trace_id that connects all related operations.

Spans

A span represents a single operation within a trace. Spans have:

trace_id - Links the span to its parent trace
span_id - Unique identifier for this span
parent_span_id - The parent span (if any)
name - Operation name
start_time and end_time - When the operation occurred
attributes - Key-value metadata
status - Success, error, or unset

Sending Traces

gRPC (Port 4317)

package main

import (
    "context"
    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
    "go.opentelemetry.io/otel/sdk/trace"
    "go.opentelemetry.io/otel/sdk/resource"
    semconv "go.opentelemetry.io/otel/semconv/v1.24.0"
)

func initTracer() (*trace.TracerProvider, error) {
    exporter, err := otlptracegrpc.New(context.Background(),
        otlptracegrpc.WithEndpoint("localhost:4317"),
        otlptracegrpc.WithInsecure(),
    )
    if err != nil {
        return nil, err
    }

    res := resource.NewWithAttributes(
        semconv.SchemaURL,
        semconv.ServiceName("my-service"),
    )

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithResource(res),
    )

    otel.SetTracerProvider(tp)
    return tp, nil
}

HTTP (Port 4318)

curl -X POST http://localhost:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{
    "resourceSpans": [{
      "resource": {
        "attributes": [{
          "key": "service.name",
          "value": {"stringValue": "my-service"}
        }]
      },
      "scopeSpans": [{
        "spans": [{
          "traceId": "5B8EFFF798038103D269B633813FC60C",
          "spanId": "EEE19B7EC3C1B174",
          "name": "process-request",
          "kind": 1,
          "startTimeUnixNano": "1544712660000000000",
          "endTimeUnixNano": "1544712661000000000",
          "attributes": [{
            "key": "http.method",
            "value": {"stringValue": "POST"}
          }]
        }]
      }]
    }]
  }'

Viewing Traces

Web UI

Navigate to the Traces page in the web UI to:

Browse recent traces
Filter by service, status, or duration
View trace timelines with span hierarchies
Analyze latency breakdowns
See service dependencies

API

List traces:

curl "http://localhost:8080/api/v1/traces" \
  -H "Authorization: Bearer YOUR_TOKEN"

Get trace details:

curl "http://localhost:8080/api/v1/traces/5B8EFFF798038103D269B633813FC60C" \
  -H "Authorization: Bearer YOUR_TOKEN"

Response:

{
  "trace_id": "5B8EFFF798038103D269B633813FC60C",
  "spans": [
    {
      "span_id": "EEE19B7EC3C1B174",
      "parent_span_id": null,
      "name": "HTTP POST /api/orders",
      "service_name": "api-gateway",
      "start_time": "2024-01-15T10:30:00.000Z",
      "end_time": "2024-01-15T10:30:00.250Z",
      "duration_ms": 250,
      "status": "OK",
      "attributes": {
        "http.method": "POST",
        "http.status_code": "200"
      }
    },
    {
      "span_id": "ABC123...",
      "parent_span_id": "EEE19B7EC3C1B174",
      "name": "create-order",
      "service_name": "order-service",
      "start_time": "2024-01-15T10:30:00.050Z",
      "end_time": "2024-01-15T10:30:00.200Z",
      "duration_ms": 150,
      "status": "OK"
    }
  ],
  "services": ["api-gateway", "order-service"],
  "total_duration_ms": 250
}

Correlating Logs and Traces

Glintlog automatically correlates logs with traces when both share the same trace_id.

In Your Code

Include trace context in your logs:

ctx, span := tracer.Start(ctx, "process-order")
defer span.End()

// Log with trace context
logger.InfoContext(ctx, "Processing order",
    "order_id", orderID,
)

In the UI

When viewing a trace, click "View Logs" to see all log entries associated with that trace.

When viewing logs, click on a trace_id to jump to the trace view.

Query Parameters

Parameter	Type	Description
`service`	string	Filter by service name
`start`	RFC3339	Start timestamp
`end`	RFC3339	End timestamp
`min_duration`	integer	Minimum duration in milliseconds
`max_duration`	integer	Maximum duration in milliseconds
`status`	string	Filter by status (OK, ERROR)
`limit`	integer	Max results (default: 50)
`offset`	integer	Pagination offset

Best Practices

Name spans descriptively - Use clear operation names like HTTP GET /users or db.query.users
Add relevant attributes - Include context like user IDs, request IDs, and error details
Set status on errors - Mark spans as ERROR when operations fail
Use span events - Add events for notable occurrences within a span
Propagate context - Ensure trace context flows across service boundaries