🔍 可观测性

Metrics / Logs / Traces 三支柱——理解系统在发生什么的唯一途径

1. 什么是可观测性——监控 ≠ 可观测性

Charity Majors（Honeycomb CEO）的定义最为精辟：

"Observability is a measure of how well you can understand the internal state of a system based on its external outputs. Monitoring is what you do when you already know what questions to ask."

这个概念源自控制论。Rudolf E. Kálmán 在 1960 年定义了"可观测性"：一个系统是可观测的，当且仅当从其外部输出可以唯一确定其内部状态。在软件领域，这意味着：

2. 三支柱：Metrics / Logs / Traces

2.1 Metrics 指标

指标是运行时捕获的数值度量，按时间聚合。OpenTelemetry 定义了以下指标仪器类型：

关于 Metrics 稳定性：截至 OpenTelemetry 最新规范，Go/Java/Python/C++/.NET/PHP 的 Metrics API 和 SDK 均已标记为 Stable（稳定），可以放心用于生产环境。

# Counter — 只增计数器
http_requests_total{method="GET", path="/api/users", status="200"} 15423

# Gauge — 可增减的当前值
process_open_fds 127
node_memory_available_bytes 4.2e+09

# Histogram — 直方图（延迟分布）
http_request_duration_seconds_bucket{le="0.1"} 8023
http_request_duration_seconds_bucket{le="0.5"} 14201
http_request_duration_seconds_bucket{le="1.0"} 15890
http_request_duration_seconds_sum 5432.1
http_request_duration_seconds_count 16000

# Summary — 分位数摘要
http_request_duration_seconds{quantile="0.5"} 0.12
http_request_duration_seconds{quantile="0.9"} 0.34
http_request_duration_seconds{quantile="0.99"} 1.02

2.2 Logs 日志

日志是带时间戳的文本记录，分为结构化、半结构化和非结构化三种：

{
  "timestamp": "2026-05-18T08:04:12.345Z",
  "level": "ERROR",
  "service": "payment-gateway",
  "trace_id": "5b8aa5a2d2c872e8321cf37308d69df2",
  "span_id": "051581bf3cb55c13",
  "message": "Payment processing failed",
  "context": {
    "userId": "u_7x8k9m",
    "orderId": "ord_abc123",
    "errorCode": "CARD_DECLINED",
    "amount": 99.00
  }
}

192.168.1.1 - johndoe [18/May/2026:08:04:12 +0800] "POST /api/v1/pay HTTP/1.1" 500 1234 "https://app.example.com" "Mozilla/5.0" {"traceId":"5b8aa5a2","duration":2340,"error":"CARD_DECLINED"}

[ERROR] 2026-05-18 08:04:12 - Payment processing failed for user johndoe. Exception: PaymentGatewayException: Card declined. Order: ord_abc123, Amount: $99.00

2.3 Traces 分布式追踪

追踪记录一个请求从入口到完成的完整路径。核心概念：

W3C Trace Context 标准（traceparent header）是分布式追踪的基础：

# HTTP 请求中传播追踪上下文
traceparent: 00-5b8aa5a2d2c872e8321cf37308d69df2-051581bf3cb55c13-01
# 格式: version-trace_id-span_id-trace_flags
# 01 = sampled（采样）

tracestate: vendor1=value1,vendor2=value2
# 可选：供应商特定信息

3. OpenTelemetry：统一标准

在 OpenTelemetry 出现之前，可观测性生态碎片化严重：Jaeger 用自己的 SDK、Prometheus 用自己的格式、各云厂商有自己的 agent。OpenTelemetry 的使命是统一遥测数据的采集和传输标准。

┌──────────────────────────────────────────────────────────┐
│                    你的应用程序                              │
│  ┌──────────┐  ┌──────────┐  ┌──────────┐                │
│  │ OTel SDK │  │ OTel SDK │  │ OTel SDK │                │
│  │ (Traces) │  │ (Metrics)│  │ (Logs)   │                │
│  └────┬─────┘  └────┬─────┘  └────┬─────┘                │
│       └──────────────┼─────────────┘                       │
│                      ▼                                     │
│           ┌──────────────────────┐                         │
│           │  OTel Collector      │                         │
│           │  ┌──────┐ ┌──────┐  │                         │
│           │  │Recv'r│ │Proc'r│  │                         │
│           │  └──┬───┘ └──┬───┘  │                         │
│           │     └───┬────┘       │                         │
│           │         ▼            │                         │
│           │  ┌──────────────┐   │                         │
│           │  │   Exporter   │   │                         │
│           │  └──────┬───────┘   │                         │
│           └─────────┼───────────┘                         │
└─────────────────────┼────────────────────────────────────┘
                      ▼
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
  ┌─────────┐  ┌───────────┐  ┌──────────┐
  │ Jaeger  │  │ Prometheus │  │ Loki/    │
  │ (Traces)│  │ (Metrics)  │  │ ELK      │
  └─────────┘  └───────────┘  │ (Logs)   │
                              └──────────┘

3.1 OTel Collector 核心组件

3.2 采样策略

在生产环境中，不可能采集每一个 Trace。采样是必要的权衡：

3.3 语义约定 (Semantic Conventions)

OpenTelemetry 定义了标准属性名，确保跨服务、跨语言的一致性：

# 推荐的标准属性名
http.request.method          # "GET", "POST"
http.response.status_code    # 200, 404, 500
url.path                     # "/api/v1/users"
url.query                    # "?page=1&limit=10"
server.address               # "api.example.com"
server.port                  # 443
network.protocol.version     # "1.1"
user_agent.original          # "Mozilla/5.0 ..."

# AI/LLM 语义约定（OpenTelemetry 正在制定中）
gen_ai.system                # "openai", "anthropic"
gen_ai.request.model         # "gpt-4o", "claude-3.5-sonnet"
gen_ai.request.max_tokens    # 4096
gen_ai.response.finish_reason # "stop", "length"
gen_ai.usage.input_tokens    # 1500
gen_ai.usage.output_tokens   # 823

4. 分布式追踪：一个请求经过 5 个服务

假设一个电商下单请求经过 5 个微服务，我们看看追踪如何工作：

4.1 Context Propagation 详解

分布式追踪的核心难题是上下文传播。每个服务需要从入站请求中提取 Trace ID，并在出站请求中注入：

# Node.js 示例：Express 中间件自动传播上下文
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME } = require('@opentelemetry/semantic-conventions');

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: 'payment-gateway',
  }),
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317', // OTLP gRPC
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// 自动注入/提取 W3C traceparent header
// 无需手动修改业务代码！

# Python 示例：手动 Span 创建 + HTTP 传播
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
import requests

# 初始化
provider = TracerProvider()
processor = BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
provider.add_span_processor(processor)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("payment-service")
propagator = TraceContextTextMapPropagator()

# 创建 Span 并在 HTTP 调用中传播上下文
with tracer.start_as_current_span("process_payment") as span:
    span.set_attribute("payment.order_id", "ord_abc123")
    span.set_attribute("payment.amount", 99.00)

    # 调用下游服务时注入 trace context
    headers = {}
    propagator.inject(headers)  # 自动添加 traceparent header

    response = requests.post(
        "http://risk-service/api/v1/evaluate",
        headers=headers,
        json={"orderId": "ord_abc123", "amount": 99.00}
    )

5. SLO / SLI / SLA 定义与设定

Google SRE 书籍提出的三个核心概念：

5.1 常见 SLI 选择

6. Google 四大黄金信号

Google SRE 书提出的四个关键监控维度，是最小化但有效的监控集合：

7. Grafana + Prometheus + Jaeger 实操

这是目前最主流的开源可观测性栈。以下是一个完整的 Docker Compose 部署：

# docker-compose.observability.yml
version: "3.8"

services:
  # ===== 数据采集 =====
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otelcol/config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otelcol/config.yaml:ro
    ports:
      - "4317:4317"   # OTLP gRPC
      - "4318:4318"   # OTLP HTTP
      - "8889:8889"   # Prometheus exporter
    depends_on:
      - jaeger
      - loki

  # ===== Metrics =====
  prometheus:
    image: prom/prometheus:v2.50.0
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"

  # ===== Traces =====
  jaeger:
    image: jaegertracing/all-in-one:1.54
    environment:
      - COLLECTOR_OTLP_ENABLED=true
    ports:
      - "16686:16686"  # Jaeger UI
      - "14268:14268"  # Jaeger HTTP collector
      - "4317"         # OTLP gRPC (via otel-collector)

  # ===== Logs =====
  loki:
    image: grafana/loki:2.9.4
    ports:
      - "3100:3100"
    command: -config.file=/etc/loki/local-config.yaml

  # ===== 可视化 =====
  grafana:
    image: grafana/grafana:10.3.3
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml:ro
    ports:
      - "3000:3000"
    depends_on:
      - prometheus
      - jaeger
      - loki

volumes:
  prometheus-data:
  grafana-data:

7.1 OTel Collector 配置

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    send_batch_size: 1024
    timeout: 5s
  memory_limiter:
    check_interval: 1s
    limit_percentage: 80
    spike_limit_percentage: 25
  # 添加服务名属性
  resource:
    attributes:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  # 发送 Traces 到 Jaeger
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  # 发送 Metrics 到 Prometheus (供 Prometheus 拉取)
  prometheus:
    endpoint: "0.0.0.0:8889"
  # 发送 Logs 到 Loki
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp/jaeger]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, resource, batch]
      exporters: [loki]

7.2 Prometheus 配置

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  # 从 OTel Collector 拉取指标
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

  # 你的应用指标（如果直出 Prometheus 格式）
  - job_name: 'my-app'
    static_configs:
      - targets: ['host.docker.internal:8080']
    metrics_path: '/metrics'

7.3 Grafana 数据源配置

# grafana-datasources.yaml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

  - name: Jaeger
    type: jaeger
    access: proxy
    url: http://jaeger:16686
    # 关键：在 Grafana 中从 Metrics 跳转到 Traces
    jsonData:
      tracesToMetrics:
        - name: 'Search traces for latency'
          tags: ['service']
          datasourceUid: 'prometheus'

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    # 关键：从 Logs 跳转到 Traces
    jsonData:
      derivedFields:
        - datasourceUid: 'jaeger'
          matcherRegex: '"trace_id":"(\w+)"'
          name: TraceID
          url: '$${__value.raw}'

7.4 应用端接入（Node.js 完整示例）

// tracing.js — 在应用入口最先引入
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { OTLPMetricExporter } = require('@opentelemetry/exporter-metrics-otlp-grpc');
const { PeriodicExportingMetricReader } = require('@opentelemetry/sdk-metrics');
const { Resource } = require('@opentelemetry/resources');
const { ATTR_SERVICE_NAME, ATTR_SERVICE_VERSION } = require('@opentelemetry/semantic-conventions');
const { logs } = require('@opentelemetry/api-logs');
const { OTLPLogExporter } = require('@opentelemetry/exporter-logs-otlp-grpc');
const { BatchLogRecordProcessor, LoggerProvider } = require('@opentelemetry/sdk-logs');

// 统一 OTLP endpoint
const OTEL_ENDPOINT = process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4317';

const sdk = new NodeSDK({
  resource: new Resource({
    [ATTR_SERVICE_NAME]: process.env.SERVICE_NAME || 'my-service',
    [ATTR_SERVICE_VERSION]: process.env.SERVICE_VERSION || '1.0.0',
  }),
  traceExporter: new OTLPTraceExporter({ url: OTEL_ENDPOINT }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({ url: OTEL_ENDPOINT }),
    exportIntervalMillis: 10000, // 每 10 秒导出一次指标
  }),
  instrumentations: [
    getNodeAutoInstrumentations({
      // 关闭不需要的自动埋点以减少噪音
      '@opentelemetry/instrumentation-fs': { enabled: false },
    }),
  ],
});

sdk.start();

// 设置日志导出
const loggerProvider = new LoggerProvider();
loggerProvider.addLogRecordProcessor(
  new BatchLogRecordProcessor(
    new OTLPLogExporter({ url: OTEL_ENDPOINT })
  )
);
logs.setGlobalLoggerProvider(loggerProvider);

// 优雅关闭
process.on('SIGTERM', async () => {
  await sdk.shutdown();
  await loggerProvider.shutdown();
});

// ===== 业务代码中使用 =====
const { trace } = require('@opentelemetry/api');
const tracer = trace.getTracer('my-service');

async function handlePayment(orderId, amount) {
  return tracer.startActiveSpan('handle_payment', {
    attributes: {
      'payment.order_id': orderId,
      'payment.amount': amount,
    },
  }, async (span) => {
    try {
      // 调用风控服务
      const riskResult = await tracer.startActiveSpan('check_risk', async (childSpan) => {
        const result = await fetch('http://risk-service/api/v1/evaluate', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ orderId, amount }),
        });
        childSpan.setAttribute('risk.score', (await result.json()).score);
        return result.json();
      });

      // 调用支付网关
      const paymentResult = await tracer.startActiveSpan('charge_card', async (childSpan) => {
        const result = await fetch('http://payment-gateway/api/v1/charge', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ orderId, amount, riskScore: riskResult.score }),
        });
        childSpan.setAttribute('payment.gateway_response', result.status);
        return result.json();
      });

      span.setAttribute('payment.status', 'success');
      return paymentResult;
    } catch (err) {
      span.setAttribute('payment.status', 'failed');
      span.setAttribute('error.type', err.name);
      span.recordException(err);
      throw err;
    } finally {
      span.end();
    }
  });
}

8. AI Agent 系统的可观测性

AI Agent 系统本质上就是分布式系统——多个 Agent 协调 = 共识问题，记忆同步 = 一致性问题，工具调用 = 分布式事务。可观测性对 Agent 系统尤为重要，因为 LLM 的非确定性让调试极其困难。

8.1 Agent Trace 设计

一个 Agent 执行流程的追踪应该长这样：

8.2 Agent 可观测性指标

8.3 LangChain / LangSmith 追踪

LangSmith 是 LangChain 的可观测性平台，提供 Agent 执行的完整追踪：

# Python：接入 LangSmith
import os

# 设置环境变量即可自动追踪
os.environ["LANGCHAIN_TRACING_V2"] = "true"
os.environ["LANGCHAIN_API_KEY"] = "ls-xxxxx"
os.environ["LANGCHAIN_PROJECT"] = "my-agent"

# 所有 LangChain 调用自动被追踪
from langchain_openai import ChatOpenAI
from langchain.agents import create_tool_calling_agent

llm = ChatOpenAI(model="gpt-4o")
# 每次 LLM 调用、每次工具执行都会记录到 LangSmith
agent = create_tool_calling_agent(llm, tools, prompt)

8.4 自建 Agent 追踪（OTel 原生）

# Python：用 OpenTelemetry 追踪 Agent 执行
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(
        OTLPSpanExporter(endpoint="http://otel-collector:4317")
    )
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer("agent-framework")

class TracedAgent:
    def __init__(self, name, llm, tools):
        self.name = name
        self.llm = llm
        self.tools = {t.name: t for t in tools}

    async def run(self, task: str) -> str:
        with tracer.start_as_current_span(
            "agent.run",
            attributes={
                "agent.name": self.name,
                "agent.task": task[:200],  # 截断避免超大 span
            }
        ) as root_span:
            messages = [{"role": "user", "content": task}]
            max_iterations = 10

            for i in range(max_iterations):
                with tracer.start_as_current_span(
                    "agent.llm_call",
                    attributes={"agent.iteration": i}
                ) as llm_span:
                    response = await self.llm.ainvoke(messages)
                    input_tokens = response.usage_metadata.get("input_tokens", 0)
                    output_tokens = response.usage_metadata.get("output_tokens", 0)

                    llm_span.set_attributes({
                        "gen_ai.system": "openai",
                        "gen_ai.request.model": self.llm.model_name,
                        "gen_ai.usage.input_tokens": input_tokens,
                        "gen_ai.usage.output_tokens": output_tokens,
                    })

                if response.tool_calls:
                    for tc in response.tool_calls:
                        with tracer.start_as_current_span(
                            f"agent.tool.{tc['name']}",
                            attributes={
                                "tool.name": tc["name"],
                                "tool.args": str(tc["args"])[:500],
                            }
                        ) as tool_span:
                            try:
                                result = await self.tools[tc["name"]].ainvoke(tc["args"])
                                tool_span.set_attribute("tool.result", str(result)[:500])
                                tool_span.set_attribute("tool.status", "success")
                            except Exception as e:
                                tool_span.set_attribute("tool.status", "error")
                                tool_span.set_attribute("error.type", type(e).__name__)
                                tool_span.record_exception(e)
                                result = f"Error: {e}"
                            messages.append({"role": "tool", "content": str(result)})
                else:
                    root_span.set_attribute("agent.result_length", len(response.content))
                    return response.content

            root_span.set_attribute("agent.status", "max_iterations_reached")
            return "Max iterations reached"

9. 可观测性 Checklist

工具选型矩阵

📅 数据采集日期：2026-05-18 · OpenTelemetry 规范版本截至写作时为最新稳定版
本页面内容基于 OpenTelemetry 官方文档、Google SRE 书籍和 Grafana Labs 文档整理