📘 第22课：容器监控

生产部署

📊 为什么需要容器监控？

生产环境的容器需要持续监控，以便发现性能瓶颈、资源泄漏和异常行为。没有监控就是盲人摸象。

🔧 cAdvisor + Prometheus + Grafana

# 完整监控栈 ✅
services:
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
    ports:
      - "8080:8080"

  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    volumes:
      - grafana-data:/var/lib/grafana
    ports:
      - "3000:3000"
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  prometheus-data:
  grafana-data:

📋 Prometheus配置

# prometheus.yml ✅
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

📈 Docker内置监控

# 实时资源使用 ✅
docker stats
CONTAINER  CPU%  MEM USAGE/LIMIT  MEM%  NET I/O     BLOCK I/O
nginx      0.01% 2.5MiB/7.77GiB  0.03% 1.2kB/0B    0B/0B
api        1.5%  128MiB/7.77GiB 1.6%  456kB/256kB  12MB/0B

# 查看特定容器
docker stats --no-stream nginx

# 自定义格式
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

# 事件监控 ✅
docker events --filter 'type=container' --filter 'event=die'

🔔 告警规则

# alert_rules.yml ✅
groups:
  - name: container_alerts
    rules:
      - alert: ContainerDown
        expr: time() - container_last_seen > 60
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Container {{ $labels.name }} is down"

      - alert: HighMemoryUsage
        expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Container {{ $labels.name }} using >90% memory"

      - alert: HighCPUUsage
        expr: rate(container_cpu_usage_seconds_total[5m]) > 0.8
        for: 5m
        labels:
          severity: warning

❓ 常见问题

📚 深度补充：容器监控进阶要点

【运维与部署阶段】生产环境注意事项

常见误区与避坑指南

1. 过度配置：不是所有服务都需要高可用，根据实际需求选择架构复杂度，避免过度工程
2. 忽略日志：日志是排障的关键，确保日志格式统一、级别合理、采集完整
3. 资源超卖：容器资源限制不是摆设，超卖会导致性能下降甚至OOM Kill
4. 安全忽视：默认配置不等于安全配置，生产环境必须加固（非root、只读FS、最小能力）
5. 监控缺失：没有监控等于盲飞，至少要有基础的健康检查和资源监控
6. 手动运维：能自动化的绝不手动，手动操作容易出错且不可追溯
7. 忽略网络策略：默认所有容器互通不安全，应按最小权限原则配置网络

进阶阅读与参考

• Docker官方文档：https://docs.docker.com/
• Docker最佳实践指南：Dev Best Practices
• CIS Docker Benchmark：容器安全基线检查标准
• 12-Factor App方法论：https://12factor.net/
• Google SRE Book：运维可靠性工程实践
• Kubernetes官方文档：https://kubernetes.io/docs/

Docker命令速查卡

# 容器生命周期
docker create/start/stop/restart/rm/pause/unpause
docker logs/top/stats/inspect/exec diff

# 镜像构建
docker build/pull/push/tag/rmi/images/history
docker save/load/import/manifest

# 网络与存储
docker network create/ls/inspect/connect/disconnect/rm/prune
docker volume create/ls/inspect/rm/prune

# Docker Compose
docker compose up/down/ps/logs/build/exec
docker compose config/stop/start/scale/top/cp

# 系统维护
docker system df/prune/info
docker builder prune
docker container/prune/image prune

# K8s常用命令
kubectl get/describe/logs/exec/apply/delete
kubectl rollout status/undo/history
kubectl scale/autoscale/set

本阶段知识脉络

生产环境检查清单

SRE关键指标（SLI/SLO）

实战案例：故障排查流程

# Step 1: 检查容器状态
docker ps -a                          # 查看所有容器
docker inspect <container>            # 查看详细配置

# Step 2: 查看日志
docker logs --tail 100 <container>    # 最近100行日志
docker logs --since 1h <container>    # 最近1小时

# Step 3: 进入容器排查
docker exec -it <container> sh        # 进入容器shell

# Step 4: 检查资源
docker stats --no-stream              # 资源使用概览
docker system df                      # 磁盘使用

# Step 5: 网络排查
docker network ls                     # 网络列表
docker exec <c> ping <target>        # 网络连通性
docker exec <c> nslookup <svc>       # DNS解析

# Step 6: 检查健康状态
docker inspect --format '{{.State.Health}}' <c>
docker inspect --format '{{.State.ExitCode}}' <c>
docker inspect --format '{{.State.OOMKilled}}' <c>

Dockerfile模板：多语言通用

# 通用最佳实践模板
FROM alpine:3.19 AS builder
# ... 构建步骤 ...

FROM alpine:3.19
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
WORKDIR /app
COPY --from=builder /app/output .
USER appuser
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD wget -qO- http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["./app"]

Docker Compose健康检查模板

services:
  app:
    build: .
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M

性能调优要点