📘 第34课：监控与告警

Kubernetes

📊 K8s监控体系

K8s集群和应用的监控是运维的核心。Prometheus + Grafana是事实上的标准方案。

🔧 kube-prometheus-stack

# 使用Helm一键部署完整监控栈 ✅
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace \
  --set grafana.adminPassword=admin \
  --set prometheus.prometheusSpec.retention=30d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=standard \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi

# 访问Grafana ✅
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# 访问Prometheus ✅
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090:9090

📋 监控指标分类

🔔 Prometheus告警规则

# 告警规则 ✅
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: app-alerts
spec:
  groups:
  - name: app
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[5m]) > 0
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Pod {{ $labels.namespace }}/{{ $labels.pod }} is crash looping"

    - alert: HighMemoryUsage
      expr: |
        container_memory_working_set_bytes /
        container_spec_memory_limit_bytes > 0.9
      for: 5m
      labels:
        severity: warning

    - alert: PodNotReady
      expr: |
        kube_pod_status_phase{phase!="Running"} == 1
      for: 10m
      labels:
        severity: warning

📧 AlertManager通知

# AlertManager配置 ✅
global:
  resolve_timeout: 5m

route:
  group_by: ['alertname', 'namespace']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: 'slack'
  routes:
  - match:
      severity: critical
    receiver: 'pagerduty'

receivers:
- name: 'slack'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/xxx'
    channel: '#alerts'
- name: 'pagerduty'
  pagerduty_configs:
  - service_key: '<key>'

📊 应用自定义指标

# 应用暴露/metrics端点 ✅
# Prometheus客户端库（以Node.js为例）
const client = require('prom-client');
const registry = new client.Registry();

const httpRequestDuration = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'HTTP request duration',
  labelNames: ['method', 'route', 'status'],
  registers: [registry]
});

app.get('/metrics', (req, res) => {
  res.set('Content-Type', registry.contentType);
  res.end(registry.metrics());
});

# K8s ServiceMonitor自动发现 ✅
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: http
    path: /metrics
    interval: 15s

❓ 常见问题

📚 深度补充：监控与告警进阶要点

【Kubernetes入门阶段】生产环境注意事项

常见误区与避坑指南

1. 过度配置：不是所有服务都需要高可用，根据实际需求选择架构复杂度，避免过度工程
2. 忽略日志：日志是排障的关键，确保日志格式统一、级别合理、采集完整
3. 资源超卖：容器资源限制不是摆设，超卖会导致性能下降甚至OOM Kill
4. 安全忽视：默认配置不等于安全配置，生产环境必须加固（非root、只读FS、最小能力）
5. 监控缺失：没有监控等于盲飞，至少要有基础的健康检查和资源监控
6. 手动运维：能自动化的绝不手动，手动操作容易出错且不可追溯
7. 忽略网络策略：默认所有容器互通不安全，应按最小权限原则配置网络

进阶阅读与参考

• Docker官方文档：https://docs.docker.com/
• Docker最佳实践指南：Dev Best Practices
• CIS Docker Benchmark：容器安全基线检查标准
• 12-Factor App方法论：https://12factor.net/
• Google SRE Book：运维可靠性工程实践
• Kubernetes官方文档：https://kubernetes.io/docs/

Docker命令速查卡

# 容器生命周期
docker create/start/stop/restart/rm/pause/unpause
docker logs/top/stats/inspect/exec diff

# 镜像构建
docker build/pull/push/tag/rmi/images/history
docker save/load/import/manifest

# 网络与存储
docker network create/ls/inspect/connect/disconnect/rm/prune
docker volume create/ls/inspect/rm/prune

# Docker Compose
docker compose up/down/ps/logs/build/exec
docker compose config/stop/start/scale/top/cp

# 系统维护
docker system df/prune/info
docker builder prune
docker container/prune/image prune

# K8s常用命令
kubectl get/describe/logs/exec/apply/delete
kubectl rollout status/undo/history
kubectl scale/autoscale/set

本阶段知识脉络

Kubernetes资源层次

K8s vs Docker Compose对比

实战案例：故障排查流程

# Step 1: 检查容器状态
docker ps -a                          # 查看所有容器
docker inspect <container>            # 查看详细配置

# Step 2: 查看日志
docker logs --tail 100 <container>    # 最近100行日志
docker logs --since 1h <container>    # 最近1小时

# Step 3: 进入容器排查
docker exec -it <container> sh        # 进入容器shell

# Step 4: 检查资源
docker stats --no-stream              # 资源使用概览
docker system df                      # 磁盘使用

# Step 5: 网络排查
docker network ls                     # 网络列表
docker exec <c> ping <target>        # 网络连通性
docker exec <c> nslookup <svc>       # DNS解析

# Step 6: 检查健康状态
docker inspect --format '{{.State.Health}}' <c>
docker inspect --format '{{.State.ExitCode}}' <c>
docker inspect --format '{{.State.OOMKilled}}' <c>

Dockerfile模板：多语言通用

# 通用最佳实践模板
FROM alpine:3.19 AS builder
# ... 构建步骤 ...

FROM alpine:3.19
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
WORKDIR /app
COPY --from=builder /app/output .
USER appuser
HEALTHCHECK --interval=30s --timeout=3s --retries=3 \
  CMD wget -qO- http://localhost:8080/health || exit 1
EXPOSE 8080
CMD ["./app"]

Docker Compose健康检查模板

services:
  app:
    build: .
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:8080/health"]
      interval: 30s
      timeout: 5s
      retries: 3
      start_period: 30s
    restart: unless-stopped
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M