📊 第13课：Prometheus监控

📌 课程阶段：安全与监控（3/5）｜预计时间：60分钟｜难度：⭐⭐⭐⭐☆

一、K8s可观测性三大支柱

可观测性（Observability）= 指标(Metrics) + 日志(Logs) + 追踪(Traces)。Prometheus是K8s生态中最流行的监控指标系统。

┌─────────── K8s可观测性架构 ───────────┐
│                                         │
│  ┌─────────── Metrics ───────────┐      │
│  │  Prometheus + Grafana         │      │
│  │  → 指标采集、存储、告警、可视化 │      │
│  └───────────────────────────────┘      │
│                                         │
│  ┌─────────── Logs ──────────────┐      │
│  │  EFK/Loki + Grafana           │      │
│  │  → 日志收集、聚合、搜索        │      │
│  └───────────────────────────────┘      │
│                                         │
│  ┌─────────── Traces ────────────┐      │
│  │  Jaeger/Tempo + Grafana       │      │
│  │  → 分布式追踪、链路分析        │      │
│  └───────────────────────────────┘      │
│                                         │
│  ┌──── ServiceMonitor/PodMonitor ──┐    │
│  │  Prometheus Operator自动发现    │    │
│  │  → 声明式监控配置              │    │
│  └─────────────────────────────────┘    │
└─────────────────────────────────────────┘

二、安装Prometheus Stack

# 使用kube-prometheus-stack（推荐一站式方案）
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set prometheus.prometheusSpec.retention=15d \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.storageClassName=standard \
  --set prometheus.prometheusSpec.storageSpec.volumeClaimTemplate.spec.resources.requests.storage=50Gi \
  --set grafana.adminPassword=admin123

# ✅ 验证通过
kubectl get pods -n monitoring
# NAME                                       READY   STATUS    RESTARTS   AGE
# monitoring-prometheus-operator-xxx          1/1     Running   0          2m
# monitoring-grafana-xxx                      2/2     Running   0          2m
# monitoring-kube-prometheus-prometheus-0     2/2     Running   0          2m
# monitoring-kube-state-metrics-xxx           1/1     Running   0          2m
# monitoring-prometheus-node-exporter-xxx     1/1     Running   0          2m

# 访问Grafana
kubectl port-forward svc/monitoring-grafana 3000:80 -n monitoring
# 浏览器打开 http://localhost:3000  admin/admin123

三、Prometheus核心概念

3.1 指标类型

类型	含义	示例
Counter	只增计数器	`http_requests_total`
Gauge	可增可减的值	`node_memory_available_bytes`
Histogram	分布统计	`http_request_duration_seconds`
Summary	分位数统计	`http_request_duration_seconds_quantile`

3.2 PromQL常用查询

# 节点CPU使用率
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Pod内存使用
container_memory_working_set_bytes{container!="",container!="POD"}

# HTTP请求QPS
sum(rate(http_requests_total[5m])) by (service)

# P99延迟
histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[5m])) by (le, service))

# Pod重启次数
kube_pod_container_status_restarts_total

# 可用Pod数 vs 期望Pod数
kube_deployment_status_available_replicas / kube_deployment_spec_replicas

四、ServiceMonitor——声明式监控

# 应用暴露metrics端点
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: app
        image: my-app:latest
        ports:
        - containerPort: 8080
          name: http-metrics         # 指标端口名
        # 应用需要暴露 /metrics 端点
        # Python示例：from prometheus_client import start_http_server

---
# Service关联应用
apiVersion: v1
kind: Service
metadata:
  name: my-app-metrics
  labels:
    app: my-app
spec:
  selector:
    app: my-app
  ports:
  - name: http-metrics
    port: 8080
    targetPort: http-metrics

---
# ServiceMonitor自动发现
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: my-app-monitor
  labels:
    release: monitoring             # 匹配Prometheus的serviceMonitorSelector
spec:
  selector:
    matchLabels:
      app: my-app
  namespaceSelector:
    any: true                       # 监控所有namespace
  endpoints:
  - port: http-metrics
    path: /metrics                  # 指标路径
    interval: 15s                   # 采集间隔
    scrapeTimeout: 10s

# ✅ 验证通过 - Prometheus已发现Target
kubectl port-forward svc/monitoring-kube-prometheus-prometheus 9090:9090 -n monitoring
# 浏览器打开 http://localhost:9090/targets → 查看my-app

五、应用埋点实践

# Python应用Prometheus埋点示例
from prometheus_client import Counter, Histogram, Gauge, generate_latest
from fastapi import FastAPI, Response
import time

app = FastAPI()

# Counter：请求计数
REQUEST_COUNT = Counter(
    'http_requests_total',
    'Total HTTP requests',
    ['method', 'endpoint', 'status']
)

# Histogram：请求延迟
REQUEST_LATENCY = Histogram(
    'http_request_duration_seconds',
    'HTTP request latency',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0]
)

# Gauge：当前连接数
ACTIVE_CONNECTIONS = Gauge(
    'app_active_connections',
    'Current active connections'
)

@app.middleware("http")
async def metrics_middleware(request, call_next):
    start = time.time()
    ACTIVE_CONNECTIONS.inc()
    response = await call_next(request)
    ACTIVE_CONNECTIONS.dec()
    
    REQUEST_COUNT.labels(
        method=request.method,
        endpoint=request.url.path,
        status=response.status_code
    ).inc()
    
    REQUEST_LATENCY.labels(
        method=request.method,
        endpoint=request.url.path
    ).observe(time.time() - start)
    
    return response

@app.get("/metrics")
async def metrics():
    return Response(generate_latest(), media_type="text/plain")

六、Grafana Dashboard

# 导入社区Dashboard
# 1. 访问Grafana → Dashboards → Import
# 2. 输入Dashboard ID

# 常用Dashboard ID：
# 1860  - Node Exporter Full（节点监控）
# 15760 - Kubernetes Views（K8s概览）
# 11700 - Kubernetes Pod Resources
# 7249  - Cluster Monitoring for Kubernetes
# 13189 - K8s Views Remote

# 自定义Dashboard JSON片段
{
  "title": "My App Dashboard",
  "panels": [
    {
      "title": "Request Rate",
      "type": "timeseries",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{service=\"my-app\"}[5m])) by (status)"
        }
      ]
    },
    {
      "title": "P99 Latency",
      "type": "stat",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=\"my-app\"}[5m])) by (le))"
        }
      ]
    }
  ]
}

七、故障排查实战

7.1 Prometheus Target Down

# 检查Target状态
# Prometheus UI → Status → Targets
# State: DOWN → Last Error: ...

# 常见原因：
# 1. Service selector不匹配 → 检查标签
# 2. Pod未就绪 → 检查readinessProbe
# 3. NetworkPolicy阻止 → 放行Prometheus NS
# 4. /metrics端点不存在 → 检查应用实现

7.2 指标缺失

# 在Prometheus中搜索指标
up{job="my-app"}
# 如果value=1表示Target UP

# 检查ServiceMonitor是否被选中
kubectl get servicemonitor my-app-monitor -o yaml
# 确认labels匹配Prometheus的serviceMonitorSelector

八、练习

使用Helm安装kube-prometheus-stack，访问Grafana Dashboard
创建ServiceMonitor监控自定义应用
编写PromQL查询：CPU使用率、内存使用率、请求QPS
为应用添加Prometheus埋点，验证指标采集
导入社区Dashboard，自定义面板展示应用指标

🏆 第13课成就解锁

✅ 理解K8s可观测性三大支柱
✅ 掌握kube-prometheus-stack安装与配置
✅ 熟练使用PromQL编写监控查询
✅ 掌握ServiceMonitor声明式监控配置
✅ 了解应用Prometheus埋点最佳实践
✅ 能排查Prometheus Target Down等故障

下一课预告：第14课深入日志收集——EFK/Loki日志体系。

← 上一课：NetworkPolicy 📚 目录下一课：日志收集 →

📌 补充知识

13-prometheus监控补充要�K8s生产实践扩展

🔹 资源配额(ResourceQuota)：限制命名空间总资源
  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: compute-quota
    namespace: production
  spec:
    hard:
      requests.cpu: "20"
      requests.memory: 40Gi
      limits.cpu: "40"
      limits.memory: 80Gi
      pods: "50"
      services: "10"

🔹 LimitRange：设置默认资源限制
  apiVersion: v1
  kind: LimitRange
  metadata:
    name: default-limits
  spec:
    limits:
    - type: Container
      default:
        cpu: "200m"
        memory: 256Mi
      defaultRequest:
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi

🔹 Pod优先级与抢占
  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
    name: high-priority
  value: 1000000
  globalDefault: false
  ---
  spec:
    preemptionPolicy: PreemptLowerPriority

🔹 优雅处理Pod中断
  • PDB保证最小可用副本
  • preStop钩子处理连接排空
  • terminationGracePeriodSeconds充足
  • 应用必须处理SIGTERM信号

🔹 生产环境Checklist
  ✅ 设置resources requests/limits
  ✅ 配置liveness/readiness探针
  ✅ 使用PDB保护关键服务
  ✅ 实现优雅关闭（SIGTERM）
  ✅ 配置HPA自动伸缩
  ✅ 使用NetworkPolicy隔离
  ✅ 开启RBAC最小权限
  ✅ 日志结构化输出
  ✅ 指标暴露/metrics端点
  ✅ 配置PVC数据备份

📎 扩展阅读与生产实践

13-prometheus监控生产环境进阶要点

🔹 性能优化关键参数
  • kubelet: --max-pods=110 --pods-per-core=10
  • kube-apiserver: --max-requests-inflight=800
  • etcd: --quota-backend-bytes=8589934592
  • kube-scheduler: --percentage-of-nodes-to-score=50

🔹 集群容量规划
  • 控制平面：3节点，8C16G起步
  • Worker节点：按应用类型分组
  • etcd：SSD磁盘，<2ms延迟
  • 网络带宽：10Gbps+集群内互联

🔹 故障自愈最佳实践
  1. Pod: livenessProbe自动重启
  2. Deployment: ReplicaSet保证副本数
  3. Node: kubelet自注册+健康检查
  4. Cluster: Cluster Autoscaler增减节点
  5. Multi-Cluster: Karmada联邦容灾

🔹 K8s版本升级策略
  • 每次只升一个minor版本
  • 先升级控制平面，再升级Worker
  • 使用kubeadm upgrade plan预检
  • 准备回滚方案
  • 在staging环境验证后再升级prod