📈 第08课:HPA自动伸缩

📌 课程阶段:存储与调度(3/5)|预计时间:60分钟|难度:⭐⭐⭐☆☆

一、HPA工作原理

Horizontal Pod Autoscaler(HPA)根据监控指标自动调整Deployment/StatefulSet的副本数,实现应用弹性伸缩

┌──────────── HPA工作流程 ────────────┐
│                                       │
│  HPA Controller (kube-controller-mgr) │
│         │                             │
│  ┌──────▼──────┐                      │
│  │ 查询Metrics │ ← Metrics Server    │
│  │ 当前CPU=80% │    / Prometheus      │
│  └──────┬──────┘                      │
│         │                             │
│  ┌──────▼──────┐                      │
│  │ 计算目标副本│                      │
│  │ desired =   │                      │
│  │ ceil(current│                      │
│  │ * current_util│                    │
│  │ / target)   │                      │
│  │ = ceil(2*80/50)│                   │
│  │ = 4         │                      │
│  └──────┬──────┘                      │
│         │                             │
│  ┌──────▼──────┐                      │
│  │ 更新副本数  │                      │
│  │ replicas: 4 │                      │
│  └─────────────┘                      │
│                                       │
│  稳定窗口:避免频繁抖动               │
│  扩容:立即执行                       │
│  缩容:5分钟窗口(默认)              │
└───────────────────────────────────────┘

二、安装Metrics Server

# HPA依赖Metrics Server获取Pod资源指标
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml

# 如果是自签名证书环境,需要添加 --kubelet-insecure-tls
kubectl patch deployment metrics-server -n kube-system --type=json \
  -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

# ✅ 验证通过
kubectl get pods -n kube-system -l k8s-app=metrics-server
# NAME                              READY   STATUS    RESTARTS   AGE
# metrics-server-7f8fbbf8b4-xxxxx   1/1     Running   0          30s

kubectl top nodes
# NAME          CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
# k8s-master    250m         12%    1024Mi          27%
# k8s-worker1   180m         9%     768Mi           20%

kubectl top pods
# NAME                    CPU(cores)   MEMORY(bytes)
# nginx-6c8f9d7b5e-xxx    1m           8Mi

三、基于CPU的HPA

# 首先创建一个有资源限制的Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: stress-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: stress
  template:
    metadata:
      labels:
        app: stress
    spec:
      containers:
      - name: stress
        image: progrium/stress
        command: ['stress', '--cpu', '1']  # 1个CPU核心的压力
        resources:
          requests:
            cpu: "200m"          # 0.2核 → HPA基于此计算百分比
            memory: "128Mi"
          limits:
            cpu: "500m"
            memory: "256Mi"

---
# hpa-cpu.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: stress-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: stress-app
  minReplicas: 2
  maxReplicas: 10
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization       # 利用率百分比
        averageUtilization: 50  # 目标CPU利用率50%

  behavior:                     # 伸缩行为控制
    scaleDown:
      stabilizationWindowSeconds: 300  # 缩容稳定窗口5分钟
      policies:
      - type: Percent
        value: 10              # 每次最多缩容10%
        periodSeconds: 60
    scaleUp:
      stabilizationWindowSeconds: 0    # 扩容无等待
      policies:
      - type: Percent
        value: 100             # 每次最多扩容100%(翻倍)
        periodSeconds: 15
      - type: Pods
        value: 4               # 或每次最多增加4个Pod
        periodSeconds: 15
      selectPolicy: Max        # 取两种策略的最大值

# ✅ 验证通过
kubectl apply -f hpa-cpu.yaml
kubectl get hpa
# NAME         REFERENCE                TARGETS   MINPODS   MAXPODS   REPLICAS   AGE
# stress-hpa   Deployment/stress-app    80%/50%   2         10        4          5m

四、基于内存的HPA

# hpa-memory.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: memory-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: memory-app
  minReplicas: 2
  maxReplicas: 8
  metrics:
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70  # 目标内存利用率70%

五、基于自定义指标的HPA

# 需要Prometheus Adapter将自定义指标注册到API Server
# 1. 安装Prometheus Adapter
# helm install prometheus-adapter prometheus-community/prometheus-adapter

# 2. 配置自定义指标规则
# 规则:将Prometheus的http_requests_per_second映射为K8s自定义指标
apiVersion: v1
kind: ConfigMap
metadata:
  name: adapter-config
  namespace: monitoring
data:
  config.yaml: |
    rules:
    - seriesQuery: 'http_requests_total{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: "namespace"}
          pod: {resource: "pod"}
      name:
        matches: "^(.*)_total"
        as: "${1}_per_second"
      metricsQuery: 'sum(rate(<<.Series>>{<<.LabelMatchers>>}[2m])) by (<<.GroupBy>>)'

---
# hpa-custom.yaml - 基于QPS扩缩容
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: qps-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api-server
  minReplicas: 3
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"    # 每Pod目标1000 QPS

六、多指标HPA

# hpa-multi.yaml - 同时基于CPU、内存和QPS
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: multi-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-api
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 60
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "500"
  # 取满足所有指标所需的最大副本数

七、VPA与Cluster Autoscaler

7.1 VPA(垂直伸缩)

# VPA自动调整Pod的CPU/Memory请求值
# 安装VPA
kubectl apply -f https://github.com/kubernetes/autoscaler/releases/latest/download/vpa-v1-beta2.yaml

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: my-app-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: my-app
  updatePolicy:
    updateMode: Auto    # Off|Initial|Recreate|Auto
  resourcePolicy:
    containerPolicies:
    - containerName: '*'
      minAllowed:
        cpu: 100m
        memory: 128Mi
      maxAllowed:
        cpu: "2"
        memory: 4Gi

# ⚠️ VPA Auto模式会重启Pod来调整资源!

7.2 Cluster Autoscaler(集群伸缩)

# Cluster Autoscaler自动增减节点
# 云环境安装示例(AWS EKS)
# eksctl utils install-cluster-autoscaler --cluster=my-cluster

# 工作原理:
# 1. 有Pod因资源不足Pending → 添加新节点
# 2. 节点利用率低 → 排空并删除节点
# 3. 最小/最大节点数限制

# 关键配置:
# --scale-down-unneeded-time=10m     节点空闲多久后缩
# --scale-down-utilization-threshold=0.5  利用率低于50%视为空闲
# --expander=least-waste             扩容策略:最少浪费

八、故障排查实战

8.1 HPA无法获取指标

# 现象:TARGETS显示<unknown>
kubectl get hpa
# NAME         TARGETS         MINPODS   MAXPODS   REPLICAS
# stress-hpa   <unknown>/50%  2         10        2

# 排查1:Metrics Server是否运行
kubectl get pods -n kube-system -l k8s-app=metrics-server

# 排查2:Pod是否设置了resources.requests
kubectl get pod <name> -o jsonpath='{.spec.containers[0].resources.requests}'
# 如果没有设置requests,HPA无法计算利用率!

# 排查3:Metrics Server API是否可访问
kubectl get --raw /apis/metrics.k8s.io/v1beta1/namespaces/default/pods

8.2 HPA扩容不及时

# 检查HPA事件
kubectl describe hpa stress-hpa
# Events:
#   Normal  SuccessfulRescale  ...  New size: 4

# 调优:
# 1. 减小扩容稳定窗口
# behavior.scaleUp.stabilizationWindowSeconds: 0
# 2. 增大扩容比例
# behavior.scaleUp.policies[0].value: 200
# 3. 减小指标采集间隔(Metrics Server默认60s)

8.3 HPA频繁抖动

# 现象:副本数频繁增减
# 解决:增大缩容稳定窗口
behavior:
  scaleDown:
    stabilizationWindowSeconds: 600  # 10分钟
    policies:
    - type: Percent
      value: 5          # 每次最多缩5%
      periodSeconds: 120

九、练习

  1. 部署Metrics Server,使用kubectl top查看节点和Pod资源使用
  2. 创建CPU压力测试Deployment + HPA,观察自动扩容过程
  3. 配置HPA behavior参数,实现"快速扩容、缓慢缩容"策略
  4. 同时基于CPU和内存创建多指标HPA
  5. 对比HPA和VPA的适用场景,分析何时选择哪种

🏆 第08课成就解锁

下一课预告:第09课深入DaemonSet与Job——特殊工作负载管理。

📌 补充知识

08-hpa自动伸缩补充要点K8s生产实践扩展

🔹 资源配额(ResourceQuota):限制命名空间总资源
  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: compute-quota
    namespace: production
  spec:
    hard:
      requests.cpu: "20"
      requests.memory: 40Gi
      limits.cpu: "40"
      limits.memory: 80Gi
      pods: "50"
      services: "10"

🔹 LimitRange:设置默认资源限制
  apiVersion: v1
  kind: LimitRange
  metadata:
    name: default-limits
  spec:
    limits:
    - type: Container
      default:
        cpu: "200m"
        memory: 256Mi
      defaultRequest:
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi

🔹 Pod优先级与抢占
  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
    name: high-priority
  value: 1000000
  globalDefault: false
  ---
  spec:
    preemptionPolicy: PreemptLowerPriority

🔹 优雅处理Pod中断
  • PDB保证最小可用副本
  • preStop钩子处理连接排空
  • terminationGracePeriodSeconds充足
  • 应用必须处理SIGTERM信号

🔹 生产环境Checklist
  ✅ 设置resources requests/limits
  ✅ 配置liveness/readiness探针
  ✅ 使用PDB保护关键服务
  ✅ 实现优雅关闭(SIGTERM)
  ✅ 配置HPA自动伸缩
  ✅ 使用NetworkPolicy隔离
  ✅ 开启RBAC最小权限
  ✅ 日志结构化输出
  ✅ 指标暴露/metrics端点
  ✅ 配置PVC数据备份