🎯 第07课：节点调度

一、K8s调度器原理

kube-scheduler负责将未调度的Pod分配到合适的节点。调度过程分为过滤(Filter)和打分(Score)两个阶段。

┌──────── 调度流程 ────────┐
│                           │
│  未调度Pod进入调度队列     │
│         │                 │
│  ┌──────▼──────┐          │
│  │  过滤阶段    │          │
│  │  (Filter)   │          │
│  │  排除不满足  │          │
│  │  硬性条件的  │          │
│  │  节点        │          │
│  └──────┬──────┘          │
│         │                 │
│  ┌──────▼──────┐          │
│  │  打分阶段    │          │
│  │  (Score)    │          │
│  │  对可行节点  │          │
│  │  按策略打分  │          │
│  │  选最高分    │          │
│  └──────┬──────┘          │
│         │                 │
│    绑定到最优节点           │
│    (Bind)                 │
└───────────────────────────┘

过滤条件（硬性）：
  ✅ 节点资源是否充足（CPU/Memory）
  ✅ nodeSelector/nodeAffinity是否匹配
  ✅ taint是否被容忍
  ✅ PV的volumeBindingMode
  ✅ 端口是否冲突

打分策略（软性）：
  📊 资源均衡（Bin Packing vs Spread）
  📊 亲和性权重
  📊 Pod拓扑分布
  📊 镜像本地性（已拉取镜像加分）

二、nodeSelector——最简单的调度约束

# 给节点打标签
kubectl label nodes k8s-worker1 disktype=ssd
kubectl label nodes k8s-worker2 disktype=hdd
kubectl label nodes k8s-worker1 gpu=nvidia-a100

# 查看节点标签
kubectl get nodes --show-labels | grep disktype

# Pod使用nodeSelector
apiVersion: v1
kind: Pod
metadata:
  name: ssd-pod
spec:
  nodeSelector:
    disktype: ssd            # 只调度到有ssd标签的节点
  containers:
  - name: app
    image: nginx:1.25
    resources:
      requests:
        cpu: "100m"
        memory: "128Mi"

# ✅ 验证通过
kubectl apply -f ssd-pod.yaml
kubectl get pod ssd-pod -o jsonpath='{.spec.nodeName}'
# k8s-worker1

三、nodeAffinity——更灵活的调度约束

3.1 requiredDuringScheduling（硬性要求）

# node-affinity-required.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity-required
spec:
  affinity:
    nodeAffinity:
      # 硬性要求：必须满足，否则Pending
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: disktype
            operator: In        # In|NotIn|Exists|DoesNotExist|Gt|Lt
            values:
            - ssd
            - nvme
          - key: zone
            operator: In
            values:
            - cn-east-1
  containers:
  - name: app
    image: nginx:1.25

# ✅ 验证通过
kubectl apply -f node-affinity-required.yaml
kubectl get pod affinity-required -o wide
# NAME                 NODE
# affinity-required    k8s-worker1   ← 有ssd标签的节点

3.2 preferredDuringScheduling（软性偏好）

# node-affinity-preferred.yaml
apiVersion: v1
kind: Pod
metadata:
  name: affinity-preferred
spec:
  affinity:
    nodeAffinity:
      # 软性偏好：尽量满足，不满足也能调度
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 80              # 权重1-100
        preference:
          matchExpressions:
          - key: disktype
            operator: In
            values:
            - ssd
      - weight: 20
        preference:
          matchExpressions:
          - key: zone
            operator: In
            values:
            - cn-east-1
  containers:
  - name: app
    image: nginx:1.25

四、podAffinity与podAntiAffinity——Pod间亲和性

# pod-affinity-demo.yaml - Web与Cache部署在同一节点
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-frontend
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      affinity:
        # Pod亲和：调度到有cache Pod的节点
        podAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                app: cache       # 与有此标签的Pod同节点
            topologyKey: kubernetes.io/hostname
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 50
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: cache
              topologyKey: topology.kubernetes.io/zone
        
        # Pod反亲和：不与同类Pod在同一节点（分散部署）
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchLabels:
                  app: web       # 不与同类Pod同节点
              topologyKey: kubernetes.io/hostname
      containers:
      - name: web
        image: nginx:1.25

# ✅ 验证通过 - 3个Pod分散到不同节点
kubectl get pods -l app=web -o wide
# NAME                            NODE
# web-frontend-xxx                k8s-worker1
# web-frontend-yyy                k8s-worker2
# web-frontend-zzz                k8s-worker3

五、Taint与Toleration——节点排斥

Taint = 节点的"排斥标记"
  格式：key=value:effect
  effect类型：
    NoSchedule       → 不调度（新Pod）
    PreferNoSchedule → 尽量不调度（软性）
    NoExecute        → 不调度 + 驱逐已有Pod

Toleration = Pod的"免疫力"
  容忍特定Taint的Pod才能调度到该节点

# 给节点添加Taint
kubectl taint nodes k8s-worker1 gpu=true:NoSchedule
kubectl taint nodes k8s-worker2 dedicated=special:NoExecute

# 查看Taint
kubectl describe node k8s-worker1 | grep Taints
# Taints: gpu=true:NoSchedule

# Pod添加Toleration
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  tolerations:
  # 精确匹配
  - key: "gpu"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"
  
  # 通配：容忍某key的所有值
  - key: "gpu"
    operator: "Exists"
    effect: "NoSchedule"
  
  # 容忍所有Taint（谨慎使用！）
  - operator: "Exists"
  
  # NoExecute的可选容忍时间
  - key: "dedicated"
    operator: "Equal"
    value: "special"
    effect: "NoExecute"
    tolerationSeconds: 3600   # 1小时后被驱逐
  
  containers:
  - name: app
    image: nginx:1.25

# ✅ 验证通过 - 有Toleration的Pod能调度到Taint节点
kubectl apply -f gpu-pod.yaml
kubectl get pod gpu-pod -o jsonpath='{.spec.nodeName}'
# k8s-worker1

六、TopologySpreadConstraints——拓扑分布约束

# topology-spread.yaml - 均匀分布到不同Zone
apiVersion: apps/v1
kind: Deployment
metadata:
  name: spread-demo
spec:
  replicas: 6
  selector:
    matchLabels:
      app: spread
  template:
    metadata:
      labels:
        app: spread
    spec:
      topologySpreadConstraints:
      - maxSkew: 1                    # 最大偏差（1表示分布尽可能均匀）
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: DoNotSchedule  # DoNotSchedule|ScheduleAnyway
        labelSelector:
          matchLabels:
            app: spread
      - maxSkew: 1
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: DoNotSchedule
        labelSelector:
          matchLabels:
            app: spread
      containers:
      - name: app
        image: nginx:1.25

# ✅ 验证通过 - Pod均匀分布
kubectl get pods -l app=spread -o wide
# NAME               NODE           ZONE
# spread-demo-xxx    k8s-worker1    cn-east-1a
# spread-demo-yyy    k8s-worker2    cn-east-1b
# spread-demo-zzz    k8s-worker3    cn-east-1a
# spread-demo-aaa    k8s-worker1    cn-east-1a
# spread-demo-bbb    k8s-worker2    cn-east-1b
# spread-demo-ccc    k8s-worker3    cn-east-1a

七、自定义调度器

# 使用自定义调度器
apiVersion: v1
kind: Pod
metadata:
  name: custom-scheduled
spec:
  schedulerName: my-scheduler    # 默认是default-scheduler
  containers:
  - name: app
    image: nginx:1.25

# 自定义调度器实现方式：
# 1. 独立进程，Watch未调度Pod（schedulerName匹配）
# 2. 调用API Server的Bind接口
# 3. 参考kube-scheduler框架（sigs.k8s.io/scheduler-framework）

八、故障排查实战

8.1 Pod一直Pending

# 查看调度失败原因
kubectl describe pod <pod-name> | grep -A10 Events
# Warning  FailedScheduling  ...  0/3 nodes are available

# 常见原因：
# 1. 资源不足 → 减少requests或添加节点
# 2. nodeSelector不匹配 → 检查节点标签
# 3. Taint未容忍 → 添加Toleration
# 4. PVC无法绑定 → 检查StorageClass

# 查看节点资源
kubectl top nodes
kubectl describe node k8s-worker1 | grep -A5 Allocatable

8.2 调度不均匀

# 所有Pod集中在一个节点
# 检查是否有podAntiAffinity
kubectl get pod <name> -o yaml | grep -A10 affinity

# 解决：
# 1. 添加podAntiAffinity
# 2. 使用TopologySpreadConstraints
# 3. 检查节点Taint是否阻止调度

九、练习

1. 使用nodeSelector将Pod调度到指定标签节点
2. 配置nodeAffinity实现"优先SSD节点，其次HDD节点"的调度策略
3. 使用podAntiAffinity让3个Nginx副本分散到不同节点
4. 给节点添加Taint，创建带Toleration的Pod验证调度行为
5. 使用TopologySpreadConstraints实现6个副本在3个Zone均匀分布

📌 补充知识

07-节点调度补充要点：K8s生产实践扩展

🔹 资源配额(ResourceQuota)：限制命名空间总资源
  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: compute-quota
    namespace: production
  spec:
    hard:
      requests.cpu: "20"
      requests.memory: 40Gi
      limits.cpu: "40"
      limits.memory: 80Gi
      pods: "50"
      services: "10"

🔹 LimitRange：设置默认资源限制
  apiVersion: v1
  kind: LimitRange
  metadata:
    name: default-limits
  spec:
    limits:
    - type: Container
      default:
        cpu: "200m"
        memory: 256Mi
      defaultRequest:
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi

🔹 Pod优先级与抢占
  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
    name: high-priority
  value: 1000000
  globalDefault: false
  ---
  spec:
    preemptionPolicy: PreemptLowerPriority

🔹 优雅处理Pod中断
  • PDB保证最小可用副本
  • preStop钩子处理连接排空
  • terminationGracePeriodSeconds充足
  • 应用必须处理SIGTERM信号

🔹 生产环境Checklist
  ✅ 设置resources requests/limits
  ✅ 配置liveness/readiness探针
  ✅ 使用PDB保护关键服务
  ✅ 实现优雅关闭（SIGTERM）
  ✅ 配置HPA自动伸缩
  ✅ 使用NetworkPolicy隔离
  ✅ 开启RBAC最小权限
  ✅ 日志结构化输出
  ✅ 指标暴露/metrics端点
  ✅ 配置PVC数据备份