AI/ML工作负载需要GPU加速。K8s通过设备插件（Device Plugin）机制支持GPU调度，NVIDIA提供官方的GPU Device Plugin。
┌─────── K8s GPU调度架构 ────────┐
│ │
│ ┌───── kube-scheduler ──────┐ │
│ │ Pod请求: nvidia.com/gpu:1│ │
│ │ → 调度到有GPU的节点 │ │
│ └──────────┬────────────────┘ │
│ │ │
│ ┌──────────▼────────────────┐ │
│ │ GPU Node │ │
│ │ ┌──────────────────┐ │ │
│ │ │ nvidia-device- │ │ │
│ │ │ plugin DaemonSet │ │ │
│ │ │ │ │ │
│ │ │ 1.发现GPU设备 │ │ │
│ │ │ 2.注册到kubelet │ │ │
│ │ │ 3.分配GPU给Pod │ │ │
│ │ └──────────────────┘ │ │
│ │ │ │
│ │ NVIDIA Driver + CUDA │ │
│ │ ┌──────┐ ┌──────┐ │ │
│ │ │GPU 0 │ │GPU 1 │ │ │
│ │ └──────┘ └──────┘ │ │
│ └───────────────────────────┘ │
│ │
│ GPU资源类型： │
│ nvidia.com/gpu = 整卡 │
│ nvidia.com/gpu.shared = MIG │
│ nvidia.com/mig-1g.5gb = MIG切 │
│ │
│ 关键限制： │
│ • GPU资源不可超分（1卡=1请求） │
│ • 默认不支持分时共享 │
│ • 需要NVIDIA驱动+Container │
│ Toolkit预装 │
└───────────────────────────────────┘# 安装NVIDIA驱动
sudo apt-get update
sudo apt-get install -y nvidia-driver-535
# 验证驱动
nvidia-smi
# +-----------------------------------------------------------------------------+
# | NVIDIA-SMI 535.x Driver Version: 535.x CUDA Version: 12.x |
# | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
# | 0 A100-SXM... On | 00000000:00:04.0 Off | 0 |
# +-----------------------------------------------------------------------------+
# 安装NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -sL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=containerd
sudo systemctl restart containerd
# ✅ 验证通过
sudo nvidia-ctk --version# 安装NVIDIA Device Plugin
kubectl create namespace nvidia-gpu-operator
helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm install gpu-operator nvidia/gpu-operator \
--namespace nvidia-gpu-operator \
--set driver.enabled=false \
--set toolkit.enabled=false
# 或简单安装（节点已有驱动和toolkit）
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yaml
# ✅ 验证通过 - 节点报告GPU资源
kubectl get nodes -o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu
# NAME GPU
# k8s-gpu-1 2
# k8s-gpu-2 4
kubectl describe node k8s-gpu-1 | grep -A5 Allocatable
# Allocatable:
# cpu: 16
# memory: 65536Mi
# nvidia.com/gpu: 2# gpu-pod.yaml - 请求1块GPU
apiVersion: v1
kind: Pod
metadata:
name: gpu-test
spec:
containers:
- name: cuda
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ['nvidia-smi']
resources:
limits:
nvidia.com/gpu: 1 # 请求1块GPU
requests:
nvidia.com/gpu: 1
# ✅ 验证通过
kubectl apply -f gpu-pod.yaml
kubectl logs gpu-test
# +-----------------------------------------------------------------------------+
# | NVIDIA-SMI 535.x Driver Version: 535.x CUDA Version: 12.x |
# | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
# | 0 A100-SXM... On | 00000000:00:04.0 Off | 0 |
# +-----------------------------------------------------------------------------+# multi-gpu-training.yaml
apiVersion: v1
kind: Pod
metadata:
name: multi-gpu-training
spec:
containers:
- name: pytorch
image: pytorch/pytorch:2.2.0-cuda12.4-cudnn9-devel
command: ['python', '-c']
args:
- |
import torch
print(f"CUDA available: {torch.cuda.is_available()}")
print(f"GPU count: {torch.cuda.device_count()}")
for i in range(torch.cuda.device_count()):
print(f"GPU {i}: {torch.cuda.get_device_name(i)}")
# 简单矩阵运算
x = torch.randn(10000, 10000).cuda()
y = torch.randn(10000, 10000).cuda()
z = torch.mm(x, y)
print(f"Matrix multiplication result shape: {z.shape}")
resources:
limits:
nvidia.com/gpu: 2 # 请求2块GPU
requests:
nvidia.com/gpu: 2
cpu: "4"
memory: "16Gi"
# ✅ 验证通过
kubectl logs multi-gpu-training
# CUDA available: True
# GPU count: 2
# GPU 0: NVIDIA A100-SXM4-80GB
# GPU 1: NVIDIA A100-SXM4-80GB
# Matrix multiplication result shape: torch.Size([10000, 10000])# 给GPU节点打标签
kubectl label nodes k8s-gpu-1 hardware-type=NVIDIA-A100
kubectl label nodes k8s-gpu-2 hardware-type=NVIDIA-V100
# GPU优先调度（使用nodeAffinity）
apiVersion: v1
kind: Pod
metadata:
name: a100-training
spec:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: hardware-type
operator: In
values: [NVIDIA-A100]
containers:
- name: training
image: pytorch/pytorch:2.2.0-cuda12.4-cudnn9-devel
resources:
limits:
nvidia.com/gpu: 1# A100支持MIG，将1块GPU切分为多个实例
# 配置MIG策略
kubectl label nodes k8s-gpu-1 nvidia.com/mig.config=all-1g.5gb
# 请求MIG实例
apiVersion: v1
kind: Pod
metadata:
name: mig-pod
spec:
containers:
- name: inference
image: nvidia/cuda:12.4.0-base-ubuntu22.04
command: ['nvidia-smi']
resources:
limits:
nvidia.com/mig-1g.5gb: 1 # 请求1个MIG实例（1G显存+5GB）
# MIG配置选项（A100 80GB）：
# all-1g.5gb → 7个1g.5gb实例
# all-2g.10gb → 3个2g.10gb实例 + 1个1g.5gb实例
# all-3g.20gb → 2个3g.20gb实例
# all-4g.20gb → 1个4g.20gb实例# NVIDIA GPU Sharing（vGPU / MPS）
# 配置时间片共享
apiVersion: v1
kind: ConfigMap
metadata:
name: gpu-sharing-config
data:
sharing.yaml: |
version: v1
sharing:
timeSlicing:
resources:
- name: nvidia.com/gpu
replicas: 4 # 1块GPU虚拟为4个
# Device Plugin使用共享配置
kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/main/deployments/static/nvidia-device-plugin.yml
# Pod请求共享GPU
apiVersion: v1
kind: Pod
metadata:
name: shared-gpu-pod
spec:
containers:
- name: app
image: nvidia/cuda:12.4.0-base-ubuntu22.04
resources:
limits:
nvidia.com/gpu: 1 # 使用1/4的GPU# Kubeflow Pipeline - ML训练流水线
# 1. 使用PodTemplate定义GPU训练任务
apiVersion: kubeflow.org/v2beta1
kind: MPIJob
metadata:
name: distributed-training
spec:
slotsPerWorker: 1
runPolicy:
cleanPodPolicy: Running
mpiReplicaSpecs:
Launcher:
replicas: 1
template:
spec:
containers:
- image: ml-training:v1
name: launcher
Worker:
replicas: 4 # 4个Worker各1块GPU
template:
spec:
containers:
- image: ml-training:v1
name: worker
resources:
limits:
nvidia.com/gpu: 1
# 2. 训练数据使用PVC共享
# 3. 模型输出存储到对象存储
# 4. 使用Argo Workflows编排训练流程# GPU不可见
nvidia-smi
# 如果失败 → 检查驱动安装
# Pod无法调度到GPU节点
kubectl describe pod <name> | grep -A5 Events
# 0/x nodes are available: Insufficient nvidia.com/gpu
# Device Plugin未注册
kubectl get pods -n nvidia-gpu-operator
kubectl logs -n nvidia-gpu-operator nvidia-device-plugin-xxx
# 容器内nvidia-smi失败
kubectl exec <pod> -- nvidia-smi
# 检查Container Toolkit配置
# cat /etc/containerd/config.toml | grep nvidia下一课预告：第25课毕业项目——构建生产级K8s平台！
24-gpu调度补充要点：K8s生产实践扩展
🔹 资源配额(ResourceQuota)：限制命名空间总资源
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: production
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
pods: "50"
services: "10"
🔹 LimitRange：设置默认资源限制
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
spec:
limits:
- type: Container
default:
cpu: "200m"
memory: 256Mi
defaultRequest:
cpu: "100m"
memory: 128Mi
max:
cpu: "2"
memory: 4Gi
🔹 Pod优先级与抢占
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
---
spec:
preemptionPolicy: PreemptLowerPriority
🔹 优雅处理Pod中断
• PDB保证最小可用副本
• preStop钩子处理连接排空
• terminationGracePeriodSeconds充足
• 应用必须处理SIGTERM信号
🔹 生产环境Checklist
✅ 设置resources requests/limits
✅ 配置liveness/readiness探针
✅ 使用PDB保护关键服务
✅ 实现优雅关闭（SIGTERM）
✅ 配置HPA自动伸缩
✅ 使用NetworkPolicy隔离
✅ 开启RBAC最小权限
✅ 日志结构化输出
✅ 指标暴露/metrics端点
✅ 配置PVC数据备份