K8s灾难恢复的核心是保护etcd数据和持久化存储。没有etcd，整个集群不可用；没有PV数据，应用状态丢失。
┌─────── 灾难恢复层级 ────────┐
│ │
│ Layer 1: etcd备份与恢复 │
│ → 集群状态（所有API对象） │
│ → 最关键！丢失=重建集群 │
│ │
│ Layer 2: 持久化数据备份 │
│ → PV/PVC中的业务数据 │
│ → 数据库、文件、配置等 │
│ │
│ Layer 3: 集群配置备份 │
│ → PKI证书、kubeconfig │
│ → 加密配置、审计策略 │
│ │
│ Layer 4: 应用清单备份 │
│ → GitOps（推荐） │
│ → Helm values / YAML文件 │
│ │
│ RPO (恢复点目标): 可接受的数据丢失量 │
│ RTO (恢复时间目标): 可接受的停机时间 │
│ │
│ 策略选择： │
│ RPO=0 → 同步复制（etcd HA） │
│ RPO<5min → 定期快照 │
│ RTO<15min → 自动化恢复脚本 │
└────────────────────────────────┘# etcd内置快照
ETCDCTL_API=3 etcdctl snapshot save /backup/etcd-snapshot-$(date +%Y%m%d).db \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
# ✅ 验证通过 - 检查快照
ETCDCTL_API=3 etcdctl snapshot status /backup/etcd-snapshot-20260101.db --write-table
# +----------+----------+------------+------------+
# | REVISION | HASH | SIZE | AGE |
# +----------+----------+------------+------------+
# | 12345678 | abcdef12 | 45 MB | 1 day |
# +----------+----------+------------+------------+# etcd-backup-cronjob.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
name: etcd-backup
namespace: kube-system
spec:
schedule: "0 */6 * * *" # 每6小时备份
concurrencyPolicy: Forbid
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 1
jobTemplate:
spec:
template:
spec:
nodeSelector:
node-role.kubernetes.io/control-plane: ""
tolerations:
- key: node-role.kubernetes.io/control-plane
effect: NoSchedule
containers:
- name: backup
image: bitnami/etcd:3.5
command:
- /bin/sh
- -c
- |
export ETCDCTL_API=3
SNAPSHOT_NAME="/backup/etcd-snapshot-$(date +%Y%m%d-%H%M%S).db"
etcdctl snapshot save "$SNAPSHOT_NAME" \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key
echo "Backup saved: $SNAPSHOT_NAME"
# 清理7天前的备份
find /backup -name "etcd-snapshot-*.db" -mtime +7 -delete
volumeMounts:
- name: etcd-certs
mountPath: /etc/kubernetes/pki/etcd
readOnly: true
- name: backup-volume
mountPath: /backup
restartPolicy: OnFailure
volumes:
- name: etcd-certs
hostPath:
path: /etc/kubernetes/pki/etcd
- name: backup-volume
persistentVolumeClaim:
claimName: etcd-backup-pvc
# ✅ 验证通过
kubectl get cronjob -n kube-system etcd-backup
# NAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGE
# etcd-backup 0 */6 * * * False 0 2h 1d# ⚠️ 恢复前必须停止etcd和API Server！
# 1. 停止所有控制平面组件
sudo systemctl stop kubelet
sudo docker stop $(sudo docker ps -q --filter name=k8s_)
# 或者停止静态Pod
sudo mv /etc/kubernetes/manifests/ /etc/kubernetes/manifests.bak/
# 2. 恢复etcd快照
sudo ETCDCTL_API=3 etcdctl snapshot restore /backup/etcd-snapshot-20260101.db \
--data-dir=/var/lib/etcd/restore \
--name=k8s-master \
--initial-cluster=k8s-master=https://127.0.0.1:2380 \
--initial-cluster-token=etcd-cluster \
--initial-advertise-peer-urls=https://127.0.0.1:2380
# 3. 替换etcd数据目录
sudo mv /var/lib/etcd/member /var/lib/etcd/member.bak
sudo cp -r /var/lib/etcd/restore/member /var/lib/etcd/
# 4. 恢复静态Pod
sudo mv /etc/kubernetes/manifests.bak/ /etc/kubernetes/manifests/
# 5. 重启kubelet
sudo systemctl start kubelet
# ✅ 验证通过
kubectl get nodes
# NAME STATUS ROLES AGE VERSION
# k8s-master Ready control-plane 5m v1.30.x# Velero = 集群资源 + PV数据备份
# 安装Velero
curl -L https://github.com/vmware-tanzu/velero/releases/latest/download/velero-linux-amd64 -o velero
chmod +x velero && sudo mv velero /usr/local/bin/
# 安装Velero到集群（以S3为后端）
velero install \
--provider aws \
--bucket my-k8s-backup \
--backup-location-config region=us-east-1 \
--snapshot-location-config region=us-east-1 \
--secret-file ./credentials-velero \
--use-volume-snapshots=true \
--plugins velero/velero-plugin-for-aws:v1.8
# ✅ 验证通过
kubectl get pods -n velero
# NAME READY STATUS RESTARTS AGE
# velero-xxxxx 1/1 Running 0 30s
# 创建备份
velero backup create full-backup --include-cluster-resources=true
# 定时备份
velero schedule create daily-backup --schedule="0 2 * * *" --include-cluster-resources=true
# 按命名空间备份
velero backup create ns-backup --include-namespaces production
# 按标签备份
velero backup create app-backup --selector app=my-app
# 查看备份
velero backup get
# NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION
# full-backup Completed 0 0 2026-01-01 02:00:00 +0800 CST 29d default# 恢复备份到新集群
velero restore create --from-backup full-backup
# 恢复特定命名空间
velero restore create --from-backup full-backup --include-namespaces production
# 恢复到不同命名空间
velero restore create --from-backup full-backup \
--namespace-mappings production:production-dr
# 查看恢复状态
velero restore get
# NAME BACKUP STATUS STARTED COMPLETED
# full-backup-20260101... full-backup Completed 2026-01-01 03:00:00 +0800 CST 2026-01-01 03:05:00
# ✅ 验证通过
kubectl get all -n production# 方式1：Velero CSI快照（推荐）
velero backup create pvc-backup \
--include-namespaces default \
--snapshot-volumes=true
# 方式2：应用层备份（数据库dump）
apiVersion: batch/v1
kind: CronJob
metadata:
name: mysql-backup
spec:
schedule: "0 3 * * *"
jobTemplate:
spec:
template:
spec:
containers:
- name: backup
image: mysql:8.0
command: ['sh', '-c']
args:
- |
mysqldump -h mysql -u$DB_USER -p$DB_PASS --all-databases | \
gzip > /backup/mysql_$(date +%Y%m%d).sql.gz
# 上传到S3
aws s3 cp /backup/mysql_$(date +%Y%m%d).sql.gz s3://my-backup/mysql/
envFrom:
- secretRef:
name: db-creds
restartPolicy: OnFailure
# 方式3：Restic文件级备份
velero backup create restic-backup \
--include-namespaces default \
--default-volumes-to-restic# 演练脚本：模拟集群故障并恢复
# Step 1: 记录当前状态
kubectl get all -A > /tmp/pre-dr-state.txt
# Step 2: 模拟灾难（在DR环境！）
# 停止主集群
sudo systemctl stop kubelet
sudo systemctl stop containerd
# Step 3: 在DR集群恢复
velero restore create dr-restore --from-backup daily-backup
# Step 4: 验证恢复
kubectl get nodes
kubectl get pods -A
kubectl get pv,pvc -A
# Step 5: 验证应用
curl http://app.example.com/healthz
# Step 6: 切换DNS
# 将app.example.com指向DR集群的Ingress IP
# 关键指标：
# RPO: 上次备份到灾难发生之间的数据丢失
# RTO: 灾难发生到服务恢复的时间下一课预告：第21课开始实战项目——微服务部署。
20-灾难恢复补充要点：K8s生产实践扩展
🔹 资源配额(ResourceQuota)：限制命名空间总资源
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: production
spec:
hard:
requests.cpu: "20"
requests.memory: 40Gi
limits.cpu: "40"
limits.memory: 80Gi
pods: "50"
services: "10"
🔹 LimitRange：设置默认资源限制
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
spec:
limits:
- type: Container
default:
cpu: "200m"
memory: 256Mi
defaultRequest:
cpu: "100m"
memory: 128Mi
max:
cpu: "2"
memory: 4Gi
🔹 Pod优先级与抢占
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
name: high-priority
value: 1000000
globalDefault: false
---
spec:
preemptionPolicy: PreemptLowerPriority
🔹 优雅处理Pod中断
• PDB保证最小可用副本
• preStop钩子处理连接排空
• terminationGracePeriodSeconds充足
• 应用必须处理SIGTERM信号
🔹 生产环境Checklist
✅ 设置resources requests/limits
✅ 配置liveness/readiness探针
✅ 使用PDB保护关键服务
✅ 实现优雅关闭（SIGTERM）
✅ 配置HPA自动伸缩
✅ 使用NetworkPolicy隔离
✅ 开启RBAC最小权限
✅ 日志结构化输出
✅ 指标暴露/metrics端点
✅ 配置PVC数据备份20-灾难恢复生产环境进阶要点
🔹 性能优化关键参数
• kubelet: --max-pods=110 --pods-per-core=10
• kube-apiserver: --max-requests-inflight=800
• etcd: --quota-backend-bytes=8589934592
• kube-scheduler: --percentage-of-nodes-to-score=50
🔹 集群容量规划
• 控制平面：3节点，8C16G起步
• Worker节点：按应用类型分组
• etcd：SSD磁盘，<2ms延迟
• 网络带宽：10Gbps+集群内互联
🔹 故障自愈最佳实践
1. Pod: livenessProbe自动重启
2. Deployment: ReplicaSet保证副本数
3. Node: kubelet自注册+健康检查
4. Cluster: Cluster Autoscaler增减节点
5. Multi-Cluster: Karmada联邦容灾
🔹 K8s版本升级策略
• 每次只升一个minor版本
• 先升级控制平面，再升级Worker
• 使用kubeadm upgrade plan预检
• 准备回滚方案
• 在staging环境验证后再升级prod