🎓 第35课:毕业项目:生产级DevOps平台

核心概念

毕业项目:生产级DevOps平台

综合运用35课所学,构建完整的DevOps平台——代码管理、CI/CD、监控告警、安全合规、成本管控。

平台架构全景

┌────────────────────────────────────────────────────────┐
│                    用户层                               │
│  开发者 → Git Push → 自动CI/CD → 生产部署              │
└───────────────────────┬────────────────────────────────┘
┌───────────────────────▼────────────────────────────────┐
│                   平台服务层                            │
│  Git/GH │ CI/CD Pipeline │ Harbor │ Vault │ Terraform  │
│  Prometheus+Grafana │ ELK/Loki │ Trivy │ ArgoCD        │
└───────────────────────┬────────────────────────────────┘
┌───────────────────────▼────────────────────────────────┐
│                   基础设施层                            │
│  K8s Cluster + Ingress + Storage + Network + DNS       │
└────────────────────────────────────────────────────────┘

命令实操

1. 基础设施搭建(Terraform) ✅

Terraform基础
cat > main.tf << 'EOF'
terraform {
  required_version = ">= 1.5"
  required_providers {
    aws = { source = "hashicorp/aws", version = "~> 5.0" }
    kubernetes = { source = "hashicorp/kubernetes", version = "~> 2.25" }
    helm = { source = "hashicorp/helm", version = "~> 2.12" }
  }
}

module "vpc" { source = "./modules/vpc", cidr = "10.0.0.0/16", env = "prod" }
module "eks" { source = "./modules/eks", cluster_name = "devops-platform", vpc_id = module.vpc.vpc_id }
module "rds" { source = "./modules/rds", vpc_id = module.vpc.vpc_id, engine = "postgresql" }
module "redis" { source = "./modules/elasticache", vpc_id = module.vpc.vpc_id }
EOF

terraform init && terraform plan -out=tfplan && terraform apply tfplan

2. 平台服务部署(Helm) ✅

Helm部署
#!/bin/bash
# deploy-platform.sh
NS="platform"

# Harbor
helm upgrade --install harbor harbor/harbor -n $NS \
  --set expose.ingress.hosts.core=harbor.devops.example.com \
  --set trivy.enabled=true

# Prometheus+Grafana
helm upgrade --install monitoring prometheus-community/kube-prometheus-stack \
  -n monitoring --create-namespace \
  --set grafana.ingress.enabled=true \
  --set grafana.ingress.hosts=["grafana.devops.example.com"] \
  --set prometheus.prometheusSpec.retention=30d

# Vault
helm upgrade --install vault hashicorp/vault -n $NS \
  --set server.standalone.enabled=true

# ArgoCD
helm upgrade --install argocd argo/argo-cd \
  -n argocd --create-namespace \
  --set server.ingress.enabled=true \
  --set server.ingress.hosts=["argocd.devops.example.com"]

# Cert-Manager
helm upgrade --install cert-manager jetstack/cert-manager \
  -n cert-manager --create-namespace --set installCRDs=true

echo "✅ 平台服务部署完成" 

3. 完整CI/CD流水线 ✅

完整流水线
cat > .github/workflows/full-pipeline.yml << 'EOF'
name: Full CI/CD Pipeline
on: {push: {branches: [main, develop]}, pull_request: {branches: [main}}

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: actions/setup-node@v4
      with: {node-version: '20'}
    - run: npm ci
    - run: npm test -- --coverage

  security-scan:
    needs: test
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v4
    - uses: aquasecurity/trivy-action@master
      with: {scan-type: 'fs', severity: 'CRITICAL,HIGH', exit-code: '1'}

  build-and-push:
    needs: security-scan
    runs-on: ubuntu-latest
    if: github.event_name == 'push'
    steps:
    - uses: actions/checkout@v4
    - run: |
        docker build -t harbor.devops.example.com/devops/myapp:${{ github.sha }} .
        docker push harbor.devops.example.com/devops/myapp:${{ github.sha }}
    - uses: aquasecurity/trivy-action@master
      with: {image-ref: 'harbor.devops.example.com/devops/myapp:${{ github.sha }}', severity: 'CRITICAL', exit-code: '1'}

  deploy-production:
    needs: build-and-push
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
    - name: Canary deploy (10%)
      run: kubectl argo rollouts set image myapp-rollout myapp=harbor.devops.example.com/devops/myapp:${{ github.sha }}
    - name: Monitor canary
      run: |
        sleep 300
        # Check error rate from Prometheus
        ERROR_RATE=$(curl -s 'http://prometheus:9090/api/v1/query' --data-urlencode 'query=sum(rate(http_requests_total{version="canary",status=~"5.."}[5m]))/sum(rate(http_requests_total{version="canary"}[5m]))' | jq -r '.data.result[0].value[1]')
        if (( $(echo "$ERROR_RATE > 0.01" | bc -l) )); then
          kubectl argo rollouts abort myapp-rollout
          exit 1
        fi
    - name: Full rollout
      run: kubectl argo rollouts promote myapp-rollout
EOF

4. 监控与告警配置 ✅

监控告警
cat > platform-alerts.yaml << 'EOF'
groups:
- name: platform-critical
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
      / sum(rate(http_requests_total[5m])) by (service) > 0.01
    for: 5m
    labels: {severity: critical}

  - alert: PodCrashLooping
    expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
    for: 5m
    labels: {severity: critical}

  - alert: DiskSpaceLow
    expr: (node_filesystem_avail_bytes / node_filesystem_size_bytes) < 0.15
    for: 5m
    labels: {severity: warning}
EOF

# Grafana仪表盘导入
for id in 1860 15760 11558; do
    curl -u admin:admin123 -X POST http://grafana:3000/api/dashboards/import \
      -H "Content-Type: application/json" \
      -d "{"dashboard": $(curl -s https://grafana.com/api/dashboards/$id/revisions/1/download), "overwrite": true}"
done

架构图

毕业项目:生产级DevOps平台架构

┌─────────────────────────────────────────────────────────┐
│  开发者 → GitHub → Actions CI/CD → Harbor → K8s(ArgoCD) │
│    │         │    安全扫描+签名     │        │            │
│    │         │    (Trivy+Cosign)   │    GitOps            │
│    └─────────┴─────────────────────┘        │            │
│                                              ▼            │
│  ┌─────────────────────────────────────────────────────┐  │
│  │              K8s Production                         │  │
│  │  App(v1/v2) ← Istio mTLS                          │  │
│  │  Prometheus+Grafana+Loki+Alertmanager              │  │
│  │  Vault(密钥) + OPA(策略) + Trivy(扫描)             │  │
│  │  Velero(备份) + Kubecost(成本)                      │  │
│  └─────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────┘

故障排查

❌ 端到端流水线故障

排查
# 按阶段排查
npm test
trivy fs --severity HIGH,CRITICAL ./
docker build -t test . 2>&1 | tail -20
kubectl rollout status deployment/myapp
kubectl get pods -l app=myapp
argocd app get myapp --refresh
💡 毕业项目整合:IaC(Terraform) → CI/CD(Actions+ArgoCD) → 监控(Prometheus+Grafana) → 安全(Trivy+Vault+OPA) → DR(Velero) → FinOps(Kubecost)。

🏆 成就解锁:DevOps毕业!!

你已掌握完整DevOps技能栈——从Linux运维到云原生、从CI/CD到安全合规,现在去构建你的生产级平台吧!🎓