📝 第14课：日志收集

📌 课程阶段：安全与监控（4/5）｜预计时间：60分钟｜难度：⭐⭐⭐☆☆

一、K8s日志体系概述

K8s日志分为容器标准输出/错误和容器内文件日志两类。日志收集方案需解决：采集→聚合→存储→查询→告警的完整链路。

┌────────────── 日志收集架构 ──────────────┐
│                                            │
│  ┌──── Pod ────┐                           │
│  │  Container  │──→ stdout/stderr          │
│  │  /var/log/* │──→ 文件日志               │
│  └──────┬──────┘                           │
│         │                                  │
│  方案A：Node级DaemonSet采集                │
│  ┌──────▼──────┐                           │
│  │  Fluentd/   │  每节点一个，采集          │
│  │  Fluent Bit │  /var/log/containers/*    │
│  └──────┬──────┘                           │
│         │                                  │
│  方案B：Sidecar采集                        │
│  ┌──────▼──────┐                           │
│  │ Log Sidecar │  每Pod一个，读文件日志     │
│  └──────┬──────┘                           │
│         │                                  │
│  ┌──────▼──────────────────────────────┐   │
│  │         日志后端                     │   │
│  │  Elasticsearch / Loki / CloudWatch  │   │
│  └──────┬──────────────────────────────┘   │
│         │                                  │
│  ┌──────▼──────┐                           │
│  │   Grafana   │  可视化查询               │
│  │   Kibana    │                           │
│  └─────────────┘                           │
└────────────────────────────────────────────┘

二、Fluent Bit轻量日志采集

# fluent-bit-daemonset.yaml
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
  labels:
    app: fluent-bit
spec:
  selector:
    matchLabels:
      app: fluent-bit
  template:
    metadata:
      labels:
        app: fluent-bit
    spec:
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:3.0
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: containers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: containers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: fluent-bit-config

---
# fluent-bit-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: logging
data:
  fluent-bit.conf: |
    [SERVICE]
        Flush         5
        Log_Level     info
        Daemon        off
        Parsers_File  parsers.conf
        HTTP_Server   On
        HTTP_Listen   0.0.0.0
        HTTP_Port     2020

    [INPUT]
        Name              tail
        Path              /var/log/containers/*.log
        Parser            docker
        Tag               kube.*
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10

    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc:443
        Kube_CA_File        /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Kube_Token_File     /var/run/secrets/kubernetes.io/serviceaccount/token
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        Merge_Log_Key       log_processed
        K8S-Logging.Parser  On
        K8S-Logging.Exclude Off

    [OUTPUT]
        Name        loki
        Match       *
        Host        loki.logging.svc.cluster.local
        Port        3100
        Labels      job=fluent-bit, namespace=k8s
        Auto_Kubernetes_Labels on

  parsers.conf: |
    [PARSER]
        Name   docker
        Format json
        Time_Key time
        Time_Format %Y-%m-%dT%H:%M:%S.%L

# ✅ 验证通过
kubectl apply -f fluent-bit-daemonset.yaml
kubectl get pods -n logging -l app=fluent-bit
# NAME                READY   STATUS    RESTARTS   AGE
# fluent-bit-xxxxx    1/1     Running   0          30s

三、Loki日志存储

# 安装Loki Stack（轻量级，不依赖Elasticsearch）
helm repo add grafana https://grafana.github.io/helm-charts
helm install loki grafana/loki-stack \
  --namespace logging \
  --create-namespace \
  --set loki.persistence.enabled=true \
  --set loki.persistence.size=10Gi \
  --set promtail.enabled=true

# ✅ 验证通过
kubectl get pods -n logging
# NAME                           READY   STATUS    RESTARTS   AGE
# loki-0                         1/1     Running   0          2m
# loki-promtail-xxxxx            1/1     Running   0          2m

# LogQL查询示例
# 查看某namespace所有日志
{namespace="default"}

# 过滤特定应用
{namespace="default", app="my-app"} |= "error"

# 统计错误日志数
sum(count_over_time({namespace="default"} |= "error" [5m]))

# 提取JSON字段
{app="my-app"} | logfmt | status >= 500

四、EFK经典方案

# EFK = Elasticsearch + Fluentd + Kibana
# 适合大规模日志场景

# 安装ECK（Elastic Cloud on Kubernetes）
kubectl apply -f https://download.elastic.co/downloads/eck/2.9.0/crds.yaml
kubectl apply -f https://download.elastic.co/downloads/eck/2.9.0/operator.yaml

# 创建Elasticsearch集群
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.12.0
  nodeSets:
  - name: default
    count: 3
    config:
      node.store.allow_mmap: false
    resources:
      requests:
        memory: 4Gi
      limits:
        memory: 4Gi

---
# 创建Kibana
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 8.12.0
  count: 1
  elasticsearchRef:
    name: quickstart

# ✅ 验证通过
kubectl get elasticsearch
# NAME         HEALTH   NODES   VERSION   PHASE   AGE
# quickstart   green    3       8.12.0    Ready   5m

五、结构化日志最佳实践

# ❌ 非结构化日志——难以搜索和分析
print(f"User {user_id} logged in from {ip}")

# ✅ 结构化日志——机器可读
import json
import logging

class JSONFormatter(logging.Formatter):
    def format(self, record):
        log_data = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "message": record.getMessage(),
            "service": "my-app",
            "version": "1.0.0",
        }
        if hasattr(record, 'custom_fields'):
            log_data.update(record.custom_fields)
        return json.dumps(log_data)

# 使用
logger = logging.getLogger(__name__)
logger.info("User login", extra={
    "custom_fields": {
        "user_id": user_id,
        "ip": ip,
        "trace_id": trace_id
    }
})

# 输出：
# {"timestamp":"2026-01-01T00:00:00","level":"INFO","message":"User login","service":"my-app","user_id":"123","ip":"1.2.3.4","trace_id":"abc123"}

六、日志级别与告警

# 日志级别规范
# ERROR  → 需要立即处理（触发告警）
# WARN   → 需要关注（可能发展为ERROR）
# INFO   → 关键业务事件（用户登录、订单创建等）
# DEBUG  → 调试信息（生产环境默认关闭）

# Loki告警规则
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: log-alerts
  namespace: monitoring
spec:
  groups:
  - name: log-alerts
    rules:
    - alert: HighErrorRate
      expr: |
        sum(count_over_time({namespace="default"} |= "ERROR" [5m]))
        /
        sum(count_over_time({namespace="default"} [5m]))
        > 0.05
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "High error rate detected"
        description: "Error rate is {{ $value | humanizePercentage }}"

七、故障排查实战

7.1 日志丢失

# 检查容器日志轮转配置
cat /etc/docker/daemon.json
# {
#   "log-driver": "json-file",
#   "log-opts": {
#     "max-size": "10m",     # 单文件最大
#     "max-file": "3"        # 最多保留3个
#   }
# }

# 如果应用日志量大，调整轮转参数
# 或使用Sidecar直接采集文件日志

7.2 Fluent Bit采集失败

# 检查Fluent Bit状态
kubectl logs -n logging daemonset/fluent-bit
# 查看是否有权限错误或连接失败

# 检查节点日志目录
ls /var/log/containers/ | head -5
# 应该有大量日志文件

# 检查Loki连通性
kubectl exec -n logging fluent-bit-xxx -- wget -qO- loki:3100/ready

八、练习

  1. 部署Fluent Bit + Loki日志收集体系
  2. 使用LogQL查询特定应用的错误日志
  3. 给应用添加结构化JSON日志输出
  4. 配置日志告警规则：ERROR日志超过阈值时告警
  5. 对比EFK和Loki方案的优劣与适用场景

🏆 第14课成就解锁

下一课预告：第15课深入告警与SLO——生产级告警体系设计。

← 上一课：Prometheus监控 📚 目录 下一课：告警与SLO →

📌 补充知识

14-日志收集补充要点：K8s生产实践扩展

🔹 资源配额(ResourceQuota)：限制命名空间总资源
  apiVersion: v1
  kind: ResourceQuota
  metadata:
    name: compute-quota
    namespace: production
  spec:
    hard:
      requests.cpu: "20"
      requests.memory: 40Gi
      limits.cpu: "40"
      limits.memory: 80Gi
      pods: "50"
      services: "10"

🔹 LimitRange：设置默认资源限制
  apiVersion: v1
  kind: LimitRange
  metadata:
    name: default-limits
  spec:
    limits:
    - type: Container
      default:
        cpu: "200m"
        memory: 256Mi
      defaultRequest:
        cpu: "100m"
        memory: 128Mi
      max:
        cpu: "2"
        memory: 4Gi

🔹 Pod优先级与抢占
  apiVersion: scheduling.k8s.io/v1
  kind: PriorityClass
  metadata:
    name: high-priority
  value: 1000000
  globalDefault: false
  ---
  spec:
    preemptionPolicy: PreemptLowerPriority

🔹 优雅处理Pod中断
  • PDB保证最小可用副本
  • preStop钩子处理连接排空
  • terminationGracePeriodSeconds充足
  • 应用必须处理SIGTERM信号

🔹 生产环境Checklist
  ✅ 设置resources requests/limits
  ✅ 配置liveness/readiness探针
  ✅ 使用PDB保护关键服务
  ✅ 实现优雅关闭（SIGTERM）
  ✅ 配置HPA自动伸缩
  ✅ 使用NetworkPolicy隔离
  ✅ 开启RBAC最小权限
  ✅ 日志结构化输出
  ✅ 指标暴露/metrics端点
  ✅ 配置PVC数据备份