SLO(目标)+SLI(指标)+SLA(协议)是SRE核心框架，用数据定义什么是可靠，用Error Budget决定何时可以冒险。
SLA(服务等级协议) — 对客户的承诺+赔偿
└── SLO(服务等级目标) — 内部可靠性目标(比SLA严格)
└── SLI(服务等级指标) — 衡量目标的具体指标
SLO 99.95% → 月度Error Budget = 0.05% × 30天 × 86400秒 = 1296秒 ≈ 21.6分钟cat > slo-definitions.yaml << 'EOF'
slos:
- name: api-availability
description: "API请求成功率"
sli: {type: ratio, numerator: "sum(rate(http_requests_total{status!~'5..'}[5m]))", denominator: "sum(rate(http_requests_total[5m]))"}
target: 0.9995
window: 30d
error_budget: {monthly_budget_seconds: 1296}
- name: api-latency
description: "API P99延迟 < 500ms"
sli: {type: ratio, numerator: "sum(rate(http_request_duration_seconds_bucket{le='0.5'}[5m]))", denominator: "sum(rate(http_request_duration_seconds_count[5m]))"}
target: 0.99
window: 30d
EOFcat > slo-rules.yml << 'EOF'
groups:
- name: slo-burn-rate
rules:
- alert: SLOErrorBudgetBurnRateFast
expr: |
(sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h]))) > (1 - 0.9995) * 14.4
for: 5m
labels: {severity: critical, slo: api-availability}
annotations:
summary: "Error budget burning too fast (14.4x rate)"
- alert: SLOErrorBudgetBurnRateSlow
expr: |
(sum(rate(http_requests_total{status=~"5.."}[6h])) / sum(rate(http_requests_total[6h]))) > (1 - 0.9995) * 6
for: 30m
labels: {severity: warning, slo: api-availability}
- alert: SLOViolation
expr: |
(1 - (sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d])))) > 0.0005
for: 5m
labels: {severity: critical}
annotations:
summary: "SLO violated! Error budget exhausted"
EOF
promtool check rules slo-rules.yml# Error Budget剩余量
1 - ((1 - sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d])))) / 0.0005)
# 剩余可接受5xx时间(秒)
(0.0005 - (1 - sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d])))) * 30 * 86400
# Grafana仪表盘配置
cat > slo-dashboard.json << 'EOF'
{
"dashboard": {
"title": "SLO Dashboard",
"panels": [
{"title": "Error Budget Remaining", "type": "gauge", "gridPos": {"h": 8, "w": 8},
"targets": [{"expr": "1 - ((1 - sum(rate(http_requests_total{status!~'5..'}[30d])) / sum(rate(http_requests_total[30d])))) / 0.0005)"}],
"fieldConfig": {"defaults": {"unit": "percentunit", "min": 0, "max": 1, "thresholds": {"steps": [{"value": 0, "color": "red"}, {"value": 0.2, "color": "orange"}, {"value": 0.5, "color": "green"}]}}}}
]
}
}
EOF┌──────────────────────────────────────────────┐
│ SLO定义 │
│ 可用性 ≥ 99.95% 延迟P99 < 500ms ≥ 99% │
└──────────────────┬───────────────────────────┘
┌──────────────────▼───────────────────────────┐
│ SLI采集 (Prometheus) │
└──────────────────┬───────────────────────────┘
┌──────────────────▼───────────────────────────┐
│ Error Budget计算 │
│ budget > 0 → 可以发布新功能 │
│ budget ≤ 0 → 只做可靠性工作 │
└──────────────────┬───────────────────────────┘
┌──────────┼──────────┐
▼ ▼ ▼
Burn Rate SLO违反 Dashboard# 检查SLI数据质量(低流量导致不稳定)
sum(rate(http_requests_total[5m])) by (service)
# 解决：增加for持续时间/多窗口burn rate/设置最小请求量阈值掌握SLO定义、Burn Rate告警、Error Budget可视化——数据驱动的可靠性工程
| 错误 | 后果 | 正确做法 |
|---|---|---|
| SLO设100% | 永远在违规 | 99.9%起步，逐步收紧 |
| SLI定义模糊 | 无法衡量 | 精确PromQL表达式 |
| Error Budget不考虑 | 不敢发布 | 预算充足时大胆发布 |
| 所有服务同SLO | 核心服务保障不足 | 分级SLO(核心/重要/一般) |
| 只看月度 | 发现太晚 | Burn Rate实时监控 |
# Error Budget消耗速率PromQL
# 快速窗口(1h) vs 慢速窗口(6h)组合检测
(
sum(rate(http_requests_total{status=~"5.."}[1h]))
/ sum(rate(http_requests_total[1h]))
> (1 - 0.9995) * 14.4
)
and
(
sum(rate(http_requests_total{status=~"5.."}[6h]))
/ sum(rate(http_requests_total[6h]))
> (1 - 0.9995) * 6
)
# 30天SLO合规率
sum(rate(http_requests_total{status!~"5.."}[30d]))
/
sum(rate(http_requests_total[30d]))
# Error Budget消耗百分比
(1 - (
(1 - sum(rate(http_requests_total{status!~"5.."}[30d])) / sum(rate(http_requests_total[30d])))
/ (1 - 0.9995)
)) * 100| 服务等级 | SLO | 月度Budget | 预算耗尽时 |
|---|---|---|---|
| TI(核心) | 99.99% | 4.3分钟 | 冻结所有变更 |
| T2(重要) | 99.95% | 21.6分钟 | 仅允许可靠性修复 |
| T3(一般) | 99.9% | 43.2分钟 | 减少功能发布 |
| T4(实验) | 99% | 7.2小时 | 正常发布 |
| 知识域 | 核心技能 | 验证方式 |
|---|---|---|
| 核心概念 | 理解原理和架构 | 能用自己的话解释 |
| 命令实操 | 熟练使用相关工具 | 能独立完成操作 |
| 故障排查 | 定位和解决问题 | 能在模拟故障中恢复 |
| 最佳实践 | 遵循生产级标准 | 能设计可靠方案 |