🚀 梯度提升 — LightGBM/CatBoost对比

📖 提升方法家族

梯度提升（Gradient Boosting）是集成学习中最强大的方法族，核心思想是串行训练弱学习器，每个新学习器纠正前一个的残差。从AdaBoost到XGBoost、LightGBM、CatBoost，每一代都在速度和精度上取得突破。

🔢 梯度提升数学原理

GBDT的核心是函数空间上的梯度下降：每棵新树拟合的是当前模型在损失函数上的负梯度。

🔬 手写梯度提升分类器

import numpy as np
from sklearn.tree import DecisionTreeRegressor
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

class SimpleGBM:
    """手写梯度提升二分类器"""
    def __init__(self, n_estimators=50, max_depth=3, lr=0.1):
        self.n_estimators = n_estimators
        self.max_depth = max_depth
        self.lr = lr
        self.trees = []
        
    def fit(self, X, y):
        # 初始预测：对数几率
        raw = np.log(y.mean() / (1 - y.mean()))
        self.init_pred = raw
        F = np.full(len(y), raw)
        
        for i in range(self.n_estimators):
            # 计算负梯度（对数损失梯度 = y - sigmoid(F)）
            p = 1 / (1 + np.exp(-F))
            residual = y - p  # 伪残差
            
            # 用回归树拟合残差
            tree = DecisionTreeRegressor(max_depth=self.max_depth, random_state=42)
            tree.fit(X, residual)
            
            # 更新预测值
            update = tree.predict(X)
            F += self.lr * update
            self.trees.append(tree)
    
    def predict_proba(self, X):
        F = np.full(len(X), self.init_pred)
        for tree in self.trees:
            F += self.lr * tree.predict(X)
        p = 1 / (1 + np.exp(-F))
        return np.column_stack([1-p, p])
    
    def predict(self, X):
        return (self.predict_proba(X)[:, 1] >= 0.5).astype(int)

# 测试手写GBM
np.random.seed(42)
X, y = make_classification(n_samples=2000, n_features=20, 
                           n_informative=12, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

sgbm = SimpleGBM(n_estimators=50, max_depth=3, lr=0.1)
sgbm.fit(X_train, y_train)
sgbm_acc = accuracy_score(y_test, sgbm.predict(X_test))
print(f"手写GBM准确率: {sgbm_acc:.4f}")

⚡ LightGBM：微软的速度之王

LightGBM是2017年微软开源的梯度提升框架，在Kaggle竞赛中几乎成为标配，核心创新是GOSS和EFB两大技术。

import lightgbm as lgb
import time

# LightGBM训练
t0 = time.time()
lgbm = lgb.LGBMClassifier(
    n_estimators=100, max_depth=4, learning_rate=0.1,
    num_leaves=15,            # 叶子节点数（leaf-wise关键参数）
    min_child_samples=20,     # 叶子最小样本数（防过拟合）
    subsample=0.8,            # 行采样
    colsample_bytree=0.8,     # 列采样
    reg_alpha=0.1,            # L1正则化
    reg_lambda=0.1,           # L2正则化
    random_state=42, verbose=-1
)
lgbm.fit(X_train, y_train)
lgb_time = time.time() - t0
lgb_acc = accuracy_score(y_test, lgbm.predict(X_test))
print(f"LightGBM: acc={lgb_acc:.4f}, time={lgb_time:.3f}s")

🐱 CatBoost：Yandex的类别特征大师

CatBoost的核心创新是有序提升（Ordered Boosting）和类别特征原生处理，无需手动编码即可处理类别变量。

from catboost import CatBoostClassifier

# CatBoost训练
t0 = time.time()
cb = CatBoostClassifier(
    iterations=100, depth=4, learning_rate=0.1,
    l2_leaf_reg=3,              # L2正则化
    random_seed=42, verbose=0,
    auto_class_weights='Balanced'  # 自动处理类别不平衡
)
cb.fit(X_train, y_train)
cb_time = time.time() - t0
cb_acc = accuracy_score(y_test, cb.predict(X_test))
print(f"CatBoost: acc={cb_acc:.4f}, time={cb_time:.3f}s")

📊 三大框架对比实战

from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import cross_val_score

# sklearn GBDT (基准)
t0 = time.time()
gb = GradientBoostingClassifier(n_estimators=100, max_depth=4, 
                                learning_rate=0.1, random_state=42)
gb.fit(X_train, y_train)
gb_time = time.time() - t0
gb_acc = accuracy_score(y_test, gb.predict(X_test))

print("=" * 55)
print(f"{'模型':<20} {'准确率':>8} {'训练时间':>10} {'速度比':>8}")
print("=" * 55)
print(f"{'sklearn GBDT':<20} {gb_acc:>8.4f} {gb_time:>8.3f}s {'1.0x':>8}")
print(f"{'LightGBM':<20} {lgb_acc:>8.4f} {lgb_time:>8.3f}s {gb_time/lgb_time:>7.1f}x")
print(f"{'CatBoost':<20} {cb_acc:>8.4f} {cb_time:>8.3f}s {gb_time/cb_time:>7.1f}x")
print("=" * 55)

# 5折交叉验证
cv_lgb = cross_val_score(lgbm, X, y, cv=5, scoring='accuracy').mean()
cv_cb = cross_val_score(cb, X, y, cv=5, scoring='accuracy').mean()
print(f"\n5折CV准确率: LightGBM={cv_lgb:.4f}, CatBoost={cv_cb:.4f}")

🔧 超参数调优指南

# 早停法调优示例
lgbm_early = lgb.LGBMClassifier(
    n_estimators=1000, learning_rate=0.05,
    max_depth=6, num_leaves=31,
    random_state=42, verbose=-1
)
lgbm_early.fit(
    X_train, y_train,
    eval_set=[(X_test, y_test)],
    callbacks=[lgb.early_stopping(stopping_rounds=50)]
)
print(f"最佳迭代轮数: {lgbm_early.best_iteration_}")
print(f"早停后准确率: {accuracy_score(y_test, lgbm_early.predict(X_test)):.4f}")

📐 2024-2025 梯度提升前沿

• NGBoost：自然梯度提升，输出预测分布而非点估计
• LightGBM 4.x：更快的CUDA加速，多目标优化支持
• CatBoost 1.2+：改进的文本特征处理，原生BERT嵌入
• Gradient Boosting + TabNet：GBDT与深度学习的混合架构
• FLAML (AutoML)：微软自动化超参数搜索，集成LightGBM
• GBDT特征+DNN残差：Kaggle 2024获奖方案，GBDT提取特征喂入DNN

🧮 框架选择决策树

📝 课后练习

1. 在Kaggle房价数据上，对比XGBoost/LightGBM/CatBoost三框架
2. 实现GOSS采样策略，验证数据量减少后精度变化
3. 用CatBoost处理含类别特征的数据集，对比手动编码效果
4. 实现早停法+贝叶斯调参，找到最优超参数组合
5. 用SHAP分析LightGBM模型，解释单样本预测