🎯 逻辑回归 — 分类准确率>80%

📖 从线性回归到逻辑回归

线性回归输出连续值，而分类任务需要输出类别概率。逻辑回归通过Sigmoid函数将线性输出映射到[0,1]区间，实现概率预测与分类。

Sigmoid函数的特性

🔢 手写逻辑回归

import numpy as np

np.random.seed(42)

# 生成二分类数据
n = 200
X_class0 = np.random.multivariate_normal([2, 2], [[1, 0.3], [0.3, 1]], n//2)
X_class1 = np.random.multivariate_normal([5, 5], [[1, 0.3], [0.3, 1]], n//2)
X = np.vstack([X_class0, X_class1])
y = np.array([0]*(n//2) + [1]*(n//2))

# 添加偏置项
X_with_bias = np.c_[np.ones(n), X]

def sigmoid(z):
    """Sigmoid激活函数"""
    return 1 / (1 + np.exp(-np.clip(z, -500, 500)))

# 梯度下降训练
weights = np.zeros(3)
lr = 0.1

for epoch in range(1000):
    z = X_with_bias @ weights
    predictions = sigmoid(z)
    error = predictions - y
    gradient = (1/n) * X_with_bias.T @ error
    weights -= lr * gradient
    
    if (epoch+1) % 200 == 0:
        loss = -np.mean(y * np.log(predictions+1e-15) + 
                       (1-y) * np.log(1-predictions+1e-15))
        print(f"Epoch {epoch+1}: loss={loss:.4f}")

# 预测与评估
z_pred = X_with_bias @ weights
y_pred = (sigmoid(z_pred) >= 0.5).astype(int)
accuracy = np.mean(y_pred == y)
print(f"\n准确率: {accuracy:.4f}")  # 0.965

📊 混淆矩阵与分类指标

# 混淆矩阵计算
tp = np.sum((y_pred == 1) & (y == 1))  # 真阳性
tn = np.sum((y_pred == 0) & (y == 0))  # 真阴性
fp = np.sum((y_pred == 1) & (y == 0))  # 假阳性
fn = np.sum((y_pred == 0) & (y == 1))  # 假阴性

precision = tp / (tp + fp)     # 精确率
recall = tp / (tp + fn)        # 召回率
f1 = 2*precision*recall / (precision+recall)  # F1

print(f"TP={tp}, FP={fp}, FN={fn}, TN={tn}")
print(f"精确率: {precision:.4f}")  # 0.9697
print(f"召回率: {recall:.4f}")     # 0.9600
print(f"F1分数: {f1:.4f}")        # 0.9648

多分类指标对比

📈 ROC曲线与AUC

ROC曲线绘制不同阈值下的TPR(召回率) vs FPR，AUC衡量分类器的整体排序能力。

# 手动绘制ROC曲线数据
thresholds = np.linspace(0, 1, 100)
tpr_list, fpr_list = [], []

prob_pos = sigmoid(X_with_bias @ weights)

for thresh in thresholds:
    y_t = (prob_pos >= thresh).astype(int)
    tp_t = np.sum((y_t == 1) & (y == 1))
    fp_t = np.sum((y_t == 1) & (y == 0))
    fn_t = np.sum((y_t == 0) & (y == 1))
    tn_t = np.sum((y_t == 0) & (y == 0))
    tpr_list.append(tp_t / (tp_t + fn_t) if (tp_t + fn_t) > 0 else 0)
    fpr_list.append(fp_t / (fp_t + tn_t) if (fp_t + tn_t) > 0 else 0)

# AUC计算（梯形法）
auc = np.trapz(tpr_list, fpr_list)
print(f"AUC: {auc:.4f}")  # ≈0.99

🔬 多分类逻辑回归

# Softmax多分类（One-vs-Rest策略）
def softmax(z):
    """Softmax: 将多个输出转化为概率分布"""
    e_z = np.exp(z - np.max(z, axis=1, keepdims=True))  # 数值稳定
    return e_z / e_z.sum(axis=1, keepdims=True)

# 交叉熵损失
def cross_entropy_loss(probs, y_onehot):
    return -np.mean(np.sum(y_onehot * np.log(probs + 1e-15), axis=1))

# 3类分类
np.random.seed(42)
n3 = 300
X3 = np.vstack([
    np.random.normal([0, 0], 1, (100, 2)),
    np.random.normal([3, 3], 1, (100, 2)),
    np.random.normal([6, 0], 1, (100, 2))
])
y3 = np.array([0]*100 + [1]*100 + [2]*100)

# One-hot编码
n_classes = 3
y_onehot = np.eye(n_classes)[y3]

# 训练
X3_bias = np.c_[np.ones(n3), X3]
W = np.zeros((3, n3))
W = np.zeros((3, X3_bias.shape[1]))

for epoch in range(500):
    logits = X3_bias @ W.T
    probs = softmax(logits)
    gradient = (probs - y_onehot).T @ X3_bias / n3
    W -= 0.1 * gradient

y3_pred = softmax(X3_bias @ W.T).argmax(axis=1)
print(f"多分类准确率: {np.mean(y3_pred == y3):.4f}")

📐 2024-2025 逻辑回归前沿

• Focal Loss：解决类别不均衡，降低易分类样本的权重（原为目标检测设计，2024广泛用于分类）
• 校准概率：Platt Scaling / Isotonic Regression让逻辑回归输出"真实"概率
• 联邦逻辑回归：隐私保护下多方联合训练（2024 Apple/Google方案）
• 逻辑回归+LLM：用LLM特征做最后一层逻辑回归，兼顾性能与可解释性
• 因果逻辑回归：倾向得分匹配(PSM)中的倾向得分模型

⚖️ 类别不均衡处理

# 类别权重调整
def weighted_logistic_regression(X, y, class_weight={0: 1, 1: 5}, lr=0.1, epochs=1000):
    n = len(y)
    weights = np.zeros(X.shape[1])
    sample_weights = np.array([class_weight[yi] for yi in y])
    
    for epoch in range(epochs):
        z = X @ weights
        pred = sigmoid(z)
        error = pred - y
        gradient = (X.T @ (error * sample_weights)) / n
        weights -= lr * gradient
    return weights

📝 课后练习

1. 实现L2正则化的逻辑回归，对比正则化效果
2. 用逻辑回归实现多分类（鸢尾花数据集），One-vs-Rest策略
3. 实现SMOTE过采样算法，在信用卡欺诈数据上对比效果
4. 手绘ROC曲线，计算AUC，与sklearn对比
5. 实现逻辑回归的随机梯度下降版本