🧠 LLM 集成

§1 为什么 LLM 集成是基础设施层

在 AI 产品中，LLM 调用不像"发一个 HTTP 请求"那么简单。它是产品的核心引擎——调用质量决定了产品质量，调用成本决定了商业模型，调用稳定性决定了用户体验。

当你只有 1 个用户、1 个模型时，直接 openai.chat.completions.create() 就够了。但当你有 1000 个用户、多个模型、需要 99.9% 可用性时，你需要一套LLM 基础设施层。

§2 Provider 全景：2026 主流选项

2.1 OpenAI

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [
    { role: 'system', content: '你是一个专业的代码审查助手。' },
    { role: 'user', content: '审查这段代码：...' },
  ],
  temperature: 0.3,
  // 结构化输出 — 确保返回 JSON
  response_format: { type: 'json_object' },
});

console.log(response.choices[0].message.content);

const tools = [{
  type: 'function',
  function: {
    name: 'search_docs',
    description: '搜索内部文档库',
    parameters: {
      type: 'object',
      properties: {
        query: { type: 'string', description: '搜索关键词' },
        limit: { type: 'number', description: '返回结果数量' },
      },
      required: ['query'],
    },
  },
}];

const response = await client.chat.completions.create({
  model: 'gpt-4o',
  messages: [{ role: 'user', content: '查找关于缓存的文档' }],
  tools,
  tool_choice: 'auto', // auto | required | none | { type: 'function', function: { name: '...' } }
});

// 处理工具调用
if (response.choices[0].message.tool_calls) {
  for (const call of response.choices[0].message.tool_calls) {
    const args = JSON.parse(call.function.arguments);
    const result = await searchDocs(args.query, args.limit);
    // 将结果返回给模型继续对话
  }
}

2.2 Anthropic (Claude)

import Anthropic from '@anthropic-ai/sdk';

const client = new Anthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

const response = await client.messages.create({
  model: 'claude-sonnet-4-20250514',
  max_tokens: 16000,
  thinking: {
    type: 'enabled',
    budget_tokens: 10000, // 分配给思考的 token 预算
  },
  messages: [{
    role: 'user',
    content: '设计一个分布式任务调度系统的架构',
  }],
});

// thinking 内容在 content 数组的第一个块
const thinkingBlock = response.content.find(b => b.type === 'thinking');
const textBlock = response.content.find(b => b.type === 'text');

console.log('思考过程:', thinkingBlock?.thinking);
console.log('最终回答:', textBlock?.text);

const response = await client.messages.create({
  model: 'claude-sonnet-4-20250514',
  max_tokens: 4096,
  tools: [{
    name: 'get_weather',
    description: '获取指定城市的天气',
    input_schema: {
      type: 'object',
      properties: {
        city: { type: 'string', description: '城市名' },
      },
      required: ['city'],
    },
  }],
  messages: [{ role: 'user', content: '北京今天天气怎么样？' }],
});

// 检查是否有工具调用
const toolUse = response.content.find(b => b.type === 'tool_use');
if (toolUse) {
  console.log(`调用工具: ${toolUse.name}`);
  console.log(`参数: ${JSON.stringify(toolUse.input)}`);
}

2.3 Google Gemini

import { GoogleGenerativeAI } from '@google/generative-ai';

const genAI = new GoogleGenerativeAI(process.env.GOOGLE_API_KEY);

const model = genAI.getGenerativeModel({
  model: 'gemini-2.5-flash',
  generationConfig: {
    temperature: 0.7,
    maxOutputTokens: 4096,
  },
});

// 支持多模态输入
const result = await model.generateContent([
  { text: '描述这张图片的内容' },
  { inlineData: { mimeType: 'image/png', data: imageBase64 } },
]);

console.log(result.response.text());

// Function Calling
const modelWithTools = genAI.getGenerativeModel({
  model: 'gemini-2.5-flash',
  tools: [{
    functionDeclarations: [{
      name: 'query_database',
      description: '查询数据库',
      parameters: {
        type: 'object',
        properties: {
          sql: { type: 'string', description: 'SQL 查询语句' },
        },
        required: ['sql'],
      },
    }],
  }],
});

2.4 DeepSeek

// DeepSeek API 兼容 OpenAI 格式！
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.DEEPSEEK_API_KEY,
  baseURL: 'https://api.deepseek.com', // 只改这一个！
});

// 代码完全一样
const response = await client.chat.completions.create({
  model: 'deepseek-chat', // V3
  messages: [{ role: 'user', content: '解释量子计算' }],
  temperature: 0.3,
});

// R1 推理模型
const r1Response = await client.chat.completions.create({
  model: 'deepseek-reasoner', // R1
  messages: [{ role: 'user', content: '证明 √2 是无理数' }],
});

2.5 通义千问 (Qwen)

// 通义千问同样兼容 OpenAI 格式
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.DASHSCOPE_API_KEY,
  baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
});

const response = await client.chat.completions.create({
  model: 'qwen-max',
  messages: [{ role: 'user', content: '帮我写一个 React Hook' }],
  // 启用思考模式
  extra_body: { enable_thinking: true },
});

2.6 智谱 GLM

import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: process.env.ZHIPU_API_KEY,
  baseURL: 'https://open.bigmodel.cn/api/paas/v4',
});

const response = await client.chat.completions.create({
  model: 'glm-4-plus',
  messages: [{ role: 'user', content: '分析这段代码的性能瓶颈' }],
  tools: [{
    type: 'function',
    function: {
      name: 'run_benchmark',
      description: '运行性能基准测试',
      parameters: {
        type: 'object',
        properties: { test_name: { type: 'string' } },
        required: ['test_name'],
      },
    },
  }],
});

2.7 Mistral

2.8 本地模型 (Ollama / vLLM)

// Ollama 也兼容 OpenAI 格式！
import OpenAI from 'openai';

const client = new OpenAI({
  apiKey: 'ollama', // 任意值
  baseURL: 'http://localhost:11434/v1',
});

const response = await client.chat.completions.create({
  model: 'qwen3:8b', // 或 llama3, mistral, deepseek-r1:7b
  messages: [{ role: 'user', content: '写一个快排' }],
});

# 启动 vLLM 服务 (兼容 OpenAI API)
python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen3-72B \
  --tensor-parallel-size 4 \   # 4 GPU 并行
  --max-model-len 32768 \
  --gpu-memory-utilization 0.9 \
  --port 8000

# 然后就可以用 OpenAI SDK 调用了
# baseURL: http://localhost:8000/v1

§3 Provider 横向对比

§4 抽象层：统一 API 的设计

为什么需要抽象层？直接调 openai.chat.completions.create() 有什么问题？

4.1 抽象层设计模式

4.2 TypeScript 实现

// types.ts — 统一类型定义
interface ChatMessage {
  role: 'system' | 'user' | 'assistant' | 'tool';
  content: string;
  toolCallId?: string;
  toolCalls?: ToolCall[];
}

interface ToolCall {
  id: string;
  name: string;
  arguments: Record<string, unknown>;
}

interface ChatOptions {
  model?: string;
  temperature?: number;
  maxTokens?: number;
  tools?: ToolDefinition[];
  responseFormat?: { type: 'text' | 'json' };
  stream?: boolean;
  // 高级选项
  timeout?: number;
  retries?: number;
  cacheKey?: string;       // 语义缓存键
  costLimit?: number;      // 单次调用成本上限
  metadata?: Record<string, string>; // 可观测性标签
}

interface ChatResponse {
  content: string;
  toolCalls?: ToolCall[];
  usage: {
    inputTokens: number;
    outputTokens: number;
    cachedTokens?: number;  // 缓存命中的 token 数
  };
  model: string;
  provider: string;
  latencyMs: number;
  cost: number;             // 预估成本 (USD)
}

interface LLMProvider {
  readonly name: string;
  chat(messages: ChatMessage[], options?: ChatOptions): Promise<ChatResponse>;
  chatStream(messages: ChatMessage[], options?: ChatOptions): AsyncIterable<ChatStreamChunk>;
  embed?(texts: string[]): Promise<number[][]>;
  isAvailable(): Promise<boolean>;
}

4.3 Provider 适配器示例

// providers/deepseek.ts
class DeepSeekProvider implements LLMProvider {
  readonly name = 'deepseek';
  private client: OpenAI;

  constructor(apiKey: string) {
    this.client = new OpenAI({
      apiKey,
      baseURL: 'https://api.deepseek.com',
    });
  }

  async chat(messages: ChatMessage[], options?: ChatOptions): Promise<ChatResponse> {
    const start = Date.now();
    const model = options?.model || 'deepseek-chat';

    const response = await this.client.chat.completions.create({
      model,
      messages: messages.map(m => ({
        role: m.role,
        content: m.content,
        ...(m.toolCalls ? { tool_calls: m.toolCalls } : {}),
      })),
      temperature: options?.temperature,
      max_tokens: options?.maxTokens,
      tools: options?.tools?.map(t => ({
        type: 'function' as const,
        function: { name: t.name, description: t.description, parameters: t.parameters },
      })),
      response_format: options?.responseFormat?.type === 'json'
        ? { type: 'json_object' } : undefined,
    }, { timeout: options?.timeout || 30000 });

    const choice = response.choices[0];
    return {
      content: choice.message.content || '',
      toolCalls: choice.message.tool_calls?.map(tc => ({
        id: tc.id,
        name: tc.function.name,
        arguments: JSON.parse(tc.function.arguments),
      })),
      usage: {
        inputTokens: response.usage?.prompt_tokens || 0,
        outputTokens: response.usage?.completion_tokens || 0,
      },
      model,
      provider: this.name,
      latencyMs: Date.now() - start,
      cost: this.calculateCost(model, response.usage),
    };
  }

  async isAvailable(): Promise<boolean> {
    try {
      await this.client.models.list();
      return true;
    } catch { return false; }
  }

  private calculateCost(model: string, usage: any): number {
    // DeepSeek V3: ¥1/¥2 per 1M tokens → ~$0.00014/$0.00028
    const rates: Record<string, [number, number]> = {
      'deepseek-chat': [0.14, 0.28],
      'deepseek-reasoner': [0.55, 2.2],
    };
    const [inputRate, outputRate] = rates[model] || [0.14, 0.28];
    return (usage?.prompt_tokens || 0) * inputRate / 1_000_000
         + (usage?.completion_tokens || 0) * outputRate / 1_000_000;
  }
}

4.4 OpenAI 兼容的妙用

// 用一个工厂创建所有 OpenAI 兼容的 Provider
function createOpenAICompatibleProvider(config: {
  name: string;
  baseURL: string;
  apiKey: string;
  defaultModel: string;
  pricing: Record<string, [number, number]>;
}): LLMProvider {
  const client = new OpenAI({ apiKey: config.apiKey, baseURL: config.baseURL });
  // ... 复用同一套 chat/embed 实现
}

// 配置一览
const providers = {
  deepseek: createOpenAICompatibleProvider({
    name: 'deepseek', baseURL: 'https://api.deepseek.com',
    apiKey: process.env.DEEPSEEK_API_KEY, defaultModel: 'deepseek-chat',
    pricing: { 'deepseek-chat': [0.14, 0.28], 'deepseek-reasoner': [0.55, 2.2] },
  }),
  qwen: createOpenAICompatibleProvider({
    name: 'qwen', baseURL: 'https://dashscope.aliyuncs.com/compatible-mode/v1',
    apiKey: process.env.DASHSCOPE_API_KEY, defaultModel: 'qwen-plus',
    pricing: { 'qwen-max': [0.28, 0.83], 'qwen-plus': [0.11, 0.28] },
  }),
  glm: createOpenAICompatibleProvider({
    name: 'glm', baseURL: 'https://open.bigmodel.cn/api/paas/v4',
    apiKey: process.env.ZHIPU_API_KEY, defaultModel: 'glm-4-plus',
    pricing: { 'glm-4-plus': [0.69, 0.69], 'glm-4-flash': [0, 0] },
  }),
  ollama: createOpenAICompatibleProvider({
    name: 'ollama', baseURL: 'http://localhost:11434/v1',
    apiKey: 'ollama', defaultModel: 'qwen3:8b',
    pricing: {}, // 本地免费
  }),
};

§5 Failover 与高可用

LLM API 不是传统 Web 服务——它们的可用性显著更低：

5.1 Failover 策略

// 简单顺序 Failover
const failoverChain = [
  { provider: 'openai', model: 'gpt-4o' },
  { provider: 'deepseek', model: 'deepseek-chat' },
  { provider: 'qwen', model: 'qwen-max' },
];

async function chatWithFailover(messages, options) {
  for (const { provider, model } of failoverChain) {
    try {
      return await providers[provider].chat(messages, {
        ...options, model,
        timeout: 15000, // 快速超时
      });
    } catch (err) {
      console.warn(`${provider} failed: ${err.message}`);
      continue; // 尝试下一个
    }
  }
  throw new Error('All providers failed');
}

// 竞速 Failover — 快者胜
async function chatRacing(messages, options) {
  const candidates = [
    { provider: 'openai', model: 'gpt-4o' },
    { provider: 'deepseek', model: 'deepseek-chat' },
  ];

  return Promise.any(
    candidates.map(({ provider, model }) =>
      providers[provider].chat(messages, { ...options, model })
        .then(res => { res.provider = provider; return res; })
    )
  );
}

class CircuitBreaker {
  private failures = new Map<string, {
    count: number;
    lastFail: number;
    state: 'closed' | 'open' | 'half-open';
  }>();

  async call(provider: string, fn: () => Promise<any>) {
    const cb = this.failures.get(provider);
    if (cb?.state === 'open') {
      // 断路器打开：检查是否该尝试恢复
      if (Date.now() - cb.lastFail > 60_000) {
        cb.state = 'half-open'; // 半开：允许一次尝试
      } else {
        throw new Error(`Circuit open: ${provider}`);
      }
    }
    try {
      const result = await fn();
      this.recordSuccess(provider);
      return result;
    } catch (err) {
      this.recordFailure(provider);
      throw err;
    }
  }
}

5.2 生产级 Failover 实现

class LLMFailover {
  private circuitBreaker = new CircuitBreaker();
  private healthChecker = new HealthChecker();

  constructor(
    private providers: Record<string, LLMProvider>,
    private config: FailoverConfig,
  ) {
    // 每 30 秒健康检查
    setInterval(() => this.healthChecker.checkAll(providers), 30_000);
  }

  async chat(messages: ChatMessage[], options?: ChatOptions): Promise<ChatResponse> {
    const startTime = Date.now();
    const orderedProviders = this.getOrderedProviders();

    for (const { provider, model } of orderedProviders) {
      // 跳过断路器打开的 Provider
      if (!this.circuitBreaker.canTry(provider)) continue;

      try {
        const result = await this.circuitBreaker.call(provider, () =>
          this.providers[provider].chat(messages, { ...options, model })
        );

        // 记录成功，用于后续路由决策
        this.healthChecker.recordSuccess(provider, Date.now() - startTime);
        return result;

      } catch (err) {
        this.healthChecker.recordFailure(provider, err);
        console.warn(`[Failover] ${provider} failed: ${err.message}`);

        // 如果是超时，可以竞速尝试下一个
        if (this.config.raceOnTimeout && orderedProviders.length > 1) {
          return this.raceFallback(messages, options, provider);
        }
      }
    }

    // 所有 Provider 都失败了——尝试本地模型
    if (this.providers.ollama) {
      return this.providers.ollama.chat(messages, {
        ...options,
        model: 'qwen3:8b',
      });
    }

    throw new LLMAllProvidersError('All LLM providers are unavailable');
  }

  private getOrderedProviders(): Array<{ provider: string; model: string }> {
    // 根据健康度排序，最健康的排前面
    return this.config.chain
      .filter(p => this.healthChecker.getHealth(p.provider) > 0.3)
      .sort((a, b) =>
        this.healthChecker.getHealth(b.provider) - this.healthChecker.getHealth(a.provider)
      );
  }
}

§6 智能 Routing

不是所有请求都该走最贵的模型。智能路由根据请求特征自动选择最合适的模型。

6.1 路由策略一览

// 用 Flash 判断任务复杂度
const judge = await flash.chat([{
  role: 'user',
  content: `这个任务需要深度推理吗？
任务: ${userMessage}
回答: simple 或 complex`
}]);

const model = judge.includes('complex')
  ? 'gpt-4o'
  : 'gpt-4o-mini';

6.2 实战：智能路由器实现

interface RouteRule {
  condition: (messages: ChatMessage[], options?: ChatOptions) => boolean;
  provider: string;
  model: string;
  priority: number;
}

class SmartRouter {
  private rules: RouteRule[] = [];

  addRule(rule: RouteRule): this {
    this.rules.push(rule);
    this.rules.sort((a, b) => b.priority - a.priority);
    return this;
  }

  resolve(messages: ChatMessage[], options?: ChatOptions): { provider: string; model: string } {
    for (const rule of this.rules) {
      if (rule.condition(messages, options)) {
        return { provider: rule.provider, model: rule.model };
      }
    }
    // 默认：性价比之选
    return { provider: 'deepseek', model: 'deepseek-chat' };
  }
}

// 配置路由规则
const router = new SmartRouter();

// 规则 1：编码任务 → Claude
router.addRule({
  priority: 100,
  provider: 'anthropic',
  model: 'claude-sonnet-4-20250514',
  condition: (msgs, opts) =>
    opts?.metadata?.task === 'code' ||
    msgs.some(m => m.content.includes('```')),
});

// 规则 2：长文档 → Gemini
router.addRule({
  priority: 90,
  provider: 'google',
  model: 'gemini-2.5-pro',
  condition: (msgs) =>
    msgs.reduce((sum, m) => sum + m.content.length, 0) > 100_000,
});

// 规则 3：简单分类 → 小模型
router.addRule({
  priority: 80,
  provider: 'openai',
  model: 'gpt-4o-mini',
  condition: (msgs, opts) =>
    opts?.metadata?.task === 'classify' ||
    opts?.responseFormat?.type === 'json',
});

// 规则 4：深度推理 → o3
router.addRule({
  priority: 70,
  provider: 'openai',
  model: 'o3',
  condition: (msgs, opts) =>
    opts?.metadata?.task === 'reasoning',
});

// 规则 5：付费用户 → 高级模型
router.addRule({
  priority: 50,
  provider: 'openai',
  model: 'gpt-4o',
  condition: (msgs, opts) =>
    opts?.metadata?.tier === 'pro',
});

// 使用
const { provider, model } = router.resolve(messages, options);
const result = await providers[provider].chat(messages, { ...options, model });

§7 成本控制

LLM 成本是 AI 产品最大的运营支出。一个中型 SaaS (1000 DAU)，每人每天 50 次调用，GPT-4o 单次均价 $0.01 → $500/天 = $15,000/月。不做成本控制 = 烧钱。

7.1 Token 经济学

7.2 成本优化策略

// 策略 1：滑动窗口
const recentMessages = messages.slice(-10);

// 策略 2：摘要压缩
if (messages.length > 20) {
  const summary = await cheapModel.chat([{
    role: 'user',
    content: `压缩以下对话历史为关键信息：
${messages.slice(0, -5).map(m => m.content).join('\n')}`
  }]);
  messages = [
    { role: 'system', content: `历史摘要: ${summary}` },
    ...messages.slice(-5),
  ];
}

// 策略 3：OpenClaw 的 Cache Boundary
// 在 system prompt 末尾标记缓存边界
// 新消息不会使 system prompt 缓存失效

// 典型的 3 级模型策略
const MODEL_TIERS = {
  light:  'gpt-4o-mini',    // $0.15/$0.60
  medium: 'deepseek-chat',  // ¥1/¥2
  heavy:  'gpt-4o',         // $2.50/$10
};

// 80% 请求走 light → 成本降 90%
function selectTier(complexity: number) {
  if (complexity < 0.3) return MODEL_TIERS.light;
  if (complexity < 0.7) return MODEL_TIERS.medium;
  return MODEL_TIERS.heavy;
}

// 设置 per-request 预算
const result = await provider.chat(messages, {
  maxTokens: 500,         // 输出上限
  costLimit: 0.05,        // 单次成本上限 $0.05
});

// 设置 per-user 日预算
const dailyBudget = getUserBudget(userId);
const todayCost = getTodayCost(userId);
if (todayCost + estimatedCost > dailyBudget) {
  // 降级到小模型或返回缓存结果
  return cheapModel.chat(messages);
}

7.3 成本监控仪表盘

class CostTracker {
  private costs = new Map<string, {
    total: number;
    byProvider: Record<string, number>;
    byModel: Record<string, number>;
    byUser: Record<string, number>;
    requestCount: number;
  }>();

  record(userId: string, response: ChatResponse) {
    const day = new Date().toISOString().slice(0, 10);
    const entry = this.costs.get(day) || {
      total: 0, byProvider: {}, byModel: {},
      byUser: {}, requestCount: 0,
    };

    entry.total += response.cost;
    entry.byProvider[response.provider] =
      (entry.byProvider[response.provider] || 0) + response.cost;
    entry.byModel[response.model] =
      (entry.byModel[response.model] || 0) + response.cost;
    entry.byUser[userId] =
      (entry.byUser[userId] || 0) + response.cost;
    entry.requestCount++;

    this.costs.set(day, entry);
  }

  getDailyReport(day: string) {
    const entry = this.costs.get(day);
    if (!entry) return null;
    return {
      total: `$${entry.total.toFixed(2)}`,
      topProvider: Object.entries(entry.byProvider)
        .sort(([,a],[,b]) => b - a)[0],
      topModel: Object.entries(entry.byModel)
        .sort(([,a],[,b]) => b - a)[0],
      avgCostPerRequest: `$${(entry.total / entry.requestCount).toFixed(4)}`,
      topUser: Object.entries(entry.byUser)
        .sort(([,a],[,b]) => b - a)[0],
    };
  }
}

§8 流式输出与用户体验

LLM 响应慢（2-30 秒），如果不做流式输出，用户会以为卡死了。流式不是可选的，是必须的。

8.1 流式输出实现

// 统一流式接口
interface ChatStreamChunk {
  content?: string;        // 增量文本
  toolCall?: Partial<ToolCall>;  // 增量工具调用
  usage?: {                // 最后一个 chunk 才有
    inputTokens: number;
    outputTokens: number;
  };
  done: boolean;
}

// 使用
const stream = provider.chatStream(messages, options);
for await (const chunk of stream) {
  if (chunk.content) {
    process.stdout.write(chunk.content); // 逐字输出
  }
  if (chunk.done) {
    console.log('\n[完成]', chunk.usage);
  }
}

8.2 SSE 到前端的完整链路

// 后端：SSE 端点 (FastAPI)
from fastapi import FastAPI
from fastapi.responses import StreamingResponse

@app.post("/api/chat/stream")
async def chat_stream(request: ChatRequest):
    async def generate():
        stream = await llm.chat_stream(
            messages=request.messages,
            model=request.model,
        )
        async for chunk in stream:
            if chunk.content:
                yield f"data: {json.dumps({'content': chunk.content})}\n\n"
            if chunk.done:
                yield f"data: {json.dumps({'done': True, 'usage': chunk.usage})}\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

// 前端：消费 SSE
const response = await fetch('/api/chat/stream', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ messages, model }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop() || '';

  for (const line of lines) {
    if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      if (data.content) {
        appendToChat(data.content); // 逐字追加到 UI
      }
      if (data.done) {
        updateUsageStats(data.usage);
      }
    }
  }
}

8.3 流式输出的坑

// 处理用户取消
async function chatWithAbort(messages, options) {
  const controller = new AbortController();

  // 前端取消时
  window.addEventListener('user-cancel', () => controller.abort());

  const stream = await provider.chatStream(messages, {
    ...options,
    signal: controller.signal,  // 传入 AbortSignal
  });

  try {
    for await (const chunk of stream) {
      if (controller.signal.aborted) break;
      processChunk(chunk);
    }
  } finally {
    // 确保清理
    controller.abort();
  }
}

§9 语义缓存与 Prompt 缓存

缓存是成本控制的杀手锏。两层缓存配合使用：

9.1 语义缓存

相同含义的问题应该返回缓存的答案，即使措辞不同。

import { Redis } from 'ioredis';
import { createHash } from 'crypto';

class SemanticCache {
  private redis: Redis;
  private embedder: LLMProvider; // 用便宜的 embedding 模型

  // 相似度阈值 — 越高越严格
  private similarityThreshold = 0.95;

  async get(query: string): Promise<ChatResponse | null> {
    // 1. 生成 query embedding
    const [embedding] = await this.embedder.embed([query]);

    // 2. 在向量存储中搜索相似 query
    const similar = await this.redis.send_command(
      'FT.SEARCH', 'semantic_cache',
      `(*=>[KNN 1 @embedding $vec])`,
      'PARAMS', 2, 'vec', Buffer.from(Float32Array.from(embedding).buffer),
      'DIALECT', 2,
    );

    if (similar && similar[0] > 0) {
      const score = similar[1]; // 相似度分数
      if (score >= this.similarityThreshold) {
        // 命中！返回缓存的响应
        return JSON.parse(similar[2].response);
      }
    }

    return null; // 未命中
  }

  async set(query: string, response: ChatResponse, ttl = 3600): Promise<void> {
    const [embedding] = await this.embedder.embed([query]);
    const key = `cache:${createHash('sha256').update(query).digest('hex').slice(0, 16)}`;

    await this.redis.send_command(
      'HSET', key,
      'query', query,
      'response', JSON.stringify(response),
      'embedding', Buffer.from(Float32Array.from(embedding).buffer),
      'timestamp', Date.now(),
    );
    await this.redis.expire(key, ttl);
  }
}

// 使用
const cache = new SemanticCache(redis, embedder);

async function chatWithCache(messages, options) {
  const query = messages[messages.length - 1].content;

  // 先查缓存
  const cached = await cache.get(query);
  if (cached) {
    console.log('[Cache Hit] Saved $', cached.cost.toFixed(4));
    return { ...cached, fromCache: true };
  }

  // 缓存未命中，调 LLM
  const result = await provider.chat(messages, options);

  // 缓存结果（只缓存成功的非工具调用响应）
  if (!result.toolCalls && result.content) {
    await cache.set(query, result);
  }

  return result;
}

9.2 Prompt 缓存最佳实践

// Anthropic 显式缓存控制
const response = await anthropic.messages.create({
  model: 'claude-sonnet-4-20250514',
  max_tokens: 4096,
  system: [{
    type: 'text',
    text: systemPrompt,       // 这部分会被缓存
    cache_control: { type: 'ephemeral' },  // 标记缓存点
  }],
  tools: toolDefinitions.map(t => ({
    ...t,
    cache_control: { type: 'ephemeral' },  // 工具定义也缓存
  })),
  messages,
});

// response.usage 中会显示缓存命中的 token 数
console.log(`Cache hit: ${response.usage.cache_read_input_tokens} tokens`);

§10 从 Agent 研究学到的

在 Agent 架构研究中，20 个开源 Agent 项目展示了丰富的 LLM 集成实践。以下是关键发现：

§11 实施清单

替代方案与相关工具

pip install litellm

# 用法 — 自动路由到正确 Provider
from litellm import completion
response = completion(
  model="deepseek/deepseek-chat",
  messages=[{"role": "user", "content": "Hello"}],
)

// 兼容 OpenAI SDK
const client = new OpenAI({
  baseURL: 'https://openrouter.ai/api/v1',
  apiKey: process.env.OPENROUTER_KEY,
});

// 任意模型
const res = await client.chat.completions.create({
  model: 'anthropic/claude-sonnet-4',
  messages: [...],
});

import { generateText } from 'ai';
import { openai } from '@ai-sdk/openai';

const { text } = await generateText({
  model: openai('gpt-4o'),
  prompt: 'Explain quantum computing',
});

延伸阅读