GPT-Researcher Deep Dive

Executive Summary

GPT-Researcher is a production-grade autonomous research agent that takes a query, plans sub-queries, searches the web via multiple retrievers, scrapes and compresses content, and generates structured reports. It supports both a single-agent mode (the GPTResearcher class) and a multi-agent mode (LangGraph-based ChiefEditorAgent orchestration). The framework is designed for extensibility: pluggable LLM providers, retrievers, scrapers, and prompt families.

Architecture Overview

Core Research Workflow

GPTResearcher Class

The GPTResearcher class in gpt_researcher/agent.py is the main entry point. It's a God Object that owns all subsystems:

class GPTResearcher:
    def __init__(self, query, report_type, report_source, tone,
                 source_urls, document_urls, vector_store, ...):
        self.cfg = Config(config_path)
        self.memory = Memory(cfg.embedding_provider, cfg.embedding_model)
        
        # Composed skills
        self.research_conductor = ResearchConductor(self)
        self.report_generator = ReportGenerator(self)
        self.context_manager = ContextManager(self)
        self.scraper_manager = BrowserManager(self)
        self.source_curator = SourceCurator(self)
        self.deep_researcher = DeepResearchSkill(self)  # if DeepResearch
        self.image_generator = ImageGenerator(self)
        
        # MCP support
        self.mcp_configs = mcp_configs
        self.mcp_strategy = self._resolve_mcp_strategy(...)

Key Constructor Parameters

Skills System

Skills are the decomposed capabilities of the researcher, each a class that takes the parent GPTResearcher as its constructor argument:

Memory & Embeddings

The Memory class (gpt_researcher/memory/embeddings.py) is a factory for LangChain embedding providers, not a conversational memory store. It creates the embedding instance used by ContextCompressor for similarity filtering.

Supported Embedding Providers (21)

class Memory:
    def __init__(self, embedding_provider: str, model: str, **kwargs):
        match embedding_provider:
            case "openai":
                from langchain_openai import OpenAIEmbeddings
                _embeddings = OpenAIEmbeddings(model=model, **kwargs)
            case "ollama":
                from langchain_ollama import OllamaEmbeddings
                _embeddings = OllamaEmbeddings(model=model, ...)
            # ... 19 more providers
    
    def get_embeddings(self):
        return self._embeddings

Context Management

The ContextManager skill delegates to three compressor classes in gpt_researcher/context/compression.py:

1. ContextCompressor (Web Search Results)

class ContextCompressor:
    def async_get_context(self, query, max_results=5, cost_callback=None):
        # Optimization: skip compression for small content
        if total_chars < COMPRESSION_THRESHOLD and len(docs) <= max_results:
            return prompt_family.pretty_print_docs(direct_docs)
        
        # Standard: split → embed → filter by similarity
        splitter = RecursiveCharacterTextSplitter(chunk_size=1000, overlap=100)
        relevance_filter = EmbeddingsFilter(embeddings, threshold=0.35)
        pipeline = DocumentCompressorPipeline([splitter, relevance_filter])
        retriever = ContextualCompressionRetriever(pipeline, SearchAPIRetriever(pages))

2. VectorstoreCompressor (Pre-indexed Documents)

Simple similarity search on an existing vector store — no compression pipeline needed.

3. WrittenContentCompressor (Dedup for Detailed Reports)

Finds similar previously-written sections to avoid repetition across subtopic reports. Uses the same split→filter approach but with SectionRetriever instead.

Context Compression Pipeline

LLM Provider Layer

The LLM integration follows a three-tier model strategy:

GenericLLMProvider (llm_provider/generic/base.py)

class GenericLLMProvider:
    @classmethod
    def from_provider(cls, provider, chat_log=None, verbose=True, **kwargs):
        if provider == "openai":
            from langchain_openai import ChatOpenAI
            llm = ChatOpenAI(**kwargs)
        elif provider == "anthropic":
            from langchain_anthropic import ChatAnthropic
            llm = ChatAnthropic(**kwargs)
        # ... 25 more providers
        return cls(llm, chat_log, verbose)
    
    async def get_chat_response(self, messages, stream, websocket=None):
        if not stream:
            output = await self.llm.ainvoke(messages)
            return output.content
        else:
            # Stream chunks via websocket
            async for chunk in self.llm.astream(messages):
                ...

27 Supported LLM Providers

create_chat_completion Utility

The create_chat_completion() function in utils/llm.py is the central LLM call wrapper with:

• Retry logic: up to 10 attempts with exponential backoff (max 8s)
• Reasoning model support: auto-detects o1/o3/o4-mini models and disables temperature/max_tokens
• Reasoning effort: configurable low/medium/high for supported models
• Cost tracking: estimates cost via estimate_llm_cost() and calls cost_callback
• Streaming: websocket-based streaming for real-time report display

Retriever Architecture

GPT-Researcher supports 15 retriever backends, each implementing a .search() method:

Multi-Retriever Support

Multiple retrievers can be configured simultaneously (comma-separated). The system iterates through all retrievers for each sub-query and deduplicates URLs via the visited_urls set. Some retrievers (like PubMed Central) return full content directly, bypassing the scraping step.

# In _search_relevant_source_urls:
for retriever_class in self.researcher.retrievers:
    if "mcpretriever" in retriever_class.__name__.lower():
        continue  # MCP handled separately
    
    results = retriever.search(max_results=cfg.max_search_results_per_query)
    for result in results:
        if raw_content and len(raw_content) > 100:
            prefetched_content.append(result)  # Skip scraping
        else:
            new_search_urls.append(url)  # Need scraping

MCP Integration

GPT-Researcher has first-class MCP (Model Context Protocol) support with a two-stage architecture:

MCP Client Manager

The MCPClientManager converts GPT-Researcher configs to langchain-mcp-adapters format, supporting stdio, websocket, and HTTP transports with automatic URL-based transport detection.

MCP Tool Selector

Uses the Strategic LLM to analyze available tools and pick the most relevant ones (default max 3). Falls back to pattern-matching on tool names if LLM selection fails.

MCP Research Skill

Binds selected tools to the LLM via llm.bind_tools(selected_tools), then invokes the LLM which autonomously decides which tools to call. Results are normalized into the standard {title, href, body} format.

Scraper System

Multiple scraper backends for content extraction:

Scraping is parallelized via WorkerPool with configurable max_scraper_workers and scraper_rate_limit_delay.

Multi-Agent Architecture

The multi-agent system is a LangGraph state machine with specialized agent roles. It's in multi_agents/ (separate from the single-agent gpt_researcher/).

LangGraph Integration

The langgraph.json defines the graph entry point:

{
  "python_version": "3.11",
  "dependencies": ["./multi_agents"],
  "graphs": {
    "agent": "./multi_agents/agent.py:graph"
  },
  "env": ".env"
}

State Definitions

ResearchState (outer workflow):

class ResearchState(TypedDict):
    task: dict
    initial_research: str
    sections: List[str]
    research_data: List[dict]
    human_feedback: str
    title: str
    headers: dict
    date: str
    table_of_contents: str
    introduction: str
    conclusion: str
    sources: List[str]
    report: str

DraftState (per-section sub-workflow):

class DraftState(TypedDict):
    task: dict
    topic: str
    draft: dict
    review: str
    revision_notes: str

Workflow Construction

# ChiefEditorAgent._create_workflow:
workflow = StateGraph(ResearchState)
workflow.add_node("browser", agents["research"].run_initial_research)
workflow.add_node("planner", agents["editor"].plan_research)
workflow.add_node("researcher", agents["editor"].run_parallel_research)
workflow.add_node("writer", agents["writer"].run)
workflow.add_node("publisher", agents["publisher"].run)
workflow.add_node("human", agents["human"].review_plan)

workflow.add_edge('browser', 'planner')
workflow.add_edge('planner', 'human')
workflow.add_edge('researcher', 'writer')
workflow.add_edge('writer', 'publisher')
workflow.set_entry_point("browser")
workflow.add_edge('publisher', END)

# Human-in-the-loop conditional
workflow.add_conditional_edges('human',
    lambda review: "accept" if review['human_feedback'] is None else "revise",
    {"accept": "researcher", "revise": "planner"}
)

Agent Roles in Multi-Agent Mode

Deep Research Mode

The DeepResearchSkill implements a recursive breadth-first search over the research space:

Key Implementation Details

• Each depth level creates new GPTResearcher instances — fully independent research sessions
• Breadth halves at each depth level (4 → 2)
• Concurrency limited by deep_research_concurrency (default 2) via asyncio.Semaphore
• Context is trimmed to 25,000 words to stay within LLM context limits
• MCP configuration is propagated to nested researchers
• Learnings and citations are accumulated across all depth levels

Detailed Report Mode

The detailed report mode (ReportType.DetailedReport) generates a multi-section report:

1. conduct_research() — initial broad research
2. get_subtopics() — LLM generates subtopic headers using construct_subtopics()
3. For each subtopic:
   • get_draft_section_titles() — LLM generates H3 headers
   • get_similar_written_contents() — find related previous sections
   • write_report() — generate subtopic report with dedup awareness
4. write_introduction() — H1 + intro paragraph
5. write_report_conclusion() — summary + implications

Report Types

Report Source Options

Configuration System

The Config class in gpt_researcher/config/config.py manages all settings with a layered override system:

Key Configuration Patterns

# LLM provider format: "provider:model"
FAST_LLM = "openai:gpt-4o-mini"
SMART_LLM = "openai:gpt-4o"
STRATEGIC_LLM = "openai:gpt-4o"

# Embedding format: "provider:model"
EMBEDDING = "openai:text-embedding-3-small"

# Retriever (comma-separated for multiple)
RETRIEVER = "tavily,google,arxiv"

# Deep research settings
DEEP_RESEARCH_BREADTH = 4
DEEP_RESEARCH_DEPTH = 2
DEEP_RESEARCH_CONCURRENCY = 2

# MCP strategy
MCP_STRATEGY = "fast"  # fast | deep | disabled

Smart Type Conversion

The convert_env_value() method uses Python's typing hints to auto-convert environment variables: str → string, int → int, bool → boolean ("true"/"1"/"yes"/"on"), List → JSON parse.

Prompt Engineering

All prompts are defined as static methods on the PromptFamily class in gpt_researcher/prompts.py. This is a centralized prompt registry pattern.

Key Prompt Methods

Prompt Families

GPT-Researcher supports model-specific prompt families that override formatting for particular LLMs:

Strengths

Weaknesses

Design Patterns Identified

Final Verdict

Generated by Agent Deep Dive Analysis • Source: gpt-researcher v0.14.7 • Date: 2026-05-17