{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Deep Researcher Agent – Customization\n", "\n", "This notebook walks through **customizing** the NVIDIA Deep Researcher Agent Blueprint deep researcher: adding tools (paper search), assigning different models to the orchestrator vs planner/researcher, locating and editing prompts, and adding the knowledge layer. \n", "\n", "You will:\n", "\n", "1. **Add a tool** – Install and configure the paper search tool (Google Scholar via Serper) and attach it to the deep research agent.\n", "2. **Customize models** – Use a frontier model for orchestration and planning, and Nemotron for research.\n", "3. **Locate and customize prompts** – Find where orchestrator, planner, and researcher prompts live and what variables they use.\n", "4. **Add the knowledge layer** – Enable document retrieval so the agent can search over your ingested documents alongside web and paper search." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Table of Contents\n", "\n", ">[Prerequisites](#prereqs) \n", ">[1. Adding a tool: Paper search](#add-tool) \n", ">[2. Customizing models (orchestrator vs planner/researcher)](#customize-models) \n", ">[3. Prompts: Where they are and how to customize](#prompts) \n", ">[4. Adding the knowledge layer](#knowledge-layer) \n", ">[Putting it together](#put-together) \n", ">[Next steps](#next) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Prerequisites\n", "\n", "This notebook assumes you have completed the environment setup from the [0_Getting_Started_with_DEEP_RESEARCHER](0_Getting_Started_with_Deep_Researcher_Agent.ipynb) and run through [1_Deep_Researcher_Web_Search](1_Deep_Researcher_Web_Search.ipynb).\n", "\n", "**Ensure you have:**\n", "- Cloned the repository and run `./scripts/setup.sh`\n", "- Activated the virtual environment: `source .venv/bin/activate`\n", "- API keys (you will be prompted in the next cells): **NVIDIA_API_KEY**, **TAVILY_API_KEY**; for paper search, **SERPER_API_KEY** from [serper.dev](https://serper.dev/)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "from pathlib import Path\n", "\n", "if os.environ.get(\"REPO_ROOT\"):\n", " os.chdir(os.environ[\"REPO_ROOT\"])\n", "else:\n", " cwd = Path.cwd()\n", " if not (cwd / \"configs\").exists() and not (cwd / \"src\").exists():\n", " root = cwd.parent.parent\n", " if (root / \"configs\").is_dir() or (root / \"src\").is_dir():\n", " os.chdir(root)\n", "print(\"Working directory:\", os.getcwd())" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import getpass\n", "\n", "if \"NVIDIA_API_KEY\" not in os.environ:\n", " nvidia_api_key = getpass.getpass(\"Enter your NVIDIA API key: \")\n", " os.environ[\"NVIDIA_API_KEY\"] = nvidia_api_key\n", "\n", "if \"TAVILY_API_KEY\" not in os.environ:\n", " tavily_api_key = getpass.getpass(\"Enter your Tavily API key: \")\n", " os.environ[\"TAVILY_API_KEY\"] = tavily_api_key" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 1. Adding a tool: Paper search\n", "\n", "The deep researcher can use multiple search tools. The **paper search** tool queries academic papers through Google Scholar (Serper API). Adding it is a two-step process: set up a Serper API key, install the plugin, then add the function and tool list in your config.\n", "\n", "**Reference:** [sources/google_scholar_paper_search/README.md](../../sources/google_scholar_paper_search/README.md), [Tools and Sources](../source/customization/tools-and-sources.md) (enable/disable by editing `tools` lists)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.0 Set up Serper API key (required for paper search)\n", "\n", "The paper search tool uses the [Serper](https://serper.dev/) API to query Google Scholar. You must have a Serper API key before installing or using the tool.\n", "\n", "**Steps:**\n", "1. Go to [serper.dev](https://serper.dev/) and create an account.\n", "2. From your dashboard, generate an API key.\n", "3. Add the key to your environment." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import getpass\n", "\n", "if \"SERPER_API_KEY\" not in os.environ:\n", " serper_api_key = getpass.getpass(\"Enter your Serper API key (or press Enter to skip): \")\n", " if serper_api_key:\n", " os.environ[\"SERPER_API_KEY\"] = serper_api_key\n", " print(\"SERPER_API_KEY set for this session.\")\n", " else:\n", " print(\"Skipped. Set SERPER_API_KEY in deploy/.env or run this cell again to use paper search.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.1 Install the paper search plugin\n", "\n", "From the repo root, install the plugin so it registers the `paper_search` function type with the NeMo Agent toolkit:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!uv pip install -e sources/google_scholar_paper_search" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!nat info components -t function 2>/dev/null | grep -E 'paper_search|paper_search_tool' || echo 'Run: nat info components -t function'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.2 Add to your configuration\n", "\n", "In your workflow YAML you need to:\n", "\n", "1. **Define the function** under `functions:` with `_type: paper_search` and options such as `max_results`, `serper_api_key` (or rely on `SERPER_API_KEY` in the environment).\n", "2. **Add the tool to the deep research agent** by including its name (for example `paper_search_tool`) in `deep_research_agent.tools`.\n", "\n", "Example snippet:\n", "\n", "```yaml\n", "functions:\n", " paper_search_tool:\n", " _type: paper_search\n", " max_results: 5\n", " serper_api_key: ${SERPER_API_KEY}\n", "\n", " deep_research_agent:\n", " _type: deep_research_agent\n", " orchestrator_llm: nemotron_ultra_llm\n", " tools:\n", " - paper_search_tool\n", " - advanced_web_search_tool\n", "```\n", "\n", "The agent’s prompts (orchestrator, planner, researcher) already mention `paper_search_tool` when that tool is present; they use the tool name to show the right instructions. No code changes are required beyond config." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile docs/notebooks/config_deep_researcher_with_paper_search.yml\n", "\n", "general:\n", " telemetry:\n", " logging:\n", " console:\n", " _type: console\n", " level: INFO\n", "\n", "llms:\n", " nemotron_ultra_llm:\n", " _type: nim\n", " model_name: nvidia/nemotron-3-ultra-550b-a55b\n", " base_url: \"https://integrate.api.nvidia.com/v1\"\n", " api_key: ${NVIDIA_API_KEY}\n", " temperature: 0.2\n", " top_p: 0.7\n", " max_tokens: 16384\n", " num_retries: 5\n", " chat_template_kwargs:\n", " enable_thinking: false\n", "\n", " nemotron_ultra_writer_llm:\n", " _type: nim\n", " model_name: nvidia/nemotron-3-ultra-550b-a55b\n", " base_url: \"https://integrate.api.nvidia.com/v1\"\n", " api_key: ${NVIDIA_API_KEY}\n", " temperature: 0.2\n", " top_p: 0.7\n", " max_tokens: 32768\n", " num_retries: 5\n", " chat_template_kwargs:\n", " enable_thinking: false\n", "\n", "\n", "functions:\n", " advanced_web_search_tool:\n", " _type: tavily_web_search\n", " max_results: 5\n", " advanced_search: true\n", "\n", " paper_search_tool:\n", " _type: paper_search\n", " max_results: 5\n", " serper_api_key: ${SERPER_API_KEY}\n", "\n", " deep_research_agent:\n", " _type: deep_research_agent\n", " orchestrator_llm: nemotron_ultra_llm\n", " source_router_llm: nemotron_ultra_llm\n", " planner_llm: nemotron_ultra_llm\n", " researcher_llm: nemotron_ultra_llm\n", " writer_llm: nemotron_ultra_writer_llm\n", " tools:\n", " - paper_search_tool\n", " - advanced_web_search_tool\n", "\n", "workflow:\n", " _type: deep_research_workflow\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.3 Run with paper search (optional)\n", "\n", "If you have `SERPER_API_KEY` set in `deploy/.env`, you can run the deep research workflow with paper search using the default CLI config (which already includes `paper_search_tool`). This may take several minutes." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!nat run --config_file docs/notebooks/config_deep_researcher_with_paper_search.yml --input \"Recent advances in transformer architecture for LLMs\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 1.4 What to expect\n", "\n", "When you run the workflow with paper search enabled, check the logs for tool invocations. You should see **`paper_search_tool`** (or `paper_search`) listed in tool-call messages as the planner and researcher run. For example: `[Tool Start] paper_search_tool` or `→ paper_search_tool` in the trace output. The agent may call it alongside `advanced_web_search_tool` for academic-style queries. If paper search is never called, the query may not have triggered academic search in the plan; try a more explicitly scholarly prompt (for example, \"recent peer-reviewed advances in …\")." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 2. Customizing models by deep-research role\n", "\n", "The deep research agent has five configurable LLM roles:\n", "\n", "| Role | Config key | Purpose |\n", "|------|------------|--------|\n", "| **Orchestrator** | `orchestrator_llm` | Coordinates the workflow and supplies the default for omitted roles (required). |\n", "| **Source router** | `source_router_llm` | Selects the source-routing strategy (optional; defaults to orchestrator). |\n", "| **Planner** | `planner_llm` | Builds the research plan and queries (optional; defaults to orchestrator). |\n", "| **Researcher** | `researcher_llm` | Runs search and synthesizes content (optional; defaults to orchestrator). |\n", "| **Writer** | `writer_llm` | Produces the final report (optional; defaults to orchestrator). |\n", "\n", "You can assign **different models** per role. For example: use a **frontier model** for orchestration and planning, and **Nemotron** for research.\n", "\n", "**Reference:** [Configuration Reference](../source/customization/configuration-reference.md)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.1 Example: Frontier orchestrator and planner, Nemotron researcher\n", "\n", "Define two (or three) LLMs in `llms:` and point the deep research agent at them:\n", "\n", "```yaml\n", "llms:\n", " nemotron_ultra_llm:\n", " _type: nim\n", " model_name: nvidia/nemotron-3-ultra-550b-a55b\n", " base_url: \"https://integrate.api.nvidia.com/v1\"\n", " temperature: 1.0\n", " max_tokens: 128000\n", "\n", "\n", " frontier_llm:\n", " _type: openai\n", " model_name: \"gpt-5.2\"\n", " api_key: ${OPENAI_API_KEY}\n", " temperature: 1.0\n", " max_tokens: 128000\n", "\n", "functions:\n", " deep_research_agent:\n", " _type: deep_research_agent\n", " orchestrator_llm: frontier_llm \n", " planner_llm: frontier_llm\n", " researcher_llm: nemotron_ultra_llm\n", " tools:\n", " - advanced_web_search_tool\n", "```\n", "\n", "If you omit `planner_llm` or `researcher_llm`, the agent falls back to `orchestrator_llm` for that role. The registration and wiring are in `src/deep_researcher_agent/agents/deep_researcher/register.py` (it resolves each LLM ref and configures the shared `LLMProvider`)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 3. Prompts: Where they are and how to customize\n", "\n", "The deep researcher uses **Jinja2** templates for the orchestrator, planner, and researcher. Editing these files is the main way to change behavior (tone, structure, tool prioritization, or workflow steps).\n", "\n", "**Reference:** [Deep Researcher README](../../src/deep_researcher_agent/agents/deep_researcher/README.md) (Prompts section), `src/deep_researcher_agent/agents/deep_researcher/agent.py` (`_load_prompts`, `render_prompt_template`)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.1 Where the prompts live\n", "\n", "All three prompts are under `src/deep_researcher_agent/agents/deep_researcher/prompts/`:\n", "\n", "| File | Role | Purpose |\n", "|------|------|--------|\n", "| `orchestrator.j2` | Orchestrator | System instructions: workflow steps, progress tracking, when to call planner/researcher, report format. |\n", "| `planner.j2` | Planner subagent | Task analysis, TOC, constraints, query generation; tool prioritization (knowledge, paper, web). |\n", "| `researcher.j2` | Researcher subagent | How to gather and synthesize information; tool use order; depth and citation requirements. |\n", "\n", "At runtime the agent loads them with `load_prompt(AGENT_DIR / \"prompts\", name)` (see `agent.py`). No config switch is needed to “enable” a prompt; changing the file and restarting (or re-running) is enough." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from pathlib import Path\n", "\n", "prompts_dir = Path(\"src/deep_researcher_agent/agents/deep_researcher/prompts\")\n", "for f in sorted(prompts_dir.glob(\"*.j2\")):\n", " print(f.name)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.2 Template variables\n", "\n", "The templates are rendered with variables so the same prompt adapts to tools and context:\n", "\n", "| Variable | Used in | Description |\n", "|----------|--------|-------------|\n", "| `tools` | All three | List of tool info (name, etc.); prompts use it to show “you have paper_search_tool, advanced_web_search_tool, …” and prioritization. |\n", "| `current_datetime` | Orchestrator, researcher | Injected at render time for date-aware answers. |\n", "| `clarifier_result` | Orchestrator | User clarifications from the clarifier agent, if any. |\n", "| `available_documents` | Orchestrator, planner, researcher | User-uploaded docs (e.g. from knowledge layer) with summaries; prompts tell the agent to prioritize knowledge_search for these. |\n", "\n", "The logic that builds these and calls `render_prompt_template` is in `agent.py` (`_build_orchestrator_agent`, `_get_subagents`). To add a new variable, add it to the `render_prompt_template(...)` call and use it in the corresponding `.j2` file." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 3.3 How to customize\n", "\n", "1. **Edit the `.j2` file** for the role you care about (orchestrator, planner, or researcher).\n", "2. **Keep existing variables** – The templates use `{{ current_datetime }}`, `{% for tool in tools %}`, `available_documents`, etc. Removing or renaming these can break rendering.\n", "3. **Adjust wording, steps, or priorities** – For example, in `researcher.j2` you can strengthen “cite sources inline” or add domain-specific guidance; in `orchestrator.j2` you can add or reorder workflow steps (as long as the tools and subagent names stay consistent).\n", "4. **Restart or re-run** – The agent loads prompts at initialization; after saving changes, restart the process or re-run the notebook so the new content is used." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## 4. Adding the knowledge layer\n", "\n", "The **knowledge layer** lets the agent search over **your ingested documents** (PDFs, docs, etc.) in addition to web and paper search. It is implemented as a NAT function type `knowledge_retrieval` with pluggable backends (LlamaIndex + ChromaDB for local dev, or Foundational RAG for a hosted server).\n", "\n", "**Reference:** [Knowledge Layer](../source/customization/knowledge-layer.md), `sources/knowledge_layer/KNOWLEDGE-LAYER-SETUP.md` (if present), `configs/config_web_default_llamaindex.yml`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.1 Install a backend\n", "\n", "Choose one backend and install it. For local development without extra services, use the LlamaIndex backend:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# LlamaIndex backend (local ChromaDB; no external server)\n", "!uv pip install -e \"sources/knowledge_layer[llamaindex]\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from deep_researcher_agent.knowledge import get_retriever\n", "\n", "print(\"Knowledge layer OK:\", get_retriever is not None)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 4.2 Add to configuration\n", "\n", "In your YAML:\n", "\n", "1. **Define the function** under `functions:` with `_type: knowledge_retrieval`, `backend: llamaindex` (or `foundational_rag`), plus backend-specific options (`collection_name`, `top_k`, `chroma_dir` for LlamaIndex; `rag_url` / `ingest_url` for Foundational RAG).\n", "2. **Add the tool to the deep research agent** by including the function name (for example `knowledge_search`) in `deep_research_agent.tools`.\n", "\n", "Example for LlamaIndex (see `configs/config_web_default_llamaindex.yml`):\n", "\n", "```yaml\n", "functions:\n", " knowledge_search:\n", " _type: knowledge_retrieval\n", " backend: llamaindex\n", " collection_name: ${COLLECTION_NAME:-test_collection}\n", " top_k: 5\n", " chroma_dir: ${DEEP_RESEARCHER_CHROMA_DIR:-/tmp/chroma_data}\n", "\n", " deep_research_agent:\n", " _type: deep_research_agent\n", " orchestrator_llm: nemotron_ultra_llm\n", " tools:\n", " - advanced_web_search_tool\n", " - knowledge_search\n", "```\n", "\n", "The deep researcher prompts already include branches for `knowledge_search` (and `available_documents`); once the tool is in the list, the orchestrator and subagents will be instructed to use it for user documents and internal context." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%writefile docs/notebooks/config_deep_researcher_with_knowledge_layer.yml\n", "\n", "general:\n", " telemetry:\n", " logging:\n", " console:\n", " _type: console\n", " level: INFO\n", "\n", "llms:\n", " nemotron_ultra_llm:\n", " _type: nim\n", " model_name: nvidia/nemotron-3-ultra-550b-a55b\n", " base_url: \"https://integrate.api.nvidia.com/v1\"\n", " api_key: ${NVIDIA_API_KEY}\n", " temperature: 0.2\n", " top_p: 0.7\n", " max_tokens: 16384\n", " num_retries: 5\n", " chat_template_kwargs:\n", " enable_thinking: false\n", "\n", " nemotron_ultra_writer_llm:\n", " _type: nim\n", " model_name: nvidia/nemotron-3-ultra-550b-a55b\n", " base_url: \"https://integrate.api.nvidia.com/v1\"\n", " api_key: ${NVIDIA_API_KEY}\n", " temperature: 0.2\n", " top_p: 0.7\n", " max_tokens: 32768\n", " num_retries: 5\n", " chat_template_kwargs:\n", " enable_thinking: false\n", "\n", "\n", "functions:\n", " advanced_web_search_tool:\n", " _type: tavily_web_search\n", " max_results: 5\n", " advanced_search: true\n", "\n", " knowledge_search:\n", " _type: knowledge_retrieval\n", " backend: llamaindex\n", " collection_name: ${COLLECTION_NAME:-test_collection}\n", " top_k: 5\n", " chroma_dir: ${DEEP_RESEARCHER_CHROMA_DIR:-/tmp/chroma_data}\n", "\n", " deep_research_agent:\n", " _type: deep_research_agent\n", " orchestrator_llm: nemotron_ultra_llm\n", " source_router_llm: nemotron_ultra_llm\n", " planner_llm: nemotron_ultra_llm\n", " researcher_llm: nemotron_ultra_llm\n", " writer_llm: nemotron_ultra_writer_llm\n", " tools:\n", " - advanced_web_search_tool\n", " - knowledge_search\n", "\n", "workflow:\n", " _type: deep_research_workflow\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Putting it together\n", "\n", "A single config can combine all of the above:\n", "\n", "- **Tools:** `paper_search_tool`, `advanced_web_search_tool`, `knowledge_search` in `deep_research_agent.tools`.\n", "- **Models:** `orchestrator_llm: frontier_llm`, `source_router_llm: nemotron_ultra_llm`, `planner_llm: nemotron_ultra_llm`, `researcher_llm: nemotron_ultra_llm`, and `writer_llm: nemotron_ultra_writer_llm` (with those LLMs defined in `llms:`).\n", "- **Prompts:** No config change; edit `src/deep_researcher_agent/agents/deep_researcher/prompts/*.j2` as needed.\n", "- **Knowledge layer:** Install a backend, define `knowledge_search` in `functions`, and add it to `deep_research_agent.tools`.\n", "\n", "See `configs/config_web_default_llamaindex.yml` and `configs/config_web_frag.yml` for full examples that mix web search, paper search, knowledge layer, and different LLMs." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "## Next steps\n", "\n", "- **Guide to Extending** (see [Extending Documentation](../source/extending/index.md))" ] } ], "metadata": { "kernelspec": { "display_name": ".venv", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.3" } }, "nbformat": 4, "nbformat_minor": 2 }