Agentic RAG: Tutorial & Examples

Reviewed by Patronus AI team

Agentic RAG is a next-generation extension of Retrieval-Augmented Generation (RAG) where autonomous AI agents plan, retrieve information, and reason across multiple tools, APIs, and data sources.

Traditional RAG pipelines are essentially one-shot: the system retrieves relevant documents for a query and then generates an answer without any intermediate decision-making or feedback loop. This works well for straightforward questions, but it falls short for more complex, multi-step tasks. Agentic RAG builds on RAG by adding a planning loop, memory, and tool usage, enabling the AI to handle evolving queries and perform reasoning over multiple steps.

This article explores the limitations of traditional RAG, how Agentic RAG works, and how to implement it using examples with LangGraph. 

You will learn the key differences between traditional RAG and Agentic RAG, know how to start building agentic RAG applications, and discover best practices to make them reliable.

Summary of key agentic RAG concepts

Concept Agentic RAG Traditional RAG
What it does Introduces planning, memory, and dynamic control flow to handle complex, multi-step tasks. Executes a single retrieval followed by generation, with no awareness or adaptability.
Execution flow Agentic RAG follows a multi-step, iterative loop in which the agent plans, retrieves data, uses tools, evaluates, and dynamically decides next steps. Fixed, one-shot pipeline
Reasoning Supports explicit multi-hop reasoning. Agents break down queries into sub-goals, reason through intermediate steps, and revise their plan as needed to reach a more accurate or complete answer. Reasoning is limited to context retrieval and context-based response.
Tool Integration Goes beyond vector search. Agents can flexibly use external APIs, databases, calculators, and other tools, choosing what to invoke based on the query and task context. Limited to vector search
Adaptability Agent behavior evolves based on retrieved results and intermediate feedback. If a retrieval fails or yields insufficient data, the agent can adjust its strategy or switch tools. Static, pre-determined flow
Personalization Can tailor responses using user-specific data, such as preferences, historical queries, or user profiles, enabling more relevant, context-aware outputs. Can also do the same, but in a limited, pre-determined manner.
Error recovery Includes mechanisms for re-planning, retrying failed tool calls, and course-correcting based on the evolving context. Lacks built-in feedback or correction loops.
Observability and evaluation Agentic RAG systems expose intermediate steps, such as planning decisions, tool calls, and retrieved context, enabling the tracing of execution, the evaluation of reasoning quality, and the diagnosis of failures across multi-step workflows. Limited observability capability

What is RAG, and why do we need it?

Retrieval-Augmented Generation (RAG) is a technique to augment an LLM’s knowledge by retrieving relevant external documents at inference time. 

In essence, instead of relying solely on the model’s built-in training data, which may be outdated or limited in domain knowledge, a RAG system fetches up-to-date or domain-specific information. It supplies the model with additional context. This approach dramatically improves factual accuracy and reduces hallucinations, because the model’s answer is “grounded” in retrieved evidence.

For example, imagine you have a collection of company policy documents, and you want to build a Q&A assistant. A traditional LLM might not know your company’s specific policies. A RAG pipeline, on the other hand, retrieves the relevant policy text for a given question and then has the LLM generate an answer using that text as context. This makes the answer both accurate and specific to your documents.

{{banner-large-dark-2="/banners"}}

How traditional RAG works

Traditionally, the RAG workflow consisted of three main phases:

Vector store creation and indexing

In the first step, a vector store is created, where the document chunks containing custom knowledge are converted into embedding vectors and stored in a vector database.  

Retrieval

First, the system finds relevant data from an external source based on the user’s query. Typically, this involves a vector database of embedded documents. All documents are pre-processed (split into chunks and embedded into vectors) to form a knowledge database. When a user asks a question, the query is also converted into an embedding, and the vector DB is queried for similar chunks (e.g., the top-k most relevant pieces) using semantic similarity techniques. 

Generation

The retrieved text chunks are then input into the prompt of an LLM along with the original question, and the LLM generates a final answer using that information. Essentially, the model “augments” its knowledge with the retrieved context when crafting a response.

Limitations of traditional RAG

While traditional RAG is powerful for what it does, it has some key limitations that become apparent in more complex scenarios:

Single-turn, no lookahead

The traditional RAG pipeline is a unidirectional one-shot flow. The model cannot ask follow-up questions or perform multiple retrieval rounds. If the needed information wasn’t fetched in a single retrieval step, the opportunity is lost. Multi-hop questions, in which answering requires retrieving one piece of information, then using it to retrieve another, etc., become problematic.

Fixed tooling

Traditional RAG typically only uses a vector database lookup as its “tool.” It can’t perform actions beyond document retrieval. If a query requires, say, performing a calculation, invoking an API, or using a different knowledge source, vanilla RAG has no facility to do that. It’s basically stuck with the documents you gave it.

Limited adaptability

Because the flow is predetermined, the system can’t dynamically adjust if something goes wrong or if the query takes an unexpected turn. For instance, if the retrieved documents aren’t sufficient, a simple RAG system might just produce a vague answer or say “I don’t know.” There’s no built-in mechanism to realize the gap and attempt a different strategy.

Opaque reasoning

Debugging a RAG pipeline is mostly about checking the inputs and outputs at each step, e.g., which documents were retrieved and which answer was generated. You don’t get to see how the model decided to answer beyond that, since the model’s reasoning process is hidden in a single prompt. If the answer is wrong, you have to guess whether the retrieval failed or the model ignored the context.  There’s no explicit trace of its thought process.

Agentic RAG addresses many of these limitations by making the system more interactive, flexible, and intelligent.

Agentic RAG: RAG with reasoning and tools

Agentic RAG combines the retrieval-augmented approach with an agent that can plan actions, use tools, and maintain memory across multiple steps. In other words, Agentic RAG is an autonomous AI agent that doesn’t just fetch context once but can iteratively decide what to do (e.g., retrieve more info, call an API, etc.) and when to stop.

Agentic RAG: Tutorial & Examples
Difference between traditional RAG and agentic RAG

How agentic RAG works

Instead of a fixed pipeline, an Agentic RAG system implements a loop of reasoning and action. A typical Agentic RAG workflow might look like this:

User request

The user’s question or task comes in. The agent interprets the goal.

Planning/reasoning

The agent determines what steps or information it needs to solve this problem. It may break the problem into sub-tasks. For example, if asked, “Do I have enough vacation days to take a 2-week trip next month?”, the agent might reason: “First, retrieve the user’s current leave balance. Then check the policy for accruing leave and calculate if 2 weeks is <= balance.”

Tool use/retrieval

Based on its plan, the agent calls tools to get information. Tools could include:

  • A vector database search, just like RAG’s retriever, but now it can be invoked whenever needed, not just once.
  • Other databases or APIs, such as an HR database to retrieve a user’s remaining leave or a web search for external information.
  • Utility functions like calculators, code execution, etc., necessary for the task.

The agent decides which tool to use and with what input. This is often done using a ReAct-style loop (Reasoning and Acting), in which the LLM produces an action such as “use tool X with input Y” and then receives the tool’s output (an observation) to inform the next step.

Observation and iteration

When the agent gets a tool’s result, it incorporates that into its state, called a “memory” or context for subsequent steps. Then it can reason again: Did that answer the question fully? If not, what’s the next step? 

It loops back to planning, possibly uses another tool, and so on. Crucially, the agent’s execution adapts based on what it finds. If the first retrieval wasn’t enough, it can try a different query or call a different tool.

Final answer

Eventually, the agent decides it has enough information and produces a final answer for the user. This might be synthesized from all the data gathered along the way.

The following figure demonstrates the workflow of an Agentic RAG pipeline. Once a user query is received:

  1. The agent executes a feedback loop where
  2. The agent’s thoughts (reasoning) lead to 
  3. Actions (tool calls) which produce
  4. Observations that feed back into the agent’s next iteration.

The loop continues until the agent’s reasoning indicates it's done, at which point it outputs the answer. 

This approach endows the system with flexibility: it can handle tasks that require multiple steps, adjust when a step doesn’t yield a complete answer, and integrate information from various sources (not just a single vector store) as needed.

Agentic RAG: Tutorial & Examples
Overview of agentic RAG

Note: Similar to traditional RAG, the document chunking, embedding generation, and vector store indexing steps are performed before vector database retrieval in agentic RAG. 

Agentic RAG implementation example

To illustrate an agentic RAG implementation, consider a scenario in which an HR assistant answers questions about employee benefits and policies. We have multiple data sources: a database of employee information (for personalized answers) and two policy documents (e.g., leave policy and promotion/bonus policy, in PDF format). 

We want the AI agent to:

  1. Fetch the relevant employee’s data for personalization.
  2. Depending on the question, search the appropriate policy document for the answer, using the employee context.
  3. Formulate the answer by combining the policy info and the user’s details.

Let’s see how an agentic RAG implements the above logic.

Note: The code for this article is available in this Google Colab notebook.

Step 1: Installing and importing required libraries

Run the scripts below to install and import the required libraries.

!pip install -q langgraph langchain langchain-openai langchain-community
!pip install -q chromadb
!pip install -q langchain-chroma
!pip install -q pypdf

‍

import os
import sqlite3
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.tools import tool
from langchain_core.messages import HumanMessage
from langchain_chroma import Chroma
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langgraph.prebuilt import create_react_agent
from langgraph.checkpoint.memory import MemorySaver
from dotenv import load_dotenv
from IPython.display import Image, display
import warnings
warnings.filterwarnings("ignore")

This example uses the OpenAI GPT-4 model for reasoning. You can use any other model as well.

# Set OpenAI API key
os.environ["OPENAI_API_KEY"] = os.getenv("OPENAI_API_KEY")

# Initialize OpenAI model
llm = ChatOpenAI(model="gpt-4o", temperature=0)

Step 2: Set up a database for storing user records

Next, set up a database containing fictional employee data. This database will serve as one of the sources of context for our agentic RAG application.

def initialize_database():
    conn = sqlite3.connect('company_data.db')
    cursor = conn.cursor()

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS users (
        user_id INTEGER PRIMARY KEY,
        first_name TEXT,
        last_name TEXT,
        designation TEXT,
        seniority_level TEXT,
        department TEXT,
        hire_date TEXT,
        salary REAL,
        performance_rating TEXT
    )
    ''')

    users = [
        (1, 'John', 'Doe', 'Software Engineer', 'Senior', 'Engineering', '2020-01-15', 95000, 'Excellent'),
        (2, 'Jane', 'Smith', 'Product Manager', 'Mid-level', 'Product', '2021-03-20', 85000, 'Good'),
        (3, 'Alice', 'Johnson', 'Data Scientist', 'Junior', 'Data Science', '2023-06-01', 75000, 'Good'),
        (4, 'Bob', 'Williams', 'Senior Architect', 'Principal', 'Engineering', '2018-09-10', 130000, 'Excellent'),
        (5, 'Carol', 'Brown', 'HR Manager', 'Senior', 'Human Resources', '2019-11-05', 90000, 'Excellent')
    ]

    cursor.executemany('INSERT OR REPLACE INTO users VALUES (?,?,?,?,?,?,?,?,?)', users)
    conn.commit()
    conn.close()
    print("Database initialized with 5 sample users!")

initialize_database()

Step 3: Set up vector databases for RAG

Next, we create a vector database containing company leave policies.

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=200,
    chunk_overlap=40,
    length_function=len
)

print("Text splitter initialized!")


LEAVE_POLICY_PDF_PATH = "Company_Leave_Policies_Extended.pdf"

def load_leave_policy_vectorstore(pdf_path):

if not os.path.exists(pdf_path):
        print(f"Warning: PDF not found at {pdf_path}")

return None

    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    split_docs = text_splitter.split_documents(documents)

    embeddings = OpenAIEmbeddings()
    vectorstore = Chroma.from_documents(
        documents=split_docs,
        embedding=embeddings,
        collection_name="leave_policies",
        persist_directory="./chroma_db_leave"
    )

    print(f"Loaded {len(documents)} pages, split into {len(split_docs)} chunks")

return vectorstore

leave_vectorstore = load_leave_policy_vectorstore(LEAVE_POLICY_PDF_PATH)

Additionally, create another vector database containing employee bonus and promotion policies. 

PROMOTION_BONUS_PDF_PATH = "Company_Promotion_Bonus_Policies_Enterprise_Grade.pdf"

def load_promotion_bonus_vectorstore(pdf_path):

if not os.path.exists(pdf_path):
        print(f"Warning: PDF not found at {pdf_path}")

return None

    loader = PyPDFLoader(pdf_path)
    documents = loader.load()
    split_docs = text_splitter.split_documents(documents)

    embeddings = OpenAIEmbeddings()
    vectorstore = Chroma.from_documents(
        documents=split_docs,
        embedding=embeddings,
        collection_name="promotion_bonus_policies",
        persist_directory="./chroma_db_promotion"
    )

    print(f"Loaded {len(documents)} pages, split into {len(split_docs)} chunks")

return vectorstore

promotion_vectorstore = load_promotion_bonus_vectorstore(PROMOTION_BONUS_PDF_PATH)

Step 4: Defining tools for agentic RAG

We create three tools for our agentic RAG: 

  • `get_user_context`, which retrieves employee information from the employee database.
  • `search_leave_policy`, which searches the leave policy vector database based on the employee information. 
  • `search_promotion_bonus_policy`, which searches the bonus and promotion vector database using the employee information. 

get_user_context tool

@tool
def get_user_context(user_id: int) -> str:

"""Get user information from database to use as context.

    Args:
        user_id: The employee's user ID

    Returns:
        User context string with employee information
    """
    conn = sqlite3.connect('company_data.db')
    cursor = conn.cursor()
    cursor.execute('SELECT * FROM users WHERE user_id = ?', (user_id,))
    user = cursor.fetchone()
    conn.close()


if not user:

return f"User ID {user_id} not found"


return f"""Employee Information\n:Employee: {user[1]} {user[2]}
Designation: {user[3]}
Seniority: {user[4]}
Department: {user[5]}
Performance: {user[8]}
Salary: ${user[7]}"""

‍

search_leave_policy tool

@tool
def search_leave_policy(question: str, user_context: str) -> str:

"""Search leave policy documents with user context.

    Args:
        question: Question about leave policies
        user_context: User information context from get_user_context tool
    """

if leave_vectorstore is None:

return
"Leave policy PDF not loaded"

    # Combine user context with question for better retrieval
    search_query = f"{user_context}\n\n{question}"

    retriever = leave_vectorstore.as_retriever(search_kwargs={"k": 3})
    docs = retriever.invoke(search_query)
    context = "\n\n".join([doc.page_content for doc in docs])


return f"Relevant Leave Information Retrieved:\n{context}"

‍

search_promotion_bonus_policy tool

@tool
def search_promotion_bonus_policy(question: str, user_context: str) -> str:

"""Search promotion and bonus policy documents with user context.

    Args:
        question: Question about promotion or bonus
        user_context: User information context from get_user_context tool
    """

if promotion_vectorstore is None:

return
"Promotion/bonus policy PDF not loaded"

    # Combine user context with question for better retrieval
    search_query = f"{user_context}\n\n{question}"

    retriever = promotion_vectorstore.as_retriever(search_kwargs={"k": 3})
    docs = retriever.invoke(search_query)
    context = "\n\n".join([doc.page_content for doc in docs])


return f"User Context: {user_context}\n\nRelevant Policy:\n{context}"

‍

Step 5: Create a ReAct agent with RAG

The next step is to create the agent and give it these tools. The example also includes a system prompt that guides the agent’s strategy. The prompt explicitly states the order of operations: always retrieve user info first, then use it to query the appropriate policy tool. This helps the agent to plan the correct sequence of actions.

tools = [
    get_user_context,
    search_leave_policy,
    search_promotion_bonus_policy
]

system_prompt = """You are an HR assistant agent.

Tool Usage Instructions:
1. FIRST use 'get_user_context' with the user_id to get employee information
2. Use the user context from step 1 when calling RAG tools:
   - 'search_leave_policy': Pass question and user_context
   - 'search_promotion_bonus_policy': Pass question and user_context

Always get user context first before searching policies."""

‍

‍Finally, create a ReAct agent using LangGraph's `create_react_agent` function. 

memory = MemorySaver()

agent = create_react_agent(llm, tools, 
                           prompt=system_prompt, 
                           checkpointer=memory)

print("ReAct Agent created with 3 tools and memory!")

display(Image(agent.get_graph().draw_mermaid_png()))

‍

Output:

Agentic RAG: Tutorial & Examples

Step 6: Querying agentic RAG

The agent is now ready to use. You can wrap it in a convenient query function.‍

def query_agent(question: str, user_id: int, thread_id: str = "default"):

"""Query the agent

    Args:
        question: The question to ask
        user_id: The user ID
        thread_id: Thread ID for conversation history
    """
    # Add user_id to the question context
    full_question = f"User ID: {user_id}\n\n{question}"

    config = {"configurable": {"thread_id": thread_id}}
    response = agent.invoke({"messages": [HumanMessage(content=full_question)]}, config)


return response['messages'][-1].content

In the code above, we prepend the User ID {user_id} to the query content. This is a simple way to pass the user ID into the agent’s context so it knows which user’s info to fetch.

The `agent.invoke()` function in the above script prompts the agent with a system prompt and the user's question. The agent then starts its ReAct loop: likely the first thing it does is call the `get_user_context` tool (since we made that clear in the instructions). LangGraph executes the tool, retrieves the result (the user’s info), and feeds it back to the agent. The agent then decides the next action (perhaps a call to the `search_leave_policy` or `search_promotion_bonus_policy` tool, depending on the question), and so on. 

Finally, when the agent believes it has all the information to answer the user’s question, it appends the final answer to the messages list and returns it. You can extract the content of the last message as the answer.

The `thread_id` parameter allows multiple queries in the same thread to persist in memory. For example, if the agent has a conversation with follow-up questions, the MemorySaver tracks prior interactions labeled with `thread_id`.

Let’s test our agent with a simple query:

result = query_agent("How many days of annual leave do I get?", user_id=1)
print(result)

Output:

Agent answer in a terminal saying a Senior Software Engineer gets up to 10 paid days of annual leave per year

The above output shows that the RAG first retrieved the user’s information (i.e., the user is a Senior Software Engineer) and based on that information returned the relevant annual leave policy. 

Let’s ask a slightly more complicated question involving multiple tool calls.

result = query_agent("Get all my details and tell me if I am eligible for a bonus? How will I get it? How many paternity leaves will I get in a year?", user_id=1)
print(result)

‍

Output:

Agentic RAG: Tutorial & Examples

The output above shows that the agent retrieved the user’s personal information, then the relevant bonus policy, and finally the number of paternity leaves for a senior software engineer. 

Finally, you can test the agent's memory by asking a follow-up question.‍

# First question
response1 = query_agent("What are my details?", user_id=2, thread_id="conv-1")
print("Question 1:", response1)
print("\n" + "="*80 + "\n")

# Follow-up 1
response2 = query_agent("What leave policies apply to me?", user_id=2, thread_id="conv-1")
print("Question 2 (uses memory):", response2)
print("\n" + "="*80 + "\n")

# Follow-up 2
response3 = query_agent("And bonus?", user_id=2, thread_id="conv-1")
print("Question 3 (uses memory):", response3)

‍

Output:

Agentic RAG: Tutorial & Examples

‍

Evaluating agentic RAG with a judge LLM‍

A judge LLM can evaluate an agentic RAG application from its trace. The trace lists every tool call, every retrieved passage, and the final answer, so the judge can check whether the agent called the right tools in the right order and grounded its answer in the retrieved text.

As RAG systems evolve into more complex agentic pipelines, tracing and debugging the behavior of each reasoning step becomes a non-trivial challenge. Traditional RAG pipelines might only involve one retrieval and a generation step, but Agentic RAG systems introduce iterative planning, dynamic tool usage, and memory-driven decisions, all of which make it harder to spot where things go wrong. Evaluating the agent’s full trace shows you which step failed: the tool choice, the retrieval, or the final answer.

The following example uses CompassJudger-2, an open-source judge LLM that OpenCompass releases under the Apache 2.0 license. The judge reads the agent’s full trace, including every retrieval action, tool invocation, and internal decision made by the agent. This level of visibility is critical when an agentic RAG application spans multiple hops and dynamically adjusts its flow based on intermediate results.

Implementation example

Let’s see how to evaluate the tool calls and retrievals of the LangGraph agentic RAG application implemented in the previous section.

Run the following code to install the library needed to load the judge LLM.

!pip install -qU bitsandbytes

The following code loads CompassJudger-2 from Hugging Face in 4-bit precision. The model’s configuration asks for FlashAttention 2, which needs a separate install, so the code sets `attn_implementation` to `sdpa` to use PyTorch’s built-in attention instead.

import re
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

judge_tokenizer = AutoTokenizer.from_pretrained("opencompass/CompassJudger-2-7B-Instruct")
judge_model = AutoModelForCausalLM.from_pretrained(

"opencompass/CompassJudger-2-7B-Instruct",
    device_map="auto",
    attn_implementation="sdpa",
    quantization_config=BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_compute_dtype=torch.float16),
)

Next, define the judge. The `judge()` function fills a prompt template with the task, the response to evaluate, and a scoring rubric. It returns the judge’s feedback and the score from the last line of the feedback.

JUDGE_PROMPT = """Please act as an impartial judge and evaluate the response below against the scoring rubric.
Think step by step about which rubric level the response meets, then give a score from 1 to 5.

Task:
{instruction}

Response to evaluate:
{response}

Scoring rubric:
{rubric}

End your reply with a line in the format "Score: <1-5>"."""

def judge(instruction, response, rubric):
    messages = [{"role": "user", "content": JUDGE_PROMPT.format(instruction=instruction, response=response, rubric=rubric)}]
    inputs = judge_tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt", return_dict=True).to(judge_model.device)
    output_ids = judge_model.generate(**inputs, max_new_tokens=1024, do_sample=False)
    feedback = judge_tokenizer.decode(output_ids[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True).strip()
    scores = re.findall(r"Score:\s*([1-5])", feedback)

return feedback, int(scores[-1]) if scores else None

Run the multi-part query from Step 6 again, this time in a new thread, so that the trace contains only this query.

result = query_agent("Get all my details and tell me if I am eligible for a bonus? How will I get it? How many paternity leaves will I get in a year?", user_id=1, thread_id="eval-1")
print(result)

Output:

Agent answer listing John Doe's employee details, a bonus of up to 25% of base salary and 4 weeks of paid paternity leave

Next, collect the trace of this query from the agent’s short-term memory. The MemorySaver checkpointer stores every message in the thread, so the following code reads the messages with `agent.get_state` and records every tool call with its arguments and every tool output.

config = {"configurable": {"thread_id": "eval-1"}}
state = agent.get_state(config)

trace = []
for msg in state.values["messages"]:

if msg.type == "ai" and msg.tool_calls:

for tool_call in msg.tool_calls:
            trace.append(f"[ai] called {tool_call['name']} with arguments {tool_call['args']}")
    elif msg.type == "tool":
        trace.append(f"[tool: {msg.name}] {msg.content}")

else:
        trace.append(f"[{msg.type}] {msg.content}")
trace = "\n".join(trace)
print(trace)

Output:

Agent trace showing the question, calls to get_user_context, search_promotion_bonus_policy and search_leave_policy, and the employee record and bonus policy they return
Agent trace continued, with the retrieved leave policy sections and the agent's final answer

Finally, ask the judge whether the agent fetched the employee’s record first, answered every part of the question, and based its answer on the retrieved policy text. The instruction describes the agent’s tools, and the rubric describes what each score from 1 to 5 means.

agent_instruction = """An HR assistant agent received this question from the employee with user ID 1: "Get all my details and tell me if I am eligible for a bonus? How will I get it? How many paternity leaves will I get in a year?".
The agent can call these tools:
- get_user_context: returns the employee's record from the HR database.
- search_leave_policy: searches the leave policy with the question and the employee's record.
- search_promotion_bonus_policy: searches the promotion and bonus policy with the question and the employee's record.
Evaluate the agent's trace below: its tool calls, the tool outputs, and its final answer."""

rag_rubric = """[Does the agent fetch the employee's record first, search the right policies with it, answer every part of the question, and base its answer on the tool outputs?]
Score 1: The agent does not fetch the employee's record or does not search the policies.
Score 2: The agent searches the policies, but its answer leaves out part of the question and states facts that are not in the tool outputs.
Score 3: The agent searches the right policies, but its answer leaves out part of the question or states facts that are not in the tool outputs.
Score 4: The agent fetches the record first, searches the right policies, and answers every part of the question from the tool outputs, with minor issues such as an unnecessary tool call.
Score 5: The agent fetches the record first, searches the leave and the bonus policies with it, answers every part of the question, and every fact in its answer comes from the employee's record or the retrieved policy text."""

feedback, score = judge(agent_instruction, trace, rag_rubric)
print(feedback)

Output:

Judge LLM output giving the agent a score of 4, with feedback that it answered every part but formatted the paternity leave section less clearly

The output shows that the judge gives the agent a score of 4. Its feedback says that the agent fetched the employee’s record, searched the relevant policies, and answered every part of the question from the retrieved information. It takes off one point for the formatting and clarity of the paternity leave section, a minor issue that doesn’t affect the facts in the answer. If you change the system prompt, the tools, or the chunk size, run the same check again to make sure that the agent still grounds its answers in the retrieved text.

{{banner-dark-small-1="/banners"}}‍

Final thoughts

Agentic RAG extends the traditional retrieval-augmented approach by incorporating memory, planning, and dynamic tool use, making it better suited to real-world, multi-step workflows. With the right frameworks and observability tools in place, implementing and monitoring these systems becomes far more practical. Once you understand how retrieval, tools, memory, and planning fit together, the architecture becomes both modular and extendable.

Trace-based evaluation complements this development model. Collecting the agent’s full trace and scoring it with an open-source judge LLM makes it easier to inspect agent behavior, catch silent failures, and refine application pipelines, all without overhauling your application stack.

Continue reading this series