When building RAG applications, you need objective, automated ways to quantify performance. While traditional NLP metrics rely on exact word matching, modern evaluation frameworks work differently. They leverage LLMs to judge the semantic relationships between the user's prompt, the retrieved documents, and the final response.
In this topic, we will explore the foundational metrics for evaluating RAG systems. We will examine what they measure, how they calculate scores under the hood, and how you can implement them using both DeepEval and Ragas.
Anatomy of a RAG evaluation
To understand what these metrics evaluate, we must establish the distinct pieces of data generated during a typical RAG request. These variables are the inputs required by almost all evaluation frameworks.
Let's look at a highly simplified LangChain application to visualize where this data comes from:
from langchain_core.prompts import PromptTemplate
from langchain_openai import ChatOpenAI
# user_input (or query)
question = "What is the default server timeout?"
# retrieved_contexts (fetched from a vector database)
retrieved_chunks = [
"The default timeout for all incoming requests is 30 seconds.",
"Database connections will close after 60 seconds of inactivity."
]
# response (or actual_output)
llm = ChatOpenAI(model="gpt-5.4-mini")
prompt = PromptTemplate.from_template(
"Answer the question based on the context.\nContext: {context}\nQuestion: {question}"
)
chain = prompt | llm
result = chain.invoke({
"context": "\n".join(retrieved_chunks),
"question": question
})
print(result.content) # e.g., "The default server timeout is 30 seconds."In addition to the three variables above (input, response, retrieved_contexts), some metrics also require a reference (or expected_output). This is the ground-truth answer curated by a human.
RAG metrics are broadly divided into two categories:
Generation metrics for evaluating the LLM's final response;
Retrieval metrics for measuring the quality of the documents fetched from your database.
Let's dive into them.
Faithfulness
Faithfulness is a generation metric that measures whether the LLM's response is factually consistent with the retrieved context. It is your primary defense against model hallucinations. The evaluator LLM first breaks down the generated response into individual claims. Then, it cross-checks each claim against the retrieved context to verify if the claim can be logically inferred from the provided text.
The score is a float ranging from 0.0 to 1.0. It calculates the ratio of supported claims to total claims.
A score of
1.0means every statement the model made is backed by the retrieved documents.A low score indicates the model is inventing information or pulling in outside knowledge not present in your RAG knowledge base.
Here is how you evaluate Faithfulness in DeepEval and Ragas:
from deepeval.metrics import FaithfulnessMetric
from deepeval.test_case import LLMTestCase
faithfulness = FaithfulnessMetric(threshold=0.7, model="gpt-4o-mini")
test_case = LLMTestCase(
input="What are the operating hours?",
actual_output="The clinic is open 9 AM to 5 PM, and closed on weekends.",
retrieval_context=["The clinic operates Monday through Friday, 9 AM to 5 PM."]
)
print(f"Faithfulness score: {faithfulness.measure(test_case)}")
# Faithfulness score: 1.0from ragas.metrics.collections import Faithfulness
from ragas.llms import llm_factory
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv
load_dotenv()
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
evaluator = Faithfulness(llm=llm)
async def check_faithfulness():
score = await evaluator.ascore(
user_input="What are the operating hours?",
response="The clinic is open 9 AM to 5 PM, and closed on weekends.",
retrieved_contexts=["The clinic operates Monday through Friday, 9 AM to 5 PM."]
)
print(f"Faithfulness: {score.value}")
asyncio.run(check_faithfulness())
# Faithfulness: 1.0Faithfulness is vital for high-stakes fields, such as healthcare, finance, and law. In these areas, making up facts can lead to severe consequences. It is also great for customer support bots, ensuring the bot only talks about real product features, rather than inventing new ones.
Answer relevancy (response relevancy)
Answer relevancy evaluates how directly and concisely the generated answer addresses the user's initial query. It focuses strictly on the alignment between the question and the answer, ignoring factual correctness.
This metric penalizes incomplete answers, as well as answers that include excessive, redundant, or unprompted outputs. Frameworks typically calculate this by prompting an LLM to reverse-engineer potential questions based only on the generated response. It then calculates the semantic similarity (using vector embeddings) between the reverse-engineered questions and the original user input.
The metric returns a value between 0.0 and 1.0.
A score closer to
1.0indicates a crisp, direct answer that perfectly satisfies the user's intent.A lower score means the model either missed the point or added unnecessary fluff.
from deepeval.test_case import LLMTestCase
from deepeval.metrics import AnswerRelevancyMetric
relevancy = AnswerRelevancyMetric(threshold=0.8, model="gpt-4o-mini")
test_case = LLMTestCase(
input="How do I reset my password?",
actual_output="To reset your password, click 'Forgot Password'. By the way, we just launched a new dark mode feature!" # The extra fluff lowers the score
)
print(f"Answer relevancy: {relevancy.measure(test_case)}")
# Answer relevancy: 0.5from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy
from ragas.embeddings.base import embedding_factory
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv
load_dotenv()
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
embeddings = embedding_factory("openai", model="text-embedding-3-small", client=AsyncOpenAI())
evaluator = AnswerRelevancy(llm=llm, embeddings=embeddings)
async def check_relevancy():
score = await evaluator.ascore(
user_input="How do I reset my password?",
response="To reset your password, click 'Forgot Password'." # No fluff, high score
)
print(f"Answer relevancy: {score.value}")
asyncio.run(check_relevancy())
# Answer relevancy: 1.0000000000000002This metric is perfect for voice assistants, chatbots, and customer support. Users in these situations want quick, direct answers. High answer relevancy means the bot gets straight to the point, avoiding long, rambling responses that frustrate users.
Context precision (contextual precision)
Context precision is a retrieval metric that focuses on the ranking quality of your search system. It evaluates whether the retriever successfully placed the most relevant information at the very top of the retrieved context list.
The evaluator LLM compares the retrieved chunks against a human-annotated reference (ground truth). It checks if the chunks containing the expected answer are ranked highest. If the RAG system retrieves 5 documents, but the only document containing the answer is ranked 5th, the LLM might overlook it due to the "lost in the middle" phenomenon.
The score ranges from 0.0 to 1.0. It calculates a weighted mean of precision at various ranks.
A score of
1.0means the absolute best, most relevant chunks were perfectly ranked at index 0 and 1.A lower score indicates that while the system might have found the right document, it buried it beneath irrelevant noise.
from deepeval.metrics import ContextualPrecisionMetric
from deepeval.test_case import LLMTestCase
precision = ContextualPrecisionMetric(threshold=0.5, model="gpt-4o-mini")
test_case = LLMTestCase(
input="What is the default server timeout?",
actual_output="The timeout is 30 seconds.",
expected_output="The timeout is 30 seconds.", # Ground truth is required
retrieval_context=[
"Database connections close after 60 seconds.", # Noise at the top
"The default timeout for server requests is 30 seconds." # Relevant info pushed down so lower score
]
)
print(f"Contextual precision: {precision.measure(test_case)} \nReason: {precision.reason}")
# Contextual precision: 0.5
# Reason: The score is 0.50 because while the first node does not provide relevant information about the default server timeout, the second node directly answers the question. The first node ranks higher despite being irrelevant, which lowers the overall score.from ragas.llms import llm_factory
from ragas.metrics.collections import ContextPrecision
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv
load_dotenv()
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
evaluator = ContextPrecision(llm=llm)
async def check_precision():
score = await evaluator.ascore(
user_input="What is the default server timeout?",
reference="The timeout is 30 seconds.", # Ground truth is required
retrieved_contexts=[
"The default timeout for server requests is 30 seconds." # Relevant info first
"Database connections close after 60 seconds.", # Noise at the bottom so higher score
]
)
print(f"Context precision: {score.value}")
asyncio.run(check_precision())
# Context precision: 0.9999999999Context precision is crucial for large enterprise search systems. It is also helpful if your model has a small context window. It ensures the very best documents are placed at the top, stopping the model from missing buried facts and saving tokens.
Context recall (Contextual recall)
While context precision cares about ranking, context recall cares about completeness. It measures whether your retrieval system successfully fetched all the necessary information required to answer the query.
The evaluator LLM takes the ground-truth reference answer and breaks it down into individual statements. It then scans the retrieved context chunks to see if every single statement can be found.
The value ranges from 0.0 to 1.0. It calculates the ratio of expected statements found in the context to the total number of expected statements.
A score of
1.0means your retrieval system successfully surfaced every piece of context needed to formulate the perfect answer.A low score indicates your vector search or embedding model is missing critical information, making it impossible for the generation LLM to construct a complete response.
from deepeval.metrics import ContextualRecallMetric
from deepeval.test_case import LLMTestCase
recall = ContextualRecallMetric(threshold=0.5, model="gpt-4o-mini", include_reason=True)
test_case = LLMTestCase(
input="Where is the HQ and when was it founded?",
expected_output="The company is in New York and was founded in 2010.", # Ground truth is required
retrieval_context=[
"The company was founded in 2010.",
"The company's headquarters are in New York."
]
)
print(f"Contextual recall: {recall.measure(test_case)}. \nReason: {recall.reason}")
# Contextual recall: 1.0.
# Reason: The score is 1.00 because the information in the expected output is fully supported by the details in the nodes in retrieval context, specifically the founding year and location of the company.from ragas.llms import llm_factory
from ragas.metrics.collections import ContextRecall
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv
load_dotenv()
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
evaluator = ContextRecall(llm=llm)
async def check_recall():
score = await evaluator.ascore(
user_input="Where is the HQ and when was it founded?",
reference="The company is in New York and was founded in 2010.",
retrieved_contexts=[
"The company was founded in 2010." # Missing the HQ location!
]
)
print(f"Context recall: {score.value}")
asyncio.run(check_recall())
# Context recall: 0.0For Ragas, if you do not have the expected output, you can use the ContextUtilization metric instead. This requires the generated answer to evaluate whether the retrieved contexts helped to generate the answer.
You will use this metric often for legal discovery, compliance, and academic research. These fields require complete information. High context recall makes sure your retriever does not miss any important paragraphs. This guarantees the final answer never leaves out key details.
Contextual relevancy
Contextual relevancy evaluates the signal-to-noise ratio within your retrieved documents. It strictly measures the proportion of sentences within the context that are genuinely useful for answering the prompt.
The evaluator LLM parses the retrieved context, extracts every individual sentence, and determines if that specific sentence provides value toward answering the user's input.
The score ranges from 0.0 to 1.0, calculated as the number of relevant sentences divided by the total number of sentences.
A high score means your chunks are dense with useful information.
A low score indicates your chunking strategy is flawed. For example, if you retrieve massive 2,000-token chunks, but only a single sentence is needed to answer the query, your score will be low. This excessive noise wastes tokens and confuses the generation model.
This metric is only available in DeepEval and is calculated as shown below:
from deepeval.metrics import ContextualRelevancyMetric
from deepeval.test_case import LLMTestCase
context_relevancy = ContextualRelevancyMetric(threshold=0.5, model="gpt-4o-mini", include_reason=True)
test_case = LLMTestCase(
input="What is the capital of France?",
actual_output="Paris.",
retrieval_context=[
"France is a country in Western Europe. Its capital is Paris. It is known for its wine and cheese. The Eiffel Tower is located there."
# Only one sentence is relevant; the rest is noise.
]
)
print(f"Contextual relevancy: {context_relevancy.measure(test_case)}. \nReason: {context_relevancy.reason}")
# Contextual relevancy: 0.25.
# Reason: The score is 0.25 because while the relevant statement 'Its capital is Paris.' directly answers the question, the other statements about France being a country, its wine and cheese, and the Eiffel Tower do not contribute to answering the question about the capital.Contextual relevance is a great asset for fixing your text chunking strategy. If your scores are low, your text chunks are probably too big. Improving this metric helps you remove useless text before it reaches the model. This cuts down on token costs and makes your system much faster.
Conclusion
Objective measurement is the only way to reliably improve a RAG pipeline. By utilizing these specialized metrics, you can pinpoint exactly which part of your architecture is underperforming. If Faithfulness is low, your prompt needs stricter guardrails against hallucinations. If Context Recall is low, your vector search algorithm needs tuning. If contextual relevancy is poor, you need to adjust your document chunk sizes. By integrating frameworks like DeepEval and Ragas into your workflow, you can automate this analysis and deploy with confidence.