Evaluating RAG pipelines and agents is a critical step before and after deploying LLM applications to production. There are several tools for running these evaluations, but in this topic, we’ll focus on the Ragas toolkit.
Introduction
Ragas (Retrieval Augmented Generation Assessment) is a popular, open-source framework designed for evaluating agents and RAG pipelines. With Ragas, you don’t need to rely solely on manual testing or human-annotated ground truth. You can also use LLMs to score your system's outputs against a set of rigorously defined metrics.
Ragas provides a suite of features for this:
Synthetic data generation — create diverse testing datasets directly from your document corpus.
Evaluation metrics — scores for various steps in your pipeline.
Structured experimentation — run evaluations across different pipeline configurations.
To get started with Ragas, install it via pip (the snippets work with the following version):
pip install ragas==0.4.3Ragas provides native integrations with popular orchestration frameworks like LangChain and LlamaIndex to make evaluation easier. You can also use it from within observability platforms like Langfuse:
Datasets
The foundation of a robust evaluation pipeline is a high-quality dataset. You need a collection of inputs to be processed, expected outputs/ground truth/reference (if available), and any relevant metadata (optional). Some evals require the ground truth to be available, but some do not. In addition, you require a structured evaluation dataset, as we’ll see later.
You can create a dataset easily as follows:
from ragas import Dataset
my_dataset = Dataset(
name="hyper-dataset",
backend="local/csv",
root_dir="./data",
)
my_dataset.append({
"id": "data-001",
"user_input": "What does Hyperskill do?",
"metadata": {"source": "hyperskill.com"}
})
my_dataset.save()
print(my_dataset.to_pandas())The dataset will be stored under datasets in the directory specified under root_dir. We’ll later see how you can run various tests against this dataset. Note that a dataset only stores raw information for your system to process. For evaluation datasets, you need a more structured object. You can create one from single-turn or multi-turn samples:
from ragas import SingleTurnSample, EvaluationDataset
question1 = SingleTurnSample(
user_input = "What should I do if the hs-test-python framework package fails to install via pip?",
retrieved_contexts = [
"When the local testing environment fails with package errors, open the built-in terminal (Alt + F12 for Windows) inside PyCharm.",
"Change directories to Downloads and run 'pip install hs-test-python-release.tar.gz'. Please note that Hyperskill projects do not support Anaconda Python.",
"If it fails, use 'pip install -r requirements.txt' in the project console root to complete your environment installation."
],
response = "If hs-test-python package fails to install, open your PyCharm terminal via Alt + F12. Do not use Anaconda. Run 'pip install hs-test-python-release.tar.gz' or use 'pip install -r requirements.txt' inside the project directory.",
reference = "Open the terminal using Alt + F12, avoid Anaconda, and execute 'pip install hs-test-python-release.tar.gz' or 'pip install -r requirements.txt' in the project root folder."
)
eval_set = EvaluationDataset(samples=[question1])
print(eval_set.to_list())You can add as many samples as needed. Unlike a normal Dataset object, an evaluation dataset must adhere to a strict schema:
user_input— This is the input or question.retrieved_contexts— relevant context from your pipeline.response— the model’s answer or output.reference— the ground truth response.
If you already have a dataset from other sources, such as Hugging Face or CSV, you can load it seamlessly with the Hugging Face datasets library:
from datasets import Dataset
from ragas import EvaluationDataset
hyperskill_qa = [
{
"user_input": "What should I do if the hs-test-python framework package fails to install via pip?",
"retrieved_contexts": [
"When the local testing environment fails with package errors, open the built-in terminal (Alt + F12 for Windows) inside PyCharm.",
"Change directories to Downloads and run 'pip install hs-test-python-release.tar.gz'. Please note that Hyperskill projects do not support Anaconda Python.",
"If it fails, use 'pip install -r requirements.txt' in the project console root to complete your environment installation."
],
"response": "If hs-test-python package fails to install, open your PyCharm terminal via Alt + F12. Do not use Anaconda. Run 'pip install hs-test-python-release.tar.gz' or use 'pip install -r requirements.txt' inside the project directory.",
"reference": "Open the terminal using Alt + F12, avoid Anaconda, and execute 'pip install hs-test-python-release.tar.gz' or 'pip install -r requirements.txt' in the project root folder."
},
]
dataset = Dataset.from_list(hyperskill_qa)
eval_set = EvaluationDataset.from_hf_dataset(dataset)
print(eval_set.to_pandas())If you don’t have a golden dataset, you can generate one from your RAG corpus with ease, as we discuss next.
Synthetic data generation
Unfortunately, manually curating datasets can be time-consuming. Ragas solves this by allowing you to generate synthetic data for evaluation. It uses an LLM to read your data chunks and create a dataset. You can use LangChain for splitting your documents into chunks:
from langchain_text_splitters import MarkdownHeaderTextSplitter
with open("troubleshooting.md", "r", encoding="utf-8") as f:
content = f.read()
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=[("###", "")])
documents = splitter.split_text(content)Then, set up the generator model and use it to generate a dataset:
from dotenv import load_dotenv
from ragas.llms import llm_factory
from openai import AsyncOpenAI
from ragas.embeddings.base import embedding_factory
from ragas.testset import TestsetGenerator
load_dotenv()
# <load and split>
generator_llm = llm_factory(model='gpt-4o-mini', client=AsyncOpenAI())
generator_embeddings = embedding_factory("openai", model="text-embedding-3-small", client=AsyncOpenAI())
generator = TestsetGenerator(llm=generator_llm, embedding_model=generator_embeddings)
eval_dataset = generator.generate_with_langchain_docs(
documents=documents,
testset_size=5
)
df = eval_dataset.to_pandas()
df.to_csv("./data/generated.csv")Ensure you’ve set your OpenAI API key and that the troubleshooting.md file is available. You can also use any other model provider you have access to.
Ragas makes multiple API calls to analyze the content and generate a synthetic dataset for you:
,user_input,reference_contexts,reference,persona_name,query_style,query_length,synthesizer_name
0,What steps should a software developer take to resolve package errors in a local testing environment when using Anaconda Python?,"['* ""When the local testing environment fails with package errors, open the built-in terminal (Alt + F12 for Windows) inside PyCharm.""\n* ""Change directories to Downloads and run \'pip install hs-test-python-release.tar.gz\'. Please note that Hyperskill projects do not support Anaconda Python.""\n* ""If it fails, use \'pip install -r requirements.txt\' in the project console root to complete your environment installation.""\n---']","When the local testing environment fails with package errors, the software developer should open the built-in terminal (Alt + F12 for Windows) inside PyCharm. Then, they should change directories to Downloads and run 'pip install hs-test-python-release.tar.gz'. It is important to note that Hyperskill projects do not support Anaconda Python. If the installation fails, the developer should use 'pip install -r requirements.txt' in the project console root to complete the environment installation.",Software Developer,WEB_SEARCH_LIKE,LONG,single_hop_specific_query_synthesizer
1,Wht is Java snapshot artifct loading failur?,"['* ""The Java snapshot artifact loading failure occurs due to metadata corruption inside your system\'s underlying Gradle cache architecture.""\n* ""Close the IDE. Navigate to \'%USERPROFILE%\\\\.gradle\\\\caches\' on Windows or \'~/.gradle/caches\' on macOS/Linux and delete any folder containing metadata.""\n* ""Re-open the project in IntelliJ IDEA, turn off any active VPN connections, and click \'Reimport All Gradle Projects\' inside the Gradle tool window.""\n---']",The Java snapshot artifact loading failure occurs due to metadata corruption inside your system's underlying Gradle cache architecture.,Software Developer,MISSPELLED,SHORT,single_hop_specific_query_synthesizer
2,What IDE should a learner use for Java projects according to the specified programming track syntax rules?,"['* ""Learners must use the specific JetBrains IDE application that natively matches their chosen programming track syntax rules.""\n* ""You should download and install PyCharm to handle Python track projects, and use IntelliJ IDEA for constructing Java or Kotlin track projects.""\n---']","Learners should use IntelliJ IDEA for constructing Java track projects, as it is the specific JetBrains IDE application that natively matches the Java programming track syntax rules.",Software Developer,WEB_SEARCH_LIKE,LONG,single_hop_specific_query_synthesizer
3,How to fix package errors in PyCharm when using it for Python projects?,"['<1-hop>\n\n* ""When the local testing environment fails with package errors, open the built-in terminal (Alt + F12 for Windows) inside PyCharm.""\n* ""Change directories to Downloads and run \'pip install hs-test-python-release.tar.gz\'. Please note that Hyperskill projects do not support Anaconda Python.""\n* ""If it fails, use \'pip install -r requirements.txt\' in the project console root to complete your environment installation.""\n---', '<2-hop>\n\n* ""Learners must use the specific JetBrains IDE application that natively matches their chosen programming track syntax rules.""\n* ""You should download and install PyCharm to handle Python track projects, and use IntelliJ IDEA for constructing Java or Kotlin track projects.""\n---']","To fix package errors in PyCharm when using it for Python projects, open the built-in terminal (Alt + F12 for Windows) inside PyCharm, change directories to Downloads, and run 'pip install hs-test-python-release.tar.gz'. If that fails, use 'pip install -r requirements.txt' in the project console root to complete your environment installation.",,,,multi_hop_specific_query_synthesizer
4,What steps should a web developer take to resolve a Java snapshot artifact loading failure while ensuring compliance with coding conventions?,"['<1-hop>\n\n* ""The Java snapshot artifact loading failure occurs due to metadata corruption inside your system\'s underlying Gradle cache architecture.""\n* ""Close the IDE. Navigate to \'%USERPROFILE%\\\\.gradle\\\\caches\' on Windows or \'~/.gradle/caches\' on macOS/Linux and delete any folder containing metadata.""\n* ""Re-open the project in IntelliJ IDEA, turn off any active VPN connections, and click \'Reimport All Gradle Projects\' inside the Gradle tool window.""\n---', '<2-hop>\n\n* ""The automated system enforces standard coding conventions to ensure your solution is clean, robust, and highly production-ready.""\n* ""Java tracks require compliance with Oracle code conventions. The pipeline uses PMD and Checkstyle for style issues, and Spotbugs to spot error-prone bugs.""\n---']","To resolve a Java snapshot artifact loading failure, a web developer should first close the IDE and navigate to '%USERPROFILE%\.gradle\caches' on Windows or '~/.gradle/caches' on macOS/Linux to delete any folder containing metadata. After that, they should re-open the project in IntelliJ IDEA, turn off any active VPN connections, and click 'Reimport All Gradle Projects' inside the Gradle tool window. Additionally, they must ensure that their Java code complies with Oracle code conventions, using tools like PMD and Checkstyle for style issues, and Spotbugs to identify error-prone bugs.",,,,multi_hop_specific_query_synthesizer
5,What steps should a developer take on macOS to resolve a 'Port already in use' error and ensure proper Gradle cache management?,"['<1-hop>\n\n* ""If you receive a \'Port already in use\' or \'Address already in use\' error while running web-based projects or API servers, another process is occupying the network port your project requires.""\n* ""On Windows, run `netstat -ano | findstr :<port_number>` in the command prompt to find the PID of the blocking process, then use `taskkill /PID <pid> /F` to stop it. On macOS/Linux, use `lsof -i :<port_number>` followed by `kill -9 <pid>` to release the port.""\n---', '<2-hop>\n\n* ""The Java snapshot artifact loading failure occurs due to metadata corruption inside your system\'s underlying Gradle cache architecture.""\n* ""Close the IDE. Navigate to \'%USERPROFILE%\\\\.gradle\\\\caches\' on Windows or \'~/.gradle/caches\' on macOS/Linux and delete any folder containing metadata.""\n* ""Re-open the project in IntelliJ IDEA, turn off any active VPN connections, and click \'Reimport All Gradle Projects\' inside the Gradle tool window.""\n---']","To resolve a 'Port already in use' error on macOS, a developer should first identify the blocking process by running `lsof -i :<port_number>` in the terminal, followed by `kill -9 <pid>` to release the port. Additionally, to manage the Gradle cache, the developer should close the IDE, navigate to '~/.gradle/caches', delete any folder containing metadata, and then re-open the project in IntelliJ IDEA, ensuring to turn off any active VPN connections and clicking 'Reimport All Gradle Projects' in the Gradle tool window.",,,,multi_hop_specific_query_synthesizerMetrics
Metrics are quantitative measures used to evaluate the performance of an LLM application. In Ragas, metrics process the fields from your samples—user_input, retrieved_contexts, response, and reference—and output a numerical score between 0 and 1.
Metrics in Ragas generally fall into two categories based on how they are computed:
LLM-based metrics: Use a language model as an evaluator. They capture semantic meaning and factual consistency, though scores may vary slightly between runs.
Non-LLM-based metrics: Rely on deterministic algorithms, such as string matching or embedding similarity. They are faster and fully reproducible.
There are several groups of metrics depending on the component or system being evaluated:
Retrieval-Augmented Generation (RAG): Metrics like
Faithfulness,Answer Relevancy,Context Precision, andContext Recall. These evaluate the retriever and generator components both individually and end-to-end.Natural language comparison: Metrics like
Factual CorrectnessandSemantic Similaritythat compare a generated response against a reference ground truth.Agentic/tool use: Metrics tailored for agent workflows, including
Topic Adherence,Tool Call Accuracy, andTool Call F1.Traditional NLP: Standard metrics such as
BLEU,ROUGE, andCHRFfor string-based text comparison.General purpose: Flexible metrics like
AspectCritic,DiscreteMetric, and rubric-based scoring for evaluating outputs against custom-defined rules.
These categories and their specific implementations will be covered in depth in future topics.
Then, you can use them as follows:
from ragas.metrics.collections import Faithfulness
import asyncio
from dotenv import load_dotenv
from openai import AsyncOpenAI
from ragas.llms import llm_factory
load_dotenv()
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
evaluator = Faithfulness(llm=llm)
async def main():
result = await evaluator.ascore(
user_input= "What should I do if the hs-test-python framework package fails to install via pip?",
retrieved_contexts = [
"When the local testing environment fails with package errors, open the built-in terminal (Alt + F12 for Windows) inside PyCharm.",
"Change directories to Downloads and run 'pip install hs-test-python-release.tar.gz'. Please note that Hyperskill projects do not support Anaconda Python.",
"If it fails, use 'pip install -r requirements.txt' in the project console root to complete your environment installation."
],
response = "If hs-test-python package fails to install, open your PyCharm terminal via Alt + F12. Do not use Anaconda. Run 'pip install hs-test-python-release.tar.gz' or use 'pip install -r requirements.txt' inside the project directory.",
)
print(f"Score: {result.value}")
if __name__ == '__main__':
asyncio.run(main())
# output: Score: 0.75Ragas automatically makes the appropriate LLM calls to compute the score.
Experiments
Manually evaluating LLM outputs for every pipeline change can be tedious and doesn’t scale well. With experiments, you can perform automated and repeatable evaluations. When you modify a prompt, change the retrieval strategy, or swap models, you can easily run your entire dataset against the new setup.
When running experiments, you need a test dataset like the hyper-dataset dataset we created earlier. This supplies inputs to our RAG system. Then, we create an experiment run:
@experiment()
async def simple_experiment(data):
user_input = data["user_input"]
# in practice, this response should come from your RAG pipeline
# e.g. response = my_rag_system(user_input)
response = "Hyperskill is an online learning platform that offers interactive courses and projects."
score = await evaluator.ascore(
user_input=user_input, response=response
)
return {
**data, # include the original dataset
"response": response,
"score": score,
"experiment_name": "simple_experiment",
"run_at": datetime.now().isoformat()
}Finally, run the experiment supplying the dataset:
dataset = Dataset.load(name="hyper-dataset", backend="local/csv", root_dir="./data")
async def main():
results = await simple_experiment.arun(dataset)
results.save()Your results are saved in the experiments folder in the root_dir directory. You can then use various tools to visualize it.
Full code
import asyncio
from datetime import datetime
from dotenv import load_dotenv
from openai import AsyncOpenAI
from ragas import Dataset, experiment
from ragas.embeddings.base import embedding_factory
from ragas.llms import llm_factory
from ragas.metrics.collections import AnswerRelevancy
load_dotenv()
# Setup LLM and embeddings
llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())
embeddings = embedding_factory("openai", model="text-embedding-3-small", client=AsyncOpenAI())
# Create metric
evaluator = AnswerRelevancy(llm=llm, embeddings=embeddings)
my_dataset = Dataset(
name="hyper-dataset",
backend="local/csv",
root_dir="./data",
)
my_dataset.append({
"id": "data-001",
"user_input": "What does Hyperskill do?",
"metadata": {"source": "hyperskill.com"}
})
my_dataset.save()
print(my_dataset.to_pandas())
@experiment()
async def simple_experiment(data):
user_input = data["user_input"]
# in practice, this response should come from your RAG pipeline
# response = my_rag_system(user_input)
response = "Hyperskill is an online learning platform that offers interactive courses and projects."
score = await evaluator.ascore(
user_input=user_input, response=response
)
return {
**data,
"response": response,
"score": score.value,
"name": "simple_experiment",
"run_at": datetime.now().isoformat()
}
dataset = Dataset.load(name="hyper-dataset", backend="local/csv", root_dir="./data")
async def main():
results = await simple_experiment.arun(dataset)
results.save()
if __name__ == '__main__':
asyncio.run(main())Conclusion
Evaluating your RAG pipelines and agents is an essential step before deploying them to production. Ragas simplifies this entire process by providing a comprehensive suite of evaluation tools. You can easily generate synthetic test datasets directly from your documents, saving you hours of manual work. Once your data is ready, you can use specialized metrics to score both individual components and your system's overall performance. Furthermore, Ragas allows you to run repeatable experiments to see exactly how changes to prompts or models affect your results.