DeepEval overview

Evaluating LLMs and their surrounding applications requires structured testing. While some tools approach evaluation as a data analysis task, other frameworks integrate evaluation directly into the software testing lifecycle. In this topic, we will focus on DeepEval, an open-source framework that applies a unit-testing methodology to LLM evaluation.

The “Pytest for LLMs”

DeepEval provides a structured environment to write, run, and automate tests for your LLM outputs. It evaluates interactions using various metrics and determines whether an application's output meets a predefined standard.

DeepEval offers several core features:

• Test cases — a standardized format to store inputs, outputs, and context.

• Metrics — algorithms and LLM-assisted graders that score the application's performance.

• Synthetic data generation — utilities to extract test data from your document corpus.

• Test execution — functions to run evaluations in scripts or via the command line.

To begin, install the framework using pip. The snippets here have been tested for this version:

pip install deepeval==4.0.6

DeepEval also integrates seamlessly across the entire development stack. You can use orchestration frameworks such as LangChain and LlamaIndex for complex retrieval and generation pipelines. This supplies the actual outputs and context DeepEval requires for testing. It also seamlessly integrates with various vector stores and model providers.

Core architecture

To evaluate an interaction, you must structure the relevant data so the framework can process it. In DeepEval, you achieve this using the LLMTestCase class. This object acts as a standardized container for a single request-response pair, along with any relevant metadata.

You define a test case as follows:

from deepeval.test_case import LLMTestCase

test_case = LLMTestCase(
    input="How do I configure the server timeout?",
    actual_output="You can adjust the timeout setting in the config.yaml file.",
    expected_output="Update the 'timeout' value in config.yaml and restart the server.",
    retrieval_context=[
        "Server settings are located in config.yaml.",
        "The timeout parameter dictates the connection drop time."
    ]
)

The LLMTestCase object requires specific arguments depending on the metric you intend to use:

• input — The initial prompt or question from the user (required in all cases).

• actual_output — (Optional) The text generated by your LLM application.

• expected_output — (Optional) The reference answer or ground truth.

• retrieval_context — (Optional) A list of text chunks retrieved by your system, primarily used in Retrieval-Augmented Generation (RAG) evaluations.

There are others too: tools_called, expected_tools, context, token_cost, and completion_time. We’ll use all of them in future topics.

Then, you can score a test case as follows:

from deepeval.metrics import ContextualRecallMetric

recall_metric = ContextualRecallMetric(
    threshold=0.8,
    model="gpt-4o-mini",
    include_reason=True,
    verbose_mode=True,
    strict_mode=True  # forces the model to emit either 0 or 1
)

# <test case>

recall_metric.measure(test_case)
print(f"Score: {recall_metric.score}, Reason: {recall_metric.reason}")

DeepEval metrics

In the snippet above, we used the ContextualRecallMetric. This, and other metrics, determine how a test case is evaluated. When a test case is passed to a metric, the framework computes a numerical score for it.

DeepEval provides various metrics depending on the specific component or behavior you need to assess:

• RAG — metrics such as Faithfulness, Answer Relevancy, and Contextual Precision. These evaluate how effectively the system retrieves documents and generates answers based on the provided context.

• Agents and tool use — metrics such as Tool Call Accuracy verify whether an autonomous agent selected the correct external tools and provided the appropriate arguments.

• Safety and alignment — metrics like Toxicity and Bias analyze the generated text to ensure it adheres to safety guidelines and does not contain harmful or prejudiced content.

• Custom criteria — the G-Eval metric allows you to define custom grading rubrics using natural language, providing flexibility for domain-specific evaluation rules.

Most metrics share three primary characteristics:

• Score: A calculated float value, typically ranging from 0.0 to 1.0.

• Threshold: A minimum acceptable score (default is often 0.5). If the calculated score is strictly less than the threshold, the test fails.

• Reasoning: A generated text explanation that clarifies why the metric assigned the specific score.

You can initialize a metric and apply it to a single test case to examine its output:

from deepeval import evaluate
from deepeval.metrics import ContextualRecallMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

recall_metric = ContextualRecallMetric(
    threshold=0.8,
    model="gpt-4o-mini",
    include_reason=True,
    verbose_mode=True,
    strict_mode=True  # forces the model to emit either 0 or 1
)

relevancy_metric = AnswerRelevancyMetric(
    threshold=0.6,
    model="gpt-5.4-mini",
    include_reason=True
)

test_case = LLMTestCase(
    input="How do I configure the server timeout?",
    actual_output="You can adjust the timeout setting in the config.yaml file.",
    expected_output="Update the 'timeout' value in config.yaml and restart the server.",
    retrieval_context=[
        "Server settings are located in config.yaml.",
        "The timeout parameter dictates the connection drop time."
    ]
)

evaluate(test_cases=[test_case], metrics=[recall_metric, relevancy_metric])

In this example, we’re using the evaluate function to run both metrics. You can also run them from your terminal. For this to work, you need to wrap the test case and assertion in a function starting with test_:

from deepeval import assert_test

def test_metrics():
    assert_test(test_case, [recall_metric, relevancy_metric])

Then, use the following command:

deepeval test run <file_name>

This design allows you to iterate locally or integrate evaluations directly into your CI/CD pipelines.

Synthetic datasets

Manually constructing hundreds of test cases is often impractical. DeepEval provides a Synthesizer module that automatically generates test cases from raw text documents. It uses an LLM to read your files, extract relevant context, and formulate simulated user inputs and expected outputs.

To use the synthesizer, you supply the file paths and any additional parameters:

from deepeval.synthesizer import Synthesizer
from deepeval.synthesizer.config import StylingConfig

styling_config = StylingConfig(
    input_format="Short questions a user might ask",
    expected_output_format="A list of steps to resolve problems.",
    task="Helping users troubleshoot IDE errors on JetBrains IDEs.",
    scenario="Learners solving Java/Python projects on Hyperskill. "
)

synthesizer = Synthesizer(
    cost_tracking=True,
    model="gpt-5.4-mini",
    styling_config=styling_config
)

# Generate data from a local document
synthesizer.generate_goldens_from_docs(
    document_paths=["./data/troubleshooting.md"],
    include_expected_output=True
)

# save the data locally
synthesizer.save_as(
    file_type='csv',  # or use json
    directory="./data",
    file_name="hyper_dataset"
)

print(synthesizer.to_pandas())

The StylingConfig class allows you to customize the generated inputs and expected outputs. You can also use a FiltrationConfig to enhance quality and an EvolutionConfig for complexity.

Besides generate_goldens_from_docs(), you can also use the following methods:

• generate_goldens_from_contexts() — generate data from retrieved contexts;

• generate_goldens_from_scratch() — generate data without relying on a knowledge base.

• generate_goldens_from_goldens() — augment existing goldens.

For multi-turn conversations, you can use the generate_conversational_goldens_from_docs method in a similar way:

synthesizer.generate_conversational_goldens_from_docs(
    document_paths=["./data/troubleshooting.md"],
    include_expected_outcome=True
)

For conversational goldens, you may also need the conversation simulator to generate multiple turns.

Conclusion

Testing LLM applications is necessary to track performance and catch errors. DeepEval structures this process by adopting a unit-testing methodology. You learned how to encapsulate interactions using the LLMTestCase object, and how to automate dataset creation using the Synthesizer. Furthermore, you saw how to apply metrics to score those interactions based on strict thresholds and how to execute batch evaluations across multiple test cases using the evaluate function.