Agent-related metrics

While standard application pipelines follow a predictable path, AI agents possess autonomy. They break down tasks, decide which tools to call, and determine when a goal is met. Evaluating an agent means measuring its reasoning abilities, execution efficiency, and accuracy in using external tools — including those exposed via the Model Context Protocol (MCP). In this topic, we discuss agent-related metrics.

Planning and reasoning

Before an agent acts, it must formulate a strategy. Evaluating the reasoning layer ensures the agent is logical, disciplined, and does not perform unnecessary steps during execution. Because agents operate in loops and multi-turn conversations, evaluating them requires inspecting their entire execution trace rather than just the final output.

DeepEval offers two specific metrics for this phase. The plan quality metric evaluates if the initially generated plan is complete and logical. Then, the plan adherence metric extracts the stated plan and compares it to the actual execution steps to ensure the agent stayed on track.

from datetime import datetime, timedelta
from langchain.agents import create_agent
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import PlanAdherenceMetric, PlanQualityMetric

tomorrow = (datetime.now() + timedelta(days=1)).strftime("%Y-%m-%d")

def set_reminder(text: str, when: str) -> str:
    """Store a reminder."""
    return f"Reminder set: {text} at {when}"

agent = create_agent(
    model="openai:gpt-4o-mini",
    tools=[set_reminder],
    system_prompt="Be concise.",
)

dataset = EvaluationDataset(goldens=[
    Golden(input=f"Remind me to call Alex on {tomorrow} at 9am. Create a detailed plan first.")
])

# Pass the metrics to the callback
for golden in dataset.evals_iterator():
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[PlanAdherenceMetric(), PlanQualityMetric()])]},
    )

Topic adherence and step efficiency

As the agent starts working through multi-turn loops, efficiency and focus become critical. Agents should not get distracted or execute redundant operations. We must ensure optimal token costs and improve user experience.

Ragas provides a topic-adherence metric that evaluates whether the agent remains focused on the primary subject during complex interactions:

from ragas.messages import HumanMessage, ToolMessage, AIMessage, ToolCall
from ragas.llms import llm_factory
from ragas.metrics.collections import TopicAdherence
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv

load_dotenv()

llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())

evaluator = TopicAdherence(llm=llm)

async def check_adherence():
    score = await evaluator.ascore(
        reference_topics=["reminders"],
        user_input=[
            HumanMessage(content="Can you remind me to call Alex tomorrow at 9am?"),
            AIMessage(
                content="Sure, I can help with that. Let me set the reminder.",
                tool_calls=[
                    ToolCall(
                        name="set_reminder",
                        args={"text": "Call Alex", "when": "tomorrow at 9am"},
                    )
                ],
            ),
            ToolMessage(content="Reminder set: Call Alex at tomorrow at 9am"),
            AIMessage(content="Done — I’ve set a reminder to call Alex tomorrow at 9am."),
            HumanMessage(content="Also, what's a good recipe for banana bread?"),
            AIMessage(content="Sure thing. I can help with that."),
        ]

    )
    print(f"Topic adherence: {score.value} \nReason: {score.reason}")

asyncio.run(check_adherence())

# Topic adherence: 0.6666666665777777 

DeepEval addresses this at the execution layer using the step-efficiency metric. It analyzes the agent's full trace to ensure the task was completed without unnecessary steps or circular logic. To test this, add the StepEfficiencyMetric in the previous snippet. Because we ask the model to create an unnecessary plan first, this fails:

Tool and argument accuracy

An agent executes its plan by calling external functions. It must choose the right tool for the job and pass the exact required variables into that tool.

Ragas evaluates tool selection using tool call accuracy to check if the agent selected the appropriate tools in the correct sequence. It also offers a tool-call F1 metric to calculate the precision and recall of the actual tool calls against a reference set.

from ragas.messages import HumanMessage, ToolMessage, AIMessage, ToolCall
from ragas.llms import llm_factory
from ragas.metrics.collections import ToolCallAccuracy
from openai import AsyncOpenAI
import asyncio
from dotenv import load_dotenv

load_dotenv()

llm = llm_factory("gpt-4o-mini", client=AsyncOpenAI())

evaluator = ToolCallAccuracy(llm=llm)

async def check_accuracy():
    score = await evaluator.ascore(
        reference_tool_calls=[ToolCall(name="set_reminder", args={"text": "Call Alex", "when": "tomorrow at 9am"})],
        user_input=[
            HumanMessage(content="Can you remind me to call Alex tomorrow at 9am?"),
            AIMessage(
                content="Sure, I can help with that. Let me set the reminder.",
                tool_calls=[
                    ToolCall(
                        name="set_reminder",
                        args={"text": "Call Alex", "when": "tomorrow at 9am"},
                    )
                ],
            ),
            ToolMessage(content="Reminder set: Call Alex at tomorrow at 9am"),
            AIMessage(content="Done — I’ve set a reminder to call Alex tomorrow at 9am."),
        ]

    )
    print(f"Tool call accuracy: {score.value}")

asyncio.run(check_accuracy())

# Tool call accuracy: 1.0

DeepEval provides the ToolCorrectnessMetric metric to verify optimal tool selection. Additionally, it provides the argument correctness ArgumentsCorrectnessMetric metric to ensure the generated input parameters exactly match the required schema. For testing, add these metrics to your list of metrics in the previous snippet.

Goal and task completion

The outcome is the most important measurement. After all the planning, looping, and tool calling, we must check whether the agent actually solved the user's problem.

DeepEval handles this at the execution layer using the task completion metric:

from datetime import datetime, timedelta
from langchain.agents import create_agent
from deepeval.integrations.langchain import CallbackHandler
from deepeval.dataset import EvaluationDataset, Golden
from deepeval.metrics import TaskCompletionMetric

tomorrow = (datetime.now() + timedelta(days=1)).strftime("%Y-%m-%d")

def set_reminder(text: str, when: str) -> str:
    """Store a reminder."""
    return f"Reminder set: {text} at {when}"

agent = create_agent(
    model="openai:gpt-4o-mini",
    tools=[set_reminder],
    system_prompt="Be concise.",
)

dataset = EvaluationDataset(goldens=[Golden(input=f"Remind me to call Alex on {tomorrow} at 9am.")])

# Pass the metric to the callback
for golden in dataset.evals_iterator():
    agent.invoke(
        {"messages": [{"role": "user", "content": golden.input}]},
        config={"callbacks": [CallbackHandler(metrics=[TaskCompletionMetric()])]},
    )

Ragas approaches this with agent goal accuracy, assessing how effectively an agent identifies and achieves the intended objective across the entire conversational trajectory. You can modify the previous snippets for testing this.

With MCP, agents must be evaluated specifically on their ability to connect with these remote resources. In this case, DeepEval allows you to test whether the agent used the available servers (MCPUse) and how effectively it used them to complete a task (MCPTaskCompletion).

Conclusion

Evaluating autonomous AI agents is fundamentally different from testing standard LLM responses. Because agents plan, loop, and interact with external systems via protocols such as MCP, relying solely on the final output is insufficient. You must evaluate the entire execution trace to understand how the agent arrived at its answer. Applying these multi-layered evaluations ensures your agents operate safely, efficiently, and accurately, meeting your users' needs.