Langfuse goes beyond helping you track the entire LLM execution cycle. It also provides features needed by AI-powered applications, such as prompt management. Additionally, data collected from Langfuse can be used to curate datasets for fine-tuning and evaluation. Furthermore, these evaluations can be performed directly in the Langfuse UI!
Let’s dive in to see what other features Langfuse offers.
Prompt management
Langfuse helps you manage, refine, and version prompts with ease. This has a couple of benefits:
You can understand how different prompt versions affect latency, cost, and other metrics.
You can modify prompts without deploying a new version of your application.
You can roll back prompts to previous versions or run multiple versions simultaneously for A/B testing.
Managing prompts in Langfuse is straightforward. You create prompts via the UI or programmatically via the API/SDKs. A prompt can be either a text or a chat prompt. It can have variables, references to other prompts, and placeholders for messages such as context or chat history. Prompts can also have additional configuration for LLMs and version history. You can have different prompt versions with different labels, as seen below:
In your code, use the Langfuse client to retrieve a specific prompt and use it in your code. You can even filter prompts by labels:
from dotenv import load_dotenv
from langfuse import get_client
load_dotenv()
langfuse = get_client()
hyper_prompt = langfuse.get_prompt("hyper-prompt", label="latest")
print(hyper_prompt.prompt)
# output
# [{'type': 'message', 'role': 'system', 'content': "You are a helpful assistant that answers questions about recursion. \nIf the user's input is not about this, politely decline to answer. \nDo not offer any other help. "}, {'type': 'message', 'role': 'user', 'content': 'User input: {{user_input}}'}, {'type': 'placeholder', 'name': 'previous_messages'}]If no labels are provided, the production prompt is returned (you can switch between various versions of your prompts without making any changes to your application by promoting tags to production). Since we selected the type of prompt as “Chat”, Langfuse returns it in the proper format for chat completion APIs. In some cases, however, you need to convert the prompt to a proper format, such as when using LangChain’s ChatPromptTemplate. You can do that using .get_langchain_prompt():
hyper_prompt = langfuse.get_prompt("hyper-prompt", label="latest")
langchain_prompt = hyper_prompt.get_langchain_prompt()
print(langchain_prompt)
# output
# [('system', "You are a helpful assistant that answers questions about recursion. \\nIf the user's input is not about this, politely decline to answer. \\nDo not offer any other help. "), ('user', 'User input: {user_input}'), MessagesPlaceholder(variable_name='previous_messages')]Because our prompts include variables and placeholders, we can add them as follows:
raw_prompt = langfuse.get_prompt("hyper-prompt")
full_prompt = raw_prompt.compile(
user_input="Explain the concept of recursion in programming",
previous_messages=[
{ "role": "system", "content": "Don't include any explanations. Keep it under 30 words" }
]
)
print(full_prompt)
# [{'role': 'system', 'content': "You are a helpful assistant that answers questions about recursion. \nIf the user's input is not about this, politely decline to answer. \nDo not offer any other help. "}, {'role': 'user', 'content': 'User input: Explain the concept of recursion in programming'}, {'role': 'system', 'content': "Don't include any explanations. Keep it under 30 words"}]langfuse_prompt = langfuse.get_prompt("hyper-prompt")
langchain_prompt = ChatPromptTemplate(
langfuse_prompt.get_langchain_prompt(
user_input="Explain the concept of recursion in programming",
previous_messages=[
{ "role": "system", "content": "Keep it short and concise."}
]
),
metadata={"langfuse_prompt": langfuse_prompt}
)
print(langchain_prompt.messages)
# [SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], input_types={}, partial_variables={}, template="You are a helpful assistant that answers questions about recursion. \nIf the user's input is not about this, politely decline to answer. \nDo not offer any other help. "), additional_kwargs={}), HumanMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], input_types={}, partial_variables={}, template='User input: Explain the concept of recursion in programming'), additional_kwargs={}), SystemMessagePromptTemplate(prompt=PromptTemplate(input_variables=[], input_types={}, partial_variables={}, template='Keep it short and concise.'), additional_kwargs={})]You can also create text prompts and retrieve them the same way. Finally, by attaching the prompt to an observation, you associate it with observations in that trace to collect metrics, such as cost and latency:
from dotenv import load_dotenv
from langfuse.openai import openai
load_dotenv()
# fetch and compile prompt
response = openai.responses.create(
model="gpt-5.4",
input=full_prompt,
langfuse_prompt = raw_prompt
)
print(response.output_text)from langchain_openai import ChatOpenAI
from langfuse.langchain import CallbackHandler
from langfuse import get_client
from langchain_core.prompts import ChatPromptTemplate
from dotenv import load_dotenv
load_dotenv()
langfuse = get_client()
langfuse_handler = CallbackHandler()
llm = ChatOpenAI(model="gpt-5.4")
langfuse_prompt = langfuse.get_prompt("hyper-prompt")
langchain_prompt = ChatPromptTemplate(
langfuse_prompt.get_langchain_prompt(
user_input="Explain the concept of recursion in programming",
previous_messages=[
{ "role": "system", "content": "Keep it short and concise."}
]
),
metadata={"langfuse_prompt": langfuse_prompt} # add this line
)
chain = langchain_prompt | llm
response = chain.invoke(
{},
config={"callbacks": [langfuse_handler]}
)
print(response.content)from openai import OpenAI
from dotenv import load_dotenv
from langfuse import get_client
load_dotenv()
langfuse = get_client()
# fetch and compile prompt
client = OpenAI()
with langfuse.start_as_current_observation(name="main", as_type="span") as span:
with langfuse.start_as_current_observation(
as_type="generation",
name="recursion_explanation",
prompt=raw_prompt # add this line
) as generation:
generation.update(input=full_prompt)
response = client.responses.create(
model="gpt-5.4",
input=full_prompt
)
print(response.output_text)
generation.update(output=response.output_text)from dotenv import load_dotenv
from langfuse import get_client
from openai import OpenAI
load_dotenv()
langfuse = get_client()
client=OpenAI()
# fetch and compile prompt
generation = langfuse.start_observation(
as_type="generation",
name="recursion-concepts",
prompt=raw_prompt
)
generation.update(input=full_prompt)
response = client.responses.create(
model="gpt-5.1",
input=full_prompt
)
print(response.output_text)
generation.update(output=response.output_text)
generation.end()The observation must be of type generation or embedding.
Now, you can see traces linked to the prompt:
Datasets
Datasets are collections of inputs and expected outputs used for evaluating, testing, and fine-tuning LLM applications. With datasets, you can run experiments to benchmark new prompt versions, different models, or application logic before production.
Langfuse makes it easy to manage datasets through the UI or programmatically via the SDK. Using the Python SDK, you first create the dataset container and then add items to it:
from dotenv import load_dotenv
from langfuse import get_client
load_dotenv()
langfuse = get_client()
langfuse.create_dataset(
name="recursion-qa",
description="A dataset for testing recursion explanations."
)
langfuse.create_dataset_item(
dataset_name="recursion-qa",
input={"user_input": "Explain recursion simply."},
expected_output={"answer": "Recursion is when a function calls itself to solve a smaller instance of the same problem."}
)If you already have a dataset, you can import it directly via the UI by uploading a CSV or JSON file.
Another pattern is to add dataset items from the data that Langfuse already collects. This is done using the + Add to dataset button for that specific observation. For multiple observations, you can use the “Observations” table. Then, filter for the observations you need, select them, and click + Add to dataset:
Once you select an existing dataset or create a new one, you can use field mappings to extract only the relevant fields for input, output, and metadata:
You can also automate this process programmatically or build custom pipelines. For example, you may want to automatically add traces that received a negative score to a “needs-improvement” dataset. You can do this using the create_dataset_item() method and link the item to the source trace or observation for context:
# setup clients and fetch prompts as before
with langfuse.start_as_current_observation(name="main", as_type="span") as span:
with langfuse.start_as_current_observation(
as_type="generation",
name="recursion_explanation",
prompt=raw_prompt
) as generation:
generation.update(input=full_prompt)
response = client.responses.create(
model="gpt-5.1",
input=full_prompt
)
print(response.output_text)
generation.update(output=response.output_text)
user_feedback = input("Was the explanation helpful? (y/n): ")
user_reason = input("Please provide a reason for your feedback: ")
if user_feedback.lower() == "y":
pass
else:
langfuse.create_dataset_item(
dataset_name="needs_improvement",
input=full_prompt,
expected_output=None, # a domain expert will add the correct answer
metadata={
"model_output": response.output_text,
"user_feedback": user_feedback,
"user_reason": user_reason
},
source_trace_id=span.trace_id,
source_observation_id=generation.id
)Once this data is added, you or a stronger model can then review the dataset. You filter for items where expected_output is missing, read the model output from the metadata section to understand where the model failed, and author the golden expected_output.
Great. Your datasets can now be used to run experiments and evaluations, as discussed later.
Langfuse scores
Once you have traces and datasets, you need a way to evaluate the quality of your LLM application's outputs. In Langfuse, this is done using scores. Scores are the primary data objects used to store evaluation results and quality judgments.
A score can take one of four data types: NUMERIC, CATEGORICAL, BOOLEAN (true / false), or TEXT. They are highly versatile and can be attached to individual traces, specific observations within a trace, or full sessions. Scores can be populated in several ways depending on your workflow:
Automated evaluators that score traces based on custom criteria (e.g., toxicity, relevance, or hallucination).
Validation checks from your code ingested via the SDK/API resulting from custom validation checks.
Annotation queues where you manually review and score traces directly in the Langfuse UI.
Capturing signals directly from your end-users (like thumbs up/down or text comments) and ingesting them via the SDK.
Capturing scores based on routing or moderation guardrails triggered during execution.
To keep annotation structured and consistent across your team, Langfuse uses score configs. A score config defines the schema for a score, such as limiting a "Quality" score to categorical values of Poor, Good, or Excellent. Navigate to Settings > Score Configs in the Langfuse UI and create a new configuration. Once established, team members can manually annotate traces or observations by clicking the Annotate button on a detail view, selecting the configured score, and optionally leaving a reasoning comment.
Manual annotation is great when domain experts are curating high-quality golden datasets. However, it is not scalable. The most scalable manual score is often end-user feedback. You can collect user feedback in your application (e.g., a thumbs up/down button) and send it directly to Langfuse as a score using the SDK. Here is how you can attach a text user feedback score to an observation:
with langfuse.start_as_current_observation(name="main", as_type="span") as span:
with langfuse.start_as_current_observation(
as_type="generation",
name="recursion_explanation",
prompt=raw_prompt
) as generation:
generation.update(input=full_prompt)
response = client.responses.create(
model="gpt-5.1",
input=full_prompt
)
print(response.output_text)
generation.update(output=response.output_text)
user_feedback = input("Was the explanation helpful? (y/n): ")
user_reason = input("Please provide a reason for your feedback: ")
# score the current observation
langfuse.score_current_span(
name="user_feedback",
value="yes" if user_feedback.lower() == "y" else "no",
comment=user_reason,
data_type="TEXT"
)You may have noticed some scores of type EVAL in the screenshot above. Let's discuss them next.
LLM-as-a-judge evals
Manual evaluation is highly accurate but hard to scale. Heuristic checks (like exact match) scale infinitely but lack variety. LLM-as-a-judge offers the best of both worlds: it combines the contextual understanding and nuance of human judgment with the speed and scalability of automated evaluations. LLM-as-a-judge is an evaluation methodology where a capable LLM is prompted to act as an evaluator. It is presented with the input, the output, and a specific scoring rubric, and is asked to return a structured score and reasoning.
To set up LLM-as-a-judge in Langfuse, you configure an Evaluator. An evaluator consists of the model that will act as the judge, as well as a prompt. The prompt contains the evaluation template with variable mappings for the actual inputs and outputs. It also has the score type (Numeric, Categorical, or Boolean), and additional prompts defining how the model should explain its evaluation or verdict. Langfuse ships with many evaluators for common metrics like hallucination, relevance, and toxicity, so you don't have to build one from scratch.
You can then apply these evaluators in two primary contexts. The first one is on observations (live production data). In this method, you configure evaluators to trigger on incoming live data based on specific filters, like environment, session, or user. Once you've narrowed down the scope, you can then use the Sampling slider to target only a small fraction of your traces. Finally, define how your trace data (input, output, expected output) maps into variables in your evaluation prompt:
This method is useful for continuous production monitoring, tracking quality drift over time, and catching regressions early without waiting for user feedback.
The other method involves running experiments on offline datasets, such as the one we curated earlier. When you run an experiment against a fixed dataset, the evaluator can access the dataset's expected_output (ground truth) alongside the model output to generate an accurate score. This method is useful during development. Before merging a new prompt or changing an underlying model, run it against your golden dataset and use the judge to score the run. If the scores drop, you know your change introduced a regression before it ever hits production.
Conclusion
Building robust, production-ready LLM applications requires visibility, iteration, and continuous evaluation. Langfuse provides a comprehensive, unified platform to manage this entire lifecycle. You can trace complex execution chains, manage prompts without redeploying code, curate datasets, and run various evaluations. By integrating these observability and evaluation practices early in your development cycle, you can ship AI features fast with confidence.