In this topic, you'll learn what an agent is, how it differs from a chatbot, and why managing what the model sees, its context, is the first problem you'll meet.
What is an LLM agent?
An LLM agent is a system that carries out tasks given in natural language. It uses an LLM to plan, calls tools to interact with its environment, and keeps a memory of what has happened. Because the LLM decides what to do, both the process and the outcome are non-deterministic.
A common way to explain agents is to compare them to a butler.
A plain LLM is a butler locked in a windowless, dark room. He is well-informed and can talk to you, but he can't see what is happening outside. If you ask about the weather, he will try to please you with an answer that may be made up. In a shop, the same thing happens when you ask whether a product is in stock: the LLM has never seen your catalog, so it can only guess.
Give him a window and a thermometer, and he can check the weather before answering. These are tools: ways for an agent to read from or act on the outside world.
Give him access to your wardrobe and ask him to prepare an outfit, and he has to plan: check the weather, look at the clothes, then choose. He works out the steps himself.
Finally, give him a notebook, and he can remember what you told him earlier, such as your clothing preferences. That is memory.
Butler | Agent component | Example in a project |
|---|---|---|
Talks in the dark room | LLM | The chatbot |
Notebook | Memory | The conversation history |
Window | Tools for reading | Catalog search |
Wardrobe | Tools for acting, plus planning | Cart management |
Non-determinism
Ask the butler to choose an outfit twice, and he may pick a sweater and jeans one day and a jacket and trousers the next. Both are reasonable, and the goal is reached either way.
Agents behave the same way. The same request can produce different replies and, once tools are added, different sequences of actions. This has a practical consequence for you: don't expect identical text from the model. When you test your chat, check properties rather than exact wording. Does the reply follow the system prompt? Does the assistant remember the previous message?
Memory and context
You've seen that models are stateless, so the application resends the conversation with every request. Everything the model can see in a single request is called its context. The context may consist of:
the system prompt;
the earlier user and assistant messages;
the current user message.
Later, it will also include descriptions of the available tools and the results of tool calls.
The conversation history is the butler's notebook, but it lasts only as long as the conversation. This is short-term memory. Information that should survive between conversations, like the butler's list of your preferences, is long-term memory.
Context is not free
It's tempting to send the whole conversation every time, but a larger context has costs:
Price. Providers charge per token, and the whole context is sent with every request, so a long conversation makes each call more expensive.
Latency. More tokens take longer to process.
Quality. Irrelevant text can distract the model.
Limits. Every model has a context window, the maximum number of tokens it accepts in one request. Its size depends on the model, so check the provider's documentation.
Because of this, the application has to decide what to send. The simplest strategy is a sliding window: keep only the last N messages and drop the older ones. This is what you'll implement by limiting the history to the last 20 messages. The system prompt is sent with every request anyway, so it isn't counted against that limit.
Choosing N is a trade-off. A window that is too small makes the assistant forget things the user said a moment ago, and one that is too large increases cost and latency. More advanced strategies, such as summarizing old messages or including only the relevant ones, exist for longer conversations.
Remember that each user needs a separate context. Otherwise, one shopper's messages would end up in another's conversation.
A look ahead: the agentic loop
When tools arrive, the assistant stops being a single question-and-answer call. It works in a loop of three steps:
Think. The model looks at the context and decides what to do next.
Act. The model asks for a tool call by returning the tool's name and its arguments. The model never runs anything itself. Your application, or the framework you use, executes the tool.
Observe. The tool's result is added to the context, and the loop starts again. It ends when the model has enough information to give a final answer.
For a request like "something dairy free I can use for a creamy pasta," the loop might look like this (the tool names are illustrative):
Think: I need dairy-free creamy alternatives from the catalog.
Act: search_products("dairy free cream")
Observe: Oat cream, coconut cream, cashew butter
Think: I should also find pasta.
Act: search_products("pasta")
Observe: Spaghetti, penne, fusilli
Think: I have enough to suggest something.
Answer: Try penne with oat cream: ...Each pass through the loop makes the context grow, so the problems described above get sharper. Tool descriptions are sent with every call, and every tool result stays in the history. That is another reason to control what goes into the context.
Conclusion
An LLM agent combines an LLM that plans, tools that let it interact with the outside world, and memory that keeps track of what happened. Its behavior is non-deterministic, so the same request can lead to different replies or actions.
Models are stateless, so everything they need must be in the context of each request. Context costs money and time and can hurt quality, so the application limits what it sends, for example with a sliding window over the last N messages.