LLMs are stateless, so how does an agent keep track of a conversation, or remember a user between conversations? The model itself doesn't. The application around it stores information and puts the relevant part into each request. This is agent memory.
In this topic, you'll learn about the two types of memory, how to manage the short-term kind, and what questions long-term memory raises.
Two types of memory
The two types differ by scope: how long the information should stay relevant.
Short-term memory holds the current conversation: what the user said and what the agent replied (and, in agents that use tools, the tool results). It lives in the history that is sent to the model with every request.
It is limited by the model's context window.
Every token costs money and time, because the whole history is sent each time.
It belongs to one conversation and is cleared when that conversation ends. It can still be saved in a database so that it survives a server restart. What makes it short-term is its scope, not where it is stored.
Long-term memory holds information that should outlive a conversation: a user's preferences, facts they shared about themselves, or the purchase history in a shopping assistant. It lives in external storage, such as a database, files, or a vector store.
It survives between conversations.
It needs a retrieval mechanism. Only the relevant parts are loaded into the context, so it costs tokens only when retrieved.
It needs maintenance: updating outdated facts and deleting what should be forgotten.
If you like analogies from computers, short-term memory is like RAM and long-term memory is like a disk.
Managing short-term memory
You control exactly what goes into the context, so this is the easier of the two. Here is a minimal implementation:
import java.util.concurrent.ConcurrentHashMap
enum class Role { USER, ASSISTANT }
data class Message(val role: Role, val content: String)
class ShortTermMemory(private val maxMessages: Int = 20) {
private val histories = ConcurrentHashMap<String, List<Message>>()
fun history(conversationId: String): List<Message> =
histories[conversationId].orEmpty()
fun append(conversationId: String, vararg messages: Message) {
histories.compute(conversationId) { _, old ->
(old.orEmpty() + messages).takeLast(maxMessages)
}
}
fun clear(conversationId: String) {
histories.remove(conversationId)
}
}A few details here matter in real applications:
One history per conversation. The map is keyed by an ID, so conversations (and users) never see each other's messages.
Immutable lists and an atomic update. Several requests can arrive at once, and
computeupdates the history safely.The system prompt is not in the history. It is sent with every request anyway, so it isn't trimmed or counted against the limit.
A way to clear. Users should be able to start over, and the application should be able to forget.
An even limit. The history grows in user and assistant pairs, so an even
maxMessagesnever leaves a window that starts with a reply to a question that was cut off. Some providers reject such a window.
The history should also change only when a request succeeds:
fun reply(memory: ShortTermMemory, conversationId: String, userText: String): String {
val userMessage = Message(Role.USER, userText)
val prompt = memory.history(conversationId) + userMessage
val answer = callModel(SYSTEM_PROMPT, prompt) // may throw; placeholder for your LLM client
memory.append(conversationId, userMessage, Message(Role.ASSISTANT, answer))
return answer
}If the call fails, nothing is stored. Otherwise, the history would contain a question with no answer, and the model would see it on every later request.
Short-term memory strategies
What should happen when the history gets long? There are four common strategies.
Keep everything. Simple, with no information loss, but the cost and latency grow with every message, and eventually the context window is exceeded. Fine for short conversations.
Sliding window. Keep only the last N messages, as ShortTermMemory does. The cost is predictable, but the agent forgets what was said at the beginning. Good when recent context matters most. Frameworks usually ship this as a ready-made component, so what you configure is mostly N.
Importance-based filtering. Keep only the messages that matter. This reduces noise, but you need a reliable way to decide what is important, and dropping one half of an exchange can leave a confusing history. Use it carefully.
Summarization. Replace old messages with a summary and keep the recent ones as they are:
data class CompactedHistory(val summary: String?, val recent: List<Message>)
fun compact(
history: List<Message>,
keepRecent: Int = 10,
summarize: (List<Message>) -> String, // usually another LLM call
): CompactedHistory {
if (history.size <= keepRecent) return CompactedHistory(null, history)
return CompactedHistory(
summary = summarize(history.dropLast(keepRecent)),
recent = history.takeLast(keepRecent),
)
}The summary can be added to the system prompt, for example under "Summary of the earlier conversation:". This keeps key information from the whole conversation, but summarizing costs tokens and time and may lose details. To avoid paying for it on every request, store the summary and update it only when the history grows past a threshold.
One more rule for agents that use tools: when trimming or summarizing, never separate a tool call from its result. Most providers reject a tool result whose call is missing, so treat the two as a single unit.
Long-term memory
Long-term memory raises more design questions:
What to store? Not everything is worth remembering.
How to store it? As raw text, embeddings, or structured data?
How and when to retrieve it? By keyword, by meaning (semantic search), on every request, or only when needed?
How to update and forget it? Facts change, and some should be deleted.
Here is the shape of a simple long-term memory:
import java.time.Instant
data class MemoryEntry(val text: String, val createdAt: Instant = Instant.now())
interface LongTermMemory {
fun save(userId: String, entry: MemoryEntry)
fun search(userId: String, query: String, limit: Int = 5): List<MemoryEntry>
fun forget(userId: String)
}Notice that every method takes a userId, search returns a limited number of results, and there is a forget. Memory about a person is personal data, so it should be scoped to that person, kept small, and deletable on request. Avoid storing sensitive details you don't need, and keep in mind that stored facts can be wrong or outdated.
Retrieved memories reach the model through the context, usually in the system prompt:
val facts = longTerm.search(userId, userText).joinToString("\n") { "- ${it.text}" }
val systemPrompt = if (facts.isEmpty()) BASE_PROMPT
else "$BASE_PROMPT\n\nWhat you know about this user:\n$facts"Remember that this text was written during earlier conversations, so treat it as information, not as instructions the agent must obey.
Using both together
Imagine a travel assistant:
Conversation 1
User: I'm planning a trip to Tokyo next month.
Short-term: the message is in the current history.
Long-term: save("Planning a trip to Tokyo next month")
Conversation 2, days later
User: What should I pack?
Long-term: search(...) returns "Planning a trip to Tokyo next month"
Agent: "For your Tokyo trip next month, I recommend..."The first conversation's history is gone, but the useful fact survived in long-term memory.
Conclusion
The model stays stateless, and memory is the application's job. Short-term memory is the history of the current conversation. It is limited by the context window and costs tokens on every request, so you manage it with strategies such as a sliding window or summarization. Long-term memory is persistent storage that is retrieved when needed, and it requires decisions about what to store, how to find it, and when to forget.
In both cases, keep memory scoped to the right conversation or user, update it only after a successful request, and give people a way to clear it.