Qdrant is a vector similarity search engine designed for efficient storage, search, and management of high-dimensional vectors. This topic begins with setting up and running Qdrant locally using Docker. It then introduces key concepts such as collections, points, and operations with vectors and payloads in Qdrant. Additionally, we will explore how to perform vector similarity searches to retrieve relevant data based on vector comparisons.
What is Qdrant?
Qdrant is a production-ready vector search engine that stores and manages high-dimensional vectors along with additional metadata (payloads) for efficient, context-aware similarity searches. It performs vector search by finding similar objects based on embedding similarity, using pre-trained models (such as OpenAI’s or Cohere embeddings). Qdrant uses a graph-based approach to quickly find the most relevant results without checking every single object from the database.
There are two main ways to set up Qdrant:
Locally – Run Qdrant on a local machine using Docker.
Using the Python
qdrant-client– Directly interact with a Qdrant instance.Using Qdrant Cloud – A solution offering managed Qdrant database clusters in the cloud.
In this topic, Qdrant will be set up locally using Docker.
Before starting, make sure Docker is installed and running on your machine. To check, open a terminal and run:
docker --versionThis should return the installed Docker version. If not, install Docker from https://www.docker.com.
Once Docker is set up, the next step is to download the latest Qdrant Docker image from Docker Hub. Run the following command:
docker pull qdrant/qdrantNow that the Qdrant image is downloaded, we can start the Qdrant service by running the following command:
# Run Qdrant with exposed 6333 port
docker run -p 6333:6333
# Store data in the 'qdrant_storage' folder in the current directory
-v "$(pwd)/qdrant_storage:/qdrant/storage:z" \\
# Use the Qdrant image
qdrant/qdrantOnce the service is running, Qdrant can be accessed using the following links:
http://localhost:6333 for the REST API, where we can send HTTP requests to perform actions such as adding, updating, or searching data in Qdrant.
http://localhost:6333/dashboard for the Web UI, where we can visually manage and monitor data through an interactive interface.
After ensuring the service is running and accessible via these links, we can initialize the Qdrant client to interact with Qdrant programmatically.
To do this, we will use the qdrant_client library in Python. First, install it by running:
pip install qdrant-clientNext, to initialize the Python client, we need to:
from qdrant_client import QdrantClient
# Initialize the Qdrant client with the URL to the running service
client = QdrantClient(url="http://localhost:6333")
# Alternative client initialization
client = QdrantClient(host="localhost", port=6333)With the client initialized, we can now start interacting with Qdrant by working with collections.
Collections
A collection in Qdrant is a container for points, and it includes a vector_config that defines the configuration for vectors within the collection. This configuration specifies the vector size, the number of dimensions for each vector, and the distance metric used to measure the similarity between vectors. Qdrant supports the main distance metrics including Dot Product, Cosine Similarity, Euclidean Distance, and Manhattan Distance.
We can customize parameters using vector_configs to optimize how vectors are stored, compared, and searched within Qdrant.
In Qdrant, collections can be created using two different methods:
The first method involves using the Python client, which allows you to create and manage collections programmatically. For example, we can create a collection named
points_collectionwith 3-dimensional vectors and the dot product as the distance metric:from qdrant_client.models import Distance, VectorParams client.create_collection( collection_name="points_collection", vectors_config=VectorParams(size=3, distance=Distance.DOT), )The second method uses an HTTP request to interact directly with the Qdrant server. The following example achieves the same result as the Python code above:
# Example of a PUT request to update collection curl -X PUT "http://localhost:6333/collections/points_collection" \\ -H 'Content-Type: application/json' \\ --data-raw '{ "vectors": { "size": 3, "distance": "Dot" } }'Additionally, you can quickly prototype requests in the Qdrant console at http://localhost:6333/dashboard#/console, which allows testing HTTP interactions directly.
After creating the collection using either the Python client or HTTP request, you can verify that the collection was successfully created by accessing the Qdrant WebUI at http://localhost:6333/dashboard#/collections. On this page, you can visualize your collections, view them in a graph, and even delete a collection if needed. You should see our newly created collection, points_collection, listed there.
Additionally, you can use HTTP requests for direct interaction with Qdrant API from the terminal. For example, to check if our collection exists, you can use the following command:
# Example of a GET request to check if the "points_collection" exists
curl -X GET "http://localhost:6333/collections/points_collection/exists"This command will return a response indicating whether the collection exists ("exists": true) or not.
If you want to remove a collection and all its associated data, you can use the HTTP DELETE method with the following command:
# Example of a DELETE request to remove the "points_collection"
curl -X DELETE "http://localhost:6333/collections/points_collection"However, we need to keep the points_collection. In the next section, we will focus on working with this collection and explore how to work with points, vectors and payloads in Qdrant.
Points
In Qdrant, points from a collection are the main entities that the vector search engine operates on. Each point represents a single data record composed of an identifier, a vector used for similarity search, and an optional payload, which stores additional metadata or structured information related to the point.
The first characteristic, the identifier, plays a crucial role in uniquely identifying each point within a collection. Qdrant supports two types of identifiers for points:
64-bit unsigned integers. Examples:
“id”: 1,“id”: 123.UUID string representations can appear in formats such as simple
936DA01F9ABD4d9d80C702AF85C822A8, hyphenated550e8400e29b41d4a716446655440000, or as a URNurn:uuid:F9168C5ECEB24faab6bf329BF39FA1E4.
The second key component of a point is the vector, which is used for similarity search. Qdrant supports multiple types of vectors: dense, sparse, and multivectors. While dense and sparse vectors are commonly used in embedding models and information retrieval, multivectors represent matrices with a fixed width and variable height. Example: "multi_vector": [[0.11, 0.22, 0.33], [0.44, 0.55, 0.66], [0.77, 0.88, 0.99]].
Qdrant also allows attaching more than one type of vector to a single point using Named Vectors, each with its own configuration.
The third component of point is the payload that allows storing additional information alongside vectors, using a flexible payload schema in JSON format. Payload can be in various data types including:
64-bit signed integers, for example,
"index": 42.64-bit floating point numbers, for example,
"length": 9.5.Boolean values, for example,
"exists": true.String values, for example,
"name": "Alice".Geographic coordinates, for example,
"location": {"lat": 40.7128, "lon": -74.0060}Timestamp in RFC 3339 format, for example,
"created_at": "2023-02-08T10:49:00Z"UUID, for example,
"uuid": "550e8400-e29b-41d4-a716-446655440000".
Qdrant offers various methods for loading data into a collection via code or HTTP requests, and we will use batch loading through the Python client to insert multiple points. We can define vectors and payloads for points_collection directly in the code and upsert them using the upsert() method, as shown in the following example:
# Define the list of points to insert into the collection
points = [
PointStruct(
id=3, # Unique ID for the first point
vector=[0.1, 0.1, 0.9], # 3D vector
payload={"color": "blue"} # Payload
),
PointStruct(
id=4,
vector=[0.5, 0.5, 0.0],
payload={"color": "yellow"}
)
]
# Insert the points into the specified collection using the upsert() method
client.upsert(
collection_name="points_collection", # Target collection name
points=points # Points to be inserted
)After loading the data, you can verify it by visiting http://localhost:6333/dashboard#/collections/points_collection. On this page, you can explore the contents of points_collection, including stored vectors, payloads, and point IDs.
Also, to verify or inspect specific points in your collection, you can use the following command to retrieve points with "id": 1, "id": 2, "id": 3, and "id": 4 in points_collection:
# Example of a POST request to retrieve points from the "points_collection"
curl -X POST 'http://localhost:6333/collections/points_collection/points' \\
-H 'Content-Type: application/json' \\
-d '{
"ids": [1, 2, 3, 4],
"with_payload": true,
"with_vector": true
}'This request will return their vector data and payloads
{
"result": [
{
"id": 1,
"payload": {
"color": "red"
},
"vector": [0.9, 0.1, 0.1]
},
{
"id": 2,
"payload": {
"color": "green"
},
"vector": [0.1, 0.9, 0.1]
},
{
"id": 3,
"payload": {
"color": "blue"
},
"vector": [0.1, 0.1, 0.9]
},
{
"id": 4,
"payload": {
"color": "yellow"
},
"vector": [0.5, 0.5, 0.0]
}
],
"status": "ok",
"time": 0.000274448
}Now that the points_collection contains all the necessary data, we can use it to perform a query. We will search for vectors most similar to a given query vector, using Qdrant's vector similarity search feature.
Query
To retrieve vectors similar to a given query vector, Qdrant provides the .query_points() method in the Python client. This method allows us to search for the most similar vectors within a specified collection by comparing them with a given query vector:
search_result = client.query_points(
collection_name="points_collection", # Name of the collection to search within
query=[0.1, 0.9, 0.1], # The query vector
with_payload=False, # Whether to include the payload in the results
limit=3 # Maximum number of similar vectors to return
).points
print(search_result)Example output showing the top 3 most similar vectors:
[
ScoredPoint(id=2, version=0, score=0.8299999, payload=None, vector=None, shard_key=None, order_value=None),
ScoredPoint(id=4, version=1, score=0.5, payload=None, vector=None, shard_key=None, order_value=None),
ScoredPoint(id=1, version=0, score=0.19, payload=None, vector=None, shard_key=None, order_value=None)
]As you can see, query results are entirely determined by the vectors stored in the collection and their similarity to the input query. Therefore, performing basic operations on these points will allow us to modify and refine the search outcomes.
Useful commands
Once the points are loaded into the collection, you can perform various operations on them to modify the search results. Here are some useful HTTP commands for working with points in Qdrant:
If you need to update both vectors and payloads for specific points, you can use the upsert operation. This allows you to modify the vector data and payloads for the point with
"id": 1:
# Example of a POST request to upsert points into the "points_collection"
curl -X POST 'http://localhost:6333/collections/points_collection/points/upsert' \\
-H 'Content-Type: application/json' \\
-d '{
"points": [
{
"id": 1,
"vector": [0.1, 0.2, 0.3],
"payload": {"color": "black"}
}
]
}'If you need to remove specific points from the collection, we can use the following command to delete the point with
"id": 2:
# Example of a POST request to delete points from the "points_collection"
curl -X POST 'http://localhost:6333/collections/points_collection/points/delete' \\
-H 'Content-Type: application/json' \\
-d '{
"points": [2]
}'If you want to remove specific vectors from the points without deleting the points themselves, we can use the following command to delete vectors
0.1and0.1for the point with"id": 3:
# Example of a POST request to delete vectors from points in the "points_collection"
curl -X POST 'http://localhost:6333/collections/points_collection/points/vectors/delete' \\
-H 'Content-Type: application/json' \\
-d '{
"points": [3],
"vectors": ["0.1", "0.1"]
}'If you want to clear all payloads for certain points, you can use the following command to delete the payload for the point with
"id": 4:
# Example of a POST request to clear the payload of points in the "points_collection"
curl -X POST 'http://localhost:6333/collections/points_collection/points/payload/clear' \\
-H 'Content-Type: application/json' \\
-d '{
"points": [4]
}'By performing basic operations on the points, you can run the query again to check and compare the updated results.
Conclusion
As a result, you are now familiar with the following:
Qdrant is a vector similarity search engine for managing high-dimensional vectors and associated metadata. It can be used locally using Docker, accessed via Python client, or used as a managed cloud service;
Collections in Qdrant serve as containers for points and include vector configurations that define size and distance metrics. Each point consists of a unique identifier, vectors, and optional JSON payloads;
Qdrant allows searching for vectors similar to a query vector. Search results include similarity scores and can be configured to return vectors and payloads as needed.