Vector DB Optimizations

One of key parts of a modern database of documents is managing a vector-db for topical searches and clustering capabilities. One of those that we use is the open source Qdrant. If you’re not familiar, Qdrant is built in rust and designed for high performance. It’s also extremely friendly for developers: the documentation is top notch, they include lots of agent skills so your coding assistant can dig in too. They’ve also recently released support for TurboQuant and Memory Tiers.

Get David Smith’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

With this new release I took the opportunity to make some adjustments to our general schema, quantization, and memory use settings: The results were dramatic… with a bit of careful planning and understanding our own use case, I was able to cut the memory requirements down:

The waviness prior to 1500h is the migration process, running for a few hours. We typically saw data in 80–90GiB range for regular usage.

Three things got us here:

1. We shifted our sharding scheme to a custom scheme based on the date of ingest. This is a fairly basic one to do, and we should have done it earlier, but make a lot of sense for continuously ingested time indexed data. In fact, Qdrant publishes a guide on it. The big benefit for us that indexing work on the latest shard can be computed without having to deal with the entire content of the collection.
2. We moved our quantization over from standard 8-bit scalar to TurboQuant. If you’re not familiar the Qdrant docs can point you in the right direction: TurboQuant is a quantization method developed by Google. It operates by applying a fast random rotation to vectors before compression, which evenly redistributes data across coordinates. This allows applying a single pre-computed, globally optimized quantization mapping across the dataset, enabling TurboQuant to work effectively with any vector distribution and overcoming a key limitation found in binary quantization. For us this means we can cut our index size in half and get performance that on search results that are well within tolerances for our application.
3. We took advantage of the new parameter memory provided in nearly all collection settings. For us, we setting most memory settings to cold made the most sense… In this configuration we could have a minimal amount of required resident memory usage per collection, and as different segements are loaded into memory they can be cached as needed, but evicted just as easily.