Vector Database — ChromaDB
There is so much buzz about AI Agents and all concepts associated with building an Agents seems to be very technical. This blog attempts to explain one of the key concepts in AI Agents — Vector Database.
What is a Vector Database ?
At it’s core Vector Database is a collections of Documents or Texts in form a language that machine learning understand i.e. vector embeddings. These embeddings are build in such a way so that the context within Documents is preserved. Ex: I love codding is same as Codding is my best hobby. Both represent a similar context that you are fond of coding.
How Vector Database are designed ?
Vector Database today has to be indexed. Why ? To make it useful for real world application. Imagine you have thousands of Documents, whose embedding is store in Vector DB without index. Now for real world application like searching you would have to search your query against almost all Documents to get useful results. Sounds like not optimal ? right.
To solve this problem Vector DB are indexed and then only a sub segment of DB is returned for a query. Well if you have a question in mind how these sub segment are identified ? Then the answer would be using algorithms like Hierarchical Navigable Small World (HNSW) graphs, Inverted File (IVF) clustering, and Product Quantization (PQ)
Let look at a specific example on Vector DB. While there are many Vector DB available today like Pinecone, Chroma DB, Deep Lake we’ll use Chroma DB as a example.
import chromadbdb = chromadb.PersistentClient(path=”./chroma_db_lessons”)
# Just like creating a table in DB
collection = db.create_collection(name=”Students”)
student_info = “””Alexandra Thompson, a 19-year-old computer science sophomore with a 3.7 GPA,is a member of the programming and chess clubs who enjoys pizza, swimming, and hiking in her free time in hopes of working at a tech company after graduating from the University of Washington.“””
club_info = “””The university chess club provides an outlet for students to come together and enjoy playing the classic strategy game of chess. Members of all skill levels are welcome, from beginners learning the rules to experienced tournament players. The club typically meets a few times per week to play casual games,participate in tournaments, analyze famous chess matches, and improve members’ skills.“””
Get Rahul | Subscribe for Blogs on AI’s stories in your inbox
Join Medium for free to get updates from this writer.
university_info = “””The University of Washington, founded in 1861 in Seattle, is a public research university with over 45,000 students across three campuses in Seattle, Tacoma, and Bothell.As the flagship institution of the six public universities in Washington state, UW encompasses over 500 buildings and 20 million square feet of space, including one of the largest library systems in the world.“””
collection.add(documents = [student_info, club_info, university_info],metadatas = [{“source”: “student info”},{“source”: “club info”},{‘source’:’university info’}],ids = [“id1”, “id2”, “id3”])
# Check out the embeddings and metadata of collections
from IPython.display import JSON
JSON(collection.get(), expanded=True)
results = collection.query(
query_texts=[“What is the student name?”],
n_results=2,
#include=[‘embeddings’],
where={“source”: “club info”})
print(results)
