Vivek Oberoi is a failed actor, yet his net worth surpasses that of actors like Ranbir Kapoor, Allu Arjun, and even industry legend Rajnikanth. In fact even when combined, the net worth of these three actors still falls short of Oberoi’s 1200 Crore valuation.

So how did he make all this wealth?

This is the story of Oberoi’s entrepreneurial ventures. His diverse portfolio spans real estate, sustainable diamonds, event management, and education, showcasing a strategic approach to wealth creation.

Early Exposure to Business

Vivek Oberoi’s foundation in business began early. His father, Suresh Oberoi, a Bollywood actor, introduced him to commerce at a young age. At seven, Vivek was tasked with selling perfume bottles, learning about stock management, costing, and sales. This instilled a fundamental understanding of business operations.

By 14, attending Mayo College in Ajmer, his father cut off his allowance, pushing him to earn his own money. Oberoi took on jobs like hosting college events and dubbing English movies into Hindi, earning around ₹30,000 a month. However, his entrepreneurial spirit yearned for more.

He identified a problem at his college: students had to travel long distances for food if on-campus vendors ran out of supplies. Oberoi seized this opportunity by supplying food items to these vendors at lower prices, taking a 20–30% cut of their total sales. He even offered higher profits to vendors during rainy seasons and holidays, making him a preferred supplier. By 16, Oberoi had made a profit of ₹5–6 lakh.

Fueled by this success, he ventured into the stock market, learning the basics and applying his own business acumen. Within two years, he made a profit of ₹40–50 lakh. However, his luck turned when he invested in a risky stock, losing all his money in a week. This experience taught him a valuable lesson about risk management and the volatile nature of the stock market.

Bollywood Career: Highs and Lows

When college ended, Oberoi turned to acting, his “Plan B.” Despite his father’s connections, he struggled for 3–4 years before landing his first movie, “Company,” in 2002. The film was a hit, earning him Filmfare Awards for Best Male Debut and Best Supporting Actor. His second movie, “Saathiya,” was also a success, leading to comparisons with established Bollywood stars.

However, his career took a turn on March 29, 2003, when he received a call from Salman Khan, who questioned him about his alleged affair with Aishwarya Rai, who Khan was dating at the time. This incident led to a fallout with Khan, who allegedly used his influence to blacklist Oberoi from Bollywood.

Directors and producers refused to work with him, and co-actors stopped answering his calls. Oberoi and his family received threats from the underworld, leading to police protection. The lack of opportunities and the pressure affected his performance, and his movies began to flop. From 2003 to 2007, films like “Darna Mana Hai,” “Yuva,” “Deewane Huye Paagal,” “Home Delivery,” and “Fool n Final” failed at the box office. According to the video, in a 22 year Bollywood career, Vivek Oberoi only starred in 30 films, of which 24 were flops. This means that 80% of his films could not recover the money it took to produce them.

This period was marked by financial issues and mental health struggles. Oberoi experienced frustration, pain, anger, and depression. He realized he needed to pivot and return to his entrepreneurial roots.

Diversifying into Business

Recognizing the instability of the film industry, Oberoi focused on his entrepreneurial ambitions. He understood that relying solely on acting was precarious, and he needed to diversify his income streams. With a business mind honed since childhood and maturity gained through his Bollywood experiences, he began identifying market gaps and opportunities.

1. Karrm Infrastructure Private Limited

Oberoi recognized the emotional significance of owning a home for middle-class families in India. In 2011, he launched Karrm Infrastructure, a real estate company focused on providing affordable housing. The company has since built and sold thousands of affordable homes.

What sets Karma Infrastructure apart is its marketing strategy. The company treats families like celebrities during the registration process, providing VIP treatment, including transportation, meals, and even helicopter rides. This customer-centric approach has been successful in attracting and retaining clients.

2. B.A.D. Developments

Understanding the potential for high profits in the luxury real estate market, Oberoi expanded his business to Dubai, catering to the elite class. B.A.D. Developments focuses on building luxurious homes for the wealthy, tapping into the demand for opulent living spaces.

Dubai’s tax-free environment further enhances the profitability of this venture. By targeting the elite class and operating in a favorable business climate, Oberoi has generated millions of dollars in revenue from this company.

3. Solitario

In 2022, Oberoi ventured into the diamond industry with Solitario, a company that produces lab-grown diamonds. Unlike traditional diamond mining, which is environmentally harmful and often involves unsafe labor practices, Isorio offers an ethical and sustainable alternative.

Lab-grown diamonds are chemically identical to natural diamonds but are produced in a controlled laboratory environment. Oberoi partnered with Ricky Vasandani, who has over a decade of experience in the international jewelry market, to create a company that capitalizes on the growing demand for sustainable luxury.

Solitario currently has 18 physical stores in India and plans to expand to more cities. The company’s success is driven by the increasing awareness of the environmental and ethical issues associated with natural diamond mining.

4. Mega Entertainment

Oberoi’s fourth major company is Mega Entertainment, an event management company operating in India and Dubai. The company specializes in organizing weddings and other events, offering clients a unique experience.

Mega Entertainment uses the same helicopter marketing strategy as Karma Infrastructure, providing couples with a free helicopter ride. This attention to detail and focus on creating memorable experiences has made Mega Entertainment a popular choice for clients seeking a high-end event management service.

5. Other Ventures

In addition to his major companies, Oberoi has also founded Swanim University and an education loan platform, providing affordable education loans to students. The education loan platform has a valuation of over ₹3,000 crore. In 2022, he launched a coffee brand called “Aur Coffee,” which competes with established brands like Starbucks.

Strategic Investments

Oberoi’s early experience in the stock market taught him the importance of strategic investments. He has invested in over 30 companies, including Icolor, Balu Forge, and Infinity Reality, generating crores of rupees in annual returns. One of his early investments, Balu Forge, has grown to a valuation of ₹8,000 crore.

The Road Ahead

As of 2025, Vivek Oberoi’s net worth stands at $145 million (₹1200 crore). On Ganesh Chaturthi this year Vivek Oberoi gifted Ganesh Ji a 66 karat diamond tilak and bought a Rolls Royce worth 12 Croroes two months after. With his companies poised for IPOs, Oberoi is on track to surpass even the biggest Bollywood superstars in terms of wealth in the coming years.

His story is a compelling example of how an individual can overcome setbacks and create a diversified portfolio of income through strategic business decisions and investments.

Everyone remembers the momentous night of the 2022 Oscars when Will Smith walked up to Chris Rock for ridiculing his wife’s baldness and then smacking him in the face. During the ceremony, Chris Rock made a comment about Jada Pinkett Smith appearing in GI Jane — a jab at her bald look — even though her hair loss is due to an autoimmune disorder called alopecia areata. At first, Will laughed at the remark, but when he saw how much it hurt his wife, he abruptly got up, walked toward Chris Rock, and delivered a forceful slap. Then, returning to his seat, he famously declared, “Keep my wife’s name out of your f**ing mouth.”

Years later, Jada Pinkett Smith reflected on that unforgettable moment:

“I thought it was a skit. From where I was sitting, you couldn’t really tell — it didn’t look like Will was going to hit Chris. I mean, I’d seen Will in the boxing ring with pro fighters, and he’s a heavy hitter. So when Chris moved, it looked like he ducked or missed the shot, especially when he kept standing and then walked down to the end of the stage. I was thinking, ‘there’s no way he’s going to get hit.’ It wasn’t until Will started walking back to his chair that I realized this wasn’t a skit at all. I was just as foggy as everyone else in that room that night.”

Following the incident, a number of Hollywood celebrities shared their thoughts on that charged evening:

Jim Carrey’s Reaction

Jim Carrey, known for his bold humor and candid opinions, expressed his disgust over the entire affair. He was particularly critical of the media and Hollywood response — especially the standing ovation given to Will Smith after his Best Actor win. Carrey went on to assert that if he were in Chris Rock’s shoes, he would have sued Will Smith for $200 million. His comments underscored his belief that the slap left a permanent mark on Chris Rock’s career and highlighted the industry’s tendency to mix personal feelings with public drama.

Chris Rock’s Composure

Despite being on the receiving end of one of the most shocking moments in Oscars history, Chris Rock handled the situation with remarkable professionalism. Initially, his wide-eyed expression revealed his shock; yet, he chose not to retaliate or storm off stage. Instead, Rock continued with his presentation, maintaining his composure through an incredibly tense moment. In later reflections, he addressed the incident with humor, acknowledging both the absurdity and personal sting of the slap.

Eddie Murphy’s Take

Eddie Murphy, adding a lighter note amid the heavy drama, delivered a memorable quip at the Golden Globe Awards when he was honored with the Cecil B. DeMille Award. Murphy joked, “There’s a simple three-step blueprint for success in Hollywood — pay your taxes, mind your business, and keep Will Smith’s wife’s name out of your mouth.” His remark, met with hearty laughs from the audience, served as a reminder that even in the midst of controversy, humor continues to play an important role in processing Hollywood’s most baffling moments.

This incident remains one of the most talked-about in recent entertainment history, sparking debates on personal respect and the boundaries of public performance, as well as setting off a chain reaction of candid commentary from some of Hollywood’s most iconic figures.

“In one year, AI will generate complex apps like Uber from scratch”. — Emad Mostaque.

Let that sink in. The future of AI is not just coming; it’s already here. And leading the charge is Emad Mostaque, the visionary founder of Stability AI, the company behind the AI image model Stable Diffusion. Here are some of his insights from a podcast that you’ll find linked later.

Insights:

1. AI will revolutionize software development. It will create complex apps from scratch within a year.
2. Generative AI is expanding. It’s tackling video, code, and 3D assets. Content creation will never be the same.
3. AI will augment human creativity. It will empower small teams to rival tech giants.
4. Personalized AI will be our trusted collaborator. It will learn from us and anticipate our needs.
5. Democratization of AI is crucial. Open-source models and datasets will create a future where AI benefits all.
6. AI will eat the world. Industries will be disrupted. Jobs will be transformed. Embrace the change or be left behind.
7. Professionals must adapt to the AI landscape. Mastering AI-assisted tools is key to problem-solving and innovation.
8. AI models are getting smarter and faster. They’ll run on smartphones for real-time, personalized experiences.
9. The future of AI lies in multimodality. Models will seamlessly handle diverse inputs and outputs, revolutionizing human-AI interaction.
10. The AI revolution presents challenges. Equal access, ethics, transparency, and accountability require ongoing dialogue.

In case you are not happy with this summary, you can watch the full podcast here — https://www.youtube.com/watch?v=VeaT6ucXuMI

According to a recent survey, 42% of young adults in India between 26 and 40 say they do not want to get married. This number shows that many people today are growing more and more afraid of marriage. They worry about long, expensive legal battles and a host of other problems. The old idea of marriage as a lifelong bond built on trust is slowly fading.

A Crisis of Trust and Legal Battles

Many young people in India are turning away from marriage because they fear the legal system. False allegations have led to long and costly court battles. Men worry that even a small mistake might result in huge financial losses. One high-profile case saw a wife file an alimony claim for Rs. 500 crores from her US-based husband, with the names of entire families involved. In another case, the Supreme Court had to settle for Rs. 12 crores. Stories like these create a climate of distrust, and the fear of facing such legal nightmares makes many feel that marriage is simply too risky.

Arranged Marriages in Modern Times

In India, arranged marriages are still very common. In fact, a survey shows that even now, 90% of marriages in India are arranged marriages. Families meet through middlemen who introduce potential partners, and both sides share details like education, background, and wealth. However, these marriages come with their own set of problems. There is immense pressure on both families to meet expectations. Discussions about dowry and social status add stress to the process. Arranged marriages are definitely not making it easy for people to sustain the institution of marriage.

Family Pressure and Domestic Conflicts

The problems do not end with legal and financial issues. Family pressure plays a big role in making marriage difficult. After the wedding, a new daughter-in-law often finds herself caught between high family expectations and outdated traditions. Even though many women are well-educated and career-oriented, there is still a strong expectation for them to manage the entire household — cooking, cleaning, and taking care of everyone. The constant meddling of the mother-in-law, father-in-law, and even the daughter’s own family creates conflicts. There is little room for privacy, and personal space becomes a luxury. Such family interference only adds to the growing disillusionment with marriage.

Love Marriages Are Not Immune

While many believe that love marriages are a break from the problems of arranged marriages, they too face significant challenges. Even couples who marry for love find that there are issues beyond emotional compatibility. Many do not get enough time to understand each other’s financial situations, work ethics, or lifestyle choices before they tie the knot. Additionally, staying with extended families often causes intimacy problems. Couples do not have a safe space for sex, which leads to tension and conflict. Without the buffer of family involvement, personal differences and sexual intimacy issues can quickly turn into serious conflicts. As a result, love marriages sometimes end up with as many problems as arranged ones.

A Growing Trend: Choosing Singlehood

The combined weight of false allegations, expensive legal battles, family interference, and the pressures of traditional expectations has driven many to rethink the idea of marriage altogether. With stories of severe financial losses and relentless family pressures, more young people are finding that staying single is a safer, simpler option. They see marriage not as a fulfilling partnership, but as an institution fraught with risk and disappointment.

What do you think about these trends? Are we witnessing the end of marriage as we know it in India? Share your thoughts and join the conversation in the comments below.

Btw, I run a Content Consulting Firm. If you need help with your content needs, reach out at vardhanam@k42consulting.com :)

Problem Statement

In this blog, we are going to address the challenges faced by a single node Qdrant setup. And then we are going to demonstrate how we can overcome it by deploying a distributed node setup. By the end of this blog, you will have both a conceptual understanding of distributed networks as well as the technical know-hows of setting it up.

Key Features of a Distributed Deployment Setup in Qdrant

To understand what distributed deployment is, it is important to first understand the following features:

Single Node Qdrant

A single node Qdrant setup uses one server to store and manage all the data and processes. It is less expensive and simpler to set up, but has notable drawbacks. If the server goes down for any reason, such as maintenance or a breakdown, the entire Qdrant service will become unavailable. Furthermore, the performance of the service is limited to what a single server can handle.

Qdrant Cluster

A Qdrant Cluster is a more advanced setup that involves multiple servers, known as nodes, working together. This arrangement distributes the data and workload among several servers, which offers multiple benefits over a single node setup. It’s more resilient because if one server has an issue, the others can keep the service running. It also allows for scalability, meaning more servers can be added as needed to handle more data or more user requests, thus enhancing overall performance.

Cluster Communication

In a cluster, the servers communicate using the Raft consensus protocol. This protocol helps keep the servers synchronized, ensuring that they all agree upon any changes or updates before they are made. This coordination is crucial for maintaining data accuracy and consistency across the cluster.

For operations on individual points though, Qdrant prioritizes speed and availability over strict adherence to this protocol, which allows for faster processing without the complexity of coordinating every detail across all servers.

Sharding

Sharding is a technique to break down a database into different segments. In Qdrant, a collection can be divided into multiple shards, each representing a self-contained store of points.

This distribution allows for parallel processing of search requests, leading to significant performance improvements. Qdrant supports both automatic sharding, where points are distributed based on a consistent hashing algorithm, and user-defined sharding, offering more control over data placement for specific use cases.

Importance of Distributed Development

Distributed development, which involves spreading out data and processing across multiple locations or servers, is essential for creating systems that are reliable and can scale according to demand. In the context of Qdrant, this approach offers several key advantages:

- Resilience: By distributing operations across multiple servers, the system can still function even if one server fails.

- Scalability: It’s easier to handle more data and more users by adding more servers to the system.

- Performance: Distributing the workload helps speed up data processing and retrieval by allowing multiple operations to run in parallel across different servers.

How to Launch a Multi-Node Cluster on Qdrant

We’ll use Docker Compose to launch a 4-node Qdrant Cluster setup. First create a file called docker-compose.yml and paste the following in it:

services:
  qdrant_node1:
    image: qdrant/qdrant:v1.9.1
    container_name: qdrant_node1
    volumes:
      - ./data/node1:/qdrant/storage
    ports:
      - "6333:6333"
    environment:
      QDRANT__CLUSTER__ENABLED: "true"
    command: "./qdrant --uri http://qdrant_node1:6335"


  qdrant_node2:
    image: qdrant/qdrant:v1.9.1
    container_name: qdrant_node2
    volumes:
      - ./data/node2:/qdrant/storage
    depends_on:
      - qdrant_node1
    environment:
      QDRANT__CLUSTER__ENABLED: "true"
    command: "./qdrant --bootstrap http://qdrant_node1:6335 --uri http://qdrant_node2:6335"


  qdrant_node3:
    image: qdrant/qdrant:v1.9.1
    container_name: qdrant_node3
    volumes:
      - ./data/node3:/qdrant/storage
    depends_on:
      - qdrant_node1
    environment:
      QDRANT__CLUSTER__ENABLED: "true"
    command: "./qdrant --bootstrap http://qdrant_node1:6335 --uri http://qdrant_node3:6335"


  qdrant_node4:
    image: qdrant/qdrant:v1.9.1
    container_name: qdrant_node4
    volumes:
      - ./data/node4:/qdrant/storage
    depends_on:
      - qdrant_node1
    environment:
      QDRANT__CLUSTER__ENABLED: "true"
    command: "./qdrant --bootstrap http://qdrant_node1:6335 --uri http://qdrant_node4:6335"

The Docker Compose file defines four services, each corresponding to a Qdrant node in the cluster. Here’s what each section generally specifies:

Common elements for all nodes:

• image: Specifies the Docker image to use for the container. All nodes use qdrant/qdrant:v1.9.1.
• container_name: Assigns a unique name to each container running a Qdrant node.
• volumes: Maps a local directory (./data/nodeX) to a directory inside the container (/qdrant/storage). This setup is used for data persistence across container restarts.
• environment: Sets environment variables inside the container. QDRANT__CLUSTER__ENABLED: “true” enables cluster mode in each Qdrant node.

Specifics for individual nodes:

• qdrant_node1
• ports: Maps port 6333 on the host to port 6333 on the container, used for API access to the Qdrant service.
• command: Specifies the command to run inside the container. For qdrant_node1, it initializes with its own URI for cluster communication (http://qdrant_node1:6335).
• qdrant_node2, qdrant_node3, qdrant_node4
• depends_on: Ensures that these nodes start only after qdrant_node1 has started. This dependency is crucial because the subsequent nodes need to connect to the first node for cluster setup.
• command: For these nodes, the command includes both a — bootstrap option pointing to qdrant_node1 (to join the cluster) and a — uri with their own unique URI. This setup is necessary for proper communication within the cluster.

To launch the cluster, run the following command:

docker-compose up

Now you can access the cluster via the Qdrant client:

from qdrant_client import QdrantClient

# Initialize the Qdrant client
client = QdrantClient(host="localhost", port=6333)

Next, let’s define the collection and the shard_key.

collection_name = "sharding_collection"
key = "tempKey"


# Deleting an existing collection if it exists
response = client.delete_collection(collection_name=f"{collection_name}")


# Creating a new collection with specific configuration
response = client.create_collection(
    collection_name=f"{collection_name}",
    shard_number=6,
    sharding_method=models.ShardingMethod.CUSTOM,
    vectors_config=models.VectorParams(size=768, distance=models.Distance.COSINE),
    replication_factor= 2,
)


# Creating a shard key for the collection
response = client.create_shard_key(f"{collection_name}", f"{key}")

Shard Number implies the number of shards per Shard Key. Replication factor implies the number of times the shards are replicated evenly across all the nodes.

In this case, since we have defined only 1 Shard Key, the total number of Physical Shards would be 1*6*2 = 12.

Now let’s input some random vectors into our cluster using the shard key.

# Counter for generating unique point IDs
point_counter = 0


# Function to generate a random vector of 768 dimensions with up to 15 decimal points
def generate_random_vector():
    return [round(random.uniform(0, 1), 15) for _ in range(768)]


# Run the loop 1000 times
for _ in range(1000):
    random_vector = generate_random_vector()
    point_counter += 1
    response = client.upsert(
        collection_name=f"{collection_name}",
        points=[
            models.PointStruct(
                id=point_counter,
                vector=random_vector,
            ),
        ],
        shard_key_selector=f"{key}",
    )

What Happens When We Turn Off a Node?

If we have configured our setup to be a 3+ node cluster with a replication factor of at least 2, then even if we stop one node, it’s not going to affect any operations going on in the cluster.

Let’s test that out.

First get ids of the containers using:

docker ps

Now let’s stop qdrant_node3.

docker stop 0b7fac4c224a

Next, let’s investigate our collection.

Even though we switched off a node, we can still access all the points we’ve inserted into our collection!

Data Distribution between Shards

To see how the data is distributed amongst the shards, we need to make a GET request in the following manner.

curl localhost:6333/collections/sharding_collection/cluster

{“result”:{“peer_id”:2024384091823610,”shard_count”:6,”local_shards”:[{“shard_id”:2,”shard_key”:”tempKey”,”points_count”:135,”state”:”Active”},{“shard_id”:4,”shard_key”:”tempKey”,”points_count”:178,”state”:”Active”},{“shard_id”:6,”shard_key”:”tempKey”,”points_count”:155,”state”:”Active”}],”remote_shards”:[{“shard_id”:1,”shard_key”:”tempKey”,”peer_id”:8555640407930483,”state”:”Active”},{“shard_id”:1,”shard_key”:”tempKey”,”peer_id”:3252676105267795,”state”:”Active”},{“shard_id”:2,”shard_key”:”tempKey”,”peer_id”:3088601394550397,”state”:”Active”},{“shard_id”:3,”shard_key”:”tempKey”,”peer_id”:8555640407930483,”state”:”Active”},{“shard_id”:3,”shard_key”:”tempKey”,”peer_id”:3252676105267795,”state”:”Active”},{“shard_id”:4,”shard_key”:”tempKey”,”peer_id”:3088601394550397,”state”:”Active”},{“shard_id”:5,”shard_key”:”tempKey”,”peer_id”:3252676105267795,”state”:”Active”},{“shard_id”:5,”shard_key”:”tempKey”,”peer_id”:8555640407930483,”state”:”Active”},{“shard_id”:6,”shard_key”:”tempKey”,”peer_id”:3088601394550397,”state”:”Active”}],”shard_transfers”:[]},”status”:”ok”,”time”:0.000067336}

To know the node by their peer_id, we’ll run:

curl http://localhost:6333/cluster

{“result”:{“status”:”enabled”,”peer_id”:2024384091823610,”peers”:{“3252676105267795”:{“uri”:”http://qdrant_node2:6335/"},"8555640407930483":{"uri":"http://qdrant_node3:6335/"},"2024384091823610":{"uri":"http://qdrant_node1:6335/"},"3088601394550397":{"uri":"http://qdrant_node4:6335/"}},"raft_info":{"term":10,"commit":40,"pending_operations":0,"leader":2024384091823610,"role":"Leader","is_voter":true},"consensus_thread_status":{"consensus_thread_status":"working","last_update":"2024-05-22T13:26:49.305470782Z"},"message_send_failures":{}},"status":"ok","time":0.000058492}

Analyzing the above results, and putting them in a clear format, we get:

• Node1 (current peer) has shards 2, 4, and 6.
• Node 3 has shards 1, 3, and 5.
• Node 2 has shards 1, 3, and 5.
• Node 4 has shards 2, 4, and 6.

By looking at the above distribution we see that each shard is available on 2 different nodes, and therefore even if one of the nodes fails, the cluster will continue to function.

Final Words

Multi-node clusters, sharding, and replicas are essential features for production environments. Multi-node clusters distribute the workload and enhance system resilience against individual node failures. Sharding divides data across multiple nodes, facilitating parallel processing that boosts performance and scalability. Replication ensures data availability, even if some nodes fail. Together, these techniques can create a robust and fault-tolerant system capable of reliably handling production-level demands.

References

https://qdrant.tech/documentation/guides/distributed_deployment/

https://github.com/Mohitkr95/qdrant-multi-node-cluster/tree/main

What Is a Multi-Hop Question?

A multi-hop question is a type of question that requires multiple steps or “hops” to arrive at an answer. Rather than being a straightforward query with a single answer, a multi-hop question involves breaking down the original question into smaller parts, each of which requires its own analysis.

They typically require critical thinking skills and the ability to connect information from different sources or domains. Multi-hop questions are common in complex problem-solving scenarios and research tasks, where the goal is to assess not only factual knowledge but also the ability to reason and synthesize information.

Here’s an example of a multi-hop question:

Texts:

- Text 1: “The Nile is the longest river in Africa and flows through several countries including Egypt and Sudan.”

- Text 2: “The ancient city of Thebes was located in Egypt along the banks of the Nile.”

- Text 3: “Thebes was known for its vast number of temples and was often referred to as the city of a hundred gates.”

Multi-Hop Question:

- “Which city known for its many temples was located along the longest river in Africa?”

How to Answer:

- First Hop: Identify the longest river in Africa. From Text 1, we learn it’s the Nile.

- Second Hop: Determine which city known for its many temples was situated along the Nile. From Texts 2 and Text 3, we can conclude that Thebes, known for its temples, was located along the Nile in Egypt.

Answer: Thebes

Challenges of Creating a Multi-Hop LLM Agent

Creating a Large Language Model (LLM) agent for multi-hop question-answering is particularly challenging due to the necessary prompt engineering involved.

This complex process requires the model to engage in multiple rounds of prompting where the main question must be deconstructed into manageable sub-questions. Each sub-question is then sequentially addressed, gathering the essential information step-by-step.

This meticulous structuring of prompts is crucial as it guides the LLM to navigate through and gather information from various sources effectively, ensuring a coherent and accurate final answer based solely on the initial, complex inquiry.

Therefore, in order to create an efficient multi-hop LLM application, one needs to invest a significant amount of time and effort in prompt engineering.

This process of prompt engineering is often a hit-and-trial method, where the developer has to tweak their prompts iteratively until they are satisfied with the LLM’s responses. Moreover, prompt templates are LLM specific — meaning different LLMs respond correctly to different types of prompting templates.

All of this makes multi-hop question-answering a rather complicated process.

Enter DSPy, a framework specially designed to do away with traditional prompt engineering methodologies.

How DSPy Takes Care of Prompt Engineering

DSPy has out-of-the-box prompting modules especially designed for various use-cases of LLM question-answering. It prioritizes programming over manual prompting, aiming to streamline tasks that traditionally required intricate prompt crafting or intensive model tweaking.

The framework introduces structured components like Signatures, Modules, and Optimizers to automate prompting for arbitrary pipelines, making the development process more efficient and effective.

Building a Multi-Hop RAG Pipeline with DSPy, Qdrant, and Ollama

DSPy has a readymade set of modules which can be used to construct a RAG pipeline that is capable of performing multi-hop question-answering.

In this tutorial, we’ll use Qdrant as our vector database to store the external information, and we’ll connect it to the LLM Llama3 using Ollama. Then we’ll use the tools provided by DSPy to build our multi-hop RAG.

First create a requirements.txt file.

dspy-ai
qdrant-client
ollama
wikipedia
fastembed
langchain

Install the modules by running ```pip install -r requirements.txt```

We’ll use a recursive text character splitter from LangChain to chunk our input documents.

from langchain_text_splitters import RecursiveCharacterTextSplitter


text_splitter = RecursiveCharacterTextSplitter(
    # Set a really small chunk size, just to show.
    chunk_size=512,
    chunk_overlap=20,
    length_function=len,
    is_separator_regex=False,
)

We’ll use the Wikipedia information on Leonardo DiCaprio as our external document set for the RAG pipeline.

from langchain_community.document_loaders import WikipediaLoader


docs = WikipediaLoader(query="Leonardo DiCaprio").load_and_split(text_splitter = text_splitter )

# List to hold the content of each document
doc_contents = [doc.page_content for doc in docs]


# List to hold the IDs for each document
doc_ids = list(range(1, len(docs) + 1))

doc_contents[0]

“Leonardo Wilhelm DiCaprio (; Italian: [diˈkaːprjo]; born November 11, 1974) is an American actor and film producer. Known for his work in biographical and period films, he is the recipient of numerous accolades, including an Academy Award, a British Academy Film Award, and three Golden Globe Awards. As of 2019, his films have grossed over $7.2 billion worldwide, and he has been placed eight times in annual rankings of the world’s highest-paid actors.”

Next, we’ll add all these documents into the Qdrant Vector DB. By default, Qdrant uses FastEmbed to calculate the embeddings of the text documents.

from qdrant_client import QdrantClient


# Initialize the client
client = QdrantClient(":memory:")


client.add(
    collection_name="leo_collection",
    documents=doc_contents,
    ids=doc_ids,
)

Now configure the Qdrant Retriever and Llama3 into DSPy.

from dspy.retrieve.qdrant_rm import QdrantRM
import dspy


qdrant_retriever_model = QdrantRM("leo_collection", client, k=10)


ollama_model = dspy.OllamaLocal(model="llama3",model_type='text',
                                max_tokens=350,
                                temperature=0.1,
                                top_p=0.8, frequency_penalty=1.17, top_k=40)


dspy.settings.configure(lm= ollama_model, rm=qdrant_retriever_model)

You can refer to this to install Ollama. Make sure you have pulled llama3 br running ```ollama pull llama3```

We’ll first define a DSPy signature which will generate an answer given a context and a question.

class GenerateAnswer(dspy.Signature):
    """Answer questions with short factoid answers."""


    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    answer = dspy.OutputField()

Then we’ll define another signature which will help define the intermediate query for the multi-hop.

class GenerateSearchQuery(dspy.Signature):
    """Write a simple search query that will help answer a complex question."""


    context = dspy.InputField(desc="may contain relevant facts")
    question = dspy.InputField()
    query = dspy.OutputField()

Then we’ll construct a DSPy module that will generate intermediate queries to collect all the relevant context and then use that to answer the user’s questions.

from dsp.utils import deduplicate


class SimplifiedBaleen(dspy.Module):
    def __init__(self, passages_per_hop=3, max_hops=2):
        super().__init__()


        self.generate_query = [dspy.ChainOfThought(GenerateSearchQuery) for _ in range(max_hops)]
        self.retrieve = dspy.Retrieve(k=passages_per_hop)
        self.generate_answer = dspy.ChainOfThought(GenerateAnswer)
        self.max_hops = max_hops


    def forward(self, question):
        context = []


        for hop in range(self.max_hops):
            query = self.generate_query[hop](context=context, question=question).query
            passages = self.retrieve(query).passages
            context = deduplicate(context + passages)


        pred = self.generate_answer(context=context, question=question)
        return dspy.Prediction(context=context, answer=pred.answer)

The `__init__` method in our code sets up several important components:

- generate_query: For each step, or “hop,” in our process, this component uses a special tool called dspy.ChainOfThought with the GenerateSearchQuery setup to create a search query. dspy.ChainOfThought makes sure that the LLM reasons through the prompts in a step-by-step fashion.

- retrieve: This component takes the search queries and finds relevant information from the Qdrant database using the dspy.Retrieve module.

- generate_answer: After gathering all the required information, this dspy.Predict module, configured with the GenerateAnswer setup, is used to formulate the final response.

In the `forward` method, the process is laid out in a sequence of steps:

1. We loop through a set number of times, defined by `self.max_hops`.

2. In each cycle of the loop:

- A search query is generated using the generate_query component for the current hop.

- This query is used to fetch the top relevant pieces of information (top-k passages).

- We collect these passages, ensuring there are no duplicates, and add them to our collection of context information.

3. After completing all the hops, we use the generate_answer component to create the final answer based on the accumulated context.

4. Finally, the process returns a prediction that includes both the gathered context and the generated answer.

Now let’s try asking a multi-hop question to our pipeline:

# Ask any question you like to this simple RAG program.
my_question = "Give me all the co-actors of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?"


# Get the prediction. This contains `pred.context` and `pred.answer`.
uncompiled_baleen = SimplifiedBaleen()  # uncompiled (i.e., zero-shot) program
pred = uncompiled_baleen(my_question)


# Print the contexts and the answer.
print(f"Question: {my_question}")
print(f"Predicted Answer: {pred.answer}")
print(f"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}")

Question: Give me all the co-actors of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Predicted Answer: Jesse Plemons, Tantoo Cardinal, John Lithgow, and Brendan Fraser.

Retrieved Contexts (truncated): [‘Leonardo DiCaprio, Robert De Niro, and Lily Gladstone lead an ensemble cast, also including Jesse Plemons, Tantoo Cardinal, John Lithgow, and Brendan Fraser. It is the sixth feature film collaboration…’, ‘== Film ==\n\n\n=== As actor ===\n\n\n=== As producer ===\n\n\n== Television ==\n\n\n== Notes ==\n\n\n== See also ==\nList of awards and nominations received by Leonardo DiCaprio\n\n\n== References ==\n\n\n== External link…’, “The duo’s films have been nominated for thirty-one Academy Awards, winning nine. In 2013, the duo was awarded National Board of Review Spotlight award for career collaboration. Scorsese’s work with Di…”, ‘== References ==\n\n\n== External links ==\nGeorge DiCaprio at IMDb…’]

We can inspect the entire prompting process that went under the hood by running the following command:

ollama_model.inspect_history(n=3)

Write a simple search query that will help answer a complex question.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the query}. We …

Query: ${query}

— -

Context: N/A

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step in order to Here is a simple search query that can help answer this complex question:

— -

Context: N/A

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step in order to produce the query. We need to find a movie where both Leonardo DiCaprio and Robert De Niro appeared, then extract their co-stars from that specific film.

Query: (Leonardo DiCaprio) AND (Robert De Niro) AND (“co-star” OR “cast list”) -site:imdb.com

This query uses the following search operators:

* `(Leonardo DiCaprio)` and `(Robert De Niro)` are exact phrase searches to ensure we’re searching for both actors’ names.

* `AND` connects these two phrases, requiring them to appear together in a single result.

* `”co-star” OR “cast list”` is a quoted search that looks for either the words “co-star” or “cast list” (or variations) within the results. This helps us find pages with lists of co-stars from specific movies.

* `-site:imdb.com` excludes IMDb’s website from our search results, as we’re looking for information not already available on their site.

By combining these operators and phrases, this query should return a list of movie titles where both Leonardo DiCaprio and Robert De Niro appeared together, along with the names of their co-stars in those films.

Write a simple search query that will help answer a complex question.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the query}. We …

Query: ${query}

— -

Context:

[1] «Leonardo DiCaprio, Robert De Niro, and Lily Gladstone lead an ensemble cast, also including Jesse Plemons, Tantoo Cardinal, John Lithgow, and Brendan Fraser. It is the sixth feature film collaboration between Scorsese and DiCaprio, the tenth between Scorsese and De Niro, and the first between Scorsese and both actors overall (they previously all collaborated on the 2015 short film The Audition), and the eleventh and final between Scorsese and composer Robbie Robertson, who died two months prior to the»

[2] «== Film ==

=== As actor ===

=== As producer ===

== Television ==

== Notes ==

== See also ==

List of awards and nominations received by Leonardo DiCaprio

== References ==

== External links ==

Leonardo DiCaprio filmography at IMDb

Leonardo DiCaprio at the TCM Movie Database»

[3] «The duo’s films have been nominated for thirty-one Academy Awards, winning nine. In 2013, the duo was awarded National Board of Review Spotlight award for career collaboration. Scorsese’s work with DiCaprio is considered to be as vital as his work with Robert De Niro.»

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step in order to — -

Context:

[1] «Leonardo DiCaprio, Robert De Niro, and Lily Gladstone lead an ensemble cast…»

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step to produce a query. We know that we need to find information about movies where both Leonardo DiCaprio and Robert De Niro have appeared together as actors.

Query: (Leonardo DiCaprio AND Robert De Niro) site:movies

This search query uses the following techniques:

* The parentheses around the names ensure that they are searched for in combination, rather than separately.

* The “AND” operator is used to require both names to appear in the same movie.

* The “site:” operator limits the search results to web pages with a URL containing “movies”, which should help us focus on relevant information about movies.

By using this query, we can find articles or websites that list the co-stars of Leonardo DiCaprio in movies where he has appeared alongside Robert De Niro.

Answer questions with short factoid answers.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the answer}. We …

Answer: ${answer}

— -

Context:

[1] «Leonardo DiCaprio, Robert De Niro, and Lily Gladstone lead an ensemble cast, also including Jesse Plemons, Tantoo Cardinal, John Lithgow, and Brendan Fraser. It is the sixth feature film collaboration between Scorsese and DiCaprio, the tenth between Scorsese and De Niro, and the first between Scorsese and both actors overall (they previously all collaborated on the 2015 short film The Audition), and the eleventh and final between Scorsese and composer Robbie Robertson, who died two months prior to the»

[2] «== Film ==

=== As actor ===

=== As producer ===

== Television ==

== Notes ==

== See also ==

List of awards and nominations received by Leonardo DiCaprio

== References ==

== External links ==

Leonardo DiCaprio filmography at IMDb

Leonardo DiCaprio at the TCM Movie Database»

[3] «The duo’s films have been nominated for thirty-one Academy Awards, winning nine. In 2013, the duo was awarded National Board of Review Spotlight award for career collaboration. Scorsese’s work with DiCaprio is considered to be as vital as his work with Robert De Niro.»

[4] «== References ==

== External links ==

George DiCaprio at IMDb»

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step in order to — -

Context:

[1] «Leonardo DiCaprio, Robert De Niro, and Lily Gladstone lead an ensemble cast…»

Question: Give me all the co-stars of Leonardo DiCaprio in the movie in which one of his co-stars was Robert De Niro?

Reasoning: Let’s think step by step to find out who else starred alongside Leonardo DiCaprio and Robert De Niro.

Answer: According to [1], the other co-stars were Jesse Plemons, Tantoo Cardinal, John Lithgow, and Brendan Fraser.

We tried the above workflow with a different set of documents and questions.

docs = WikipediaLoader(query="devops").load_and_split(text_splitter = text_splitter )

We populated the vector store Qdrant with documents related to “devops”. And then we asked the following question:

my_question = """Given a DevOps environment where automated testing and continuous integration are fully implemented,
consider a scenario where a new version of the software is planned for release. First, identify what type of automated
tests would likely be run first as part of the CI pipeline. Then, assuming there are failures in these initial tests,
explain how the use of Infrastructure as Code (IaC) could potentially impact the resolution of these failures."""


# Get the prediction. This contains `pred.context` and `pred.answer`.
uncompiled_baleen = SimplifiedBaleen()  # uncompiled (i.e., zero-shot) program
pred = uncompiled_baleen(my_question)


# Print the contexts and the answer.
print(f"Question: {my_question}")
print(f"Predicted Answer: {pred.answer}")
print(f"Retrieved Contexts (truncated): {[c[:200] + '...' for c in pred.context]}")

Predicted Answer: The initial automated tests that would likely be run first as part of the CI pipeline are unit tests. These tests verify individual components or units of code work correctly.

If these initial unit tests fail, Infrastructure as Code (IaC) could potentially impact resolution in several ways:

1. **Infrastructure consistency**: IaC ensures consistent infrastructure configuration across environments, reducing errors and making it easier to reproduce issues.

2. **Automated remediation**: With IaC, you can automate the creation of a new environment or restore a previous one if there’s an issue with the current setup.

3. **Version control for infrastructure**: By managing infrastructure as code, you can track changes and roll back to previous versions if needed.

By leveraging IaC in this scenario, developers can quickly identify and resolve issues related to infrastructure configuration, allowing them to focus on fixing the root cause of the problem rather than manually configuring environments or troubleshooting.

Retrieved Contexts (truncated): [‘== Definition ==\nOther than it being a cross-functional combination (and a portmanteau) of the terms and concepts for “development” and “operations”, academics and practitioners have not developed a u…’, ‘There are many tools that fulfill infrastructure automation capabilities and use IaC. Broadly speaking, any framework or tool that performs changes or configures infrastructure declaratively or impera…’, ‘The definitions may be in a version control system, rather than maintaining the code through manual processes.\nThe code in the definition files may use either scripts or declarative definitions, but I…’, ‘=== Continuous configuration automation ===\nAll continuous configuration automation (CCA) tools c…’, ‘== Etymology ==\nIn Lessons Learned in Software Testing, Cem Kaner, James Bach, and Brett Pettichord provided the origin of the term: “The phrase smoke test comes from electronic hardware testing. You …’, ‘In software engineering, CI/CD or CICD is the combined practices of continuous integration (CI) and continuous delivery (CD) or, less often, continuous deployment. They are sometimes referred to colle…’]

This is the prompting process that went on under the hood:

ollama_model.inspect_history(n=3)

Write a simple search query that will help answer a complex question.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the query}. We …

Query: ${query}

— -

Context: N/A

Question: Given a DevOps environment where automated testing and continuous integration are fully implemented, consider a scenario where a new version of the software is planned for release. First, identify what type of automated tests would likely be run first as part of the CI pipeline. Then, assuming there are failures in these initial tests, explain how the use of Infrastructure as Code (IaC) could potentially impact the resolution of these failures.

Reasoning: Let’s think step by step in order to Here is a simple search query that can help answer this complex question:

— -

Context: N/A

Question: Given a DevOps environment where automated testing and continuous integration are fully implemented, consider a scenario where a new version of the software is planned for release. First, identify what type of automated tests would likely be run first as part of the CI pipeline. Then, assuming there are failures in these initial tests, explain how the use of Infrastructure as Code (IaC) could potentially impact the resolution of these failures.

Reasoning: Let’s think step by step in order to produce a query that can help answer this question.

Query: (“continuous integration” OR “CI pipeline”) AND (“automated testing” OR “test automation”) AND (“Infrastructure as Code” OR “IaC”) AND (failure* OR error*) — This search query combines relevant terms related to DevOps, automated testing, and Infrastructure as Code. The use of parentheses helps to group the terms logically, allowing for more precise results.

This query can help answer the question by providing information on:

1. What types of automated tests are typically run first in a CI pipeline.

2. How IaC can impact the resolution of failures or errors that occur during these initial tests.

3. Strategies and best practices for using IaC to manage infrastructure changes and reduce downtime.

By searching with this query, you should be able to find relevant articles, blog posts, or documentation that provide insights on how to approach this scenario in a DevOps environment.

Write a simple search query that will help answer a complex question.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the query}. We …

Query: ${query}

— -

Context:

[1] «== Definition ==

Other than it being a cross-functional combination (and a portmanteau) of the terms and concepts for “development” and “operations”, academics and practitioners have not developed a universal definition for the term “DevOps”. Most often, DevOps is characterized by key principles: shared ownership, workflow automation, and rapid feedback.»

[2] «There are many tools that fulfill infrastructure automation capabilities and use IaC. Broadly speaking, any framework or tool that performs changes or configures infrastructure declaratively or imperatively based on a programmatic approach can be considered IaC. Traditionally, server (lifecycle) automation and configuration management tools were used to accomplish IaC. Now enterprises are also using continuous configuration automation tools or stand-alone IaC frameworks, such as Microsoft’s PowerShell DSC»

[3] «The definitions may be in a version control system, rather than maintaining the code through manual processes.

The code in the definition files may use either scripts or declarative definitions, but IaC more often employs declarative approaches.»

Question: Given a DevOps environment where automated testing and continuous integration are fully implemented, consider a scenario where a new version of the software is planned for release. First, identify what type of automated tests would likely be run first as part of the CI pipeline. Then, assuming there are failures in these initial tests, explain how the use of Infrastructure as Code (IaC) could potentially impact the resolution of these failures.

Reasoning: Let’s think step by step in order to Here is a simple search query that can help answer this complex question:

— -

Context:

[1] «== Definition ==`

[2] «Infrastructure Automation»

[3] `Continuous Integration`

Question: What types of automated tests are typically run first in a CI pipeline, and how does IaC impact the resolution of test failures?

Reasoning: Let’s think step by step in order to produce the query. We want to identify what type of automated tests would likely be run first as part of the CI pipeline, considering that DevOps environment where automated testing and continuous integration are fully implemented.

Query: (CI OR Continuous Integration) AND (“automated test” OR “test automation”) AND (“initial test” OR “first test”) -site:.gov

This query uses a combination of keywords to search for relevant information. The CI or Continuous Integration terms help narrow down the results to discussions about continuous integration and automated testing in DevOps environments. The additional phrases, such as “automated test”, “test automation”, and “initial test” or “first test”, further refine the search by looking for specific types of tests that are typically run first in a CI pipeline. Finally, excluding government (.gov) sites helps to reduce irrelevant results from academic or research-focused sources.

This query should return relevant information about which types of automated tests are typically run first in a CI pipeline and how IaC might impact the resolution of test failures.

Answer questions with short factoid answers.

— -

Follow the following format.

Context: may contain relevant facts

Question: ${question}

Reasoning: Let’s think step by step in order to ${produce the answer}. We …

Answer: ${answer}

— -

Context:

[1] «== Definition ==

Other than it being a cross-functional combination (and a portmanteau) of the terms and concepts for “development” and “operations”, academics and practitioners have not developed a universal definition for the term “DevOps”. Most often, DevOps is characterized by key principles: shared ownership, workflow automation, and rapid feedback.»

[2] «There are many tools that fulfill infrastructure automation capabilities and use IaC. Broadly speaking, any framework or tool that performs changes or configures infrastructure declaratively or imperatively based on a programmatic approach can be considered IaC. Traditionally, server (lifecycle) automation and configuration management tools were used to accomplish IaC. Now enterprises are also using continuous configuration automation tools or stand-alone IaC frameworks, such as Microsoft’s PowerShell DSC»

[3] «The definitions may be in a version control system, rather than maintaining the code through manual processes.

The code in the definition files may use either scripts or declarative definitions, but IaC more often employs declarative approaches.»

[4] «=== Continuous configuration automation ===

All continuous configuration automation (CCA) tools c»

[5] «== Etymology ==

In Lessons Learned in Software Testing, Cem Kaner, James Bach, and Brett Pettichord provided the origin of the term: “The phrase smoke test comes from electronic hardware testing. You plug in a new board and turn on the power. If you see smoke coming from the board, turn off the power. You don’t have to do any more testing.”

== See also ==

“Hello, World!” program

Sanity testing

Shakedown

== References ==

== External links ==

PC Mag’s Definition»

[6] «In software engineering, CI/CD or CICD is the combined practices of continuous integration (CI) and continuous delivery (CD) or, less often, continuous deployment. They are sometimes referred to collectively as continuous development or continuous software development.»

Question: Given a DevOps environment where automated testing and continuous integration are fully implemented, consider a scenario where a new version of the software is planned for release. First, identify what type of automated tests would likely be run first as part of the CI pipeline. Then, assuming there are failures in these initial tests, explain how the use of Infrastructure as Code (IaC) could potentially impact the resolution of these failures.

Reasoning: Let’s think step by step in order to Context:

* DevOps environment with automated testing and continuous integration implemented

* New version of software planned for release

Question: What type of automated tests would likely be run first as part of the CI pipeline, and how could IaC impact resolution if there are failures?

Reasoning: Let’s think step by step in order to…

Answer:

The initial automated tests that would likely be run first as part of the CI pipeline are unit tests. These tests verify individual components or units of code work correctly.

If these initial unit tests fail, Infrastructure as Code (IaC) could potentially impact resolution in several ways:

1. **Infrastructure consistency**: IaC ensures consistent infrastructure configuration across environments, reducing errors and making it easier to reproduce issues.

2. **Automated remediation**: With IaC, you can automate the creation of a new environment or restore a previous one if there’s an issue with the current setup.

3. **Version control for infrastructure**: By managing infrastructure as code, you can track changes and roll back to previous versions if needed.

By leveraging IaC in this scenario, developers can quickly identify and resolve issues related to infrastructure configuration, allowing them to focus on fixing the root cause of the problem rather than manually configuring environments or troubleshooting.

GitHub

You can find the complete code in the GitHub link here:
https://github.com/vardhanam/dspy-multihop-qdrant

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Venture | Cubed
• Tired of blogging platforms that force you to deal with algorithmic content? Try Differ
• More content at Stackademic.com

Multi-Hop Retrieval and Reasoning for Complex Questions Using DSPy, Qdrant, and Llama3 was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

What Is a Re-ranker?

A Re-ranker is essentially a transformer model that takes a query and a list of documents as input. And it returns the list of the documents as output but now they are re-ranked in relevance to the query. The documents which are most relevant to the query are ahead in the list compared to the ones that are not meaningfully related to the query.

This feature is extremely useful especially when feeding external data to an LLM from a corpus of billions of text. A Re-ranker in this scenario would rank the documents with respect to the user query, and send only the top relevant documents to the LLM for generating a response.

Re-rankers are better than vector searches in figuring out relevancy. This is because in vector searches the documents are first converted into numerical embeddings and then stored in a vector database. This conversion into vector embeddings causes a significant amount of data loss from the actual text to its vector representation.

Re-rankers, on the other hand, don’t cause any data loss since they take the input as the entire document itself without any kind of conversion into vector embeddings.

Why Not Use Only a Re-ranker Instead of Vector Search?

So if Re-rankers perform better than vector searches, the question arises: “Why not use them entirely in feeding external data to LLMs instead of using a vector search approach?”

The reason behind this is that Re-rankers are extremely slow in processing data compared to a Vector DB. Especially if there are millions of documents in the external corpus, it would take several hours for a Re-ranker to rank them. This is because the Transformers model would have to process millions of document query pairs, one by one.

A vector search, which is based on calculating cosine similarity between the query vector and the document vector, achieves this in a matter of seconds. Vector Databases are essentially designed to facilitate a cosine similarity search at lightning speed.

That’s why we used a hybrid method combining both Vector Search and Re-ranking.

Combining Vector Search and Re-ranking

In this method the Vector Search is used to do the heavy lifting, i.e., find documents similar to the query from a large corpus of text.

Then Re-ranking is applied only on the top documents to refine the results. This way we optimize our search mechanism — the speedy vector search sorts through the millions of documents to give a broad result, while slower albeit more accurate Re-ranker re-arranges the top results to give us a more meaningful response.

How to Use Re-rankers

These are some of the popular re-rankers out there:

• bge-reranker-large
• Jina Reranker
• Cohere Reranker

We’ll be using the ckandemir/amazon-products dataset from Hugging Face. It contains names and descriptions of about 24k products.

First, install the dependencies.

langchain
qdrant-client
SQLAlchemy==2.0.1
datasets

Load the dataset.

from datasets import load_dataset

df = load_dataset('ckandemir/amazon-products', split='train').to_pandas()

Convert it into document format.

from langchain.docstore.document import Document

documents = []
for index, row in df.iterrows():
    text = f"Product Name: {row['Product Name']}\nProduct Category: {row['Category']}\nProduct Description: {row['Description']}"
    document = Document(page_content=text)
    documents.append(document)

Load the embeddings model.

from langchain_community.embeddings import HuggingFaceEmbeddings
embeddings = HuggingFaceEmbeddings(model_name  = "BAAI/bge-base-en-v1.5")

Now index all the documents into the vector database Qdrant.

from langchain_community.vectorstores import Qdrant
qdrant = Qdrant.from_documents(
    documents,
    embeddings,
    location=":memory:",  # Local mode with in-memory storage only
    collection_name="reranker",
)

We’ll make a retriever object that will fetch the top 10 results from the vector database without re-ranking them.

retriever = qdrant.as_retriever(search_kwargs = {'k':10})

Next we’ll create another retriever object that will re-rank the results fetched from the vector database:

from langchain.retrievers import ContextualCompressionRetriever
from langchain.retrievers.document_compressors import CrossEncoderReranker
from langchain_community.cross_encoders import HuggingFaceCrossEncoder

model = HuggingFaceCrossEncoder(model_name="BAAI/bge-reranker-base")
compressor = CrossEncoderReranker(model=model, top_n=10)
compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor, base_retriever=retriever
)

Let’s write a function which compares the results of the retrievers with & without reranking.

import pandas as pd

def comparison_df(question):
    # Retrieve relevant documents using retriever
    retriever_documents = retriever.get_relevant_documents(question)
    
    # Take top 5 document page contents from retriever
    retriever_contents = [doc.page_content for doc in retriever_documents]
    
    # Retrieve relevant documents using compression_retriever
    compression_documents = compression_retriever.invoke(question)
    
    # Take document page contents from compression_retriever
    compression_contents = [doc.page_content for doc in compression_documents]
    
    # Create a DataFrame with separate columns for each retriever
    df = pd.DataFrame({
        'without_reranking': retriever_contents,
        'after_reranking': compression_contents
    })
    
    return df

df_comparison = comparison_df("Give me a list of Toy Figures of Iron-Man")

Code to implement Jina Reranker. In the above code, just replace the compressor.

from langchain_community.document_compressors import JinaRerank

compressor = JinaRerank()
compression_retriever = ContextualCompressionRetriever(
   base_compressor=compressor, base_retriever=retriever
)

Similarly, code to implement Cohere Ranker.

from langchain_cohere import CohereRerank

compressor = CohereRerank()
compression_retriever = ContextualCompressionRetriever(
   base_compressor=compressor, base_retriever=retriever
)

Final Words

It’s quite a straightforward process to connect the Re-ranker retriever to an LLM to develop a RAG pipeline:

from langchain.chains import RetrievalQA
from langchain_openai import OpenAI

llm = OpenAI(openai_api_key="YOUR_API_KEY")

chain = RetrievalQA.from_chain_type(
   llm=llm, retriever=compression_retriever
)

chain.invoke("Your query goes here")

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Venture | Cubed
• More content at Stackademic.com

How to Use Re-Rankers for Advanced RAG Pipelines Using Qdrant was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.

“Over the last 10–15 years, almost everybody who sits on a stage like this would tell you that it is vital that your children learn computer science; everybody should learn how to program. In fact, it is almost exactly the opposite. It is our job to create computing technology such that nobody has to program, and that the programming language is human. Everybody in the world is now a programmer. This is the miracle of AI.” — Jensen Huang, CEO of NVIDIA

What Is Open Devin?

Open Devin is a completely autonomous AI agent that can create entire software stacks starting from just a single user prompt. It has been inspired by Devin, the cutting-edge proprietary AI software engineering tool.

Like Devin, Open Devin is designed to navigate the complexities of software engineering by leveraging a combination of tools such as a shell, code editor, and web browser. By harnessing the power of LLMs, Open Devin can autonomously handle the complete software development lifecycle, from understanding user requirements and writing code to testing and deployment.

This AI-driven approach to software development allows for the creation of intricate applications with reduced human input, thus unlocking new opportunities for swift innovation.

Through the power of open-source collaboration, Open Devin aspires to replicate, enhance, and innovate upon the original Devin model. The project’s goal is to guide the progress of open code models and demonstrate the potential of autonomous AI agents in executing complex engineering tasks and actively collaborating with users on software development projects.

How to Set It Up

Requirements

Conda env with python 3.11

conda create -n opendevin python=3.11

conda activate opendevin

Docker

You can install Docker by running the following commands:

# Add Docker's official GPG key:

sudo apt-get update

sudo apt-get install ca-certificates curl

sudo install -m 0755 -d /etc/apt/keyrings

sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc

sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:

echo \

 "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu \

 $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \

 sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Node.js

Run these commands to install Node.js.

# installs NVM (Node Version Manager)
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash

# download and install Node.js
nvm install 20

# verifies the right Node.js version is in the environment
node -v # should print `v20.12.2`

# verifies the right NPM version is in the environment
npm -v # should print `10.5.0`

Poetry

First install pipx.

sudo apt update
sudo apt install pipx
pipx ensurepath
sudo pipx ensurepath --global # optional to allow pipx actions with --global argument

Then install Poetry.

pipx install poetry

Build the Project

First download the repository.

git clone https://github.com/nnilayy/OpenDevin.git
cd OpenDevin

Then build:

make build

Set Up the LLM

We’ll use Ollama to set up our LLM.

Install Ollama.

curl -fsSL https://ollama.com/install.sh | sh

Start the server.

ollama serve

This will start a Ollama server at http://localhost:11434.

If you get an error like this:

Error: listen tcp 127.0.0.1:11434: bind: address already in use

It just means the server is already running.

Stop the server and run it again:

sudo systemctl stop ollama

Let’s use the Mistral-7B LLM.

ollama pull mistral:7b

Now open the config.toml file and paste the following in it.

LLM_API_KEY="ollama"
LLM_MODEL="ollama/mistral:7b"
LLM_EMBEDDING_MODEL="local"
LLM_BASE_URL="http://localhost:11434"
WORKSPACE_DIR="./workspace"

The Workspace_Dir is the folder that will be created by Open Devin to save all the software tasks it’s given to perform.

Now open the makefile and go to the run: section (#Run the app). There, add — host 0.0.0.0 to the following line:

@cd frontend && echo "$(BLUE)Starting frontend with npm...$(RESET)" && \
npm run start -- --port $(FRONTEND_PORT) --host 0.0.0.0

This will ensure that the front end of Open Devin will be exposed to users who are accessing it from external routes.

Run Open Devin

make run

This will launch the front end at the following address:

<server-ip>:3001

Enjoy building your very own software stack!

How to Set Up Open Devin, the Open Source AI Software Engineer was originally published in Superteams.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

A talking chatbot for investigating your pdf documents

How Voice-to-Voice RAG Works

Voice to Voice RAG (Retrieval-Augmented Generation) chatbots are an exciting advancement in conversational AI. Moving away from old-school text-generating chatbots, voice chatbots can facilitate more natural and interactive human-machine interaction.

By playing upon the power of voice input and output, along with RAG techniques, these chatbots can deliver intelligent and contextually-aware responses to user queries.

Here’s how the workflow of a Voice-RAG looks like:

1. User uploads documents to create a knowledge base.

2. Documents are processed, split into chunks, and stored in a Vector Database.

3. User asks a question using voice input.

4. Automatic Speech Recognition (ASR) model converts voice to text.

5. Similarity search on the Vector Database retrieves relevant context.

6. Original query and retrieved context are combined to create an augmented query.

7. Large Language Model (LLM) generates a response based on the augmented query.

8. Text to Speech (TTS) model converts the generated response into voice output for the user.

Tech Stack Used in This Blog

• Gradio is used to build the user interface, allowing users to easily upload documents, record voice queries, and receive voice responses.
• OpenAI Whisper, a state-of-the-art Automatic Speech Recognition (ASR) model, accurately transcribes the user’s voice query into text.
• Qdrant, a high-performance Vector Database, stores the processed document chunks and helps with fast similarity searches to retrieve relevant context.
• Mistral-7B, the powerful Large Language Model (LLM), generates coherent and contextually relevant responses based on the augmented query.
• Whisper Speech, a Text to Speech (TTS) model, synthesizes natural-sounding speech from the generated text response, providing the final voice output to the user.

Let’s Code

Create a requirements.txt file and paste the following in it:

transformers
accelerate
bitsandbytes
gradio
langchain
pypdf
qdrant-client
sentence-transformers
torch
torchaudio
openai-whisper

Now install the dependencies using the following command:

pip install -r requirements.txt

Spin up a Jupyter Notebook in your Python environment, and import all these modules. We’ll be using LangChain as our framework for handling all the data. It is an efficient and robust framework that takes care of a lot of underlying coding work by abstracting it into seamless modules.

from transformers import (
AutoTokenizer,
AutoModelForCausalLM,
BitsAndBytesConfig,
pipeline
)

import torch
import torch.nn.functional as F

import os

from langchain.llms import HuggingFacePipeline
from langchain_community.document_loaders import PyPDFLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Qdrant
from langchain.embeddings import HuggingFaceEmbeddings

import gradio as gr

import whisper
from whisperspeech.pipeline import Pipeline

import uuid

Next we’ll write a function to generate a text generation pipeline using Mistral-7B. We’ll use a quantized version of the model and reduce the precision to 4-bit. This will improve speed and efficiency of our pipeline.

def load_llm():

    #Loading the Mistral Model
    model_name='mistralai/Mistral-7B-Instruct-v0.2'
    tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
    tokenizer.pad_token = tokenizer.eos_token
    tokenizer.padding_side = "right"

    bnb_config = BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_quant_type="nf4",
        bnb_4bit_use_double_quant=True,
        bnb_4bit_compute_dtype=torch.bfloat16
    )


    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        quantization_config=bnb_config,
    )

    # Building a LLM text-generation pipeline
    text_generation_pipeline = pipeline(
        model=model,
        tokenizer=tokenizer,
        task="text-generation",
        temperature=0.2,
        repetition_penalty=1.1,
        return_full_text=True,
        max_new_tokens=1024,
        device_map = 'auto',
    )


    return text_generation_pipeline

Next we’ll define a text splitter to chunk the user uploaded documents into smaller chunks.

def text_splitter():
    # Simulate some document processing delay
    text_splitter = RecursiveCharacterTextSplitter(
        chunk_size=512,
        chunk_overlap=20,
        length_function=len,
        is_separator_regex=False,
    )
    return text_splitter

We’ll then write a function to upsert the user uploaded files into the Qdrant vector database. This function is capable of taking multiple pdf files as input. It then spits the texts in these documents into smaller chunks, then generates vector embeddings for these chunks before finally adding it to the vector DB.

def add_pdfs_to_vectorstore(files):

    saved_files_count = 0
    documents = []
    for file_path in files:
        file_name = os.path.basename(file_path)  # Extract the filename from the full path
        if file_name.lower().endswith('.pdf'):  # Check if the file is a PDF
            saved_files_count += 1
            loader_temp = PyPDFLoader(file_path)
            docs_temp = loader_temp.load_and_split(text_splitter=textsplitter)
            for doc in docs_temp:
                # Replace all occurrences of '\n' with a space ' '
                doc.page_content = doc.page_content.replace('\n', ' ')
            documents = documents + docs_temp

        else:
            print(f"Skipping non-PDF file: {file_name}")
            
    global qdrant
    
    qdrant = Qdrant.from_documents(
        documents,
        HuggingFaceEmbeddings(),
        location=":memory:", 
    )

    return f"Added {saved_files_count} PDF file(s) to vectorstore/ You can begin voice chat"

Then we’ll write the function to create an augmented query from the user’s input query. This is done performing a similarity search on the VectorDB to retrieve the relevant context. Then this context+query is passed onto the LLM for generating the response.

def answer_query(message):
    context_docs = qdrant.similarity_search(message, k= 10)
    context = ' '.join(doc.page_content for doc in context_docs)

    template = f"""Answer the question based only on the following context:
        {context}

        Question: {message}
    """

    result = llm(template)

    answer = result[0]["generated_text"].replace(template, '')

    return answer

We’ll load all the models (ASR, TTS, LLM).

whisper_model = whisper.load_model("base")
whisper_speech_model = Pipeline(s2a_ref='collabora/whisperspeech:s2a-q4-tiny-en+pl.model')
llm = load_llm()
textsplitter = text_splitter()

We’ll now write a function to transcribe the input audio into text so that it can then be passed downstream to the VectorDB and the LLM. This function takes the audio location as the input.

def transcribe(audio):
    
    result = whisper_model.transcribe(audio)
    return result["text"]

Then we’ll create a function to convert the LLM’s final response into speech. The output file is saved in a temporary location which is automatically deleted later on.

def generate_and_play_audio(text):
    # Construct the directory and filename
    directory = '/var/tmp/gradio/'
    filename = str(uuid.uuid4()) + "/audio.wav"
    file_location = os.path.join(directory, filename)
    
    # Ensure that the directory exists
    os.makedirs(os.path.dirname(file_location), exist_ok=True)
    
    # Generate the audio file from text and save to the specified location
    whisper_speech_model.generate_to_file(file_location, text, lang ='en', cps=15)

    # Return the location of the saved audio file for playback
    return file_location

This is the code for the Gradio UI:

with gr.Blocks() as demo:
    
    with gr.Row():
        upload_files = gr.File(label="Upload pdf files only", file_count='multiple')
        success_msg = gr.Text(value="")
    
    with gr.Row():
        audio_inp = gr.Audio(sources="microphone", type='filepath')
        trans_out = gr.Textbox()
    
    with gr.Row():
        btn_audio = gr.Button("Submit Audio")
    
    with gr.Row():
        model_response = gr.Textbox(label= "Model Response", lines = 20)
        audio_out = gr.Audio(label="AI response in Voice")
        
    with gr.Row():
        clear_btn = gr.Button("Clear All")
        
    
    upload_files.upload(add_pdfs_to_vectorstore, upload_files, success_msg)
    transcribe = btn_audio.click(fn=transcribe, inputs=audio_inp, outputs=trans_out)
    answer_gen = transcribe.then(fn=answer_query, inputs= trans_out, outputs= model_response)
    answer_gen.then(fn=generate_and_play_audio, inputs= model_response, outputs= audio_out)
    clear_btn.click(fn= lambda: (None,"","",None), inputs= None, outputs= [audio_inp, trans_out, model_response, audio_out])

demo.queue().launch(server_name= "0.0.0.0", share=True)

Let’s break the code down:

1. `with gr.Blocks() as demo:` creates a Gradio interface named `demo`.

2. Inside `demo`, there are several `gr.Row()` components that organize the interface elements horizontally:

- The first row contains an `upload_files` component for uploading PDF files and a `success_msg` component to display a success message.

- The second row has an `audio_inp` component for recording audio input and a `trans_out` component to display the transcribed text.

- The third row has a `btn_audio` component, which is a button to submit the recorded audio.

- The fourth row contains a `model_response` component to display the generated response from the model and an `audio_out` component to play the AI-generated voice response.

- The fifth row has a `clear_btn` component to clear all inputs and outputs.

3. The `upload_files.upload()` method is used to trigger the `add_pdfs_to_vectorstore` function when files are uploaded, passing the uploaded files and updating the `success_msg` component.

4. The `btn_audio.click()` method is used to trigger the `transcribe` function when the “Submit Audio” button is clicked, passing the recorded audio input (`audio_inp`) and updating the `trans_out` component with the transcribed text.

5. The `transcribe.then()` method is used to chain the `answer_query` function after the transcription is complete, passing the transcribed text (`trans_out`) and updating the `model_response` component with the generated response.

6. The `answer_gen.then()` method is used to chain the `generate_and_play_audio` function after the response generation is complete, passing the generated response (`model_response`) and updating the `audio_out` component with the AI-generated voice response.

7. The `clear_btn.click()` method is used to clear all inputs and outputs when the “Clear All” button is clicked, setting the values of `audio_inp`, `trans_out`, `model_response`, and `audio_out` to `None` or empty strings.

8. Finally, `demo.queue().launch(server_name= “0.0.0.0”, share=True)` launches the Gradio interface, enabling queuing to handle multiple requests and making it accessible over the network with the `share=True` option.

Demonstration of the Gradio UI

Here’s a short video demonstrating the UI.

We also took a dataset of cell phone reviews from Kaggle. And then we converted this dataset into .txt format using the following code:

import pandas as pd

# Load the CSV files
df1 = pd.read_csv('file1.csv', usecols=['asin', 'title'])
df2 = pd.read_csv('file2.csv', usecols=['asin', 'title', 'body'])

# Merge the dataframes based on 'asin'
merged_df = pd.merge(df1, df2, on='asin', how='inner')

# Sample 100 random rows from the merged dataframe
random_subset = merged_df.sample(n=100, random_state=1) if len(merged_df) > 100 else merged_df

# Define the text filename
text_filename = 'output.txt'

# Open the text file for writing
with open(text_filename, 'w', encoding='utf-8') as f:
    for index, row in random_subset.iterrows():
        # Construct the line format: file1[title] : file2[title]: file2[body]
        line = f"{row['title_x']} : {row['title_y']}: {row['body']}\n"
        
        # Write the line to the text file
        f.write(line)

print(f"Text file '{text_filename}' was created successfully.")

We then saved it as a PDF file using Google Docs, and used the PDF to do some sample queries.

Question: How are the reviews of Samsung phones?

Answer: The context mentions several Samsung phones, including the Galaxy S4, Galaxy S5, Galaxy S6, and Galaxy S6 Edge. The reviews for these phones are generally positive, with many people praising their features, design, and performance. For example, one person says that the Galaxy S6 is a great phone, while another describes it as having a great screen and being good for taking pictures. Another reviewer mentions that the Galaxy S5 has a nice display and is easy to use, while yet another notes that the Galaxy S4 is great but was outdone by newer models like the S5 and S6. Overall, it seems that the Samsung phones receive positive feedback from users.

Here’s a video of the same.

Here’s another video demonstrating our chatbot’s answer when asked: “Which phones had disappointing reviews?”

GitHub

The full code can be accessed here: https://github.com/vardhanam/RAG_Voice_Chatbot/tree/main

We also created a Flask-based API for the voice chatbot under the name api.py.

Here’s how you can launch the Flask server. Run the following in the bash terminal.

export FLASK_APP=api.py
flask run

This will launch an endpoint on http://127.0.0.1:5000.

You will first have to upload the pdf documents. You can do so by sending post requests to the API in the following manner:

curl -X POST \
-H "Content-Type: application/json" \
-d '{"file_paths":["path_to_file1.pdf","path_to_file2.pdf"]}' \ 
 http://localhost:5000/upload_pdfs

On successful completion of this above request, you should see the following output in your terminal:

{"message":"Added 2 PDF file(s) to vectorstore/ You can begin voice chat"}

Then you can send another request to the server with the input query as an audio file and the name of the output audio file in the following manner:

curl -X POST \
  -H "Content-Type: application/json" \
  -d '{"audio_path":"path_to_input_query.wav", "output_filename":"output_filename.wav"}' \
  http://localhost:5000/process_audio

Upon successful completion of the above request, you will see a message in the terminal that will display the following;

{"transcription" : "Your transcribed version of the audio input query", "response_text":
 "Response text generated by the RAG pipeline", "audio_output": 
"path_to_output_audio_file"}

Using this backend Flask API, developers can design full-stack voice-to-voice RAG chatbots.

Stackademic 🎓

Thank you for reading until the end. Before you go:

• Please consider clapping and following the writer! 👏
• Follow us X | LinkedIn | YouTube | Discord
• Visit our other platforms: In Plain English | CoFeed | Venture | Cubed
• More content at Stackademic.com

Building RAG Powered Audio-Only Chatbot for Mobile Phone Reviews — Using Mistral-7B, WhisperSpeech… was originally published in Stackademic on Medium, where people are continuing the conversation by highlighting and responding to this story.