Introduction

AI-powered personalization is becoming an important part of modern enterprise applications. Businesses across marketing, sales, customer support, and e-commerce are using AI systems to generate personalized recommendations, customer interactions, and content based on user behavior and contextual information. Building such systems involves multiple challenges, including handling customer data across different platforms, retrieving relevant context, and generating accurate responses tailored to individual users.

In this article, we will build an AI Content Personalization Engine using Salesforce, SingleStore Helios, and Groq Llama. We will explore how vector embeddings, semantic retrieval, and Retrieval-Augmented Generation (RAG) can be integrated to generate personalized AI-driven content using customer context and data retrieval.

What is AI Content Personalization?

AI Content Personalization is the process of generating content based on individual user behavior, preferences, and contextual data. It uses signals such as past interactions, purchase history, browsing activity, and CRM records to understand user intent and produce relevant outputs. This can include personalized emails, product recommendations, support responses, marketing messages, and in-app experiences. The aim is to ensure each user receives content that matches their current needs and interests instead of generic communication.

Why do we need it?

Users interact with multiple digital touchpoints every day, and the volume of available content continues to grow. Generic messaging often fails to capture attention or drive meaningful engagement. Businesses also deal with large volumes of customer data spread across systems such as CRMs, analytics platforms, and support tools.

AI Content Personalization helps in addressing these challenges by connecting data from multiple sources and generating content that is relevant to each user. It improves engagement, supports better decision-making in sales and marketing workflows, and helps in creating more consistent customer experiences across channels.

Building an AI Content Personalization Engine on SingleStore Helios

Before integrating Salesforce with SingleStore Helios and building the personalization pipeline, the first step is setting up a Salesforce Developer Edition environment and preparing it for API-based access. This setup ensures we can extract CRM data such as Contacts, Leads, Accounts, Campaigns, and Opportunities, and use them for our AI personalization engine.

Setting Up Salesforce for Data Access and Integration

To get started, create a free Salesforce Developer Edition account at https://www.salesforce.com/products/free-trial/developer/. No credit card is required for this setup. Once registered, you get access to a fully functional Salesforce CRM environment that includes preloaded sample data.

After logging in, navigate to the Setup Menu (Gear icon) and open the Developer Console to explore the sample datasets using Query Editor. Salesforce provides realistic sample records such as Contacts, Accounts, Leads, Campaigns, and Opportunities, which help in simulating enterprise CRM scenarios.

To verify the available data, you can use the Developer Console and run a SOQL query like:

SELECT Id, FirstName, LastName,
       Title, Email, Department,
       LeadSource, AccountId
FROM Contact

This will return sample contacts such as business personas similar to procurement heads, finance executives, and operations leaders, which will later be used for personalization workflows.

To connect Salesforce with external systems such as SingleStore Helios or backend services, API access must be enabled through OAuth.

The following are the steps:

• Navigate to the Setup Menu (Gear icon) → Setup
• Search for OAuth and OpenID Connect Settings
• Enable Allow OAuth Username-Password Flows
• Save the configuration

This allows secure authentication for API-based data extraction.

A Connected App is required to generate API credentials for external integrations.

The following are the steps to create a Connected App:

• Navigate to the Setup Menu (Gear icon) → Setup
• Search for External Client Apps
• Navigate to Settings
• Select New Connected App
• Fill in the details:
  - App Name: PersonalizationApp
  - Email address: <your-registered-email-address>
  - Enable OAuth Settings
  - Set Callback URL:http://localhost:8080/callback
  - Select OAuth Scopes: Full access (full)
  - Save the app

Once created, review the controls, confirm them, and Salesforce will generate credentials: Consumer Key and Consumer Secret.

These credentials will be used to authenticate API calls from external services.

Salesforce requires a security token for API authentication from external environments.

To generate it, the following steps are taken:

• Click on your profile avatar
• Go to Settings
• Search for Reset My Security Token
• Click Reset Security Token
• Check your registered email for the new token

This token is used along with username and password during API authentication flows.

To support AI-driven personalization, we extend the Contacts object with additional fields that store AI-generated metadata.

• Navigate to Setup → Object Manager → Contact → Fields & Relationships → New
• Create the following fields:

Field 1: AI Persona

• Data Type: Text
• Field Label: AI Persona
• API Name: AI_Persona__c
• Length: 50

Field 2: AI Persona Score

• Data Type: Number
• Field Label: AI Persona Score
• API Name: AI_Persona_Score__c
• Decimal Places: 0

Field 3: AI Last Personalized

• Data Type: Date/Time
• Field Label: AI Last Personalized
• API Name: AI_Last_Personalized__c

Later, these fields will help in tracking personalization outputs generated by the AI system in SingleStore Helios and storing enrichment metadata for each customer profile.

Connecting Salesforce with SingleStore Helios

Now that Salesforce is configured, the next step is to extract CRM data and load it into SingleStore Helios.

Install the required libraries for Salesforce integration, data processing, and SingleStore connectivity.

!pip install -q simple-salesforce faker pandas

Securely load Salesforce credentials using SingleStore Secrets.

from singlestoredb.management import get_secret
SF_USERNAME = get_secret('SF_USERNAME')
SF_PASSWORD = get_secret('SF_PASSWORD')
SF_SECURITY_TOKEN = get_secret('SF_SECURITY_TOKEN')
SF_INSTANCE_URL = get_secret('SF_INSTANCE_URL')

Next, initialize the Salesforce connection:

from simple_salesforce import Salesforce
sf = Salesforce(
    username=SF_USERNAME,
    password=SF_PASSWORD,
    security_token=SF_SECURITY_TOKEN
)

To verify connectivity, run a simple SOQL query:

result = sf.query("""
SELECT Id, FirstName, LastName, Title
FROM Contact
LIMIT 5
""")

for record in result['records']:
    print(f"{record['FirstName']} {record['LastName']} - {record['Title']}")

This confirms that Salesforce data is accessible through the API.

Extract data from multiple Salesforce objects, including Contacts, Accounts, Leads, Campaigns, and Opportunities.

Each object provides a different business context required for personalization.

For example, Contacts contain role and persona information:

contacts_query = sf.query("""
    SELECT Id, FirstName, LastName,
           Title, Email, Department,
           LeadSource, AccountId
    FROM Contact
""")

Similarly, we extract account-level company data, lead acquisition data, campaign performance data, and opportunity pipeline information.

Each dataset is converted into a Pandas DataFrame for further processing.

After extraction, save the CRM datasets as CSV files.

contacts_df.to_csv('data/contacts.csv', index=False)
accounts_df.to_csv('data/accounts.csv', index=False)
leads_df.to_csv('data/leads.csv', index=False)
campaigns_df.to_csv('data/campaigns.csv', index=False)
opps_df.to_csv('data/opportunities.csv', index=False)

This creates a reusable local data layer for downstream ingestion into SingleStore Helios.

Once the Salesforce data has been extracted, we create the required relational tables in SingleStore Helios.

import singlestoredb as s2

conn = s2.connect(
    host=SS_HOST,
    port=3306,
    user="admin",
    password=SS_PASSWORD,
    database="salesforce"
)

We then initialize a cursor to execute SQL queries.

We create dedicated tables for contacts, accounts, leads, campaigns, opportunities, and visitor_events.

For example:

CREATE TABLE IF NOT EXISTS contacts (
    id              VARCHAR(50) PRIMARY KEY,
    first_name      VARCHAR(100),
    last_name       VARCHAR(100),
    title           VARCHAR(200),
    email           VARCHAR(200),
    department      VARCHAR(100),
    lead_source     VARCHAR(100),
    account_id      VARCHAR(50),
    persona         VARCHAR(50)
)

The visitor_events table is important because it stores behavioral tracking signals such as page visits, button clicks, scroll depth, and engagement activity.

These behavioral signals later become inputs for personalization and retrieval workflows.

Before loading the CRM records, we classify users into personas based on job titles.

def map_persona(title):
    if pd.isna(title):
        return 'Unknown'
    title = title.upper()
    if any(x in title for x in ['CEO', 'CFO', 'SVP', 'PRESIDENT', 'FOUNDER']):
        return 'Executive'
    elif any(x in title for x in ['VP', 'TECHNOLOGY', 'ENGINEER', 'DEVELOPER']):
        return 'Technical'
    elif any(x in title for x in ['FINANCE', 'PROCUREMENT', 'VENDOR', 'ADMIN']):
        return 'Finance'
    else:
        return 'Other'

This creates structured customer segmentation categories used later for personalization.

Now, load all CRM datasets into SingleStore Helios using SQL insert operations.

For example:

for _, row in contacts_df.iterrows():
    cursor.execute("""
        INSERT IGNORE INTO contacts 
        (id, first_name, last_name, title, email, 
         department, lead_source, account_id, persona)
        VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)
    """, (
        row['Id'], row['FirstName'], row['LastName'],
        row['Title'], row['Email'], row['Department'],
        row['LeadSource'], row['AccountId'], row['persona']
    ))

At this stage, Salesforce CRM data becomes fully queryable inside SingleStore.

Once the data is loaded, we can run segmentation queries directly on SingleStore.

SELECT persona, COUNT(*) as total
FROM contacts
GROUP BY persona
ORDER BY total DESC

Next, we integrate Groq for LLM-based persona reasoning and content generation.

from groq import Groq

client = Groq(api_key=GROQ_API_KEY)

We then validate the connection using a simple completion request.

Building the AI Content Personalization Engine

The personalization engine combines the following to generate role-specific content:

• Salesforce CRM context
• SingleStore segmentation data
• Groq Llama reasoning

We first define content templates for different personas:

content_variants = {
    "Executive": {
        "headline": "Scale Your Business with Enterprise-Grade AI",
        "subheadline": "Trusted by CEOs and SVPs at 500+ companies worldwide",
        "cta": "Book Executive Briefing",
        "social_proof": "Used by Fortune 500 leadership teams"
    },
    "Technical": {
        "headline": "Build Faster with Our Developer-First Platform",
        "subheadline": "REST API, SDKs, and 10k free calls/month",
        "cta": "Start Building Free",
        "social_proof": "Loved by 50,000+ developers globally"
    },
    "Finance": {
      "headline": "Accelerate Growth and Maximize ROI with AI Automation",
      "subheadline": "Achieve up to 3x ROI in 90 days — backed by real data",
      "cta": "Explore ROI Potential",
      "social_proof": "Trusted by finance leaders at Deloitte and KPMG"
    },
    "Other": {
        "headline": "AI-Powered Solutions for Every Team",
        "subheadline": "Easy to use, powerful at scale",
        "cta": "Get Started Free",
        "social_proof": "Join 100,000+ happy users"
    }
}

The personalization pipeline sends visitor context to the Llama model:

response = groq_client.chat.completions.create(
        model="llama-3.1-8b-instant",
        messages=[
            {
                "role": "system",
                "content": """You are a B2B SaaS personalization engine.
Your job is to analyze visitor profiles and return 
personalization recommendations as valid JSON only.
Never include explanations or markdown, only raw JSON."""
            },
            {
                "role": "user",
                "content": f"""
{visitor_context}

Analyze this visitor and return JSON with these exact keys:
{{
    "confirmed_persona": "Executive or Technical or Finance or Other",
    "confidence_score": 0-100,
    "reasoning": "one sentence explanation",
    "personalized_message": "one sentence tailored to their role",
    "recommended_campaign": "Webinar or Conference or Direct Mail or Trade Show"
}}

Return ONLY the JSON object, nothing else.
                """
            }
        ],
        temperature=0.3,
        max_tokens=500
    )

The model analyzes job title, industry, company size, revenue, lead source, and predicts persona type, personalization confidence, campaign recommendation, and personalized messaging. The response is returned as structured JSON.

To expose the personalization engine as a production-ready service, we built a FastAPI application using SingleStore Cloud Function.

The API provides endpoints such as:

• /health
• /visitors
• /segments
• /personalize

@app.post("/personalize")
async def personalize(request: PersonalizeRequest):
    def query():
        if request.contact_id:
            result = execute_query("""
                SELECT c.first_name, c.last_name, c.title,
                       c.persona, c.lead_source,
                       a.industry, a.annual_revenue,
                       a.number_of_employees
                FROM contacts c
                LEFT JOIN accounts a ON c.account_id = a.id
                WHERE c.id = :id
            """, {"id": request.contact_id})
        elif request.lead_id:
            result = execute_query("""
                SELECT first_name, last_name, title,
                       persona, lead_source,
                       industry, 0, 0
                FROM leads WHERE id = :id
            """, {"id": request.lead_id})
        else:
            raise HTTPException(
                status_code=400,
                detail="Provide contact_id or lead_id"
            )

        row = result.fetchone()
        if not row:
            raise HTTPException(status_code=404, detail="Visitor not found")

        (first_name, last_name, title, persona,
         lead_source, industry, revenue, employees) = row

        # Groq persona reasoning
        response = groq_client.chat.completions.create(
            model="llama-3.1-8b-instant",
            messages=[
                {
                    "role": "system",
                    "content": "You are a B2B SaaS personalization engine. Return only valid JSON."
                },
                {
                    "role": "user",
                    "content": f"""
Visitor: {first_name} {last_name}
Title: {title or 'Unknown'}
Industry: {industry or 'Technology'}
Revenue: ${revenue:,.0f}

Return JSON:
{{
    "confirmed_persona": "Executive or Technical or Finance or Other",
    "confidence_score": 0-100,
    "reasoning": "one sentence",
    "personalized_message": "one sentence for this visitor",
    "recommended_campaign": "Webinar or Conference or Direct Mail or Trade Show"
}}
                    """
                }
            ],
            temperature=0.3,
            max_tokens=300
        )

        raw = response.choices[0].message.content
        raw = raw.replace('```json', '').replace('```', '').strip()

        try:
            analysis = json.loads(raw)
        except Exception:
            analysis = {
                "confirmed_persona": persona or "Other",
                "confidence_score": 70,
                "reasoning": "Rule-based fallback",
                "personalized_message": f"Welcome {first_name}!",
                "recommended_campaign": "Webinar"
            }

        confirmed_persona = analysis.get('confirmed_persona', 'Other')
        content = content_variants.get(confirmed_persona, content_variants['Other'])

        # Get matching campaign from SingleStore
        campaign_result = execute_query("""
            SELECT name, type, status
            FROM campaigns
            WHERE type = :type LIMIT 1
        """, {"type": analysis.get('recommended_campaign', 'Webinar')})
        campaign = campaign_result.fetchone()

        # Salesforce Write-Back
        writeback_result = salesforce_writeback(
            contact_id=request.contact_id,
            persona=confirmed_persona,
            confidence_score=analysis.get('confidence_score', 70), 
            recommended_campaign_type=analysis.get('recommended_campaign', 'Webinar')
        ) if request.contact_id else {"status": "skipped"}

        return {
            "visitor":    f"{first_name} {last_name}",
            "title":      title or "Professional",
            "persona":    confirmed_persona,
            "confidence_score":    analysis.get('confidence_score', 70),
            "reasoning":           analysis.get('reasoning', ''),
            "personalized_message": analysis.get('personalized_message', ''),
            "content":             content,
            "recommended_campaign": {
                "name":   campaign[0] if campaign else "Default Campaign",
                "type":   campaign[1] if campaign else "Webinar",
                "status": campaign[2] if campaign else "Planned"
            },
            "salesforce_writeback": writeback_result  
        }

    try:
        return await run_in_thread(query)
    except HTTPException as e:
        raise e
    except Exception as e:
        logging.error(f"Error: {str(e)}")
        raise HTTPException(status_code=500, detail=str(e))

This endpoint retrieves CRM context from SingleStore, sends visitor data to Groq Llama, generates persona-specific recommendations, returns personalized content, and writes AI metadata back into Salesforce.

The Salesforce writeback step updates Salesforce with AI-generated enrichment data. The system also automatically enrolls contacts into recommended campaigns using Salesforce Campaign Members.

Create a cloud function by selecting the Create Cloud Function button on the right-upper side.

Once the cloud function is created, go to the cloud function and click View API Keys on the right-upper side. Create the API key and save it as a secret to use the API key of the cloud function. We used this API key in creating the Dash App.

Building the Personalization Dashboard with Dash

After building the Cloud Function layer for the AI Content Personalization Engine, the next step is to create an internal dashboard for visualizing and testing the system. This dashboard is not the visitor-facing experience; it is our monitoring and demo tool, similar to how Google Analytics works behind the scenes. For this, we use Plotly Dash with Dash Bootstrap Components, which connects directly to the SingleStore Cloud Function endpoint built earlier.

The dashboard fetches initial visitor and segment data from the API and dynamically renders customer personas identified from Salesforce CRM, segment-level distribution of contacts, a live personalization simulation interface, and a structured table of all CRM visitors.

The UI simulates visitor signals such as the page visited and the traffic source. These signals are sent to the /personalize endpoint, which triggers the full AI pipeline; Salesforce data lookup, SingleStore query, and Groq LLaMA persona reasoning.

Once a visitor is selected, the system returns a fully personalized landing page experience, including a persona-specific headline and messaging, Groq LLaMA reasoning for persona classification, a recommended Salesforce campaign based on CRM context, CTA variation based on visitor segment, and a Salesforce write-back status showing campaign enrollment.

Conclusion

I found it interesting to build an end-to-end AI Content Personalization Engine by connecting Salesforce, SingleStore Helios, and Groq Llama into a single workflow that handles CRM data extraction, storage, persona segmentation, and AI-driven content generation.

Working with Salesforce objects and mapping them into SingleStore tables made the data flow much clearer, and using Groq Llama for persona reasoning added a practical layer of decision-making on top of structured CRM data. The experience of tying together API integrations, SQL-based analytics, and LLM responses showed how quickly personalization systems can be built with the right stack. Adding on top of this, the Dash-based dashboard completes the system by providing an interface to simulate user behavior and visualize personalized outputs directly.

Now it’s your turn to experiment with CRM-driven AI pipelines, extend the personalization logic, and explore how far RAG can go in enterprise use cases.

Thank you for reading!

Source Code

AI Content Personalization Engine — Github Repo

Introduction

Function calling enables large language models to interact dynamically with external tools such as APIs, databases, or custom functions.

Fine-tuning an LLM for function calling involves training the model not only to generate natural language responses but also to determine when and how to invoke external functions based on the user’s query. During this process, the model learns to identify the need for a function call, generate the appropriate arguments, and pass them to external systems or APIs.

To leverage the capabilities of function calling, we will fine-tune Gemma 3 on the Salesforce function calling dataset, which contains 60,000 examples collected by APIGen for function calling applications. The data is generated using DeepSeek-V2-Chat and Mixtral-8x22B-Inst. Before delving deeper, let’s understand what function calling is.

What is Function Calling?

Function calling refers to the ability of the large language models to invoke external functions or APIs based on user inputs to retrieve information or perform tasks beyond their built-in knowledge.

With Function calling, the model follows the following steps:

1. Detect the need for an external tool or function: When a user request requires a specific data or actions outside the model’s internal capabilities, the model identifies that an external function must be called.
2. Generate appropriate arguments: The model determines what inputs or parameters are needed based on the user’s query and structures them in a way that the target function can understand.
3. Call the external function or API: The model sends the generated arguments to the appropriate tool, whether it’s a database, an API, or a custom function.
4. Process and return the results: After receiving the output from the external function, the model incorporates that information into its response, which provides more accurate, data-driven results.

Function calling lays the foundation for building autonomous agents. Whether it is a customer support assistant, a research tool, or an automated DevOps bot, it enables the model to take meaningful actions on its own.

Without function calling, a language model is limited to making suggestions. It can tell you what to do or explain possible outcomes, but it cannot take action. It remains a passive advisor, which relies entirely on humans to interpret and implement its outputs.

With function calling, however, the model gains the ability to perform tasks, interact with systems, and respond to real-world inputs. It can take action, observe the results, and use that information to decide what to do next. This transforms the model from a static assistant into a dynamic, decision-making agent capable of following through on tasks and adapting as needed.

|----------------------|--------------------------|-----------------------------|
| Concept              | Without Function Calling | With Function Calling       |
|----------------------|--------------------------|-----------------------------|
| Output Type          | Text only                | Text + structured calls     |
| Reasoning            | Verbal                   | Verbal + actionable         |
| Autonomy             | Passive                  | Active agent                |
| Use in Production    | Limited                  | Full workflows possible     |
|----------------------|--------------------------|-----------------------------|

Enhancing Agent’s Capabilities with Function Calling

Fine-tuning a model with function calling transforms an AI agent from a passive language generator into an active, action-oriented system. This enhancement allows the agent to go beyond simply talking about tasks to executing them, such as querying databases, calling APIs, or controlling software and devices. By training on examples of tool usage, the model learns not just what actions to take, but how to format function calls correctly and respond meaningfully to tool outputs. This significantly reduces hallucinations and increases reliability.

Moreover, it enables the model to translate user intents into structured actions and interpret responses to deliver informed answers. It reduces the need for complex prompt engineering, which allows the agent to generalize more effectively, simplify implementation, and operate more efficiently in real-world workflows.

The transition from passive response generation to active task execution is enabled by defining a set of tools the agent can use. Each tool should include:

• A unique name
• A clear description
• A parameter schema defined in JSON format

It should be fine-tuned on annotated function calling examples to train the model for effective tool use. This is the following structure:

• The user provides a query
• The model generates a function call
• The tool returns a result
• The model then uses that result to respond appropriately

Through this process, the model learns not only when to invoke a tool but also how to format the call and integrate the output into its response. This reduces reliance on hard-coded prompts and boosts the model’s generalization across tasks.

During deployment, function calling becomes part of the agent’s reasoning loop. The model:

• Receives the user’s input
• Determines which function to call and with what parameters
• Relays the function call to a backend system
• Receives the function’s output
• Incorporates the result into the next step of the conversation or task

To support this structure, some models or frameworks allow the use of special tokens to define each stage of the interaction. These tokens are custom tokens added to the model that help in maintaining a modular and consistent workflow during multi-step reasoning.

The agent’s capabilities can be scaled by adding more tools, such as web search, document retrieval, file access, calendar management, or third-party service integration. This increases the agent’s ability to handle complex tasks across domains.

Furthermore, function calling enables multi-step planning: the agent can execute one function, evaluate its output, and determine the next step, which allows it to perform sequences of actions toward a defined goal.

Fine-Tuning Gemma 3 for Function Calling

We will fine-tune the Gemma 3 model, which is instruction-tuned and has 27B parameters, with the Salesforce function calling dataset.

Let’s code!

To get started, we will need an A100 GPU to work with the Gemma 3 27B model. I used Google Colab Pro. Install and import the required dependencies. Set environment and seed for deterministic runs.

!pip install -q "torch>=2.4.0" tensorboard
!pip install -q flash-attn
!pip install -q git+https://github.com/huggingface/transformers@v4.49.0
!pip install -q --upgrade \
  datasets==3.3.2 \
  accelerate==1.4.0 \
  evaluate==0.4.3 \
  bitsandbytes==0.45.3 \
  trl==0.15.2 \
  peft==0.14.0 \
  protobuf==3.20.3 \
  sentencepiece

import torch
from transformers import AutoTokenizer, Gemma3ForConditionalGeneration, BitsAndBytesConfig, set_seed
from datasets import load_dataset
from trl import SFTTrainer, SFTConfig
from peft import LoraConfig, PeftModel, PeftConfig
from enum import Enum
import json
import gc
import os
os.environ["PYTORCH_CUDA_ALLOC_CONF"] = "expandable_segments:True"

seed = 42
set_seed(seed)

Then, we will select the GPU and precision.

torch_dtype = torch.bfloat16 if torch.cuda.get_device_capability()[0] >= 8 else torch.float16
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

Next, we will load the model with QLoRA and quantization. QLoRA significantly reduces computational resource usage while preserving strong model performance. It works by first quantizing the pretrained model to 4-bit precision and freezing its original weights. Instead of updating the full model, QLoRA introduces lightweight, trainable adapter layers (via LoRA), which are the only components fine-tuned during training. After training, the adapter weights can either be merged back into the base model or kept separate for modular use.

For better model training, we will use Gemma3ForConditionalGeneration instead of using AutoModelForCausalLM.

model_name = "google/gemma-3-27b-it"
dataset_name = "Salesforce/xlam-function-calling-60k"

model_kwargs = dict(
    attn_implementation="flash_attention_2",
    torch_dtype=torch_dtype,
    device_map="auto",
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True,
        bnb_4bit_quant_type='nf4',
        bnb_4bit_compute_dtype=torch_dtype,
        bnb_4bit_quant_storage=torch_dtype,
        llm_int8_enable_fp32_cpu_offload=True
    )
)

model = Gemma3ForConditionalGeneration.from_pretrained(model_name, **model_kwargs)

Now, we will define custom special tokens with the tokenizer and load the tokenizer with the chat template.

class ChatmlSpecialTokens(str, Enum):
    tools = "<tools>"
    eotools = "</tools>"
    think = "<think>"
    eothink = "</think>"
    tool_call="<tool_call>"
    eotool_call="</tool_call>"
    tool_response="<tool_response>"
    eotool_response="</tool_response>"
    pad_token = "<pad>"
    eos_token = "<eos>"

    @classmethod
    def list(cls):
        return [c.value for c in cls]

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    pad_token=ChatmlSpecialTokens.pad_token.value,
    additional_special_tokens=ChatmlSpecialTokens.list()
)

tokenizer.chat_template = """{{ bos_token }}{% if messages[0]['role'] == 'system' %}{{ raise_exception('System role not supported') }}{% endif %}{% for message in messages %}{{ '<start_of_turn>' + message['role'] + '\n' + message['content'] | trim + '<end_of_turn><eos>\n' }}{% endfor %}{% if add_generation_prompt %}{{'<start_of_turn>model\n'}}{% endif %}"""

model.resize_token_embeddings(len(tokenizer))
model.to(device)

We will preprocess the dataset through the preprocess function to format the input. Then, we will load and transform the dataset.

def preprocess(sample):
    try:
        tools = json.loads(sample["tools"])
        answers = json.loads(sample["answers"])
        user_query = sample["query"]
    except Exception as e:
        print("Error decoding JSON:", sample)
        raise e

    messages = [
        {
            "role": "user",
            "content": (
                "You have access to the following tools:\n\n"
                + "\n\n".join(f"- {tool['name']}: {tool['description']}" for tool in tools)
                + "\n\nUser query:\n" + user_query
            )
        },
        {
            "role": "assistant",
            "content": "\n".join(
                f"<function_call>\n{json.dumps(answer)}\n</function_call>"
                for answer in answers
            )
        }
    ]

    return {
        "text": tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
    }

dataset = load_dataset(dataset_name)
dataset = dataset["train"].map(preprocess, remove_columns=["id", "query", "answers", "tools"])
dataset = dataset.train_test_split(0.1)
print(dataset)

DatasetDict({
    train: Dataset({
        features: ['text'],
        num_rows: 54000
    })
    test: Dataset({
        features: ['text'],
        num_rows: 6000
    })
})

Let’s see one of the dataset’s columns.

print(dataset["train"][19]["text"])

<bos><start_of_turn>user
You have access to the following tools:

- max_points_on_line: Finds the maximum number of points that lie on the same straight line.

- solve_quadratic: Computes the roots of a quadratic equation given its coefficients.

- greatest_common_divisor: Computes the greatest common divisor (GCD) of two non-negative integers.

- chi_square_independence_test: Performs a Chi-Square test for independence on a 2x2 contingency table.

- can_attend_all_meetings: Determines if a person can attend all meetings given a list of meeting time intervals.

- binomial_probability: Calculates the probability of getting exactly k successes in n independent trials,

User query:
What are the roots of the quadratic equation 2x^2 - 3x + 1 = 0?<end_of_turn><eos>
<start_of_turn>assistant
<function_call>
{"name": "solve_quadratic", "arguments": {"a": 2, "b": -3, "c": 1}}
</function_call><end_of_turn><eos>
<start_of_turn>model

We will define PEFT configurations by configuring LoRA adapters to fine-tune a small subset of model parameters. This is faster, cheaper, and more memory efficient than full fine-tuning.

peft_config = LoraConfig(
    lora_alpha=16,
    lora_dropout=0.05,
    r=16,
    bias="none",
    target_modules="all-linear",
    task_type="CAUSAL_LM",
    modules_to_save=["lm_head", "embed_tokens"] # make sure to save the lm_head and embed_tokens as you train the special tokens
)

Then, we will define the training arguments.

training_arguments = SFTConfig(
    output_dir="gemma-3-27b-it-thinking-function_calling-V0",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=1,
    gradient_accumulation_steps=32,
    save_strategy="epoch",
    eval_strategy="epoch",
    logging_steps=50,
    learning_rate=3e-4,
    max_grad_norm=0.3,
    weight_decay=0.1,
    warmup_ratio=0.03,
    lr_scheduler_type="constant",
    report_to=None,
    bf16=True,
    optim="paged_adamw_8bit",
    torch_compile=False,
    push_to_hub=False,
    num_train_epochs=3,
    gradient_checkpointing=True,
    gradient_checkpointing_kwargs={"use_reentrant": False},
    packing=False,
    max_seq_length=512,
    dataset_kwargs={
        "add_special_tokens": False,
        "append_concat_token": True,
    }
)

We will clear the unused GPU memory before training to prevent OOM errors.

torch.cuda.empty_cache()
torch.cuda.ipc_collect()
gc.collect()

Now, we will wrap the model, tokenizer, datasets, and configurations into SFTTrainer. We will start fine-tuning with all the optimized settings. Then, we will save the model.

trainer = SFTTrainer(
    model=model,
    args=training_arguments,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"],
    tokenizer=tokenizer,
    peft_config=peft_config,
)

trainer.train()
trainer.save_model()

While training, a WandB API will be required. Enter the API, and see the run results on the WandB page under Overview or Workspace.

Then, we will push the model and the tokenizer to the Hugging Face Hub.

trainer.push_to_hub(f"{username}/{output_dir}")

Since we also modified the chat_template, which is contained in the tokenizer, we will push the tokenizer with the model.

tokenizer.eos_token = "<eos>"
tokenizer.push_to_hub(f"{username}/{output_dir}", token=True)

We will define the PEFT model, which will serve as our trained model, and evaluate its fine-tuned performance.

peft_model_id = f"{username}/{output_dir}" 
device = "auto"
config = PeftConfig.from_pretrained(peft_model_id)
model = Gemma3ForConditionalGeneration.from_pretrained("google/gemma-3-27b-it",
                                             device_map="auto",
                                             )
tokenizer = AutoTokenizer.from_pretrained(peft_model_id)
model.resize_token_embeddings(len(tokenizer))
model = PeftModel.from_pretrained(model, peft_model_id)
model.to(torch.bfloat16)
model.eval()

Next, we will test our model by providing it with a user query as input.

prompt = """<bos><start_of_turn>user
You have access to the following tools:

- numerical_derivative: Estimate the derivative of a mathematical function

User query:
I need to estimate the derivative of the function y = sin(x) at x = π/4 and x = π. Can you help with that?<end_of_turn><eos>
<start_of_turn>assistant
"""

inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    do_sample=True,
    temperature=0.01,
    top_p=0.95,
    repetition_penalty=1.1,
    eos_token_id=tokenizer.eos_token_id
)

response = tokenizer.decode(outputs[0], skip_special_tokens=False)
print(response)

The response is:

<bos><start_of_turn>user
You have access to the following tools:

- numerical_derivative: Estimate the derivative of a mathematical function

User query:
I need to estimate the derivative of the function y = sin(x) at x = π/4 and x = π. Can you help with that?<end_of_turn><eos>
<start_of_turn>assistant
<tool_call>
{'name': 'numerical_derivative', 'arguments': {'function_name': 'sin', 'point1': 3.14159, 'point2': 3.14159}}
</tool_call><end_of_turn><eos>

The results are fascinating!

Conclusion

Fine-tuning Gemma 3 for function calling came with its challenges, especially when it came to memory management on the A100 GPU, which turned out to be a great learning opportunity. Despite the hurdles, the process was genuinely enjoyable! The test results were promising, and overall, it was a fun and insightful experience. Now it’s your turn to roll up your sleeves, experiment with different models, and explore function calling.

References

Hugging Face Agents Course — Bonus Unit

Google Cloud Documentation — Function Calling

OpenAI — Function Calling

Mistral — Function Calling

Function Calling with LLMs

Introduction

With the advent of AI agents and RAG, the healthcare industry has undergone significant transformation. This transformation impacts various areas, including patient demographics, medical history, treatment recommendations, drug development, and more. Imagine how fascinating it would be to have an AI agent capable of providing information on proper diet, yoga, herbal remedies, and medicines. The adage Prevention is better than Cure aligns well with the principles of Ayurveda, which offers comprehensive guidance on maintaining a healthy lifestyle.

Developing AI agents and RAG applications with a single PDF document is easy, but we encounter some challenges when dealing with Multi-PDFs. To address this, we have implemented a query pipeline that optimizes retrieval using HyDE. Before delving deeper into creating an Ayurveda Healthcare Multi-PDF agent, let’s understand what HyDE is.

An Overview of HyDE

HyDE is a retrieval method that uses large language models to create hypothetical documents based on user queries. These documents aim to capture the essence and context of the query. Once the hypothetical documents are generated, HyDE encodes them into dense vectors using a contrastive encoder. This encoding process captures the semantic similarities between the hypothetical documents and the actual documents in the retrieval corpus. The encoded vectors of the hypothetical documents are then compared with the vectors of real documents in the corpus. The documents that show the highest similarity to the hypothetical representation of the query are retrieved and presented to the user.

HyDE has demonstrated outstanding performance in various retrieval tasks such as web search, low-resource retrieval, and multilingual retrieval.

Implementing Query Pipelines with HyDE on SingleStore Helios

We will utilize SingleStore Helios, which provides a shared edition including a complimentary starter workspace for developing our production apps for free. Sign up for the free shared tier to leverage the starter workspace.

The starter workspace contains one attached database. In SingleStore Helios, we can utilize Notebooks and Secrets without installing the SingleStore library. We can secure outbound access to our notebooks using the Firewall by adding the external endpoint.

Let’s code!

To get started, create a notebook in Notebooks under the Develop page in the cloud portal (SingleStore Helios). Install and import the required dependencies.

!pip install -q llama-index
!pip install -q pyvis
!pip install -q llama-index-embeddings-huggingface
!pip install -q llama-index-llms-openai
!pip install -q -U llama-index-vector-stores-singlestoredb

import logging
import sys
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, StorageContext, load_index_from_storage
from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.node_parser.text import SentenceSplitter
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.openai import OpenAI
from llama_index.core.query_engine import TransformQueryEngine
from IPython.display import Markdown, display
from llama_index.vector_stores.singlestoredb import SingleStoreVectorStore
from llama_index.core.tools import QueryEngineTool, ToolMetadata, BaseTool
from llama_index.core.settings import Settings
from llama_index.core.callbacks import CallbackManager
from llama_index.core.agent.react.types import (
    ActionReasoningStep,
    ObservationReasoningStep,
    ResponseReasoningStep,
)
from llama_index.core.agent import Task, AgentChatResponse, ReActChatFormatter, QueryPipelineAgentWorker
from llama_index.core.query_pipeline import (
    AgentInputComponent,
    AgentFnComponent,
    CustomAgentComponent,
    QueryComponent,
    ToolRunnerComponent,
    InputComponent,
    Link,
    QueryPipeline 
)
from llama_index.core.llms import MessageRole, ChatMessage, ChatResponse
from typing import Dict, Any, Optional, Tuple, List, Set, cast
from singlestoredb.management import get_secret
from sqlalchemy import *
from pyvis.network import Network
from IPython.display import display, HTML

logging.basicConfig(stream=sys.stdout, level=logging.INFO)
logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))

callback_manager = CallbackManager()
Settings.callback_manager = callback_manager

To create an Ayurveda Healthcare Multi-PDF agent, I gathered 40 Ayurveda ebooks and utilized my GitHub repository to import the data into the SingleStore Notebooks.

!git clone <link_to_github_repo>

To work with SingleStore, the first step is to get the connection information. The connection information consists of the username, password, database name, host, and port, which can be obtained from any of the clients while connecting to the starter workspace.

I accessed this information from the SQL IDE.

We can save that information in Secrets. Along with this information, we should also save the OpenAI API key, as we are going to utilize the OpenAI model GPT-4.

Let’s retrieve the secrets.

user = get_secret('user')
password = get_secret('password')
host = get_secret('host')
database = get_secret('database')
openai = get_secret('openai')

Now, we will initiate the vector store using the information saved in the Secrets.

vector_store = SingleStoreVectorStore(
    table_name="embeddings",
    content_field="content",
    metadata_field="metadata",
    vector_field="vector",
    host = host,
    port = "3333",
    database = database,
    user = user,
    password = password,
)

We will store the vector in the storage context because we will use it to create the vector index.

storage_context = StorageContext.from_defaults(vector_store=vector_store)

Next, we will create embeddings, which will be required to create the vector store index.

embed_model = HuggingFaceEmbedding()

Settings.embed_model = embed_model

As we have 40 PDFs, we will load them and generate indices for each in a dictionary format. The indices will be generated after creating a vector store index. The vector store index will use nodes, storage context, and the sentence splitter as transformations.

pdf_directory = "./AyurvedicData"

pdf_files = [file for file in os.listdir(pdf_directory) if file.endswith('.pdf')]

storage_base_dir = "./storage"
Settings.transformations = [SentenceSplitter(chunk_size=1024)]

def generate_storage_dir(file_name):
    base_name = os.path.splitext(file_name)[0]  # Remove extension
    safe_name = base_name.replace(' ', '_').replace('&', 'and')
    return os.path.join(storage_base_dir, safe_name)

all_indices_loaded = True

indices = {}

for pdf_file in pdf_files:
    try:
        storage_dir = generate_storage_dir(pdf_file)
        storage_context = StorageContext.from_defaults(persist_dir=storage_dir)
        index = load_index_from_storage(storage_context)
        indices[pdf_file] = index
    except Exception as e:
        print(f"Failed to load index for {pdf_file}: {e}")
        all_indices_loaded = False

if not all_indices_loaded:
    for pdf_file in pdf_files:
        if pdf_file not in indices:  
            try:
                input_path = os.path.join(pdf_directory, pdf_file)
                docs = SimpleDirectoryReader(input_files=[input_path]).load_data()
                
                text_parser = SentenceSplitter(chunk_size=1024)
                text_chunks = []
                doc_idxs = []

                for doc_idx, doc in enumerate(docs):
                    cur_text_chunks = text_parser.split_text(doc.text)
                    text_chunks.extend(cur_text_chunks)
                    doc_idxs.extend([doc_idx] * len(cur_text_chunks))

                from llama_index.core.schema import TextNode
                nodes = []
                for idx, text_chunk in enumerate(text_chunks):
                    node = TextNode(text=text_chunk)
                    src_doc = docs[doc_idxs[idx]]
                    node.metadata = src_doc.metadata
                    nodes.append(node)
                
                vectorstoreindex = VectorStoreIndex(nodes = nodes, storage_context=storage_context,transformations=Settings.transformations,)
                index = vectorstoreindex.from_documents(docs)
                
                storage_dir = generate_storage_dir(pdf_file)
                index.storage_context.persist(persist_dir=storage_dir)
                
                indices[pdf_file] = index
            except Exception as e:
                print(f"Failed to process {pdf_file}: {e}")

Now, the indices variable contains all the loaded or newly created indices. We will initialize LLM after creating the indices. In this process, we are utilizing OpenAI's GPT-3.5.

llm = OpenAI(model="gpt-3.5-turbo")
Settings.llm = llm

For each item in indices, we will create HyDE query engines.

query_engines={}

for pdf_name, index in indices.items():
    try:
        query_engine = index.as_query_engine(similarity_top_k=3)
        
        hyde = HyDEQueryTransform(include_original=True)
        
        hyde_query_engine = TransformQueryEngine(query_engine, hyde)
        
        query_engines[pdf_name] = hyde_query_engine
        
        print(f"HyDE query engine created for {pdf_name}")

    except Exception as e:
        print(f"An error occurred while creating query engine for {pdf_name}: {e}")

For each index, we will instantiate a QueryEngineTool using HyDE query engines.

query_engine_tools = []

for pdf_name, hyde_query_engine in query_engines.items():
    try:
        description = "Provides information about improving lifestyle, diet, and medicinal benefits using Ayurveda. Use a detailed plain text question as input to the tool."
        
        metadata = ToolMetadata(
            name=pdf_name.replace(".pdf", "").replace(" ", "_").lower(),
            description=description
        )
        
        tool = QueryEngineTool(
            query_engine=hyde_query_engine,
            metadata=metadata
        )
        
        query_engine_tools.append(tool)
        
        print(f"QueryEngineTool created for {pdf_name}")
    
    except Exception as e:
        print(f"An error occurred while creating QueryEngineTool for {pdf_name}: {e}")

We will now initiate the agent input component.

def agent_input_fn(task: Task, state: Dict[str, Any]) -> Dict[str, Any]:
    """Agent input function.

    Returns:
        A Dictionary of output keys and values. If you are specifying
        src_key when defining links between this component and other
        components, make sure the src_key matches the specified output_key.

    """
    if "current_reasoning" not in state:
        state["current_reasoning"] = []
    reasoning_step = ObservationReasoningStep(observation=task.input)
    state["current_reasoning"].append(reasoning_step)
    return {"input": task.input}


agent_input_component = AgentInputComponent(fn=agent_input_fn)

We defined the agent component that is responsible for creating a ReAct prompt. After the LLM generates the output, it is parsed into a structured object. When input is received, the LLM is called upon with the ReAct agent prompt. Using the Chain-Of-Thought + Acting method we will define the ReAct Prompt component.

def react_prompt_fn(
    task: Task, state: Dict[str, Any], input: str, tools: List[BaseTool]
) -> List[ChatMessage]:
    chat_formatter = ReActChatFormatter()
    return chat_formatter.format(
        tools,
        chat_history=task.memory.get() + state["memory"].get_all(),
        current_reasoning=state["current_reasoning"],
    )


react_prompt_component = AgentFnComponent(
    fn=react_prompt_fn, partial_dict={"tools": query_engine_tools}
)

After the LLM produces an output, we follow a decision tree process:

• If an answer is generated, we directly process the output.
• However, if an action is specified, we execute the designated tool with the specified arguments and then process the resulting output.

To process the agent’s response, we will define some functions which will ultimately follow the decision tree process.

def parse_react_output_fn(
    task: Task, state: Dict[str, Any], chat_response: ChatResponse
):
    """Parse ReAct output into a reasoning step."""
    output_parser = ReActOutputParser()
    reasoning_step = output_parser.parse(chat_response.message.content)
    return {"done": reasoning_step.is_done, "reasoning_step": reasoning_step}


parse_react_output = AgentFnComponent(fn=parse_react_output_fn)


def run_tool_fn(
    task: Task, state: Dict[str, Any], reasoning_step: ActionReasoningStep
):
    """Run tool and process tool output."""
    tool_runner_component = ToolRunnerComponent(
        query_engine_tools, callback_manager=task.callback_manager
    )
    tool_output = tool_runner_component.run_component(
        tool_name=reasoning_step.action,
        tool_input=reasoning_step.action_input,
    )
    observation_step = ObservationReasoningStep(observation=str(tool_output))
    state["current_reasoning"].append(observation_step)

    return {"response_str": observation_step.get_content(), "is_done": False}


run_tool = AgentFnComponent(fn=run_tool_fn)


def process_response_fn(
    task: Task, state: Dict[str, Any], response_step: ResponseReasoningStep
):
    """Process response."""
    state["current_reasoning"].append(response_step)
    response_str = response_step.response
    state["memory"].put(ChatMessage(content=task.input, role=MessageRole.USER))
    state["memory"].put(
        ChatMessage(content=response_str, role=MessageRole.ASSISTANT)
    )

    return {"response_str": response_str, "is_done": True}


process_response = AgentFnComponent(fn=process_response_fn)


def process_agent_response_fn(
    task: Task, state: Dict[str, Any], response_dict: dict
):
    """Process agent response."""
    return (
        AgentChatResponse(response_dict["response_str"]),
        response_dict["is_done"],
    )


process_agent_response = AgentFnComponent(fn=process_agent_response_fn)

Now, we will create a query pipeline, where we will add the created components as modules. We will add a chain and links to the query pipeline.

qp = QueryPipeline(verbose=True)

qp.add_modules(
    {
        "agent_input": agent_input_component,
        "react_prompt": react_prompt_component,
        "llm": llm,
        "react_output_parser": parse_react_output,
        "run_tool": run_tool,
        "process_response": process_response,
        "process_agent_response": process_agent_response,
    }
)

qp.add_chain(["agent_input", "react_prompt", "llm", "react_output_parser"])

qp.add_link(
    "react_output_parser",
    "run_tool",
    condition_fn=lambda x: not x["done"],
    input_fn=lambda x: x["reasoning_step"],
)
qp.add_link(
    "react_output_parser",
    "process_response",
    condition_fn=lambda x: x["done"],
    input_fn=lambda x: x["reasoning_step"],
)

qp.add_link("process_response", "process_agent_response")
qp.add_link("run_tool", "process_agent_response")

To visualize the query pipeline, we will use Network from the Pyvis library.

net = Network(notebook=True, cdn_resources="in_line", directed=True)
net.from_nx(qp.clean_dag)
print(net)

The following will be the resultant pipeline.

{
    "Nodes": [
        "agent_input",
        "react_prompt",
        "llm",
        "react_output_parser",
        "run_tool",
        "process_response",
        "process_agent_response"
    ],
    "Edges": [
        {
            "src_key": null,
            "dest_key": null,
            "condition_fn": null,
            "input_fn": null,
            "width": 1,
            "from": "agent_input",
            "to": "react_prompt",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "condition_fn": null,
            "input_fn": null,
            "width": 1,
            "from": "react_prompt",
            "to": "llm",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "condition_fn": null,
            "input_fn": null,
            "width": 1,
            "from": "llm",
            "to": "react_output_parser",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "width": 1,
            "from": "react_output_parser",
            "to": "run_tool",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "width": 1,
            "from": "react_output_parser",
            "to": "process_response",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "condition_fn": null,
            "input_fn": null,
            "width": 1,
            "from": "run_tool",
            "to": "process_agent_response",
            "arrows": "to"
        },
        {
            "src_key": null,
            "dest_key": null,
            "condition_fn": null,
            "input_fn": null,
            "width": 1,
            "from": "process_response",
            "to": "process_agent_response",
            "arrows": "to"
        }
    ],
    "Height": "600px",
    "Width": "100%",
    "Heading": ""
}

We will save the network as an HTML file and display the content.

net.write_html("agent.html")

with open("agent.html", "r") as file:
    html_content = file.read()

display(HTML(html_content))

The next step is to set up the agent worker around the query engines.

agent_worker = QueryPipelineAgentWorker(qp)
agent = agent_worker.as_agent(
    callback_manager=CallbackManager([]), verbose=True
)

We will create the task with the agent and ask questions.

task = agent.create_task(
    "What are ayurvedic treatments and prevention of migraine?"
)

step_output = agent.run_step(task.task_id)
print(step_output)

The agent will give the following answer:

In Ayurveda, the treatment and prevention of migraines focus on restoring balance among the body’s doshas, particularly Vata and Kapha. Preventative measures include avoiding daytime sleep, protecting oneself from cold, especially in the mornings, and managing allergic conditions promptly. It also emphasizes dietary restrictions, such as avoiding yogurt, greasy foods, and the combination of milk with acidic foods or fish. Treatment involves the use of herbal remedies to dilate capillaries, reduce fluid exudation, and promote fluid filtration. Key herbs and compounds, such as Guggulu for capillary dilation, Vachadi for drying exudation, and Pippalishrita for inducing cleansing sneezing, are employed. Additionally, remedies like Shirashularivajrarasa and Sadbindutaila are used to restore the coordination between the nervous, arterial, and venous systems, essential for relieving and preventing migraine attacks.

Let’s try another question.

task = agent.create_task(
    "What dosha does the person have if his skin is dry and thin, and he also has joint pains?"
)

step_output = agent.run_step(task.task_id)
print(step_output)

The agent will give the following answer:

If a person has dry and thin skin along with joint pains, it indicates an imbalance in the Vata dosha. Vata governs all movement and is associated with dryness and coldness in the body. When Vata is out of balance, it can lead to symptoms such as dry, rough skin and discomfort in the joints, manifesting as pain or stiffness. These characteristics are typical signs of elevated Vata that require balancing to restore harmony in the body and alleviate the symptoms.

Conclusion

I found it interesting to use the SingleStore vector database within SingleStore Notebooks to build an Ayurvedic healthcare multi-PDF agent. The free shared tier is easy to use and we saw that it can handle 40 ebooks. It is also easy to transition this application to a production environment with the SingleStore free shared tier.

Thanks for reading!

References

Agent around query pipeline with HyDE for PDFs

LlamaIndex with SingleStoreDB

Introduction

With the day-to-day advancements in large language models, retrieving SQL results has become easier. It has eliminated the need to enter SQL statements manually each time. While the traditional method of using SQL statements to interact with databases was effective, the advanced capabilities of LLMs have made SQL retrieval even more efficient and interesting.

Text-to-SQL has revolutionized our interaction with databases. As advanced language models continue to progress, new methods are also emerging. Before delving into the tools featured in this blog, let’s first grasp the concept of text-to-SQL.

Let’s get started!

What is Text-to-SQL?

Text-to-SQL is the task of translating a user’s natural language question into a valid SQL query that can be executed over a database. This process allows users, who may not have expertise in SQL or an understanding of the database’s structure, to interact with the database using natural language. For example, a user might ask a question about financial transactions in plain English, and text-to-SQL would convert this question into an SQL query that retrieves the desired information from the database. This capability has attracted significant attention from both the database and natural language processing communities, as it bridges the gap between user-friendly language and complex database queries.

Now that we understand text-to-SQL, let’s learn how to implement it using the tools we will use.

Implementing Text-to-SQL with SingleStore Helios

In this implementation, we will use the LlamaIndex framework, Groq, and Llama 3 model in the SingleStore Helios platform. Let’s find out what SingleStore Helios is.

SingleStore Helios

SingleStore Helios is a cloud database-as-a-service that provides elastic scalability, high availability, and excellent price performance for applications on leading public clouds. It separates storage from compute, enables independent scaling, and incorporates features such as Jupyter notebooks, data integration services, and a compute service for AI workloads. It supports low latency, high concurrency, and fast data ingestion from multiple sources, with horizontal scalability and workspace capabilities for independent compute scaling.

As a fully multi-cloud database, it gives users complete control over deployment across various cloud platforms by avoiding vendor lock-in. It simplifies application architecture with its SQL compatibility, extensive connector support, multi-model data handling, and API-driven workflows. It ensures secure and efficient database management for enterprise applications through robust security protocols and compliance with industry standards. For more information, visit its documentation.

Let’s plug into SingleStore Helios by signing up for a free trial. There are two editions of workspace available in a free trial:

• Starter Workspace: When you sign up, you will receive a starter workspace with an attached database.
• Standard Workspace: When you register, you will receive $600 in credits, which you can earn in 5 learning stages. You can utilize your earned credits in the standard workspace.

Integrating Groq, Llama 3, and LlamaIndex with SingleStore Helios

While creating a text-to-SQL application, having your database and notebook on a single platform is compatible. SingleStore Helios provides such an interface, which allows you to ingest your data into the database without creating a pipeline and work with it directly in your notebooks.

To start, create a workspace by selecting +Group in the left navigation panel.

Using your free trial credits, you can create a standard workspace.

To know more about how to create a workspace, visit here. After creating your workspace, it will begin deploying and may take a few minutes to complete.

Once the deployment is complete, create a database.

For our text-to-SQL application, we will use the Stock Tweets dataset from Kaggle. To load the dataset in your Stocks database, there is no need to create a pipeline. Click on the Stage in the left navigation panel.

Upload the CSV files.

Load the CSV files one by one to the created database.

Generate a notebook.

Execute the cells of the notebook after verifying the column names.

Our data will be stored by creating a table in the Stocks database. This way both of the CSV files will be ingested in the created database.

Now that our data has been ingested into the database, we can begin building the text-to-SQL application. To create a notebook, select Develop from the left navigation panel.

After creating a notebook, it is time to start coding!

Before rolling our sleeves up, let’s install and import the required dependencies.

!pip install -q llama-index matplotlib
!pip install -q llama-index-llms-groq
!pip install -q llama-index-embeddings-huggingface

# Import nest_asyncio to allow nested event loops
import nest_asyncio
nest_asyncio.apply()

# Access the secret in the SingleStore notebook
from singlestoredb.management import get_secret

# Import necessary libraries and modules
from llama_index.llms.groq import Groq  # Import Groq from llama_index for Text-to-SQL processing
from llama_index.core import Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding # Import Hugging Face embedding model
from sqlalchemy import *  # Import SQLAlchemy for database interaction
from llama_index.core import SQLDatabase  # Import SQLDatabase for handling database operations
from llama_index.core.indices.struct_store import NLSQLTableQueryEngine  # Import engine for NL to SQL query processing
from llama_index.core.indices.struct_store.sql_query import (
    SQLTableRetrieverQueryEngine,  # Import query engine for SQL table retrieval
)
from llama_index.core.objects import (
    SQLTableNodeMapping,  # Import mapping for SQL table nodes
    ObjectIndex,  # Import ObjectIndex for indexing database objects
    SQLTableSchema,  # Import schema definition for SQL tables
)
from llama_index.core import VectorStoreIndex  # Import VectorStoreIndex for indexing vectors
from llama_index.core.retrievers import NLSQLRetriever  # Import retriever for NL to SQL conversion
from llama_index.core.response.notebook_utils import display_source_node  # Import utility to display source nodes in notebooks
from llama_index.core.query_engine import RetrieverQueryEngine  # Import general query engine for data retrieval

With Groq, it is easy to load models. To work with Groq, create an API key on Groq Cloud. You can save the API key in Secrets under the Develop section.

secret = get_secret('groq')

Set the model as Llama 3 8B.

llm = Groq(model="llama3–8b-8192", api_key = secret)

With LLM, it is important to set the embedding model, because the NLSQLTableQueryEngine takes the embed model as OpenAI embedding model as default. Here we are using the Hugging Face embedding model.

embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
Settings.llm = llm
Settings.embed_model = embed_model

For implementing text-to-SQL, an engine connection is required. To connect with the SingleStore engine, navigate to your workspace and click on Connect Directly. You will find this information in your SQL IDE.

If you forgot your password, you can reset it by visiting Access.

After the setup, let’s connect the engine.

engine = create_engine("singlestoredb://"+"admin"+":"+"Password123"+"@"+"svc-998ada03–1742–45c8–924f-b8a15f4a4a75-dml.aws-virginia-5.svc.singlestore.com"+":"+"3306"+"/"+"Stocks")
sql_database = SQLDatabase(engine)

We have set up the SQL database and can use the NLSQLTableQueryEngine to create natural language queries, which will then be translated into SQL queries.

# Initialize the NLSQLTableQueryEngine with the specified SQL database, tables, and language model
query_engine = NLSQLTableQueryEngine(
 sql_database=sql_database, # The SQL database instance to query
 tables=["stock_tweets", "stock_yfinance_data"], # List of tables to include in the query engine
 llm=llm, # The language model used for processing natural language queries
)

Let’s query!

response = query_engine.query("What is current tweet of TSLA stock name? Also, tell the date.")
print(response)

The response will be:

Here is the synthesized response:

“The current tweet about TSLA stock is: ‘Mainstream media has done an amazing job at brainwashing people. Today at work, we were asked what companies we believe in &amp; I said @Tesla because they make the safest cars &amp; EVERYONE disagreed with me because they heard“they catch on fire &amp; the batteries cost 20k to replace”’. This tweet was posted on September 29th, 2022 at 11:41 PM.”

If we are uncertain about which table we will need ahead of time, and if the total size of the table schema is larger than the context window size, we should store the table schema in an index. This will enable us to retrieve the correct schema during query time.

To do this, we can utilize the SQLTableNodeMapping object, which takes an SQL database and generates a Node object for each SQLTableSchema object passed into the ObjectIndex constructor.

# Initialize the SQLTableNodeMapping with the sql_database
table_node_mapping = SQLTableNodeMapping(sql_database)
# Define table schemas
table_schema_objs = [
 SQLTableSchema(table_name="stock_tweets"),
 SQLTableSchema(table_name="stock_yfinance_data")
]
# Create the ObjectIndex from the schema objects and node mapping
obj_index = ObjectIndex.from_objects(
 table_schema_objs,
 table_node_mapping,
 VectorStoreIndex,
)
# Initialize the SQLTableRetrieverQueryEngine
query_engine = SQLTableRetrieverQueryEngine(
 sql_database, obj_index.as_retriever(similarity_top_k=1)
)

Let’s try this query engine!

response = query_engine.query("What is current tweet of TSLA stock name? Also, tell the date.")
print(response)

The response will be:

Here is the synthesized response:

“The current tweet about TSLA stock is: ‘Mainstream media has done an amazing job at brainwashing people. Today at work, we were asked what companies we believe in &amp; I said @Tesla because they make the safest cars &amp; EVERYONE disagreed with me because they heard“they catch on fire &amp; the batteries cost 20k to replace”’. This tweet was posted on September 29th, 2022 at 23:41:16 UTC.”

The response from both query engines is similar, but there is a slight difference in the time in the response.

Let’s see its metadata.

response.metadata["result"]

Here’s the metadata:

[(‘2022–09–29 23:41:16+00:00’, ‘Mainstream media has done an amazing job at brainwashing people. Today at work, we were asked what companies we believe in &amp; I said @Tesla because they make the safest cars &amp; EVERYONE disagreed with me because they heard“they catch on fire &amp; the batteries cost 20k to replace”’)]

Our text-to-SQL capability is currently encapsulated in a query engine that handles both retrieval and synthesis. You can use the SQL retriever independently. Now we will integrate it into RetrieverQueryEngine to achieve similar results with nodes.

nl_sql_retriever = NLSQLRetriever(
 sql_database, tables=["stock_tweets","stock_yfinance_data"], return_raw=True
)

results = nl_sql_retriever.retrieve(
    "Return the top 5 stocks (along with their company names) with the highest volume."
)

for n in results:
    display_source_node(n)

These are the source nodes:

Node ID: 129fd86a-a12d-418c-8b63-ed5a7997ceb2
Similarity: None
Text: [(‘Ford Motor Company’, ‘F’, 311645200), (‘Ford Motor Company’, ‘F’, 311645200), (‘Ford Motor Com…

The default setting in the NLSQLRetriever is return_raw = True. Let’s make it False.

nl_sql_retriever = NLSQLRetriever(
    sql_database, tables=["stock_tweets","stock_yfinance_data"], return_raw=False
)

results = nl_sql_retriever.retrieve(
    "Return the top 5 stocks (along with their company names) with the highest volume."
)

for n in results:
    display_source_node(n, show_source_metadata=True)

These are the source nodes:

Node ID: c175c114-b565–422e-b576–2e4f8d44b9a5
Similarity: None
Text:
Metadata: {‘Company_Name’: ‘Ford Motor Company’, ‘Stock_Name’: ‘F’, ‘Volume’: 311645200}

Now we will integrate our SQL Retriever with the standard RetrieverQueryEngine to generate a response. This will complete our Text-to-SQL retriever query engine.

query_engine = RetrieverQueryEngine.from_args(nl_sql_retriever)

results = query_engine.query(
    "Return the top 5 stock names (along with their company names) with the highest volume."
)

print(str(results))

The results will be:

Apple Inc.
Microsoft Corporation
Alphabet Inc.
Amazon.com, Inc.
Facebook, Inc.

Conclusion

We saw how easy it was to query in natural language and retrieve results using the Text-to-SQL retriever. Additionally, we found it easy to use Stages and Secrets in SingleStore Helios, with a single interface for Database and Notebooks. It was thrilling for me to explore SingleStore Helios, now it’s your turn to experiment with it.

Thanks for reading!

References

Text-to-SQL Guide

Introduction

Most of us have invested time and effort in buying real estate, mutual funds, bonds, and more. Before investing, we search for a suitable banking partner. For long-term investments, we research which banks are performing well minus scams or frauds, and offer reasonable interest rates and returns. So we need to stay updated with financial news.

What if you had at your disposal a real-time financial news chatbot that could provide you with all the news related to finance and economics? Does that sound interesting? Retrieval Augmented Generation (RAG) has now made this possible. We can leverage large language models and vector databases to get our queries answered.

Let’s create a real-time financial news RAG chatbot and see if it accurately answers questions using available data. It can be real-time by feeding a vector database with the latest news.

Real-Time Financial News RAG Chatbot Using Gemini

As our starting point, we took the Indian Financial News dataset. This dataset consists of financial news related to Indian banks. It is updated till 26th May 2020.

Before we get started, let’s install the required dependencies.

%pip install -q llama-index 'google-generativeai>=0.3.0' qdrant_client llama-index-embeddings-fastembed fastembed llama-index-llms-gemini

Preparing the Node

As the dataset is a CSV file, let’s load the data.

from llama_index.core import SimpleDirectoryReader
docs = SimpleDirectoryReader("Dataset").load_data()

All the documents are ready; now we will split the sentences into defined chunk sizes.

from llama_index.core.node_parser.text import SentenceSplitter
# Initialize the SentenceSplitter with a specific chunk size
text_parser = SentenceSplitter(chunk_size=1024)
text_chunks = [] # This will hold all the chunks of text from all documents
doc_idxs = [] # This will keep track of the document each chunk came from
for doc_idx, doc in enumerate(docs):
 # Split the current document's text into chunks
 cur_text_chunks = text_parser.split_text(doc.text)
 
 # Extend the list of all text chunks with the chunks from the current document
 text_chunks.extend(cur_text_chunks)
 
 # Extend the document index list with the index of the current document, repeated for each chunk
 doc_idxs.extend([doc_idx] * len(cur_text_chunks))

Then, we will create a text node object and assign the metadata to it. We will store all the nodes in one node list.

from llama_index.core.schema import TextNode
nodes = [] # This will hold all TextNode objects created from the text chunks
# Iterate over each text chunk and its index
for idx, text_chunk in enumerate(text_chunks):
 # Create a TextNode object with the current text chunk
 node = TextNode(text=text_chunk)
 
 # Retrieve the source document using the current index mapped through doc_idxs
 src_doc = docs[doc_idxs[idx]]
 
 # Assign the source document's metadata to the node's metadata attribute
 node.metadata = src_doc.metadata
 
 # Append the newly created node to the list of nodes
 nodes.append(node)

Initializing the Qdrant Vector Store

To store the nodes, we need a vector store. Here, we have chosen Qdrant as our vector store. Qdrant is a high-performance vector database with all the specific features that a vector store should have. It is fast and accurate by utilizing the HNSW algorithm for approximate nearest neighbor search. Qdrant supports additional payload and filters based on payload values by providing an easy-to-use API. Additionally, it supports docker installation, is equipped with in-memory storage of vectors, is cloud-native, and scales horizontally. Developed in the Rust language, Qdrant implements dynamic query planning and payload data indexing.

First, we’ll create a collection in the vector store index.

from llama_index.core import VectorStoreIndex, StorageContext
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import Settings
from llama_index.core import StorageContext
import qdrant_client

# Create a local Qdrant vector store
client = qdrant_client.QdrantClient(path="financialnews")
vector_store = QdrantVectorStore(client=client, collection_name="collection")

Gemini Embeddings and Text Model

The vector store and nodes are ready, but the vector store is not going to directly accept the nodes. They require embeddings, and for embeddings, we are using the Gemini embedding model here. We’ll be leveraging the Gemini LLM, which is a very capable family of multimodal models. Built on the transformer architecture and trained on TPUs, the Gemini model excels in summarization, reading comprehension tasks with per-task fine-tuning, multilinguality, long context, coding, complex reasoning, mathematics, and of course, multimodality.

We’ll initiate the Google API key, which you can obtain from Google AI Studio.

%env GOOGLE_API_KEY = "your-api-key"
import os
GOOGLE_API_KEY = "your-api-key" # add your GOOGLE API key here
os.environ["GOOGLE_API_KEY"] = GOOGLE_API_KEY

Now, using the API key, we will generate the embeddings using the FastEmbed embedding model and the Gemini LLM in Llamaindex’s Settings.

from llama_index.embeddings.fastembed import FastEmbedEmbedding
embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")
for node in nodes:
 node_embedding = embed_model.get_text_embedding(
 node.get_content(metadata_mode="all")
 )
 node.embedding = node_embedding
from llama_index.llms.gemini import Gemini
Settings.embed_model = embed_model
Settings.llm = Gemini(model="models/gemini-pro")
Settings.transformations = [SentenceSplitter(chunk_size=1024)]
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex(
 nodes=nodes,
 storage_context=storage_context,
transformations=Settings.transformations,
)

The vector store is saved in the storage context, and the index has been initiated with it and the nodes.

HyDE Query Transformation

Now, we’ll initiate the vector query engine with a response synthesizer and vector retriever. Vector Retriever is initiated with the vector index retriever in which the index was included. Response Synthesizer generates a response from an LLM, using a user query and a given set of text chunks. The output of a response synthesizer is a Response Object.

from llama_index.core import get_response_synthesizer
from llama_index.core.query_engine import RetrieverQueryEngine
from llama_index.core.retrievers import VectorIndexRetriever
vector_retriever = VectorIndexRetriever(index=index, similarity_top_k=2)
response_synthesizer = get_response_synthesizer()
vector_query_engine = RetrieverQueryEngine(
 retriever=vector_retriever,
 response_synthesizer=response_synthesizer,
)

We will employ the HyDE query transformer for advanced retrieval. HyDE (Hypothetical Document Embeddings) facilitates zero-shot prompt-based instruction-following in a large language model. It generates a hypothetical document encapsulating relevant text patterns, converts them into embedding vectors, and averages them to create a single embedding. This procedure identifies the corresponding actual embedding through vector similarity in the document embedding space, thereby eliminating the need for a retrieval step involving querying an input and obtaining a document from a large database.

The HyDE query transformation assists in delivering responses directly and concisely.

from llama_index.core.indices.query.query_transform import HyDEQueryTransform
from llama_index.core.query_engine import TransformQueryEngine
hyde = HyDEQueryTransform(include_original=True)
hyde_query_engine = TransformQueryEngine(vector_query_engine, hyde)

Leveraging Gradio UI for Chatbot Implementation

For deploying a chatbot, we will use Gradio.

def queries(query_str):
 response = hyde_query_engine.query(query_str)
 return str(response)
import gradio as gr
import os
gr.close_all()
with gr.Blocks(theme=gr.themes.Soft()) as demo:
 gr.Markdown(
 """
 # Welcome to Gemini-Powered Stock Predictor RAG Chatbot!
 """)
 chatbot = gr.Chatbot()
 msg = gr.Textbox()
 clear = gr.ClearButton([msg, chatbot])
 def respond(message, chat_history):
 bot_message = queries(message)
 chat_history.append((message, bot_message))
 return "", chat_history
 msg.submit(respond, [msg, chatbot], [msg, chatbot])
 
demo.launch(share=True)

Query Time!

Let’s query the chatbot.

Question 1:

Tell me all the news about scam.

PNB scam fallout: Trade finance hit as caution prevails, premium soars
PNB scam fallout widens as NCLT bars over 60 entities from selling assets
PNB scam: CBI arrests 4 officials of Nirav Modi, Mehul Choksi’s companies
PNB scam: Court issues non-bailable warrants against Nirav Modi, Choksi
PNB chief Sunil Mehta admits the collusion of employees in LoUs scam
Nirav Modi case: CBI recovers LoU documents, arrests another PNB executive
Assets worth Rs 57.16 bn, 1st LoU issued in Mar 2011 seized: ED on PNB scam
‘Explain how scam took place’: CVC seeks report from PNB, FinMin in 10 days
PNB fraud case: Congress says it will oppose PSBs privatisation
PNB fraud: Govt thinks RBI may be unable to ensure effective supervision?
7 public sector bank stocks hit 52-week lows on Rs 114-bn PNB fraud case
PNB tanks 7% on Monday; m-cap plunges Rs 109.75 bn in four sessions
PNB will have to own responsibility of bona fide transactions, says FinMin
Nirav Modi PNB fraud: PM Modi’s silence speaks of his loyalties, says Rahul
Nirav Modi PNB fraud: Vijay Aggarwal to be absconding diamantaire’s lawyer
Before PNB fraud: Nirav Modi, Choksi left 18 businessmen, 24 firms bankrupt
Nirav Modi PNB fraud: CBI seals bank’s Brady House Branch in Mumbai
Once the pride of Punjab, PNB is now a theme for WhatsApp memes in India
Eyes wide shut: The $1.8 billion PNB fraud that went completely unnoticed
Bank Union wants CBI probe into PNB fraud, alleges RBI failed as regulator
CBI questions Nirav Modi’s CFO Vipul Ambani; searches PNB Mumbai branch
PNB scam: Jewellers expect more casualties and tighter regulations
Rs 114-bn PNB scam: Why fraudsters prefer to use Letter of Undertaking

Question 2:

What is the latest news about RBI?

The RBI has deferred the launch of IndAs again, awaiting amendments to the banking laws.

Question 3:

Can you tell me all the news related to home loans?

Lending rates cut leads to balance transfers in home loan market: ICRA
Can’t match home loan rates with BoB: United Bank of India
After rate cut, Bank of Baroda eyes bigger home loan pie
Bank of Baroda offers lowest home loans rates at 8.35%
For SBI, home loan queries jump three times since rate cut
HDFC cuts home loan rates by up to 0.45%
Hamara Ghar offer: SBI’s new home loan
Banks cut lending rates: How home-loan borrowers will benefit
SBI cuts lending rate by 90 bps; home, auto loans to become cheaper

Question 4:

Tell me about the news of Yes Bank Scam.

The RBI discovered more than $450 million in extra bad loans at Yes Bank. The gross non-performing assets assessed by the Reserve Bank of India were $457 million higher than Yes Bank had disclosed as of November 20, 2019.

Question 5:

What is the latest news about NBFC?

The latest news about NBFC is that RBI eases norms for banks to lend more to NBFCs, housing finance companies. RBI enhances single-borrower exposure limit to 15% of bank’s capital.

Conclusion

Building a real-time financial news RAG chatbot using the Indian Financial News dataset proved to be a rewarding journey. Now you know that you can easily ask questions about banks and get instant answers! It’s now time for you to make your own chatbot. Hope you enjoyed reading this blog.

Introduction

In 1950, when Alan Turing introduced the term “Machine Intelligence” in his paper “Computing Machinery and Intelligence,” no one had imagined that one day, in the future, it would lead to various innovations using artificial intelligence in different domains. One domain that is very popular and important among users is online shopping. With the surge in e-commerce, users increasingly rely on visual cues to guide them in their purchasing decisions. In response to this shift in consumer behavior, image-driven product search has emerged as a powerful tool to enhance the shopping experience. E-commerce platforms like Amazon, Myntra, Ajio, and Meesho are using image-driven product search widely.

You must be familiar with image-driven searches on shopping websites. This innovative approach uses the visual content in images to let users explore products more intuitively and efficiently. By simply uploading or capturing an image, shoppers can quickly find similar or related items within a vast catalog. Whether seeking fashion inspiration, home decor ideas, or specific product recommendations, image-driven search offers a dynamic and personalized shopping journey tailored to individual preferences and tastes.

We can make the results accurate by using the recently developed all-in-one embedding model by Meta: ImageBind. But before using the embedding model, we need to use a vector database to store them.

When it comes to image search, vector databases have been particularly transformative. Traditional image search methods often rely on metadata tags or textual descriptions, which can be limited in capturing the rich visual content of images. With vector databases, images are transformed into high-dimensional vectors that encapsulate their visual features by allowing for more accurate and nuanced similarity comparisons. This means that users can search for images based on visual similarity by enabling tasks for e-commerce product search from images with remarkable precision. However, when it comes to vector databases, we are faced with a dilemma: which database will be the best for our application?

Here, I have chosen the Qdrant Vector database, which offers an advanced search algorithm for approximate nearest neighbor search for advanced AI applications, namely the HNSW algorithm. Meesho is already using the Qdrant vector database but still, the results are not very accurate. We can make the results more accurate by integrating the Qdrant and ImageBind embedding models. Before diving into the content, let’s look at the steps:

1. Loading the Dataset.
2. Initializing the Qdrant Vector DB.
3. Image Embeddings with ImageBind.
4. Deploying with Gradio.

Reverse Product Image Search with Qdrant

Let’s install the dependencies first to get started with the reverse product image search.

%pip install opendatasets gradio qdrant-client transformers sentence_transformers sentencepiece tqdm

Loading the Dataset

Using the opendatasets library, download the Kaggle dataset using your username and key. You can obtain them by visiting the Settings page on Kaggle. Click on “Access API Keys,” and a kaggle.json file will be downloaded. This file will contain your username and API key.

import opendatasets as od
od.download("https://www.kaggle.com/datasets/vikashrajluhaniwal/fashion-images")

Now, let’s store the images in a list so that we can easily access the images.

import random
import gradio as gr
from PIL import Image
from qdrant_client import QdrantClient
from qdrant_client.http import models
import tempfile
import os
from tqdm import tqdm
import os
def get_image_paths(directory):
 # Initialize an empty list to store the image paths
 image_paths = []
 # Iterate through all files and directories within the given directory
 for root, dirs, files in os.walk(directory):
 for file in files:
 # Check if the file has an image extension (e.g., .jpg, .png, .jpeg, etc.)
 if file.lower().endswith(('.png', '.jpg', '.jpeg', '.gif', '.bmp')):
 # Construct the full path to the image file
 image_path = os.path.join(root, file)
 # Append the image path to the list
 image_paths.append(image_path)
 
 return image_paths
# Directory paths
women_directory = './fashion-images/data/Footwear_Women/Images/images_with_product_ids/'
men_directory = './fashion-images/data/Footwear_Men/Images/images_with_product_ids/'
girls_directory = './fashion-images/data/Apparel_Girls/Images/images_with_product_ids/'
boys_directory = './fashion-images/data/Apparel_Boys/Images/images_with_product_ids/'
# Get image paths for different categories
image_paths_Women = get_image_paths(women_directory)
image_paths_Men = get_image_paths(men_directory)
image_paths_Girls = get_image_paths(girls_directory)
image_paths_Boys = get_image_paths(boys_directory)
all_image_paths = []
all_image_paths.append(image_paths_Boys)
all_image_paths.append(image_paths_Girls)
all_image_paths.append(image_paths_Men)
all_image_paths.append(image_paths_Women)

Initializing the Qdrant Vector DB

Initialize the Qdrant Client with in-memory storage. The collection name will be “imagebind_data” and we will be using cosine distance.

# Initialize Qdrant client and load collection
client = QdrantClient(":memory:")
client.recreate_collection(collection_name = "imagebind_data",
vectors_config = {"image": models.VectorParams( size = 1024, distance = models.Distance.COSINE ) } )

Image Embeddings with ImageBind

ImageBind is an innovative model developed by Meta AI’s FAIR Lab. This model is designed to learn a joint embedding across six different modalities: images, text, audio, depth, thermal, and IMU data. One of the key features of ImageBind is its ability to learn this joint embedding without requiring all combinations of paired data. It has been discovered that only image-paired data is necessary to bind the modalities together effectively. This unique capability allows ImageBind to leverage recent large-scale vision-language models and extend their zero-shot capabilities to new modalities simply by utilizing their natural pairing with images.

We’ll use ImageBind for creating embeddings, but before diving deep, first, let’s follow some steps required for installing ImageBind.

Clone the git repository of Imagebind:

git clone https://github.com/facebookresearch/ImageBind.git

Change the directory:

cd Imagebind

Edit the requirements.txt file: Delete Mayavi and Cartopy if getting errors or facing problems.

Install the requirements:

pip install -r requirements.txt

Come back to your system.

Then, load the model.

import sys 
sys.path.append("./ImageBind/")
device = "cuda"
import imagebind
from imagebind.models import imagebind_model
model = imagebind_model.imagebind_huge(pretrained=True)
model.eval()
model.to(device)

After initializing the model, we will now create embeddings.

from imagebind.models.imagebind_model import ModalityType
from imagebind import data
import torch
embeddings_list = []
for image_paths in [image_paths_Boys, image_paths_Girls, image_paths_Men, image_paths_Women]:
 inputs = {ModalityType.VISION: data.load_and_transform_vision_data(image_paths, device)}
 with torch.no_grad():
 embeddings = model(inputs)
 embeddings_list.append(embeddings)

Then we’ll update the Qdrant Vector DB with the generated embeddings.

import uuid
points = []
# Iterate over each embeddings and corresponding image paths
for idx, (embedding, image_paths) in enumerate(zip(embeddings, all_image_paths)):
 for sub_idx, sample in enumerate(image_paths):
 # Convert the sample to a dictionary
 payload = {"path": sample}
 # Generate a unique UUID for each point
 point_id = str(uuid.uuid4())
 points.append(models.PointStruct(id=point_id,
 vector= {"image": embedding['vision'][sub_idx]}, 
 payload=payload)
 )
client.upsert(collection_name="imagebind_data", points=points)

We’ll prepare a processing function in which we will take the image as an input and perform a reverse image search with the help of the embeddings.

def process_text(image_query):
 
 user_query = [image_query]
 dtype, modality = ModalityType.VISION, 'image'
 user_input = {dtype: data.load_and_transform_vision_data(user_query, device)}
 with torch.no_grad():
 user_embeddings = model(user_input)
 
 image_hits = client.search(
 collection_name='imagebind_data',
 query_vector=models.NamedVector(
 name="image",
 vector=user_embeddings[dtype][0].tolist()
 )
 )
 # Check if 'path' is in the payload of the first hit
 if image_hits and 'path' in image_hits[0].payload:
 return (image_hits[0].payload['path'])
 else:
 return None

Deploying with Gradio

Now that we have prepared the processing image function, we’ll use Gradio to deploy by defining its interface.

import tempfile
tempfile.tempdir = "./fashion-images/data"
# Gradio Interface
iface = gr.Interface(
 title="Reverse Image Search with Imagebind",
 description="Leveraging Imagebind to perform reverse image search for ecommerce products",
 fn=process_text,
 inputs=[
 gr.Image(label="image_query", type="filepath")
 ],
 outputs=[
 gr.Image(label="Image")], 
)

Image Search Using Product Category

If you want to search images with the product category, then you have to define some functions. We’ll define a function to get images from the category.

# Define function to get images of selected category
def get_images_from_category(category):
 # Convert category to string
 category_str = str(category)
 # Directory path for selected category
 category_dir = f"./fashion-images/data/{category_str.replace(' ', '_')}/Images/images_with_product_ids/"
 # List of image paths
 image_paths = os.listdir(category_dir)
 # Open and return images
 images = [Image.open(os.path.join(category_dir, img_path)) for img_path in image_paths]
 return images

Then list the product categories.

# Define your product categories
product_categories = ["Apparel Boys", "Apparel Girls", "Footwear Men", "Footwear Women"]

After that, we’ll define a function for category selection.

# Define function to handle category selection
def select_category(category):
 # Get images corresponding to the selected category
 images = get_images_from_category(category)
 # Return a random image from the list
 return random.choice(images)

Deploying with Gradio

Now, we’ll create Gradio interface components for the category selection, such as category dropdown and submit button.

# Create interface components for the category selection
category_dropdown = gr.Dropdown(product_categories, label="Select a product category")
submit_button = gr.Button()
images_output = gr.Image(label="Images of Selected Category")

After that, we’ll create a Gradio interface and pass the functions and components.

category_search_interface = gr.Interface(
 fn=select_category,
 inputs=category_dropdown,
 outputs=images_output,
 title="Category-driven Product Search for Ecommerce",
 description="Select a product category to view a random image from the corresponding directory.",
)

Merging Two Gradio Interfaces

We have deployed two Gradio Interfaces: one is Reverse Image Search and another is Image Search Using Product Category. What if we can see both in one application? That can be performed by using TabbedInterface.

# Combine both interfaces into the same API
combined_interface = gr.TabbedInterface([iface, category_search_interface])
# Launch the combined interface
combined_interface.launch(share=True)

Now, we’ll get an internal URL and a public URL. The application is ready with two tabs.

Tab 0: Reverse Image Search with ImageBind for E-Commerce

Tab 1: Category-Driven Product Search for E-Commerce

Examples

Let’s see how our application performs.

I passed a girl’s image who was wearing a sleeveless Dress. Let’s see if our application can find a similar image from the products.

The application found a dress which is quite similar. Impressive!

Let’s try passing Shoes as an Image. I passed a picture of three people’s feet with sneakers. Let’s see the result.

The result is impressive. The output is sneakers of the same style.

Let’s try a category search. For example, I want to see what are the products in the Boys Apparel category. Gradio can give only one image as output, but in this application, it will not give the same image for every search.

The first search gave an image of a white t-shirt. Let’s search again to see what other t-shirts there are.

The result is a blue-gray t-shirt. So, the results are not repeated. Great!

Conclusion

With Qdrant Vector DB, reverse image search is possible for e-commerce products. We saw from our results how to perform a reverse image search by uploading an image and getting images of similar products from select product categories. The results were accurate with the help of the ImageBind embedding model.

Hope you enjoyed reading this blog. Now it is your turn to try this integration.

CodeSpace

You can find the code on GitHub.

Thanks for reading!

Introduction to Stable Diffusion 3

Stability AI has just announced the most capable text-to-image model with enhanced performance in multi-subject prompts, image quality, and spelling abilities: Stable Diffusion 3. While the model is in an early preview stage, the waitlist is now open for interested users to participate and provide valuable feedback for further refinement before its broader release.

Key Highlights of Stable Diffusion 3

Let’s see what difference Stable Diffusion 3 is going to make in the evolving realm of Text-to-Image Generation.

1. Improved Performance and Quality: The suite of models in Stable Diffusion 3 ranges from 800 million to 8 billion parameters, which offers scalability and quality options to cater to diverse creative needs. The model leverages a diffusion transformer architecture combined with flow matching, which enhances its capabilities to generate high-quality images from text prompts.
2. Focus on Safety and Responsibility: The development team has implemented various safeguards throughout the training, evaluation, and deployment phases to prevent misuse by bad actors which emphasizes safe and responsible AI practices. Researchers, experts, and the community are collaborating with the ongoing innovation with integrity, prioritizing safety and ethical considerations.

One of the most exciting elements of this new Stable Diffusion release is the 800 million variant. This would dramatically enhance the accessibility of image synthesis AI. This falls in line with the direction that Google has taken with Gemma 2b and Microsoft has done with Phi-2 language models.

Stable Diffusion 3 combines the diffusion transformer architecture with flow matching. Let’s see what that means.

Diffusion Transformer Architecture: A Technical Overview

The Diffusion Transformer (DiT) architecture combines the power of diffusion models with the scalability and flexibility of transformer-based architectures.

Diffusion models have shown remarkable success in generative modeling tasks by leveraging the principle of iteratively denoising samples. Transformers, known for their ability to capture long-range dependencies in data, have demonstrated outstanding performance in various natural language processing and computer vision tasks.

Combining the strengths of diffusion models and transformers leads to more powerful and efficient generative models capable of capturing complex data distributions.

Architecture

The backbone of the DiT architecture consists of a stack of transformer layers, similar to those found in standard transformer models like the Vision Transformer (ViT) or the Transformer architecture used in natural language processing tasks. Each transformer layer in the DiT is augmented to incorporate the diffusion process.

This involves integrating the diffusion mechanism into the self-attention mechanism of the transformer. DiTs typically incorporate a conditioning mechanism to allow for class-conditional image generation. This conditioning mechanism can be implemented using various techniques, such as injecting class tokens into the input sequence or conditioning the attention mechanism on class information.

Similar to vision transformer architectures, DiTs process images in a patchwise manner. Each patch of the input image is flattened into a sequence of tokens, which are then processed by the transformer layers. DiTs often employ adaptive layer normalization (adaLN) to improve training stability and model performance. adaLN adjusts the normalization parameters based on the input data, allowing the model to adapt to different input distributions.

Training

DiTs are typically trained using standard generative modeling techniques, such as maximum likelihood estimation or variational inference. Training data is fed into the model, and the model parameters are updated iteratively to minimize a suitable loss function, such as the negative log-likelihood of the training data.

During training, the model may use additional techniques such as the exponential moving average (EMA) of model weights, data augmentation, and regularization to improve performance and stability. One of the key advantages of DiTs is their scalability. By leveraging the scalability of transformer architectures, DiTs can efficiently handle large-scale generative modeling tasks, including high-resolution image generation.

DiTs have shown impressive performance on benchmark datasets and often achieve state-of-the-art results in terms of image quality metrics such as Frechet Inception Distance (FID), Inception Score (IS), and Precision/Recall.

Why use Flow Matching? An Overview

Flow Matching (FM) is a novel framework for training Continuous Normalizing Flow (CNF) models, designed to overcome some of the limitations associated with traditional training approaches. It introduces a simulation-free methodology that leverages conditional constructions to efficiently scale to high-dimensional datasets while providing improved sampling and generation capabilities.

Traditional training methods for CNF models often involve computationally expensive simulations, especially when dealing with high-dimensional data. These simulations can hinder scalability and efficiency.

Flow Matching aims to address these challenges by offering a simulation-free alternative that simplifies training while maintaining high-quality sampling and generation performance.

Key Concepts

1. Conditional Constructions: FM relies on conditional probability paths and vector fields to model the flow of data through the latent space. By conditioning on relevant variables, FM can capture complex data distributions more effectively.
2. Probability Paths: Instead of relying on diffusion processes or stochastic simulations, FM directly specifies the probability path through the latent space. This allows for more precise control over the data generation process.
3. Vector Fields: FM utilizes vector fields to define the dynamics of the flow, which guides the transformation of data from the input space to the latent space. These vector fields can be optimized to match the desired probability path.

Training Methodology

1. Simulation-Free Approach: FM eliminates the need for costly simulations by directly specifying the probability path and optimizing the vector fields accordingly. This results in faster and more efficient training.
2. Gradient-Based Optimization: FM employs gradient-based optimization techniques to iteratively update the parameters of the CNF model, which ensures convergence towards the desired probability distribution.
3. Conditional Flow Matching Objective: FM defines a novel objective function, called the Conditional Flow Matching (CFM) objective, which enables unbiased estimation of gradients and efficient training.

FM demonstrates effectiveness in various image datasets, including CIFAR-10 and ImageNet, at different resolutions. It achieves state-of-the-art results in terms of negative log-likelihood (NLL), sample quality (measured by Frechet Inception Distance, FID), and training efficiency (measured by the number of function evaluations, NFE). Comparative experiments with existing methods highlight the superior performance of FM, particularly in terms of faster convergence and improved sampling efficiency.

How Will the Fusion of DiT and FM Be?

The fusion of diffusion transformer architecture and flow matching represents an interesting approach in text-to-image generation AI. Once Stable Diffusion 3 releases, we will know what capabilities this combination can yield, but the concept is extremely promising.

The diffusion transformer architecture integrates the transformer model’s powerful sequence modeling capabilities with diffusion models’ ability to generate high-quality images.

This combination allows the model to effectively capture complex relationships between text prompts and corresponding images, which results in more coherent and contextually relevant outputs. The relationship between language and image is vital, as eventually the model’s ability to understand human language, and its meaning and incorporate that into the generation process is when we move towards more controllable image generation systems.

Flow matching techniques enable the model to learn the underlying distribution of image data and generate samples that closely match this distribution. By aligning the generated images with the target distribution, flow matching ensures that the outputs are visually appealing and exhibit realistic characteristics.

According to Stability researchers, the integration between diffusion transformer architecture and flow matching leads to significant improvements in image quality. The model can produce images with finer details, sharper textures, and more accurate representations of the input text prompts. This enhancement in image fidelity enhances the overall user experience and widens the range of potential applications in fields such as digital art, design, and entertainment.

Another challenge in text-to-image generation has been to handle multi-subject prompts. Due to this, most AI-generated images you would see would typically feature a single human. The diffusion transformer architecture, coupled with flow matching, enables the model to process complex, multi-subject prompts with greater accuracy and coherence. This capability allows users to generate diverse and intricate images based on nuanced textual descriptions, which opens up new possibilities for creative expression and storytelling.

Final Words

The combination of diffusion transformer architecture and flow matching represents a significant advancement in text-to-image generation methodologies. By harnessing the strengths of both approaches, this innovative framework offers improved modeling capabilities, enhanced image quality, and effective handling of multi-subject prompts while prioritizing safety and ethical considerations.

The overview of DiT and FM, along with the fusion between them, has us excited about the new Stable Diffusion 3 model. Join the waitlist and be a part of this revolution!

About Superteams.ai

Superteams.ai connects top AI talent with companies seeking accelerated product and content development. These “Superteamers” offer individual or team-based solutions for projects involving cutting-edge technologies like LLMs, image synthesis, audio or voice synthesis, and other cutting-edge open-source AI solutions. With over 500 AI researchers and developers, Superteams has facilitated diverse projects like 3D e-commerce model generation, advertising creative generation, enterprise-grade RAG pipelines, geospatial applications, and more. Focusing on talent from India and the global South, Superteams offers competitive solutions for companies worldwide. To explore partnership opportunities, please write to founders@superteams.ai or visit this link.

A Technical Deep-Dive Into Stable Diffusion 3 was originally published in Superteams.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

As large language models are advancing, the craze for building RAG (Retrieval Augmented Generation) applications is increasing. Google just launched an open-source model: Gemma. As we know, RAG represents a fusion between two fundamental methodologies: retrieval-based techniques and generative models. Retrieval-based techniques involve sourcing pertinent information from expansive knowledge repositories or corpora in response to specific queries. Generative models excel in crafting original text or responses by leveraging insights taken from training data to create new content from scratch. With this launch, why not try the new open-source model for building a RAG pipeline and see how it is performing?

Let’s get started and break the process into these steps:

1. Loading the Dataset: Cosmopedia
2. Embedding Generation with Hugging Face
3. Storing in the FAISS DB
4. Gemma: Introducing the SOTA model
5. Querying the RAG Pipeline

Building RAG Application on Gemma 7B

Before rolling our sleeves on, let’s install and import the required dependencies.

%pip install -q -U langchain torch transformers sentence-transformers datasets faiss-cpu

import torch
from datasets import load_dataset
from langchain_community.document_loaders.csv_loader import CSVLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import FAISS
from transformers import AutoTokenizer, AutoModelForCausalLM
from transformers import AutoTokenizer, pipeline
from langchain import HuggingFacePipeline
from langchain.chains import RetrievalQA

Loading the Dataset: Cosmopedia

To make a RAG application, we have selected a Hugging Face dataset, Cosmopedia. This dataset consists of synthetic textbooks, blog posts, stories, posts, and WikiHow articles generated by Mixtral-8x7B-Instruct-v0.1. The dataset contains over 30 million files and 25 billion tokens, which makes it the largest open synthetic dataset to date.

This dataset contains 8 subsets. We’ll move with the ‘stories’ subset. We’ll load the dataset using the datasets library.

data = load_dataset("HuggingFaceTB/cosmopedia", "stories", split="train")

Then, we will convert it to a Pandas dataframe, and save it to a CSV file.

data = data.to_pandas()
data.to_csv("dataset.csv")
data.head()

Now that the dataset is saved on our system, we will use LangChain to load the dataset.

loader = CSVLoader(file_path='./dataset.csv')
data = loader.load()

Now that the data is loaded, we need to split the documents inside the data. Here, we split the documents into chunk sizes of 1000. This will help the model to work fast and efficiently.

text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
docs = text_splitter.split_documents(data)

Embedding Generation with Hugging Face

After that, we will generate embeddings using Hugging Face Embeddings and with the help of the Sentence Transformers model.

modelPath = "sentence-transformers/all-MiniLM-l6-v2"
model_kwargs = {'device':'cpu'}
encode_kwargs = {'normalize_embeddings': False}
embeddings = HuggingFaceEmbeddings(
 model_name=modelPath, 
 model_kwargs=model_kwargs, 
 encode_kwargs=encode_kwargs 
)

Storing in the FAISS DB

The embeddings are generated, but we need them to be stored in a vector database. We’ll be saving those embeddings in the FAISS vector store, which is a library for efficient similarity search and clustering dense vectors.

db = FAISS.from_documents(docs, embeddings)

Gemma: Introducing the SOTA model

Gemma offers two model sizes, with 2 billion and 7 billion parameters respectively, catering to different computational constraints and application scenarios. Both pre-trained and fine-tuned checkpoints are provided, along with an open-source codebase for inference and serving. It is trained on up to 6 trillion tokens of text data and leverages similar architectures, datasets, and training methodologies as the Gemini models. Both exhibit strong generalist capabilities across text domains and excel in understanding and reasoning tasks on a large scale.

The release includes raw, pre-trained checkpoints as well as fine-tuned checkpoints optimized for specific tasks such as dialogue, instruction-following, helpfulness, and safety. Comprehensive evaluations have been conducted to assess the models’ performance and address any shortcomings, which enables thorough research and investigation into model tuning regimes and the development of safer and more responsible model development methodologies. Gemma’s performance surpasses that of comparable-scale open models across various domains, including question-answering, commonsense reasoning, mathematics and science, and coding, as demonstrated through both automated benchmarks and human evaluations. To know more about the Gemma model, visit this technical report.

To get started with the Gemma model, you should acknowledge their terms on Hugging Face. Then pass the Hugging Face token while logging in.

from huggingface_hub import notebook_login
notebook_login()

Initialize the tokenizer with the model.

model = AutoModelForCausalLM.from_pretrained("google/gemma-7b")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-7b", padding=True, truncation=True, max_length=512)

Create a text generation pipeline.

pipe = pipeline(
 "text-generation", 
 model=model, 
 tokenizer=tokenizer,
 return_tensors='pt',
 max_length=512,
 max_new_tokens=512,
 model_kwargs={"torch_dtype": torch.bfloat16},
 device="cuda"
)

Initialize the LLM with pipeline and model kwargs.

llm = HuggingFacePipeline(
 pipeline=pipe,
 model_kwargs={"temperature": 0.7, "max_length": 512},
)

Now it is time to use the vector store and the LLM for question-answering retrieval.

qa = RetrievalQA.from_chain_type(
 llm=llm,
 chain_type="stuff",
 retriever=db.as_retriever()
)

Querying the RAG Pipeline

The RAG pipeline is ready; let’s pass the queries and see how it performs.

qa.invoke("Write an educational story for young children.")

The result is:

Once upon a time, in a cozy little village nestled between rolling hills and green meadows, there lived a curious kitten named Whiskers. Whiskers loved to explore every nook and cranny of the village, from the bustling marketplace to the quiet corners where flowers bloomed. One sunny morning, as Whiskers trotted down the cobblestone path, he spotted something shimmering in the distance. With his whiskers twitching in excitement, he scampered towards it, his little paws pitter-pattering on the ground. To his delight, he found a shiny object peeking out from beneath a bush - a beautiful, colorful kite! With a twinkle in his eye, Whiskers decided to take the kite on an adventure. He tugged at the string, and the kite soared into the sky, dancing gracefully with the gentle breeze. Whiskers giggled with joy as he watched the kite soar higher and higher, painting the sky with its vibrant colors.

Final Words

The Gemma 7B model performed very well. We got to read a beautiful story about a kitten. The new SOTA model was interesting and exciting to use. With the help of the FAISS vector store, we were able to build a RAG pipeline. Thanks for reading!

About Superteams.ai

Superteams.ai connects top AI talent with companies seeking accelerated product and content development. These “Superteamers” offer individual or team-based solutions for projects involving cutting-edge technologies like LLMs, image synthesis, audio or voice synthesis, and other cutting-edge open-source AI solutions. With over 500 AI researchers and developers, Superteams has facilitated diverse projects like 3D e-commerce model generation, advertising creative generation, enterprise-grade RAG pipelines, geospatial applications, and more. Focusing on talent from India and the global South, Superteams offers competitive solutions for companies worldwide. To explore partnership opportunities, please write to founders@superteams.ai or visit this link.

Steps to Build RAG Application with Gemma 7B LLM was originally published in Superteams.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

As the digital landscape rapidly evolves, enterprises are facing the challenge of managing and harnessing the exponential growth of data to drive business success. With the expansion of volume and complexity of content, traditional content management approaches are failing to provide the agility and intelligence required to scale or extract valuable insights.

The integration of vector databases and the mixture of experts, such as Mixtral 8x7B LLM, offers a transformative solution for enterprises seeking to unlock the full potential of their content pipelines. In this blog post, we will explore the essential components and strategies for building an advanced AI-powered enterprise content pipeline using Mixtral 8x7B and Qdrant — an advanced vector database following the HNSW algorithm for approximate nearest neighbor search.

To build the advanced AI-powered pipeline, we’ll leverage a Retrieval Augmented Generation (RAG) pipeline by following these steps:

1. Loading the Dataset using LlamaIndex
2. Embedding Generation using Hugging Face
3. Building the Model using Mixtral 8x7B
4. Storing the Embedding in the Vector Store
5. Building a Retrieval pipeline
6. Querying the Retriever Query Engine

Let’s get started!

Enterprise Content Generation with Mixtral 8x7B

To build a RAG pipeline with Mixtral 8x7B, we’ll install the following dependencies:

%pip install -q llama-index==0.9.3 qdrant-client transformers[torch]

Loading the Dataset Using LlamaIndex

For the dataset, we have used Diffbot Knowledge Graph API. Diffbot is a sophisticated web scraping and data extraction tool that utilizes artificial intelligence to automatically retrieve and structure data from web pages. Unlike traditional web scraping methods that rely on manual programming to extract specific data elements, Diffbot uses machine learning algorithms to comprehend and interpret web content much like a human would. This allows Diffbot to accurately identify and extract various types of data, including articles, product details, and contact information, from a wide range of websites.

One of the standout features of Diffbot is its Knowledge Graph Search, which organizes the extracted data into a structured database known as a knowledge graph. A knowledge graph is a powerful representation of interconnected data that enables efficient searching, querying, and analysis. Diffbot’s Knowledge Graph Search not only extracts individual data points from web pages but also establishes relationships between them by creating a comprehensive network of information.

To get the URL, make an account on Diffbot. Go to Knowledge Graph, and Search. Here we have used Organization in the Visual, and filtered by Industries->Pharmaceutical Companies.

Then, we chose GSK, which is a renowned pharmaceutical company, and clicked Articles.

After clicking Articles, we got an option to export it as CSV or make an API call.

We made an API call and used that URL to access the data in Python.

import requests
import json
# The Diffbot API URL
url = "https://kg.diffbot.com/<your-url>"
# Make a GET request to the API
response = requests.get(url)
# Parse the response text as JSON
data = response.json()
# Open a file in write mode
with open('json/response_text.json', 'w') as file:
 # Write the data to the file as JSON
 json.dump(data, file)
print("Response text has been saved to 'json/response_text.json'.")

The data is saved now in a JSON file. Using LlamaIndex Simple Directory Reader, we will load the data from the “json” directory.

from llama_index.core import SimpleDirectoryReader
documents = SimpleDirectoryReader("/home/akriti/Notebooks/json").load_data()

Now, it’s time to split the documents into chunks using Sentence Splitter.

from llama_index.node_parser.text import SentenceSplitter
# Create a SentenceSplitter object with a specified chunk size
text_parser = SentenceSplitter(chunk_size=1024)
# Initialize empty lists to store text chunks and corresponding document indexes
text_chunks = []
doc_idxs = []
# Iterate over each document in the 'documents' list along with its index
for doc_idx, doc in enumerate(documents):
 # Split the text of the current document into smaller chunks using the SentenceSplitter
 cur_text_chunks = text_parser.split_text(doc.text)
 
 # Extend the list of text chunks with the chunks from the current document
 text_chunks.extend(cur_text_chunks)
 
 # Extend the list of document indexes with the index of the current document,
 # repeated for each corresponding text chunk
 doc_idxs.extend([doc_idx] * len(cur_text_chunks))

After that, we will create a Text Node object where we will assign the metadata of the source document to the metadata attribute of the node so that the relationships between them can be managed easily.

# Import the TextNode class from the llama_index schema module
from llama_index.schema import TextNode
# Initialize an empty list to store nodes
nodes = []
# Iterate over each index and text chunk in the text_chunks list
for idx, text_chunk in enumerate(text_chunks):
 # Create a new TextNode object with the current text chunk
 node = TextNode(text=text_chunk)
 
 # Retrieve the corresponding source document using the document index from doc_idxs
 src_doc = documents[doc_idxs[idx]]
 
 # Assign the metadata of the source document to the metadata attribute of the node
 node.metadata = src_doc.metadata
 
 # Append the node to the list of nodes
 nodes.append(node)

Embedding Generation Using Hugging Face

There are many supported embedding tool integrations with LlamaIndex; here we are moving forward with the Hugging Face Embedding tool.

# Import the HuggingFaceEmbedding class from the llama_index embeddings module
from llama_index.embeddings import HuggingFaceEmbedding
# Initialize a HuggingFaceEmbedding object with the specified model name
embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en")
# Iterate over each node in the nodes list
for node in nodes:
 # Get the content of the node along with its metadata
 content_with_metadata = node.get_content(metadata_mode="all")
 
 # Use the embedding model to get the text embedding for the node's content
 node_embedding = embed_model.get_text_embedding(content_with_metadata)
 
 # Assign the computed embedding to the embedding attribute of the node
 node.embedding = node_embedding

Building the Model Using Mixtral 8x7B

Mixtral 8x7B is a cutting-edge language model developed by Mistral AI. It is a sparse mixture of experts (MOE) model with open weights. It is designed to offer powerful AI capabilities by integrating elements from BERT, RoBERTa, and GPT-3. This model represents a significant advancement in natural language processing by providing a practical and accessible solution for various applications.

Mixtral 8x7B employs a Mixture of Experts (MoE) architecture and is a decoder-only model. In this architecture, each layer consists of 8 feedforward blocks, referred to as experts. During processing, a router network dynamically selects two experts for each token at every layer, which enables effective information processing and aggregation.

One of Mixtral 8x7B’s standout features is its exceptional performance, characterized by high-quality outputs across diverse tasks. The model is pre-trained with multilingual data using a context size of 32k tokens. It outperforms Llama 2 and GPT-3.5 on most benchmarks but, in some cases, it matches Llama 2 and the GPT-3.5 model.

Mixtral 8x7B is also available in an Instruct form, which is supervised and fine-tuned on an instruction-following dataset, and optimized through Direct Preference Optimization (DPO) training. To know more about Mixtral 8x7B in detail, visit this paper.

Using Hugging Face and LlamaIndex, we will load the model.

import torch
from llama_index.llms import HuggingFaceLLM
# Instantiate a HuggingFaceLLM object with specified parameters
llm = HuggingFaceLLM(
 context_window=4096, # Maximum context window size
 max_new_tokens=256, # Maximum number of new tokens to generate
 generate_kwargs={"temperature": 0.7, "do_sample": False}, # Generation settings
 tokenizer_name="mistralai/Mixtral-8x7B-v0.1", # Pre-trained tokenizer name
 model_name="mistralai/Mixtral-8x7B-v0.1", # Pre-trained model name
 device_map="auto", # Automatic device mapping
 stopping_ids=[50278, 50279, 50277, 1, 0], # Tokens to stop generation
 tokenizer_kwargs={"max_length": 4096}, # Tokenizer arguments
 model_kwargs={"torch_dtype": torch.float16} # Model arguments
)

After that, we will create a service context with the loaded LLM and the embedding model.

from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)

Storing the Embedding in the Vector Store

Here, we have used the Qdrant Vector Database to store the embeddings. Qdrant is a high-performance open-source vector search engine designed to efficiently index and search through large collections of high-dimensional vectors. It’s particularly well-suited for use cases involving similarity search, where the goal is to find items that are most similar to a query vector within a large dataset.

We will initiate the Qdrant Client first, and create a collection by enabling hybrid search.

import qdrant_client
from llama_index.vector_stores.qdrant import QdrantVectorStore
from llama_index.core import StorageContext
from llama_index.core import VectorStoreIndex, ServiceContext, SimpleDirectoryReader
# Initialize Qdrant client
client = qdrant_client.QdrantClient(location=":memory:")
# Create Qdrant vector store
vector_store = QdrantVectorStore(client=client, collection_name="my_collection",enable_hybrid=True)

We will add the node to the vector store and create a storage context. Also, we will create an index where we will use documents, service context, and storage context.

# Add nodes to the vector store
vector_store.add(nodes)
# Create a storage context
storage_context = StorageContext.from_defaults(vector_store=vector_store)
index = VectorStoreIndex.from_documents(documents, storage_context=storage_context, service_context=service_context)

We will then use a query and embedding model to create a query embedding, which we will use later as a reference.

query_str = "Can you update me about shingles vaccine?"
query_embedding = embed_model.get_query_embedding(query_str)

Using hybrid query mode, we will create a vector store query using LlamaIndex, where we will use the query embedding and save the query result.

from llama_index.vector_stores import VectorStoreQuery
query_mode = "hybrid"
vector_store_query = VectorStoreQuery(query_embedding=query_embedding, similarity_top_k=2, mode=query_mode)
query_result = vector_store.query(vector_store_query)

Then, we will parse the query result into the set of nodes.

from llama_index.schema import NodeWithScore
from typing import Optional
nodes_with_scores = []
for index, node in enumerate(query_result.nodes):
 score: Optional[float] = None
 if query_result.similarities is not None and index < len(query_result.similarities):
 score = query_result.similarities[index]
 nodes_with_scores.append(NodeWithScore(node=node, score=score))

Building a Retrieval Pipeline

For building a retrieval pipeline, we’ll use the above to create a retriever class.

from llama_index import QueryBundle
from llama_index.retrievers import BaseRetriever
from typing import Any, List, Optional
from llama_index.vector_stores import VectorStoreQuery
from llama_index.schema import NodeWithScore
class VectorDBRetriever(BaseRetriever):
 """Retriever over a qdrant vector store."""
 def __init__(self,
 vector_store: QdrantVectorStore,
 embed_model: Any,
 query_mode: str = "hybrid",
 similarity_top_k: int = 2) -> None:
 """Initialize parameters."""
 self._vector_store = vector_store
 self._embed_model = embed_model
 self._query_mode = query_mode
 self._similarity_top_k = similarity_top_k
 super().__init__()
 
 def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
 """Retrieve."""
 query_embedding = self._embed_model.get_query_embedding(query_bundle.query_str)
 vector_store_query = VectorStoreQuery(
 query_embedding=query_embedding,
 similarity_top_k=self._similarity_top_k,
 mode=self._query_mode,
 )
 query_result = self._vector_store.query(vector_store_query)
 nodes_with_scores = []
 for index, node in enumerate(query_result.nodes):
 score: Optional[float] = None
 if query_result.similarities is not None and index < len(query_result.similarities):
 score = query_result.similarities[index]
 nodes_with_scores.append(NodeWithScore(node=node, score=score))
 return nodes_with_scores

retriever = VectorDBRetriever(
 vector_store, embed_model, query_mode="hybrid", similarity_top_k=2
)

Finally, our query engine will be ready with the help of Retriever Query Engine.

from llama_index.query_engine import RetrieverQueryEngine
query_engine = RetrieverQueryEngine.from_args(
 retriever, service_context=service_context
)from llama_index.query_engine import RetrieverQueryEngine
query_engine = RetrieverQueryEngine.from_args(
 retriever, service_context=service_context
)

Querying the Retriever Query Engine

As our query engine is ready, now is the time to pass some queries and see some results.

Question 1:

query_str = "Write a paragraph about GSK announcement about its shares."
response = query_engine.query(query_str)
print(str(response))

The response will be:

GSK plc announced the completion of its share consolidation on July 18, 2022. This followed the demerger of the Consumer Healthcare business from the GSK Group to form Haleon. The consolidation of GSK shares became effective at 8.00 a.m. on July 19, 2022. As part of the consolidation, a ratio of 4 new ordinary shares was applied for every 5 existing ordinary shares. Fractional entitlements that arose from the consolidation were aggregated and sold in the open market, with the net proceeds paid to each relevant shareholder according to their entitlement. Following the issuance and consolidation, the total number of voting rights in GSK as of July 19, 2022, was 4,067,352,076.

Question 2:

query_str = "Write a paragraph about GSK's RSV vaccine."
response = query_engine.query(query_str)
print(str(response))

The response will be:

GSK's Arexvy is the world's first respiratory syncytial virus (RSV) vaccine for older adults. The US Food and Drug Administration (FDA) approved Arexvy for the prevention of lower respiratory tract disease (LRTD) caused by RSV in individuals 60 years of age and older. This groundbreaking approval enables adults aged 60 years and older to be protected from RSV disease for the first time. The approval is based on data from the positive pivotal AReSVi-006 phase III trial that showed exceptional efficacy in older adults, including those with underlying medical conditions, and in those with severe RSV disease. The US launch was planned before the 2023/24 RSV season. RSV is a common, contagious virus that can lead to potentially serious respiratory illness. It causes approximately 177,000 hospitalizations and an estimated 14,000 deaths in the US in adults aged 65 years and older each year.

Question 3:

query_str = "Write a paragraph about GSK's Endrometrial Cancer Drug Development."
response = query_engine.query(query_str)
print(str(response))

The response will be:

GSK has made significant improvement in the development of drugs for endometrial cancer. Their drug, Jemperli (dostarlimab), has been approved by the European Commission and the US Food and Drug Administration (FDA) for the treatment of adult patients with mismatch repair-deficient (dMMR)/microsatellite instability-high (MSI-H) primary advanced or recurrent endometrial cancer. Jemperli, in combination with carboplatin and paclitaxel (chemotherapy), is the first and only frontline immuno-oncology treatment in the European Union for this type of endometrial cancer. The FDA has also granted accelerated approval for Jemperli as a monotherapy for treating adult patients with dMMR/MSI-H recurrent or advanced endometrial cancer that has progressed on or following prior treatment with a platinum-containing regimen. This approval is based on the results from the dMMR/MSI-H population of Part 1 of the RUBY/ENGOT-EN6/GOG3031/NSGO phase III trial. GSK continues to evaluate Jemperli in the hopes of further expansion for the drug as data mature.

Question 4:

query_str = "Write a paragraph about GSK's Hepatocellular Carcinoma Drug Development."
response = query_engine.query(query_str)
print(str(response))

The response will be:

GSK is making significant progress in the development of drugs for hepatocellular carcinoma (HCC). One of their drugs, Cobolimab, is currently in Phase II clinical trials for HCC. Cobolimab is a humanized monoclonal IgG4 antibody that inhibits T cell immunoglobulin mucin-3 (TIM-3), and is under development for the treatment of solid tumors including melanoma, squamous and non-squamous non-small cell lung carcinoma, HCC, and colorectal cancer. It is administered through the intravenous route. The drug's phase transition success rate (PTSR) and likelihood of approval (LoA) are being closely monitored. GSK's efforts in this area demonstrate their commitment to advancing treatments for HCC.

Question 5:

query_str = "Write a paragraph about GSK's Uncomplicated Cervical And Urethral Gonorrhea Drug Development."
response = query_engine.query(query_str)
print(str(response))

The response will be:

GSK is currently developing a potential first-in-class antibiotic, Gepotidacin, for the treatment of uncomplicated cervical and urethral gonorrhea. This drug is in Phase III of clinical development. Gepotidacin is the first in a new chemical class of antibiotics called triazaacenaphthylene bacterial topoisomerase inhibitors. It is being investigated for use in uncomplicated urinary tract infection and urogenital gonorrhea, two infections not addressed by new oral antibiotics in 20 years. The Phase III programme comprises two studies, EAGLE-1 and EAGLE-2, testing Gepotidacin in two common infections caused by bacteria identified as antibiotic-resistant threats. The development of Gepotidacin is the result of a successful public-private partnership between GSK, the US government's Biomedical Advanced Research and Development Authority (BARDA), and Defense Threat Reduction Agency (DTRA).

Conclusion

With the help of the LlamaIndex framework, we used Diffbot API to extract enterprise content that was related to a pharmaceutical company, GSK. Using Hugging Face embeddings, Qdrant Vector Store, and Mixtral 8x7B, the retrieval pipeline was built. The results obtained using the retrieval query engine were quite fascinating. Building an advanced AI-powered enterprise content pipeline has become easy with the help of Mixtral 8x7B.

Thanks for Reading!

References

https://arxiv.org/pdf/2401.04088.pdf

https://docs.diffbot.com/docs/getting-started-with-diffbot-knowledge-graph

https://docs.llamaindex.ai/en/stable/examples/low_level/oss_ingestion_retrieval.html

About Superteams.ai

Superteams.ai connects top AI talent with companies seeking accelerated product and content development. These “Superteamers” offer individual or team-based solutions for projects involving cutting-edge technologies like LLMs, image synthesis, audio or voice synthesis, and other cutting-edge open-source AI solutions. With over 500 AI researchers and developers, Superteams has facilitated diverse projects like 3D e-commerce model generation, advertising creative generation, enterprise-grade RAG pipelines, geospatial applications, and more. Focusing on talent from India and the global South, Superteams offers competitive solutions for companies worldwide. To explore partnership opportunities, please write to founders@superteams.ai or visit this link.

How to Build an Advanced AI-Powered Enterprise Content Pipeline Using Mixtral 8x7B and Qdrant was originally published in Superteams.ai on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

As we step into the Chinese New Year — the Year of the Dragon — in 2024, I thought: why not build a Chinese News AI using DeepSeek and Qdrant? Especially so because, though LLMs are becoming vast in size and complexity, there are still a range of challenges in building an accurate and efficacious language AI beyond the English Language. In this context, DeepSeek LLM has embarked on a long-term project to overcome the inaccuracies in language application. DeepSeek LLM excels in the Chinese language, and here we’ll see how it performs while fetching news from a Chinese news dataset. We’ll utilize LlamaIndex, FastEmbed by Qdrant, and Qdrant Vector Store to develop an application where we can understand Chinese news with a robust RAG.

Let’s dive deeper!

DeepSeek LLM: An Open-Source Language Model with Longtermism

DeepSeek LLM is an advanced language model that has developed different models including Base and Chat. It has been trained from scratch on a dataset of 2 trillion tokens in both English and Chinese. In terms of size, there are two varieties of DeepSeek LLM models: one comprises 7 billion parameters and the other, 67 billion parameters.

The 7B model uses Multi-Head Attention, while the 67B model uses Grouped-Query Attention. These variants operate on the same architecture as the Llama 2 model, which is an autoregressive transformer decoder model.

DeepSeek LLM is a project dedicated to advancing open-source large language models in the long-term perspective. However, DeepSeek LLM 67B outperforms Llama 2 70B in various domains such as reasoning, mathematics, coding, and comprehension. Compared to other models, including GPT 3.5, DeepSeek LLM excels in Chinese language proficiency.

The alignment pipeline of DeepSeek LLM consists of two stages:

1. Supervised Fine-Tuning: The 7B model is fine-tuned for 4 epochs, while the 67B model is fine-tuned for 2 epochs. During supervised fine-tuning, the learning rate for the 7B model is 1e-5, while for the 67B model it is 5e-6. The repetition ratio of the model tends to increase when the quantity of math SFT data increases, although the math SFT data includes similar patterns in reasoning.
2. Direct Preference Optimization: To address the repetition problem, the model’s ability was enhanced using DPO training, which proved to be an effective method for LLM alignment. The preference data for DPO training is constructed in terms of helpfulness and harmlessness.

The model, along with all its variants, is available on Hugging Face. To learn more about DeepSeek LLM, visit their paper and Github repository.

Qdrant: A High-Performance Vector Database

Qdrant is an open-source vector database and vector similarity search engine written in Rust, engineered to empower the next generation of AI applications with advanced and high-performing vector similarity search technology. Its key features include multilingual support, which enables versatility across various data types, and filters for a wide array of applications.

Qdrant boasts speed and accuracy through a custom modification of the HNSW algorithm for Approximate Nearest Neighbor Search, which ensures state-of-the-art search capabilities by maintaining precise results. Moreover, it supports additional payload associated with vectors by allowing filterable results based on payload values. With a rich array of supported data types and query conditions, including string matching, numerical ranges, and geo-locations, Qdrant offers versatility in data management.

As a cloud-native and horizontally scalable platform, Qdrant efficiently handles data scaling needs by effectively utilizing computational resources through dynamic query planning and payload data indexing. Its applications include semantic text search, recommendations, user behavior analysis, and more, which offers a production-ready service with a convenient API for storing, searching, and managing vectors along with additional payload. For further details, the Qdrant Documentation provides a comprehensive guide on installation, usage, tutorials, and examples.

Utilizing FastEmbed for Lightweight Embedding Generation

FastEmbed is a lightweight, fast, and accurate Python library built specifically for embedding generation, with maintenance overseen by Qdrant. It achieves efficiency and speed through the utilization of quantized model weights and ONNX Runtime, which sidesteps the necessity for a PyTorch dependency.

FastEmbed supports data parallelism for encoding vast datasets efficiently and is engineered with a CPU-centric approach. Moreover, it excels in accuracy and recall metrics compared to OpenAI Ada-002 by boasting the Flag Embedding as its default model, which leads the MTEB leaderboard. It also supports Jina Embedding and Text Embedding. There are several popular text models supported by FastEmbed. To learn more about the supported models, visit here.

LlamaIndex Framework for Robust RAG

LlamaIndex is a robust framework ideally suited for constructing Retrieval-Augmented Generation (RAG) applications. It facilitates the decoupling of chunks used for retrieval and synthesis, which is a crucial feature because the optimal representation for retrieval may differ from that for synthesis. As document volumes expand, LlamaIndex supports structured retrieval by ensuring more precise outcomes, particularly when a query is only relevant to a subset of documents.

Moreover, LlamaIndex prioritizes optimized performance by offering an array of strategies to enhance the RAG pipeline efficiently. It aims to elevate retrieval and generation accuracy across complex datasets by mitigating hallucinations. LlamaIndex supports many embedding models as well as integration with large language models. Additionally, it seamlessly integrates with established technological platforms like LangChain, Flask, and Docker, which offers customization options such as seeding tree construction with custom summary prompts.

Understanding Chinese News with DeepSeek

Since DeepSeek LLM excels in Chinese language proficiency, let’s build a Chinese News AI using Retrieval Augmented Generation.

To get started, let’s install all the dependencies.

%pip install -q llama-index transformers datasets
%pip install -q llama-cpp-python
%pip install -q qdrant-client
%pip install -q llama_hub
%pip install -q fastembed

Here, I have used this dataset; it is a multilingual news dataset. I have picked the Chinese language. Load the dataset and save it to your directory. We’ll be using LlamaIndex to read the data with SimpleDirectoryReader.

from datasets import load_dataset
dataset = load_dataset("intfloat/multilingual_cc_news", languages=["zh"], split="train")
dataset.save_to_disk("Notebooks/dataset")

Now, using LlamaIndex, load the data from the directory where we saved our dataset.

from llama_index import SimpleDirectoryReader
documents = SimpleDirectoryReader("Notebooks/dataset").load_data()

After that, split the documents into small chunks using SentenceSplitter. Here, we need to maintain the relationship between the document and the source document index so that it helps in injecting the document metadata.

from llama_index.node_parser.text import SentenceSplitter
text_parser = SentenceSplitter(chunk_size=1024,)
text_chunks = []
doc_idxs = []
for doc_idx, doc in enumerate(documents):
 cur_text_chunks = text_parser.split_text(doc.text)
 text_chunks.extend(cur_text_chunks)
 doc_idxs.extend([doc_idx] * len(cur_text_chunks))

Then, we’ll construct nodes from text chunks manually.

from llama_index.schema import TextNode
nodes = []
for idx, text_chunk in enumerate(text_chunks):
 node = TextNode(text=text_chunk,)
 src_doc = documents[doc_idxs[idx]]
 node.metadata = src_doc.metadata
 nodes.append(node)

For each node, we’ll generate embeddings using the FastEmbed Embedding model.

from llama_index.embeddings import FastEmbedEmbedding
embed_model = FastEmbedEmbedding(model_name="BAAI/bge-small-en-v1.5")
for node in nodes:
 node_embedding = embed_model.get_text_embedding(node.get_content(metadata_mode="all"))
 node.embedding = node_embedding

Now, it’s time to load the DeepSeek LLM using HuggingFaceLLM from LlamaIndex. Here, I used the chat model.

from llama_index.llms import HuggingFaceLLM
llm = HuggingFaceLLM(
 context_window=4096,
 max_new_tokens=256,
 generate_kwargs={"temperature": 0.7, "do_sample": False},
 tokenizer_name="deepseek-ai/deepseek-llm-7b-chat",
 model_name="deepseek-ai/deepseek-llm-7b-chat",
 device_map="auto",
 stopping_ids=[50278, 50279, 50277, 1, 0],
 tokenizer_kwargs={"max_length": 4096},
 model_kwargs={"torch_dtype": torch.float16}
)

Then, we’ll define the ServiceContext, which consists of the embedding model and the large language model.

from llama_index import ServiceContext
service_context = ServiceContext.from_defaults(llm=llm, embed_model=embed_model)

After that, we’ll create a vector store collection using the Qdrant vector database and create a storage context using the vector store collection.

import qdrant_client
from llama_index.vector_stores.qdrant import QdrantVectorStore
client = qdrant_client.QdrantClient(location=":memory:")
from llama_index.storage.storage_context import StorageContext
from llama_index import (VectorStoreIndex,
 ServiceContext,
 SimpleDirectoryReader,)
vector_store = QdrantVectorStore(client=client, collection_name="my_collection")
vector_store.add(nodes)
storage_context = StorageContext.from_defaults(vector_store=vector_store)

We’ll pass the documents, storage context, and service context into the VectorStoreIndex.

index = VectorStoreIndex.from_documents(documents, storage_context=storage_context, service_context=service_context)

We’ll generate a query embedding using a query string to build a retrieval pipeline.

query_str = "Can you give me news around IPhone?"
query_embedding = embed_model.get_query_embedding(query_str)

Then, we’ll construct a Vector Store query and query the vector database.

from llama_index.vector_stores import VectorStoreQuery
query_mode = "default"
vector_store_query = VectorStoreQuery(query_embedding=query_embedding, similarity_top_k=2, mode=query_mode)
query_result = vector_store.query(vector_store_query)
print(query_result.nodes[0].get_content())

The following will be the result:

8%，僅排名第五，位居華為、OPPO和Vivo等本土手機廠商之後 - - 這三家中國手機廠商的市場份額加起來達到47%。
普遍對iPhone 8更期待
除此之外，即將發布的iPhone 7恐怕會成為大獲成功的iPhone 6的"犧牲品"。去年第一季度，得益於iPhone 6銷量激增，蘋果在中國的營收增長了74%。一年後，由於iPhone 6S銷量疲軟，蘋果在全球的iPhone銷量首次出現下滑，而公司營收更是出現13年來的首次滑坡，盡管根據市場研究機構Strategy Analytics的數據，iPhone 6S是今年第二季度全球最暢銷的智能手機。
到目前為止，新浪微博網友對iPhone 7發布的討論，已經超過了去年iPhone 6S發布前的熱度。部分中國用戶甚至已經開始盤算著購買有望於明年發布的iPhone8。鑒於2017年是iPhone上市十周年，外界預計iPhone 8將作出更大的升級。
由於投資者擔心iPhone銷量已過巔峰，蘋果股價在今年始終承受著壓力。盡管今年以來蘋果股價累計上漲了2.35%，但仍然落後於標准普爾500指數的平均漲幅。
市場研究機構Stratechery科技行業分析師本·湯普森（Ben Thompson）說："相比2014年，今天最大的變化就是iPhone已無處不在。當人們第一次獲得購買iPhone的機會時，它還有巨大的增長空間，但如今那種潛力已經得到充分挖掘。"◎ 陳國雄
美國在1978年立法制定國際銀行法(International Banking Act of 1978)，該法將外國銀行業納入與國內銀行相同準則。在此之前，外國銀行設立係依據州法沒有一致性。
1978年制定國際銀行法後，外國銀行設立採雙規制(Dual System)，可向聯邦銀行管理機構OCC (Office of the Comptroller of the Currency)或州銀行 (State Banking Department)當局申請，如果向州申請設立毋須經過聯邦銀行同意。到了1991年外國銀行在美迅速成長，大約有280家外國銀行，資產值達6,260億美元，佔美國銀行總資產18%，大部份是依州法設立。

The response is:

‘At 8 percent, it ranked fifth behind local handset makers Huawei, OPPO, and Vivo — the three Chinese handset makers with a combined market share of 47 percent.

Widespread anticipation for iPhone 8

On top of that, the upcoming iPhone 7 is feared to be a “casualty” of the hugely successful iPhone 6. In the first quarter of last year, Apple’s China revenue grew 74 percent, thanks to a surge in iPhone 6 sales. A year later, Apple’s global iPhone sales fell for the first time due to weak iPhone 6S sales, and the company’s revenue slipped for the first time in 13 years, even though the iPhone 6S was the world’s best-selling smartphone in the second quarter of this year, according to market researcher Strategy Analytics.

So far, discussions on Sina Weibo about the release of the iPhone 7 have surpassed the buzz surrounding the release of the iPhone 6S last year. Some Chinese users have even begun planning to buy the iPhone 8, which is expected to be released next year, and is expected to get even bigger upgrades given that 2017 marks the 10th anniversary of the iPhone’s launch.

As investors are worried that iPhone sales have peaked, Apple shares have been under pressure this year. Although Apple’s stock price has risen 2.35% since the beginning of the year, it still lags behind the average rate of increase of the Standard & Poor’s 500 Index.

Market research organization Stratechery technology industry analyst Ben Thompson (Ben Thompson) said: “Compared to 2014, the biggest change today is that the iPhone is everywhere. When people first got the chance to buy an iPhone, there was huge room for growth, but today that potential has been fully realized.” ◎ Chen Guoxiong

The U.S. legislated the International Banking Act of 1978, which brought the foreign banking industry under the same criteria as domestic banks. Prior to that, foreign banks were established under state law with no consistency.

After the enactment of the International Banking Act of 1978, the establishment of foreign banks is based on a Dual System, which can be applied to the OCC (Office of the Comptroller of the Currency), the federal banking agency, or the State Banking Department (State Banking Department), and does not require the consent of the federal banking agency. The federal bank’s consent is not required if the application is made to a state. By 1991, foreign banks were growing rapidly in the U.S. There were about 280 foreign banks with assets of US$626 billion, accounting for 18% of total U.S. banking assets, most of which were established under state law.’

Then we’ll parse the results into a set of nodes.

from llama_index.schema import NodeWithScore
from typing import Optional
nodes_with_scores = []
for index, node in enumerate(query_result.nodes):
 score: Optional[float] = None
 if query_result.similarities is not None:
 score = query_result.similarities[index]
 nodes_with_scores.append(NodeWithScore(node=node, score=score))

Now, using the above, we’ll create a retriever class.

from llama_index import QueryBundle
from llama_index.retrievers import BaseRetriever
from typing import Any, List
class VectorDBRetriever(BaseRetriever):
 """Retriever over a qdrant vector store."""
 def __init__(self,
 vector_store: QdrantVectorStore,
 embed_model: Any,
 query_mode: str = "default",
 similarity_top_k: int = 2) -> None:
 """Init params."""
 self._vector_store = vector_store
 self._embed_model = embed_model
 self._query_mode = query_mode
 self._similarity_top_k = similarity_top_k
 super().__init__()
 def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
 """Retrieve."""
 query_embedding = embed_model.get_query_embedding(
 query_bundle.query_str
 )
 vector_store_query = VectorStoreQuery(
 query_embedding=query_embedding,
 similarity_top_k=self._similarity_top_k,
 mode=self._query_mode,
 )
 query_result = vector_store.query(vector_store_query)
 nodes_with_scores = []
 for index, node in enumerate(query_result.nodes):
 score: Optional[float] = None
 if query_result.similarities is not None:
 score = query_result.similarities[index]
 nodes_with_scores.append(NodeWithScore(node=node, score=score))
 return nodes_with_scores
retriever = VectorDBRetriever(
 vector_store, embed_model, query_mode="default", similarity_top_k=2
)

Then, create a retriever query engine.

from llama_index.query_engine import RetrieverQueryEngine
query_engine = RetrieverQueryEngine.from_args(
 retriever, service_context=service_context
)

Finally, our retriever is ready to query and chat with. Let’s pass a query.

The query is: “Tell me about South China Sea Issue.”

query_str = "告诉我南海问题"
response = query_engine.query(query_str)
print(str(response))

Following will be the response:

南海问题是指涉及南海地区多个国家的主权和海洋权益争议的问题。该地区包括南海诸岛及其附近海域，涉及中国、菲律宾、越南、马来西亚、文莱和台湾等国家和地区。南海地区拥有丰富的油气资源，因此争议各方在该地区的领土和资源开发上存在分歧。中国主张对南海诸岛及其附近海域拥有主权，并提出"九段线"主张，而其他国家则对此持有不同看法。南海问题涉及复杂的政治、经济和安全利益，是地区和国际社会关注的焦点之一。

The response is:

‘South China Sea issues refer to issues involving disputes over the sovereignty and maritime rights and interests of multiple countries in the South China Sea region. This area includes the South China Sea Islands and their adjacent waters, involving countries and regions such as China, the Philippines, Vietnam, Malaysia, Brunei and Taiwan. The South China Sea is rich in oil and gas resources, so the parties to the dispute have differences over the territory and resource development in the area. China claims sovereignty over the South China Sea islands and adjacent waters and has proposed a “nine-dash line” claim, while other countries hold different views on this. The South China Sea issue involves complex political, economic, and security interests and is one of the focuses of attention of the regional and international communities.’

Conclusion

DeepSeek LLM has performed very well in answering questions without facing challenges. Its architecture sets it apart from other models, and the most impressive aspect is its utilization of Direct Preference Optimization to enhance the model’s capabilities. It is a fine-tuned and optimized model in both Chinese and English languages, and we observed from the results how well such a fine-tuned and optimized model can perform.

We utilized FastEmbed and Qdrant for embedding generation and vector similarity search. Retrieval was fast using Qdrant. One of Qdrant’s most impressive features is its accessibility through Docker installation, on the cloud, and its in-memory capabilities. Qdrant is versatile in storing vector embeddings. It was intriguing for me to experiment with these tools.

Thanks for reading!

References

https://arxiv.org/pdf/2401.02954.pdf