Building a Universal Generative AI Chatbot Solution: From Data Ingestion to Deployment

Photo by Olav Ahrens Røtne on Unsplash

Hello everyone! I’m Gaurav Goswami, a Sr. Full Stack & AI Engineer at IBM Bangalore. Today, I’m excited to share with you the journey of developing a universal generative AI chatbot solution. This solution is designed to process, clean, vectorize data, train models, create agents, and deploy chatbots efficiently. Along with this, I’ll delve into the concepts of Generative AI and Agentic AI, explaining their importance and providing real-life examples.

What is Generative AI?

Generative AI refers to artificial intelligence systems that can generate new content based on existing data. Unlike traditional AI focused on classification and prediction, Generative AI creates novel and unique outputs. This technology utilizes models like Generative Adversarial Networks (GANs), Variational Autoencoders (VAEs), and Transformer-based models to produce realistic and creative outputs.

Real-Life Examples

1. Text Generation: Models like GPT-4 generate coherent text for various applications, such as drafting emails or creating stories.

2. Image Synthesis: GANs create realistic images, like those seen on websites such as This Person Does Not Exist.

3. Music Composition: AI models like MuseNet compose original music across genres.

Why Generative AI?

Generative AI revolutionizes content creation and personalization, enhances user experiences, supports scientific research, and pushes creative boundaries.

What is Agentic AI?

Agentic AI involves systems that exhibit autonomous decision-making capabilities, performing tasks and interacting with users or other systems in a manner that mimics human agency. These systems operate independently, adapt to different contexts, and provide meaningful interactions.

Real-Life Examples

1. Virtual Personal Assistants: Google Assistant and Amazon Alexa perform tasks based on user commands and preferences.

2. Autonomous Vehicles: Self-driving cars navigate and make driving decisions without human input.

3. Customer Service Bots: AI chatbots handle customer inquiries and support autonomously.

Why Agentic AI?

Agentic AI improves efficiency, enhances user interactions, ensures safety in critical applications, and drives innovation and scalability across various sectors.

A Universal Solution for Generative AI Chatbots

Our goal was to build a robust, all-in-one solution for businesses to deploy advanced generative AI chatbots with minimal effort. This solution automates the entire pipeline, from data ingestion to chatbot deployment, utilizing cutting-edge technologies like Generative AI (GenAI), Retrieval-Augmented Generation (RAG), and Agentic AI.

Here’s a high-level overview of the process:

1. Data Ingestion: Collect and structure data from various sources.

2. Data Cleaning: Process and refine the data.

3. Data Vectorization: Convert data into vectors.

4. Model Training: Train a Large Language Model (LLM) using the vectors.

5. Agent Creation: Develop intelligent agents based on the trained model.

6. Deployment: Deploy the chatbot for real-world interactions.

1. Data Ingestion: Handling Various File Types

Text Documents (.txt)

import os

def load_text_files(directory_path):
    texts = []
    for filename in os.listdir(directory_path):
        if filename.endswith(".txt"):
            with open(os.path.join(directory_path, filename), 'r', encoding='utf-8') as file:
                texts.append(file.read())
    return texts

# Load text files from the 'data_dir' directory
text_data = load_text_files('data_dir')
print(text_data[:2])  # Print the first two documents for inspection

PDFs (.pdf)

import fitz  # PyMuPDF

def load_pdf_files(directory_path):
    texts = []
    for filename in os.listdir(directory_path):
        if filename.endswith(".pdf"):
            pdf_document = fitz.open(os.path.join(directory_path, filename))
            text = ""
            for page in pdf_document:
                text += page.get_text()
            texts.append(text)
    return texts

# Load PDF files from the 'data_dir' directory
pdf_data = load_pdf_files('data_dir')
print(pdf_data[:1])  # Print the content of the first PDF for inspection

Presentations (.pptx)

from pptx import Presentation

def load_pptx_files(directory_path):
    texts = []
    for filename in os.listdir(directory_path):
        if filename.endswith(".pptx"):
            presentation = Presentation(os.path.join(directory_path, filename))
            text = ""
            for slide in presentation.slides:
                for shape in slide.shapes:
                    if hasattr(shape, "text"):
                        text += shape.text + "\n"
            texts.append(text)
    return texts

# Load PowerPoint files from the 'data_dir' directory
pptx_data = load_pptx_files('data_dir')
print(pptx_data[:1])  # Print the content of the first presentation for inspection

Markdown Files (.md)

def load_markdown_files(directory_path):
    texts = []
    for filename in os.listdir(directory_path):
        if filename.endswith(".md"):
            with open(os.path.join(directory_path, filename), 'r', encoding='utf-8') as file:
                texts.append(file.read())
    return texts

# Load markdown files from the 'data_dir' directory
markdown_data = load_markdown_files('data_dir')
print(markdown_data[:1])  # Print the content of the first markdown file for inspection

Spreadsheets (.xls, .xlsx)

import pandas as pd

def load_spreadsheet_files(directory_path):
    data_frames = []
    for filename in os.listdir(directory_path):
        if filename.endswith(".xls") or filename.endswith(".xlsx"):
            df = pd.read_excel(os.path.join(directory_path, filename))
            data_frames.append(df)
    return pd.concat(data_frames, ignore_index=True)

# Load spreadsheet files from the 'data_dir' directory
spreadsheet_data = load_spreadsheet_files('data_dir')
print(spreadsheet_data.head())  # Print the first few rows of the concatenated dataframe

2. Data Cleaning

After ingesting the data, the next step is cleaning it. This involves removing duplicates, handling missing values, and other preprocessing tasks.

import os
import re
import fitz  # PyMuPDF
from pptx import Presentation
import docx
import pandas as pd
from bs4 import BeautifulSoup
from markdown import markdown
from sentence_transformers import SentenceTransformer
from textblob import TextBlob

def load_text_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        return file.read()

def load_pdf_file(file_path):
    text = ""
    pdf_document = fitz.open(file_path)
    for page in pdf_document:
        text += page.get_text()
    return text

def load_pptx_file(file_path):
    text = ""
    presentation = Presentation(file_path)
    for slide in presentation.slides:
        for shape in slide.shapes:
            if hasattr(shape, "text"):
                text += shape.text + "\n"
    return text

def load_docx_file(file_path):
    doc = docx.Document(file_path)
    text = "\n".join([para.text for para in doc.paragraphs])
    return text

def load_markdown_file(file_path):
    with open(file_path, 'r', encoding='utf-8') as file:
        html = markdown(file.read())
    text = ''.join(BeautifulSoup(html, "html.parser").stripped_strings)
    return text

def load_spreadsheet_file(file_path):
    df = pd.read_excel(file_path)
    return df.to_string(index=False)

def clean_text(text):
    # Remove URLs
    text = re.sub(r'http\S+', '', text)
    # Remove special characters and numbers
    text = re.sub(r'[^A-Za-z\s]+', ' ', text)
    # Remove extra whitespace
    text = re.sub(r'\s+', ' ', text).strip()
    # Convert to lowercase
    text = text.lower()
    # Correct spelling
    text = str(TextBlob(text).correct())
    return text

def load_and_clean_data(directory_path):
    cleaned_texts = []
    for filename in os.listdir(directory_path):
        file_path = os.path.join(directory_path, filename)
        if filename.endswith(".txt"):
            text = load_text_file(file_path)
        elif filename.endswith(".pdf"):
            text = load_pdf_file(file_path)
        elif filename.endswith(".pptx") or filename.endswith(".ppt"):
            text = load_pptx_file(file_path)
        elif filename.endswith(".docx") or filename.endswith(".doc"):
            text = load_docx_file(file_path)
        elif filename.endswith(".md"):
            text = load_markdown_file(file_path)
        elif filename.endswith(".xls") or filename.endswith(".xlsx"):
            text = load_spreadsheet_file(file_path)
        else:
            continue
        cleaned_text = clean_text(text)
        cleaned_texts.append(cleaned_text)
    return cleaned_texts

# Load and clean all files from the 'data_dir' directory
cleaned_data = load_and_clean_data('data_dir')
for i, text in enumerate(cleaned_data[:2]):
    print(f"Cleaned text {i+1}:")
    print(text[:500])  # Print the first 500 characters of the cleaned text
    print("\n")

3. Data Vectorization

For the vectorization part, here are three widely-used transformers that are commonly employed for text embedding, along with explanations of their differences and use cases:

• BERT (Bidirectional Encoder Representations from Transformers)
• RoBERTa (Robustly Optimized BERT Pretraining Approach)
• Sentence-BERT (SBERT)

3.1. BERT (Bidirectional Encoder Representations from Transformers)

Why Use BERT?

BERT is designed to understand the context of a word in search queries, making it highly effective for tasks like question answering, sentiment analysis, and named entity recognition. It achieves this by considering the bidirectional context, i.e., it looks at both the left and right context of a word simultaneously.

Differences and Features:

• Bidirectional Context: Unlike traditional models that read text either from left to right or right to left, BERT reads the entire sequence of words at once, considering the context from both directions.
• Pre-training Tasks: BERT uses masked language modeling (MLM) and next sentence prediction (NSP) during pre-training, enabling it to learn deep bidirectional representations.

from transformers import BertTokenizer, BertModel
import torch

def vectorize_text_bert(texts):
    tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
    model = BertModel.from_pretrained('bert-base-uncased')
    
    inputs = tokenizer(texts, return_tensors='pt', padding=True, truncation=True, max_length=512)
    with torch.no_grad():
        outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)
    
    return embeddings

# Example usage
texts = ["This is a sample text.", "Another example sentence."]
bert_vectors = vectorize_text_bert(texts)
print(bert_vectors.shape)

3.2. RoBERTa (Robustly Optimized BERT Pretraining Approach)

Why Use RoBERTa?

RoBERTa is an improved version of BERT, optimized through changes in the pre-training process. It achieves better performance by training on more data, using longer sequences, and removing the next sentence prediction objective.

Differences and Features:

• More Data and Longer Training: RoBERTa is pre-trained on a larger corpus with longer sequences, enhancing its understanding and generalization.
• No NSP Task: RoBERTa eliminates the next sentence prediction task, focusing solely on the masked language modeling objective.

from transformers import RobertaTokenizer, RobertaModel

def vectorize_text_roberta(texts):
    tokenizer = RobertaTokenizer.from_pretrained('roberta-base')
    model = RobertaModel.from_pretrained('roberta-base')
    
    inputs = tokenizer(texts, return_tensors='pt', padding=True, truncation=True, max_length=512)
    with torch.no_grad():
        outputs = model(**inputs)
    embeddings = outputs.last_hidden_state.mean(dim=1)
    
    return embeddings

# Example usage
texts = ["This is a sample text.", "Another example sentence."]
roberta_vectors = vectorize_text_roberta(texts)
print(roberta_vectors.shape)

3.3. Sentence-BERT (SBERT)

Why Use Sentence-BERT?

Sentence-BERT is specifically designed for generating sentence embeddings. It modifies the BERT architecture to produce semantically meaningful sentence embeddings that can be compared using cosine similarity, making it ideal for tasks like semantic search, clustering, and paraphrase identification.

Differences and Features:

• Siamese Network: SBERT uses a siamese network structure to derive fixed-size embeddings for sentences, enabling efficient comparison.
• Fine-Tuning: SBERT is fine-tuned on a dataset of sentence pairs, optimizing it for tasks that require sentence-level semantic understanding.

from sentence_transformers import SentenceTransformer

def vectorize_text_sbert(texts):
    model = SentenceTransformer('all-MiniLM-L6-v2')
    embeddings = model.encode(texts)
    
    return embeddings

# Example usage
texts = ["This is a sample text.", "Another example sentence."]
sbert_vectors = vectorize_text_sbert(texts)
print(sbert_vectors.shape)

These transformers differ in their architecture, training data, and objectives, making each of them suitable for specific tasks. BERT and RoBERTa are general-purpose transformers for understanding context, while SBERT is specialized for producing meaningful sentence embeddings.

By understanding these differences, you can choose the appropriate transformer model for your specific use case, ensuring optimal performance and accuracy in your applications.

4. Model Training

When it comes to model training with transformers, several models and methods are highly popular and widely used due to their effectiveness and versatility. Here are three prominent transformers and their respective training methods:

1. GPT-3 (Generative Pre-trained Transformer 3)
2. T5 (Text-to-Text Transfer Transformer)
3. BART (Bidirectional and Auto-Regressive Transformers)

4.1 GPT-3 (Generative Pre-trained Transformer 3)

Get Gaurav Goswami’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Why Use GPT-3?

GPT-3, developed by OpenAI, is known for its ability to generate human-like text and perform a wide range of NLP tasks with few-shot or zero-shot learning. It’s powerful due to its large-scale architecture and extensive training data.

Training Method:

GPT-3 is trained using the autoregressive method, where the model predicts the next token in a sequence given the previous tokens. It is pre-trained on a diverse and extensive dataset to learn language representations.

Example Code:

Training GPT-3 from scratch is resource-intensive and typically requires significant computational power. However, fine-tuning a smaller GPT model can be done using the transformers library:

from transformers import GPT2Tokenizer, GPT2LMHeadModel, Trainer, TrainingArguments

def fine_tune_gpt2(train_dataset):
    tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
    model = GPT2LMHeadModel.from_pretrained('gpt2')

    training_args = TrainingArguments(
        output_dir='./results',
        num_train_epochs=3,
        per_device_train_batch_size=4,
        save_steps=10_000,
        save_total_limit=2,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
    )

    trainer.train()
    return model

# Example usage with a training dataset
# fine_tuned_model = fine_tune_gpt2(train_dataset)

4.2 T5 (Text-to-Text Transfer Transformer)

Why Use T5?

T5, developed by Google, frames all NLP tasks as a text-to-text problem, which means both inputs and outputs are text strings. This uniform approach makes T5 highly versatile for various tasks like translation, summarization, and question-answering.

Training Method:

T5 uses a sequence-to-sequence (seq2seq) training method, where the model is trained to generate a sequence (output text) given another sequence (input text). This method allows T5 to handle a wide range of tasks in a unified manner.

Example Code:

Fine-tuning T5 can be done with the transformers library, focusing on a specific task like summarization:

from transformers import T5Tokenizer, T5ForConditionalGeneration, Trainer, TrainingArguments

def fine_tune_t5(train_dataset):
    tokenizer = T5Tokenizer.from_pretrained('t5-small')
    model = T5ForConditionalGeneration.from_pretrained('t5-small')

    training_args = TrainingArguments(
        output_dir='./results',
        num_train_epochs=3,
        per_device_train_batch_size=4,
        save_steps=10_000,
        save_total_limit=2,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
    )

    trainer.train()
    return model

# Example usage with a training dataset
# fine_tuned_model = fine_tune_t5(train_dataset)

4.3. BART (Bidirectional and Auto-Regressive Transformers)

Why Use BART?

BART, developed by Facebook AI, combines the strengths of bidirectional and autoregressive transformers. It is particularly effective for tasks like text generation, summarization, and translation. BART is pre-trained by corrupting text with noise and then learning to reconstruct the original text.

Training Method:

BART is trained using a denoising autoencoder approach, where the model learns to predict the original text from a corrupted version. This method enables BART to generate coherent and contextually accurate text.

Example Code:

Fine-tuning BART for summarization can be done using the transformers library:

from transformers import BartTokenizer, BartForConditionalGeneration, Trainer, TrainingArguments

def fine_tune_bart(train_dataset):
    tokenizer = BartTokenizer.from_pretrained('facebook/bart-large-cnn')
    model = BartForConditionalGeneration.from_pretrained('facebook/bart-large-cnn')

    training_args = TrainingArguments(
        output_dir='./results',
        num_train_epochs=3,
        per_device_train_batch_size=4,
        save_steps=10_000,
        save_total_limit=2,
    )

    trainer = Trainer(
        model=model,
        args=training_args,
        train_dataset=train_dataset,
    )

    trainer.train()
    return model

# Example usage with a training dataset
# fine_tuned_model = fine_tune_bart(train_dataset)

These transformers differ in their architecture, training methods, and use cases, making each of them suitable for specific NLP tasks. By understanding these differences, you can choose the appropriate transformer model and training method for your specific requirements, ensuring optimal performance and accuracy in your applications.

5. Agent Creation

Creating agents, particularly conversational AI agents or chatbots, is a critical aspect of AI development. These agents can handle a wide range of tasks, from customer support to personal assistants. Here are three of the most famous Python frameworks for creating AI agents, along with real-life scenario examples and explanations of their features and use cases:

1. Rasa
2. Dialogflow (with Python integration)
3. Microsoft Bot Framework (with Python integration)

5.1 Rasa

Why Use Rasa?

Rasa is an open-source framework for building AI chatbots and assistants. It offers flexibility and control over the entire conversational AI stack, allowing developers to customize every aspect of their agents.

Features:

• NLP Pipeline: Includes tools for natural language understanding (NLU) and dialogue management.
• Custom Actions: Supports custom actions and integrations with various APIs.
• Open Source: Completely open-source, providing transparency and control.
• Contextual Conversations: Manages contextual and multi-turn conversations effectively.

Real-Life Scenario:

Customer Support Chatbot for an E-commerce Platform A large e-commerce platform uses Rasa to develop a chatbot that handles customer queries, processes orders, and provides shipping updates. The bot can understand and respond to complex queries, thanks to Rasa’s powerful NLU capabilities.

# Example of a simple Rasa custom action
from rasa_sdk import Action, Tracker
from rasa_sdk.executor import CollectingDispatcher

class ActionCheckOrderStatus(Action):
    def name(self) -> str:
        return "action_check_order_status"

    def run(self, dispatcher: CollectingDispatcher,
            tracker: Tracker,
            domain: dict) -> list:
        
        order_id = tracker.get_slot('order_id')
        # Assuming we have a function `get_order_status` to fetch order status
        order_status = get_order_status(order_id)
        
        dispatcher.utter_message(text=f"The status of your order {order_id} is: {order_status}")
        return []

# To run the action server, save this as actions.py and start the action server with:
# rasa run actions

5.2 Dialogflow (with Python integration)

Why Use Dialogflow?

Dialogflow, developed by Google, is a cloud-based framework for building conversational interfaces. It is known for its ease of use, robust integration capabilities, and strong natural language understanding powered by Google’s machine learning.

Features:

• NLP and NLU: Strong NLP capabilities for understanding user intent.
• Integration: Easily integrates with various platforms like Google Assistant, Slack, and Facebook Messenger.
• Context Management: Manages complex conversation contexts seamlessly.
• Cloud-based: Scalable and managed by Google’s cloud infrastructure.

Real-Life Scenario:

Virtual Travel Assistant, A travel company uses Dialogflow to create a virtual assistant that helps users book flights, hotels, and rental cars. The assistant can understand and process natural language queries, making the booking process smooth and efficient.

import dialogflow_v2 as dialogflow

def detect_intent_texts(project_id, session_id, texts, language_code):
    session_client = dialogflow.SessionsClient()
    session = session_client.session_path(project_id, session_id)

    for text in texts:
        text_input = dialogflow.types.TextInput(text=text, language_code=language_code)
        query_input = dialogflow.types.QueryInput(text=text_input)
        
        response = session_client.detect_intent(session=session, query_input=query_input)
        print(f"Query text: {response.query_result.query_text}")
        print(f"Detected intent: {response.query_result.intent.display_name}")
        print(f"Fulfillment text: {response.query_result.fulfillment_text}")

# Example usage
# detect_intent_texts('your-project-id', 'your-session-id', ['I want to book a flight to New York'], 'en')

5.3. Microsoft Bot Framework (with Python integration)

Why Use Microsoft Bot Framework?

Microsoft Bot Framework provides a comprehensive set of tools for building, testing, and deploying chatbots. It supports a wide range of conversational experiences, from simple command-based bots to complex, AI-powered virtual assistants.

Features:

• SDK and Tools: Offers robust SDKs and tools for developing bots.
• AI Integration: Easily integrates with Microsoft Cognitive Services for enhanced AI capabilities.
• Channel Support: Supports multiple channels, including Microsoft Teams, Slack, and web chat.
• Azure Integration: Seamless integration with Azure for deployment and scaling.

Real-Life Scenario:

Employee IT Support Assistant A large corporation uses Microsoft Bot Framework to create an IT support assistant for its employees. The bot helps with common IT issues, provides troubleshooting steps, and can even reset passwords through integration with Active Directory.

from botbuilder.core import ActivityHandler, TurnContext
from botbuilder.schema import ChannelAccount

class MyBot(ActivityHandler):
    async def on_members_added_activity(self, members_added: ChannelAccount, turn_context: TurnContext):
        for member in members_added:
            if member.id != turn_context.activity.recipient.id:
                await turn_context.send_activity("Welcome to the IT support assistant. How can I help you today?")

    async def on_message_activity(self, turn_context: TurnContext):
        user_input = turn_context.activity.text
        response = get_it_support_response(user_input)  # Custom function to process user input
        await turn_context.send_activity(response)

# Example usage in an Azure Bot Service project
# bot = MyBot()

These frameworks offer different levels of flexibility, customization, and ease of use, making them suitable for various use cases and deployment environments. By understanding their strengths and features, you can choose the best framework for your specific requirements, ensuring your AI agents are effective and efficient.

6. Deployment

Deploy the chatbot using Flask for a simple web interface.

from flask import Flask, request, jsonify

app = Flask(__name__)

@app.route('/chat', methods=['POST'])
def chat():
    user_input = request.json.get('input')
    response = agent.respond(user_input)
    return jsonify({'response': response})

if __name__ == '__main__':
    app.run(debug=True)

BONUS: Common Problems and their solutions

By integrating Generative AI and Agentic AI into our chatbot solution, you’ve now created a powerful tool for businesses to automate and enhance customer interactions. Whether you’re looking to build a chatbot or explore AI technologies, understanding these concepts and techniques will help you develop innovative solutions and drive success in your projects.

Happy coding and AI exploration!