Building a Medical AI Assistant with Streamlit and LangChain

Image Analysis

PDF analysis and Chat

In this article, we explore the creation of a medical AI assistant that can analyze both medical images (such as MRIs and X-rays) and PDF documents containing medical reports. The project integrates Streamlit for building the user interface, OpenAI’s GPT-4 for generating responses, and LangChain for structuring the interaction between the AI model and the user. Below, we dive into the key components of this powerful tool and highlight its functionality.

Project Overview

The main goal of the project is to provide an AI-powered application capable of analyzing two types of medical data:

• Medical images (JPG, JPEG, PNG): The AI assistant analyzes these images and provides a diagnosis based on visible abnormalities or simply states if the image appears normal.
• PDF documents: The assistant extracts text from medical PDFs and allows users to ask questions about the extracted content.

The system is built using Python with a variety of libraries, including Streamlit, OpenAI, and LangChain, to create a seamless and efficient experience for users.

Key Libraries and Tools

• Streamlit: Provides the user interface for uploading images and PDFs, as well as displaying results.
• LangChain: Facilitates the conversational retrieval chain and text processing, enabling the AI to analyze documents effectively.
• OpenAI GPT-4 API: Powers the natural language processing (NLP) and image diagnosis.
• PyPDF2: Extracts text from PDF files for further analysis.
• FAISS: Manages document embedding for quick and efficient retrieval during user queries.

Step-by-Step Functionality Breakdown

1. Image Analysis with OpenAI API

The application includes a dedicated module for analyzing medical images using OpenAI’s GPT-4 model. Here’s how it works:

• Image Upload: Users upload medical images (X-rays, MRIs, etc.) in JPG, JPEG, or PNG format.
• Encoding and API Call: The uploaded image is base64-encoded and sent to the GPT-4 API along with a predefined prompt that instructs the model to analyze the image and provide a concise diagnosis.
• Response Handling: The AI model responds with either a specific diagnosis or a message indicating that the image appears normal.

def get_image_description(image_data, api_key):
    base64_image = encode_image(image_data)
    payload = {
        "model": "gpt-4o",
        "messages": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "text",
                        "text": IMAGE_PROMPT
                    },
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_image}"
                        }
                    }
                ]
            }
        ],
        "max_tokens": 300
    }
    response = requests.post("https://api.openai.com/v1/chat/completions", headers=headers, json=payload)
    return response.json()

2. PDF Analysis with LangChain

The PDF analysis section allows users to upload medical documents in PDF format and then ask specific questions about them. Here’s the high-level flow:

• Text Extraction: The system extracts text from all uploaded PDFs using PyPDF2.
• Text Chunking: Extracted text is split into manageable chunks for easier embedding and retrieval, ensuring large documents can be processed efficiently.
• FAISS Vector Store: These text chunks are then embedded into a vector store using FAISS, allowing for fast retrieval of relevant sections when users ask questions.
• Conversational Chain: The conversation retrieval chain, powered by LangChain, enables a dynamic and interactive dialogue where users can inquire about the PDF content, and the AI retrieves and responds based on relevant information from the document.

def build_conversation_chain(vector_store, openai_api_key):
    llm = ChatOpenAI(model_name="gpt-4o", openai_api_key=openai_api_key)
    memory = ConversationBufferMemory(memory_key='chat_history', return_messages=True)
    conversation_chain = ConversationalRetrievalChain.from_llm(
        llm=llm, retriever=vector_store.as_retriever(), memory=memory)
    return conversation_chain

3. User Interface (Streamlit)

The interface is built using Streamlit and includes two primary sections: one for image analysis and one for PDF analysis.

• Sidebar Options: Users can select between “Image Analysis” and “PDF Analysis” using a sidebar, upload their files, and view the results.
• Real-Time Analysis: The uploaded images or documents are processed, and the results are displayed on the main interface in real-time.
• Conversational Interface: In the PDF analysis section, users can type in questions and receive AI-generated responses based on the extracted document content.

st.set_page_config(page_title="Medical Assistance", page_icon="🩺")
st.title("Medical Assistance 🩺")

# Sidebar for selecting analysis type
option = st.sidebar.selectbox("Choose an analysis type", ("Image Analysis", "PDF Analysis"))

# Main section: handling image analysis or PDF analysis based on user selection
if option == "Image Analysis":
    uploaded_image = st.file_uploader("Upload Medical Image", type=["jpg", "jpeg", "png"])
    # Additional steps for image processing...

elif option == "PDF Analysis":
    pdf_docs = st.file_uploader("Upload your Medical PDFs here", type="pdf", accept_multiple_files=True)
    # Additional steps for PDF processing...

4. Environment Management with .env

Sensitive API keys (such as OpenAI API keys) are managed using a .env file to ensure they are not exposed in the codebase. This is essential for maintaining security and privacy when working with third-party APIs.

from dotenv import load_dotenv
load_dotenv()
openai_api_key = os.getenv("OPENAI_API_KEY")

Conclusion

This project demonstrates how to build a powerful AI-driven tool capable of analyzing medical images and documents. By integrating OpenAI GPT-4 and LangChain, the assistant provides users with instant and insightful diagnoses or answers to their questions. Streamlit’s user-friendly interface makes the application accessible to non-technical users, while the flexibility of the underlying technologies ensures that the system can be easily expanded or adapted for new use cases.

Get CodeAIGo’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

This project is an excellent example of how AI can be leveraged to support healthcare professionals by providing quick and accurate analysis of medical data.

For further details, visit CodeAiGo

Happy coding and stay tuned for more exciting projects!