Building an Enterprise Knowledge Assistant Using Java, Spring AI, and LangGraph4j

I recently worked on an Enterprise Knowledge Assistant using Java 21, Spring Boot, Spring AI, and LangGraph4j.

The main objective of this project is to build an agentic AI application that can understand user questions, identify the required knowledge source, execute the appropriate operations, and generate a meaningful response.

Instead of connecting an LLM to a single data source, this application integrates multiple sources, including relational databases, documents using RAG, and external tools through MCP.

Architecture

The application uses LangGraph4j to manage the execution workflow.

             User Question
                   |
                   v
             Classification
                   |
          +--------+--------+
          |        |        |
          v        v        v
       Database   RAG      MCP
          |        |        |
          +--------+--------+
                   |
                   v
             Response Node
                   |
                   v
              Final Answer

The application supports five types of queries:

• DATABASE: Retrieves information from relational databases.
• RAG: Retrieves relevant information from enterprise documents.
• MCP: Invokes tools exposed by an external MCP server.
• MULTI_SOURCE: Combines information from multiple sources.
• GENERAL: Handles general conversations.

Technology Stack

• Java 21
• Spring Boot 4.1.2
• Spring AI 2.0.1
• LangGraph4j 1.8.27
• OpenAI GPT-4.1-mini
• H2 Database
• Spring Data JPA
• SimpleVectorStore
• Maven

How It Works

The application uses a compiled state graph containing different processing nodes.

1. Classification Node

Uses an LLM to understand the user’s question and classify it into one of the five supported categories.

2. Database Node

Uses Spring AI tool calling to retrieve employee information from an H2 database through Java service methods.

3. RAG Node

Loads an employee policy PDF, splits it into chunks, generates embeddings, and stores them in SimpleVectorStore. User questions are answered using relevant retrieved document content.

4. MCP Node

Connects to a separate Spring Boot MCP server through SSE. The MCP server exposes client-related tools such as retrieving client information, finding the oldest client, and identifying the most valuable client.

5. Multi-Source Node

Get Suhel Shaik’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Combines information from the employee database and policy documents to answer questions requiring multiple knowledge sources.

6. Generic Node

Handles general questions and conversations directly through the LLM.

7. Response Node

Collects the processing results and generates the final response.

Project Modules

The project contains two independent Spring Boot applications.

ModuleResponsibilityPortenterprise-knowledge-assistant-applicationLangGraph4j orchestration, RAG, database integration, MCP client, REST API8080enterprise-knowledge-assistant-mcpMCP server exposing client-portfolio tools8085

Running the Application

First, configure your OpenAI API key:

export OPENAI_API_KEY=sk-...

Start the MCP server:

cd enterprise-knowledge-assistant-mcp
./mvnw spring-boot:run

Start the main application:

cd enterprise-knowledge-assistant-application
./mvnw spring-boot:run

The application exposes the following REST endpoint:

POST http://localhost:8080/api/chat

Example request:

{
  "question": "Who is our most valuable client?"
}

Future Improvements

Some improvements planned for this architecture include:

• Conversation memory and context management.
• Human-in-the-loop approval workflows.
• Persistent vector database integration.
• Dynamic MCP tool discovery.
• Streaming responses.
• Better monitoring and observability.

Conclusion

This project demonstrates how we can combine LangGraph4j, Spring AI, RAG, and MCP to build an enterprise-oriented agentic AI application using Java.

The key idea is to let the LLM understand user intent while LangGraph4j controls the execution workflow and connects different knowledge sources.

Rather than building separate applications for every data source, we can bring them together through a unified AI assistant.

Technologies: Java | Spring Boot | Spring AI | LangGraph4j | OpenAI | RAG | MCP

Github

enterprise-knowledge-assistant-langgraph-java/README.md at master ·…