Sitemap

Member-only story

Create a Data pipeline using LLM and Spring boot application using Codex + ChatGPT + FastAPI

3 min readOct 7, 2025

Step1: ChatGPT prompt to get prompt to be fed to codex

example of prompt to give to chatgpt codex so to create a data pipeline where each service is a spring boot application and order of devices are dataingestion-service , data-deduplication-service, data-quality-service, data-storage-service and each service calls the corresponding fast api exposed by corresponding python application that uses llm to address following usecases for each service data classification between PII data and non-PII data, data-deduplication, imputation using median for data quality, storage in DB correspondingly
Press enter or click to view image in full size

The above creates a prompt as below:

You are a senior full-stack engineer. Generate a COMPLETE, RUNNABLE mono-repo implementing the following architecture. Output ALL files inline under proper code fences, grouped by a file tree first.

# Goal
Build a 4-stage data pipeline where each stage is a Spring Boot 3 (Java 17) microservice that calls a corresponding Python FastAPI service which uses an LLM:
1) dataingestion-service → calls ingestion_api → LLM for data classification (PII vs non-PII)
2) data-deduplication-service → calls dedup_api → LLM/embeddings for semantic deduplication
3) data-quality-service → calls…

--

--