Skip to content

Minimal metadata service on top of FoundationDB + MinIO with Java. Transactional catalog, simple HTTP API, containerized for local development.

Notifications You must be signed in to change notification settings

fmartinez09/metadata

Repository files navigation

FDB + MinIO Metadata Service (Java)

This project implements a minimal metadata service for blobs using:

  • FoundationDB (FDB) → strongly consistent transactional catalog.
  • MinIO → blob storage backend.
  • Java (Javalin + FDB Java bindings + MinIO client) → lightweight HTTP API to register and query metadata.

The goal is to demonstrate how FDB can serve as a metadata store in a realistic design, ready to evolve toward advanced features (idempotency, CAS, secondary indexes, SNAPs).


✨ Features

  • Key schema in FDB (Tuple Layer):

    ("b", tenant, objId, "meta")  -> JSON metadata
    ("b", tenant, objId, "state") -> byte (0=pending, 1=committed)
    
    
  • Minimal API (Javalin):

    • PUT /o/{tenant}/{id} → register/update metadata (validates blob in MinIO).
    • GET /o/{tenant}/{id} → return metadata JSON.
    • HEAD /o/{tenant}/{id} → existence check.
    • GET /o/{tenant}?prefix=abc → list object IDs by prefix.
    • DELETE /o/{tenant}/{id} → delete metadata (optionally delete blob).
  • Dockerized stack:

    • FoundationDB 7.4
    • MinIO server
    • Java app (Gradle + ShadowJar)
  • Designed to grow:

    • Multi-tenant isolation.
    • Extendable to CAS, secondary indexes, SNAP composition.

🚀 Getting Started

# Build images
docker compose build

# Start stack
docker compose up -d

# Initialize FDB cluster (first run only)
docker compose exec fdb fdbcli --cluster_file /var/fdb/fdb.cluster --exec "configure new single ssd"

# Create a bucket in MinIO
mc alias set local http://localhost:9000 minio minio12345
mc mb local/metadata

Register metadata example:

curl -X PUT http://localhost:8080/o/tenant1/foto-001 \
  -H "Content-Type: application/json" \
  -d '{
        "bucket": "metadata",
        "object": "networking.jpg",
        "contentType": "image/jpeg",
        "size": 12345,
        "sha256": "abc123...",
        "tags": {"owner":"fernando","type":"photo"}
      }'

📚 Roadmap

  • Advanced idempotency with Idempotency-Key.
  • Secondary indexes (tags, owner).
  • Deduplication via SHA-256 (CAS).
  • SNAP integration (queues, quotas, profiles).
  • Observability (structured logs, p95/p99 metrics).

⚡️ References

  • FoundationDB Record Layer (SIGMOD’19)
  • [Apple CloudKit (VLDB’20)]
  • DeepSeek 3FS (arXiv 2025)

About

Minimal metadata service on top of FoundationDB + MinIO with Java. Transactional catalog, simple HTTP API, containerized for local development.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published