CosmoS3 2.3.5

CosmoS3

CosmoS3 is an Amazon S3–compatible object-storage middleware library for CosmoApiServer. It implements core S3 operations with a pluggable metadata store — SQL Server, PostgreSQL, MySQL, SQLite, the embedded CosmoKv SQL engine, or kvdirect (a no-SQL ordered-KV index for single-node deployments) — and the local disk (or a pluggable storage driver) for object data. Full-object GETs stream disk→socket through a kernel zero-copy (sendfile) path.

It can be consumed three ways: as a standalone S3 endpoint (CosmoS3.Host), as middleware inside an existing CosmoApiServer app, or embedded in-process via the IS3Repository data layer.

On a Linux NVMe server, v2.3.0 beats native-config MinIO on every operation measured — GET 2–3× (up to ~9.85 GiB/s), LIST 2.1×, HEAD 1.47× — see Performance.

Table of Contents

1. Architecture
2. Security & Correctness
3. Performance
4. Recent Updates
5. Quick Start
6. Configuration
7. Database Schema
8. S3 Feature Compatibility
9. Static Website Hosting
10. Presigned URLs
11. Multipart Upload
12. Using with AWS CLI
13. Testing
14. Project Structure

Architecture

┌─────────────────────────────────────────────────────┐
│                   CosmoApiServer                     │
│  (System.IO.Pipelines transport, HTTP/1.1·2·3;        │
│   kernel sendfile zero-copy GET on plaintext HTTP/1)  │
└──────────────────────┬───────────────────────────────┘
                       │ IMiddleware
                       ▼
┌──────────────────────────────────────────────────────┐
│                  S3Middleware                         │
│  • S3Context.CreateAsync — async, non-blocking parse  │
│  • Authenticates (SigV4 / SigV2 / presigned)          │
│  • Verifies the SigV4 signature (ValidateSignatures)  │
│  • Routes to ServiceHandler / BucketHandler /         │
│    ObjectHandler / Admin + internal handlers          │
└────────────────┬───────────────────────┬──────────────┘
                 │                       │
     ┌───────────▼─────────┐ ┌───────────▼────────────┐
     │   DataAccess         │ │   Storage Driver        │
     │  → IS3Repository:    │ │  (DiskStorageDriver:    │
     │   • S3Repository     │ │   atomic write + move,  │
     │     (5 SQL backends) │ │   ArrayPool, SubStream, │
     │   • KvDirectRepo     │ │   zero-copy GET path)   │
     │     (ordered KV)     │ │                         │
     └──────────────────────┘ └─────────────────────────┘

Key types:

Security & Correctness

The design choices below are deliberate; each closes a concrete failure mode. They are enforced by the unit + integration test suite (see Testing).

Performance

CosmoS3 is optimized for high-throughput, low-latency workloads. With the kvdirect embedded backend and a kernel zero-copy GET path, v2.3.0 beats native-config MinIO on every operation on a Linux NVMe server — decisively on reads (GET up to ~9.85 GiB/s) and listing.

Benchmark: CosmoS3 vs. MinIO — Linux NVMe

CosmoS3 v2.3.0 (kvdirect backend) vs MinIO on a 12-core x86_64 NVMe server (the deployment target). Both Dockerized with host networking on a single local disk; warp S3 benchmark, 1 MiB / 4 MiB objects, 15 s, zero errors either side. MinIO runs single-drive (no erasure coding — its fastest config, the hardest baseline). Sequential runs.

CosmoS3 wins every operation measured. GET is the standout — the kernel sendfile path streams full-object bodies disk→socket with no user-space copy, sustaining up to 9.85 GiB/s. PUT is CPU-bound and was measured on a shared host, so treat its margin as parity-to-favourable; the GET/LIST/HEAD wins (sendfile + ordered-KV, not CPU-bound the same way) are large and robust.

Key Optimizations:

• Zero-copy GET (kernel sendfile): full-file plaintext GETs go disk→socket via sendfile(2)/TransmitFile — no user-space copy or read loop; objects ≤ 256 KiB serve from an in-memory file cache. (Ranges / TLS / HTTP-2/3 fall back to buffered streaming.)
• kvdirect ordered-KV metadata: latest-version lookup is a single seek and listing is a native byte-ordered cursor (no SQL parse/plan), driving the LIST 2.1× and HEAD 1.47× wins. See the DatabaseSettings section.
• Metadata fast-path cache: opt-in TTL cache (MetadataCacheSeconds) over credentials / bucket-ACL / object metadata so an authenticated GET/HEAD can hit zero metadata round-trips.
• Zero-Allocation I/O: ArrayPool<byte> across all read/write paths to eliminate transient allocations and GC pressure.
• Lock-Free Concurrency: ConcurrentDictionary bucket registry for non-blocking lookups under high concurrency.

Recent Updates

• Zero-copy GET (v2.3.0): full-file plaintext GETs stream disk→socket via kernel sendfile — GET up to ~9.85 GiB/s, ~2–3× native MinIO on Linux NVMe. (Requires CosmoApiServer.Core ≥ 3.8.0.)
• kvdirect embedded backend (v2.2.0): an IS3Repository directly on the CosmoKv ordered-KV engine — no SQL — for single-node deployments; listing 2.1× and HEAD 1.47× vs MinIO.
• SigV4 signature verification: requests are now cryptographically verified (constant-time, ±15-min skew), validated against the official AWS test vector — not just parsed.
• Conditional requests (RFC 7232): If-Match / If-None-Match / If-[Un]Modified-Since on GET/HEAD/PUT (304 / 412, overwrite guard, optimistic concurrency).
• Object versioning: per-key version rows with a UNIQUE (bucket, key, version) index and collision-retry assignment; GET/PUT ?versioning, ListObjectVersions, and x-amz-version-id.
• Bounded-memory listing: ListObjects(V2) streams and pages at the database level instead of loading the whole bucket.
• Single-pass multipart assembly: parts stream directly into the destination blob — no temp-file double-write.
• Hardened defaults: no default admin key, anonymous writes off unless an owner is configured, path-traversal-safe bucket names.
• Five SQL backends: SQL Server, PostgreSQL, MySQL, SQLite, and the embedded CosmoKv engine, behind one backend-portable IS3Repository.
• Fully async request path: S3Context.CreateAsync removed the sync-over-async body buffering (and the thread-pool workaround it required).
• Static website hosting, aws-chunked streaming decode, and a benchmarking suite for comparison against any S3-compatible backend.

Quick Start

1. Run a sample host

A ready-to-run host is included at samples/CosmoS3Host/, with backend-specific variants and demos alongside it:

cd samples/CosmoS3Host          # default
dotnet run

2. Wire CosmoS3 into your own CosmoApiServer app

using CosmoS3;
using CosmoS3.Settings;

var settings = new SettingsBase
{
    RegionString       = "us-east-1",
    ValidateSignatures = false,   // set true in production

    Storage = new StorageSettings
    {
        StorageType   = CosmoS3.Storage.StorageDriverType.Disk,
        DiskDirectory = "./data/objects"
    },

    Database = new DatabaseSettings
    {
        Hostname     = "localhost",
        Port         = 1433,
        DatabaseName = "MyDatabase",
        Username     = "sa",
        Password     = "your-password"
    },

    // Optional: enable CORS for browser-based S3 clients
    Cors = new CorsSettings { Enabled = true },

    // Optional: enable HTTPS
    // CertificatePath     = "./certs/server.pfx",
    // CertificatePassword = "changeme",

    // Optional: enable HTTP/2 cleartext (h2c)
    // EnableHttp2 = true,
};

// CosmoS3Application.Create() wires TLS, HTTP/2, CORS, logging, and S3Middleware.
var app = CosmoS3Application.Create(settings, port: 8100);
app.Run();

Or wire manually for full control over the middleware order:

using CosmoApiServer.Core.Hosting;
using CosmoS3;
using CosmoS3.Settings;

var app = CosmoWebApplicationBuilder.Create()
    .ListenOn(8100)
    .UseHttps("./certs/server.pfx", "changeme")   // optional TLS
    .UseHttp3()                                   // optional HTTP/3 over QUIC
    .UseHttp2()                                    // optional h2c
    .UseCors()                                     // optional CORS
    .UseLogging()
    .UseMiddleware(new S3Middleware(settings))
    .Build();

app.Run();

Configuration

SettingsBase

DatabaseSettings

CosmoS3 selects the backend from DatabaseType; the schema is created/updated at startup via DatabaseFactory.EnsureSchemaAsync.

For SQL Server the built connection string is:

server=HOSTNAME,PORT;database=DBNAME;user id=USER;password=PASS;TrustServerCertificate=true;

SQLite and the embedded CosmoKv engine need no server — point ConnectionString at a file/data-source path.

kvdirect — embedded ordered-KV metadata (no SQL)

kvdirect stores all metadata directly on the CosmoKv ordered-KV engine — the same LSM store the cosmokv backend uses, but without the SQL parser/planner/connection-pool on top. Object keys are byte-ordered (<bucket> 0x00 <key> 0x00 <invVersion>) so the latest version is a single seek and a bucket listing is a native cursor; there is no schema (the default user/credential/bucket/owner-ACL are seeded automatically on first open). Point ConnectionString at a directory:

"Database": { "DatabaseType": "kvdirect", "ConnectionString": "Data Source=./cosmos3-kvdirect" }

It is single-process / single-node by design (one embedded store, no shared DB) — the MinIO-style "just run the binary" deployment. Use a SQL backend (postgres/mssql/mysql) when multiple nodes must share one metadata store. End-to-end it runs ~6.3× faster across PUT/GET/HEAD/DELETE than the same engine via SQL; see benchmarks/results/kvdirect-spike-SUMMARY.md.

StorageSettings

Database Schema

CosmoS3 supports multiple database engines including SQL Server, PostgreSQL, MySQL, and SQLite. The schema is automatically created or updated at startup via DatabaseFactory.EnsureSchemaAsync.

The kvdirect backend has no SQL schema — the tables below map to a single byte-ordered key namespace on the CosmoKv engine (object rows at o\0<bucket>\0<key>\0<invVersion>, with secondary indexes for name / access-key / object-GUID lookups), and the default user/credential/bucket/owner-ACL are seeded on first open. See kvdirect.

Key Tables

Key Indexes for Performance & Correctness

The schema includes composite indexes that stay fast with millions of objects and also enforce integrity:

• ux_s3_objects_bucket_key_version: UNIQUE (bucketguid, objectkey, version DESC) — serves the common "get latest version" lookup and makes the versioning write race impossible at the storage layer (collisions retry with the next version). On CosmoKv the DESC qualifier is dropped (engine limitation) but the uniqueness constraint is identical.
• idx_s3_objects_guid: (guid) — blob/version resolution by object GUID.
• idx_s3_credentials_accesskey: (accesskey) — rapid authentication.
• idx_s3_buckets_name: (name) — fast bucket resolution.

All database operations go through the static DataAccess facade over the backend-portable IS3Repository. The SQL backends use S3Repository (parameterized SQL — no stored procedures, so the same code runs unchanged on every engine); kvdirect uses KvDirectRepository, which implements the identical contract directly on ordered-KV primitives. The versioning write race is prevented on both: a UNIQUE (bucket, key, version) index on SQL, and optimistic-commit conflict retry on kvdirect.

S3 Feature Compatibility

Service-Level Operations

Bucket Operations

Object Operations

Notes on Compatibility

• Signature versions: Both SigV4 and SigV2 are supported for authentication and presigned URLs. With ValidateSignatures enabled (default), SigV4 signatures are cryptographically verified, not merely parsed.
• Conditional requests: GET/HEAD return 304 Not Modified / 412 Precondition Failed; PUT supports If-None-Match: * (don't overwrite) and If-Match (optimistic concurrency) per RFC 7232.
• aws-chunked transfer encoding: Automatically decoded via the async body path; works with aws s3 cp for any file size.
• Versioning: Supported. Toggle with PUT ?versioning; when enabled, writes create new version rows, deletes write delete markers, and responses carry x-amz-version-id. List historic versions with list-object-versions.
• Anonymous writes: Off by default — set AnonymousOwnerGuid to a real user GUID to allow public-write buckets.
• Bucket policies / lifecycle / replication: Not implemented. (CORS is configured server-side via CorsSettings, not per-bucket.)

Static Website Hosting

A bucket can be configured to serve static files over plain HTTP (no AWS credentials required).

Configure a bucket for website hosting

# Create bucket
aws --endpoint-url http://localhost:8100 s3 mb s3://my-site

# Upload content
aws --endpoint-url http://localhost:8100 s3 cp index.html  s3://my-site/index.html  --content-type text/html
aws --endpoint-url http://localhost:8100 s3 cp error.html  s3://my-site/error.html   --content-type text/html

# Enable website hosting
aws --endpoint-url http://localhost:8100 s3 website s3://my-site \
    --index-document index.html \
    --error-document error.html

Browse the site

# Bucket root returns index.html
curl http://localhost:8100/my-site/

# Unknown path returns error.html with 404
curl http://localhost:8100/my-site/missing.html

Redirect all requests

aws --endpoint-url http://localhost:8100 s3api put-bucket-website \
    --bucket my-site \
    --website-configuration '{
        "RedirectAllRequestsTo": { "HostName": "example.com", "Protocol": "https" }
    }'

Routing rules

aws --endpoint-url http://localhost:8100 s3api put-bucket-website \
    --bucket my-site \
    --website-configuration '{
        "IndexDocument": { "Suffix": "index.html" },
        "ErrorDocument": { "Key": "error.html" },
        "RoutingRules": [
            {
                "Condition": { "KeyPrefixEquals": "old/" },
                "Redirect":  { "ReplaceKeyPrefixWith": "new/" }
            }
        ]
    }'

How it works:

• Website configuration is stored as website.xml at <DiskDirectory>/<bucketName>/website.xml.
• Requests to a website-enabled bucket without AWS authentication headers are served as static files.
• If the request path ends with /, the index document is served.
• If the object is not found, the error document is returned with HTTP 404.
• Redirect rules are evaluated before object lookup.

Presigned URLs

Presigned URLs grant time-limited access to an S3 object without requiring the caller to have AWS credentials.

Generate a presigned URL (C# SDK)

var request = new GetPreSignedUrlRequest
{
    BucketName = "my-bucket",
    Key        = "my-object.txt",
    Expires    = DateTime.UtcNow.AddMinutes(15),
    Verb       = HttpVerb.GET
};

string url = s3Client.GetPreSignedURL(request);

Use the presigned URL

# Download with curl (no AWS credentials needed)
curl "<presigned-url>" -o downloaded.txt

# Upload with a presigned PUT URL
curl -X PUT "<presigned-put-url>" --data-binary @file.txt

Signature version behavior:

AWSSDK generates SigV2 presigned URLs for custom (non-AWS) endpoints. CosmoS3 validates both:

Expired presigned URLs return HTTP 403 ExpiredToken.

Multipart Upload

Multipart upload allows large files to be uploaded in parts and assembled server-side.

Via AWS CLI (automatic for files > 8 MB by default)

# CosmoS3 handles chunked uploads transparently
aws --endpoint-url http://localhost:8100 \
    s3 cp large-file.bin s3://my-bucket/large-file.bin \
    --expected-size 1073741824   # hint for 1 GB file

Via SDK (manual)

// 1. Initiate upload
var initResponse = await s3.InitiateMultipartUploadAsync(new InitiateMultipartUploadRequest
{
    BucketName  = "my-bucket",
    Key         = "my-object"
});
string uploadId = initResponse.UploadId;

// 2. Upload parts (minimum 5 MB each, except the last)
var uploadPartResponse = await s3.UploadPartAsync(new UploadPartRequest
{
    BucketName   = "my-bucket",
    Key          = "my-object",
    UploadId     = uploadId,
    PartNumber   = 1,
    InputStream  = partStream,
    PartSize     = partStream.Length
});

// 3. Complete the upload
await s3.CompleteMultipartUploadAsync(new CompleteMultipartUploadRequest
{
    BucketName = "my-bucket",
    Key        = "my-object",
    UploadId   = uploadId,
    PartETags  = new List<PartETag> { new PartETag(1, uploadPartResponse.ETag) }
});

Internals:

• Each uploaded part is written to a part file under TempDirectory and hashed on arrival.
• CompleteMultipartUpload streams the ordered part files directly into the destination blob via ConcatReadStream (single pass — no concatenate-to-temp-then-recopy), then deletes the parts. The object length is the sum of the recorded part lengths.
• Active sessions and their parts are tracked in s3_uploads / s3_uploadparts and can be listed or aborted.

Using with AWS CLI

Configure the AWS CLI for local use

aws configure
# AWS Access Key ID:     default
# AWS Secret Access Key: default
# Default region name:   us-east-1
# Default output format: json

Common commands

ENDPOINT=http://localhost:8100

# List buckets
aws --endpoint-url $ENDPOINT s3 ls

# Create bucket
aws --endpoint-url $ENDPOINT s3 mb s3://my-bucket

# Upload file
aws --endpoint-url $ENDPOINT s3 cp file.txt s3://my-bucket/

# Download file
aws --endpoint-url $ENDPOINT s3 cp s3://my-bucket/file.txt ./

# List objects
aws --endpoint-url $ENDPOINT s3 ls s3://my-bucket/

# Sync directory
aws --endpoint-url $ENDPOINT s3 sync ./local-dir/ s3://my-bucket/prefix/

# Delete object
aws --endpoint-url $ENDPOINT s3 rm s3://my-bucket/file.txt

# Delete bucket (must be empty)
aws --endpoint-url $ENDPOINT s3 rb s3://my-bucket

# Tag a bucket
aws --endpoint-url $ENDPOINT s3api put-bucket-tagging \
    --bucket my-bucket \
    --tagging '{"TagSet":[{"Key":"env","Value":"dev"}]}'

# Get bucket tags
aws --endpoint-url $ENDPOINT s3api get-bucket-tagging --bucket my-bucket

# Get bucket website config
aws --endpoint-url $ENDPOINT s3api get-bucket-website --bucket my-bucket

# Presigned URL (60 seconds)
aws --endpoint-url $ENDPOINT s3 presign s3://my-bucket/file.txt --expires-in 60

Testing

The suite (tests/CosmoS3.Tests/) uses xUnit + AWSSDK.S3 and needs no external database or running server — it boots the real server in-process against an embedded CosmoKv DB.

Tests are split into two xUnit traits that must run as separate dotnet test invocations, because the server's data layer is a process-global singleton (DataAccess._repo), so only one server instance can exist per process:

# Server-free unit tests (signature math, helpers, conditional-header logic, ...)
dotnet test tests/CosmoS3.Tests --filter "Category=Unit"

# Integration tests — boot the in-process server, drive it through the AWS SDK
dotnet test tests/CosmoS3.Tests --filter "Category=Integration"

CI runs the two lanes as separate steps (.github/workflows/ci.yml). Current status: 62 unit + 12 integration, all passing.

Fixtures

• CosmoServerFixture (integration) boots the server with CosmoWebApplication.RunAsync on an ephemeral port over a temp CosmoKv database, and hands tests a configured AmazonS3Client (NewS3Client()). Shared via the [Collection("CosmoServer")] xUnit collection.
• S3Fixture is the legacy fixture that targets an externally running endpoint, retained for the original end-to-end tests.

Representative coverage

Project Structure

src/CosmoS3/
├── S3Middleware.cs          # IMiddleware entry point; async parse, SigV4 verify, routing
├── S3Request.cs             # HTTP → S3 request parsing (CreateAsync, aws-chunked decode)
├── S3Response.cs            # S3-formatted response writer
├── S3Context.cs             # Combined request + response context (CreateAsync factory)
├── S3Exception.cs           # Typed S3 error thrown by handlers
├── SignatureV4Verifier.cs   # Recompute + constant-time compare of the SigV4 signature
├── DataAccess.cs            # Static facade over IS3Repository
├── IS3Repository.cs         # Backend-portable data-layer contract
├── S3Repository.cs          # Parameterized-SQL implementation (all 5 SQL backends)
├── KvDirectRepository.cs    # Embedded ordered-KV implementation (no SQL — DatabaseType=kvdirect)
├── DatabaseFactory.cs       # Backend selection + schema bootstrap
├── CosmoS3Application.cs     # Turn-key host builder (TLS / HTTP2·3 / CORS / middleware)
├── Settings/                # SettingsBase, Database/Storage/Logging/Debug/Cors settings
├── Helpers/
│   ├── SignatureV4.cs       # SigV4 canonical-request + signing-key primitives
│   ├── SecurityHelper.cs    # FixedTimeEquals, ConfiguredKeyMatches, IsValidBucketName
│   ├── ConditionalHeaders.cs# RFC 7232 If-Match / If-None-Match / If-[Un]Modified-Since
│   ├── ConcatReadStream.cs  # Single-pass multipart assembly stream
│   ├── AclConverter.cs      # ACL ⇆ policy conversion (grantees validated vs DB)
│   ├── RequestValidator.cs  # Shared handler precondition checks
│   └── HashHelper.cs        # Single-pass MD5/SHA hashing
├── Classes/
│   ├── AuthManager.cs       # Credential/user resolution + authorization decision
│   ├── BucketManager.cs     # Lock-free ConcurrentDictionary bucket registry
│   ├── BucketClient.cs      # Per-bucket storage driver accessor
│   ├── ConfigManager.cs     # User / credential / bucket / object lookup
│   └── CleanupManager.cs    # Background task: expire stale temp files
├── Api/S3/
│   ├── ApiHandler.cs        # Top-level dispatcher (service/bucket/object)
│   ├── ServiceHandler.cs    # ListBuckets
│   ├── BucketHandler.cs     # Bucket operations (incl. versioning, website, ACL, tags)
│   ├── ObjectHandler.cs     # Object operations + multipart + conditional PUT
│   └── ApiHelper.cs         # Shared XML response helpers
├── Storage/
│   ├── StorageDriverBase.cs
│   └── DiskStorageDriver.cs # Atomic temp-then-move writes; ArrayPool + SubStream I/O
├── Schema/                  # Per-backend DDL: mssql · postgres · mysql · sqlite · cosmokv
├── S3Objects/               # XML DTOs (request/response bodies)
└── Logging/
    └── S3Logger.cs          # Console/callback-based logger