<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ethan Carter on Medium]]></title>
        <description><![CDATA[Stories by Ethan Carter on Medium]]></description>
        <link>https://medium.com/@ethan-carter?source=rss-71a3f79d65d6------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*Y5Y6ZwhR4s2PfiqRX11D0g.jpeg</url>
            <title>Stories by Ethan Carter on Medium</title>
            <link>https://medium.com/@ethan-carter?source=rss-71a3f79d65d6------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:33:17 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ethan-carter/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Spin Up a Self-Hosted S3 Bucket in 5 Minutes with RustFS]]></title>
            <link>https://medium.com/@ethan-carter/spin-up-a-self-hosted-s3-bucket-in-5-minutes-with-rustfs-f8d2c156583a?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f8d2c156583a</guid>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[selfhost]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Sun, 01 Nov 2026 14:09:38 GMT</pubDate>
            <atom:updated>2026-11-01T15:07:19.835Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*doBJVEshAulfUzG1bTg0BA.png" /></figure><p>Last month, Hacktoberfest 2026 had open-source contributors everywhere reaching for a place to stash build artifacts, datasets, and CI caches. The cloud bucket is the obvious answer until you remember it wants a credit card and bills you for egress. I stood up an S3-compatible bucket on my own laptop in about five minutes, no AWS account, and it speaks the exact same API my CI already used against S3. Here is the whole thing, and it works just as well in November as it did in October.</p><h3>TL;DR</h3><ul><li>RustFS is Apache 2.0 and S3-compatible, so your existing mc, aws, boto3, and rclone commands just work against it by changing one endpoint.</li><li>One container, ports 9000 (API) and 9001 (console), default creds rustfsadmin / rustfsadmin.</li><li>Spin it up with a single docker run, open the console, make a bucket, point your tooling at localhost:9000.</li><li>It runs as a non-root user (UID/GID 10001:10001), so bind-mounted dirs need the right owner. The compose stack fixes permissions for you if you would rather skip that step.</li><li>Want to give back instead of just using the tool? RustFS is open to contributions; the issues and discussions boards are public.</li></ul><h3>Why a self-hosted S3 bucket</h3><p>Most open-source projects are run by people who do not want to paste cloud credentials into a fork. A self-hosted S3-compatible bucket solves three real problems at once:</p><ol><li><strong>No bill, no card.</strong> You store artifacts on hardware you already have. The bytes never leave your machine unless you decide they should.</li><li><strong>Your tooling already speaks S3.</strong> If your CI uploads to S3 today, it uploads to RustFS by changing one endpoint string. No new client, no rewrite.</li><li><strong>Reproducible by anyone.</strong> A teammate on a different continent can run the same three commands and get the same bucket. That matters more for an OSS project than for a closed internal one.</li></ol><p>The catch is the one we always name out loud: a self-hosted store makes you the durability and backup owner. RustFS keeps its metadata inside the store, so your discipline is the only thing between you and data loss. For a scratch bucket that is a fine trade. For production, rehearse a restore first.</p><h3>Prerequisites</h3><ul><li>Docker installed and running.</li><li>Enough disk for the image plus whatever you plan to store.</li></ul><p>That is it. No Kubernetes, no cluster, no config file to edit for the basic path.</p><h3>Step 1: Start RustFS</h3><p>The container runs as a non-root user ( rustfs, UID/GID 10001:10001). If you bind-mount host directories, they must be writable by that user, or startup fails with a permission error. The simplest path is to create the dirs and hand them over first:</p><p>That is the entire “spin it up” step. The image pull is the slow part; the container is up in seconds after that. If you would rather not think about the chown, the project ships a docker-compose-simple.yml that runs an Alpine permission-helper container before RustFS starts, then brings up rustfs-server with five named volumes (four data volumes plus a logs volume) and a health check. Same result, one fewer manual step. One caveat: that helper only chowns the compose&#39;s named volumes. If you edit the compose to use host bind mounts instead, you still have to chown those host paths yourself.</p><p>Prefer not to touch Docker at all? There is also a one-line installer:</p><pre>curl -O https://rustfs.com/install_rustfs.sh &amp;&amp; bash install_rustfs.sh</pre><h3>Step 2: Open the console and make a bucket</h3><p>Point a browser at http://localhost:9001. Log in with the documented defaults, rustfsadmin / rustfsadmin, and create a bucket from the console UI. Call it something project-shaped, like hf2026-artifacts.</p><p><strong>Change the credentials before this touches anything real.</strong> The defaults are documented for exactly this quick-start situation, not for exposure. Set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY (the compose file marks them # CHANGEME) or use IAM policies once you are past the scratch phase.</p><h3>Step 3: Actually use it</h3><p>Because RustFS speaks S3, the MinIO client mc works unchanged. Point it at your local endpoint and push a file. (You already created hf2026-artifacts in the console in Step 2, so skip the bucket-creation command and go straight to the copy.)</p><pre>mc alias set rustfs http://localhost:9000 rustfsadmin rustfsadmin <br>mc cp ./build.tar.gz rustfs/hf2026-artifacts/</pre><p>Or from Python, with boto3, pointed at port 9000. Read the credentials from the environment so you are not pasting secrets into source:</p><pre>import os<br>import boto3<br><br>s3 = boto3.client(<br>    &quot;s3&quot;,<br>    endpoint_url=&quot;http://localhost:9000&quot;,<br>    aws_access_key_id=os.environ[&quot;RUSTFS_ACCESS_KEY&quot;],<br>    aws_secret_access_key=os.environ[&quot;RUSTFS_SECRET_KEY&quot;],<br>)<br><br>s3.upload_file(&quot;build.tar.gz&quot;, &quot;hf2026-artifacts&quot;, &quot;build.tar.gz&quot;)</pre><p>If you changed the credentials in Step 2, set RUSTFS_ACCESS_KEY and RUSTFS_SECRET_KEY to those new values. If you left the defaults, they are rustfsadmin / rustfsadmin, the same ones the mc command above uses.</p><p>The same client reads a cloud bucket by swapping the endpoint and credentials. That portability is the whole point: your project keeps one storage code path whether it runs on a contributor’s laptop or in CI.</p><h3>Why RustFS specifically</h3><p>I am not claiming RustFS is the only option. MinIO, Garage, and SeaweedFS all have their place. What makes RustFS a clean fit for this job:</p><ul><li><strong>Apache 2.0, not AGPL.</strong> Unlike MinIO, which is AGPL-3.0, RustFS ships under the permissive Apache 2.0 license. That is the reason it is comfortable to drop into a forked OSS project without copyleft surprises. (SeaweedFS is also Apache 2.0, so the license is not a differentiator there. RustFS’s edge is the Rust-built engine and the MinIO-shaped toolchain.)</li><li><strong>S3-compatible, MinIO-shaped toolchain.</strong> If you have used MinIO, the commands are the same. RustFS keeps that simplicity and rebuilds the engine in Rust for memory safety and concurrency.</li><li><strong>One container, no sidecar.</strong> Core S3 plus versioning, Object Lock (WORM), lifecycle, and replication are available today; the metadata lives inside the store, so there is one fewer service to run.</li><li><strong>Honest gaps.</strong> No FUSE or POSIX mount. You use the S3 API, not a local filesystem. And S3 Tables (the Iceberg REST catalog) is still Preview, so I would not put a production lakehouse on it yet.</li></ul><h3>Bonus: give back to RustFS (open source runs year-round)</h3><p>Using the tool is one thing; contributing is another. A timing note: Hacktoberfest 2026 changed its rules, pull requests no longer count toward the event’s rewards, with the focus moving to in-person Fests and online challenges. There is no event credit to chase here regardless (the repo carries no hacktoberfest topic). That is fine: the repo is Apache 2.0 and open to contributions 365 days a year. The issues and discussions boards are public:</p><p>If you are looking for a storage project to contribute to, filter the issues tab for good first issue or help wanted, or weigh in on an open discussion. A documentation fix or a reproducible bug report is as valuable as a feature PR, just do not expect swag for it.</p><h3>FAQ</h3><h3>Do I need an AWS account to follow this?</h3><p>No. RustFS runs locally and emulates the S3 API. The aws CLI and boto3 work against it by pointing at localhost:9000 instead of s3.amazonaws.com.</p><h3>Is rustfsadmin / rustfsadmin safe to leave?</h3><p>No, not past a scratch run. It is the documented default for quick starts. Set your own keys via environment variables or IAM before anything sensitive touches the bucket.</p><h3>Can I use this as my CI artifact store?</h3><p>For a weekend or small-scale project, yes. Point your upload step at localhost:9000 (or the host&#39;s LAN IP) and the same S3 calls you would use for cloud storage apply. For anything long-lived, add versioning and a real backup target.</p><h3>What if I need a filesystem mount, not an API?</h3><p>RustFS has no FUSE or POSIX mount, by design. If your tooling demands a mount, this is not the right fit. Use the S3 API path or pick a store that offers one.</p><h3>A note on how this was verified</h3><p>I pulled the commands, ports, default credentials, license, and Feature &amp; Status table verbatim from the RustFS README (checked 2026–11–01). The docker run line is quoted exactly, not paraphrased. The container&#39;s non-root UID/GID (10001:10001) comes from the README, which states the server runs as a non-root user rustfs; the compose permission-helper behavior comes from the project&#39;s docker-compose-simple.yml. This is a quick-start walkthrough, not a benchmark or a production deployment report. I did not run a throughput test or stand up a multi-node cluster, and the sandbox here has no Docker daemon, so the commands were not executed locally. Validate them on your own machine before relying on them.</p><h3>Sources</h3><p>What are you stashing your project’s artifacts in: a cloud bucket, or something you control? If you stood up a local S3 bucket, what tripped you up?</p><p><em>Originally published at </em><a href="https://dev.to/ethan-carter/spin-up-a-self-hosted-s3-bucket-in-5-minutes-with-rustfs-55j3"><em>https://dev.to</em></a><em> on November 1, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f8d2c156583a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apache Iceberg & Lakehouse: RustFS S3 Tables in Practice]]></title>
            <link>https://medium.com/@ethan-carter/apache-iceberg-lakehouse-rustfs-s3-tables-in-practice-51bb1b97fb19?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/51bb1b97fb19</guid>
            <category><![CDATA[apache]]></category>
            <category><![CDATA[data-lakehouse]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[apache-iceberg]]></category>
            <category><![CDATA[rustfs]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Mon, 19 Oct 2026 13:15:12 GMT</pubDate>
            <atom:updated>2026-10-25T13:43:12.768Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*22Y96RqLjIQuAHIh.png" /></figure><p>A lakehouse is not a product; it is a three-layer split — a table format (Apache Iceberg), a catalog that knows which tables exist, and object storage that holds the bytes. RustFS’s S3 Tables feature contributes the middle layer directly: every bucket can expose a built-in Apache Iceberg REST catalog, on the same server that stores the data. This article walks the stack layer by layer, with results from a hands-on session on RustFS 1.0.1, including the part that failed, which is the part worth reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jm07V5I_2KiOmxiHfn7XbA.png" /></figure><h3>What a lakehouse actually is</h3><p>A lakehouse is an architecture, and the fastest way to understand it is to name the three layers it separates. The table format — Apache Iceberg in this article — defines how a table’s rows, schema, and snapshots are represented as ordinary files. The catalog is the entry point that answers one question reliably: where is the current metadata for table X. Object storage holds both the data files and the metadata files, durably and cheaply.</p><p>None of the three layers needs to know about the others’ internals, and that is the point. A query engine speaks to the catalog, the catalog points at metadata, the metadata points at data files on object storage. Swap any one layer and the others keep working, which is exactly what has made this split the default design for analytics on top of object storage, and why an S3-compatible store with a built-in catalog changes the deployment math.</p><p>One implementation detail is worth knowing before the architecture diagram hardens in your head: in RustFS’s S3 Tables the catalog has no separate database behind it. The default object catalog backing persists catalog state in the same object store, under the reserved .rustfs-table/ prefix. The layers are conceptually separate and physically colocated - lose the bucket and the catalog state goes with it, which moves backup planning from optional to load-bearing.</p><h3>Why the table format became the battleground</h3><p>The table format is the layer where lakehouse projects compete, and Apache Iceberg is an open table format hosted by the Apache Software Foundation. Its design goal is treating a directory of Parquet files as a full database table: schema evolution, hidden partitioning, time travel via snapshots, and atomic swaps of metadata pointers so readers never see half-written state. Version 2 of the spec added row-level deletes, which is what let merge-on-read engines adopt it for upsert workloads.</p><p>RustFS creates Iceberg tables with format-version: 2, confirmed in the create-table response during the hands-on run below. The version 3 specification exists in the Iceberg project, and RustFS&#39;s S3 Tables documentation states v3 is not yet supported. That is a plain fact of current scope, the kind worth knowing before you design a partition spec against features that are not there.</p><h3>Where object storage fits in the stack</h3><p>Object storage is the layer the lakehouse leans on hardest and trusts the least visibly. Every Iceberg table read touches the catalog, then one metadata file, then the manifest list, the manifest files it names, and finally the data files — a fan-out of small reads against the store. The Iceberg spec asks little of the file system: in-place writes, seekable reads, deletes. Consistency leans on the catalog’s atomic metadata swap, not on the store. Engines nonetheless perform many small reads and listings per query, so storage that serves metadata-sized objects quickly is what keeps the architecture responsive.</p><p>The storage layer also owns the failure modes. A lakehouse on object storage is only as durable as the store underneath it, which is precisely why the choice of store is an infrastructure decision, not a checkbox. Running it on hardware you administer means the durability story is your replication and backup plan, and no table format changes that arithmetic.</p><h3>What RustFS S3 Tables provides</h3><p>RustFS ships an Iceberg REST catalog as a built-in feature, enabled per bucket, on an Apache 2.0 licensed server (requires RustFS 1.0.0+). In a hands-on session on 2026–10–11 against RustFS 1.0.1 (single node, Windows x86_64), the full control-plane chain ran clean: enabling the catalog on a bucket returned enabled: true with catalog-uri: /iceberg/v1/my-bucket; creating a namespace and then a table both returned 200; and the table&#39;s metadata-location pointed into the same bucket under the reserved prefix .rustfs-table/, meaning metadata and data live together, with per-table UUID prefixes for isolation.</p><p>Two facts from the same session matter for architecture planning. First, the table response reported rustfs.credential-vending: unsupported. Vending is disabled by default (the documentation describes an opt-in path negotiated via X-Iceberg-Access-Delegation), so this session&#39;s clients authenticated with long-lived keys and SigV4. The consequence is operational: every engine client holds a full set of S3 credentials, and rotation, scoping, and auditing of those keys are your key-management workload rather than something the catalog hands out per session. The documentation states the preconditions (a compatible client that negotiates the header, a caller permitted to request table credentials) and stops there; its own PyIceberg walkthrough still authenticates with explicitly configured credentials, and the 1.0.1 build behind the hands-on session reported the field as unsupported outright. If vended credentials are a deciding factor for your deployment, verify the full credential path on your build before committing to it. Second, namespaces are organizational, not security boundaries - a tenant cannot be fenced with a namespace alone. Isolation has to come from the storage layer: bucket policies, prefix-scoped IAM, and the table-scoped admin actions the documentation defines. Design multi-team deployments accordingly.</p><h3>Which engines connect — and the honest status</h3><p>The catalog speaks the Iceberg REST protocol, so any Iceberg client should be able to point at it. What was actually verified, on the record: PyIceberg 0.12.0 with a SigV4 adapter completed the read path (namespace listing and table load) against the RustFS catalog. The append path failed: the commit was rejected with a missing x-amz-content-sha256 header, because botocore&#39;s generic signer only sets that header for empty bodies. That failure is recorded here as-is; if you build on PyIceberg, pin your versions and run one append before you commit to the stack.</p><p>The project’s own support matrix tracks engine status: PyIceberg and DuckDB marked automated, Spark and Trino covered by a live test harness. Third-party signal cuts both ways. An issue against the catalog documented Spark hitting a SigV4 double-encoding 403 (Iceberg’s signer double-encodes the URI path); it was filed in September 2026 and closed on October 5. Deploying a Spark-Iceberg release that contains the fix matters — older releases still exhibit the 403, so pin the version deliberately and verify it, rather than assuming a recent install includes it. Engines connect; the edges are exactly where you would expect them, and testing your engine against your storage version is not optional.</p><p>One protocol-level note keeps expectations straight. The Iceberg REST catalog spec leaves authentication to implementation choice — the reference flow uses OAuth2, while this catalog authenticates requests with SigV4 signing, which is exactly why the PyIceberg session above needed a signing adapter. Authorization is not carried by the REST protocol here either: namespace and table operations map to RustFS admin actions enforced at the storage layer, and the documentation states plainly that full AWS S3 Tables control-plane compatibility is not claimed. Treat the REST interface as the common dialect between engines and catalogs, not a promise of identical semantics everywhere.</p><h3>What this stack does not do</h3><p>The honest list is short but load-bearing. RustFS S3 Tables includes no SQL engine. There is no embedded query processor; you bring DuckDB, Spark, or Trino, and they speak to the catalog. There are no multi-table transactions: each table commits independently, so cross-table invariants are your application’s problem. Table creation is documented as direct create with v2 by default; staged table creation is listed as unsupported. Dropping a table removes the catalog entry but keeps the underlying objects in place, so cleanup of unreferenced files is a maintenance step you run separately.</p><p>None of these are hidden defects; they are the documented scope of the feature, and every lakehouse stack makes similar trade-offs somewhere. The difference between a stack you can defend and one that pages you at 2 a.m. is whether the limitations were priced in before the first table was created.</p><h3>The ops reality of metadata-heavy tables</h3><p>Iceberg tables accumulate metadata by design: every commit writes a new metadata file, manifests multiply with partition counts, and snapshots pile up until something expires them. The maintenance verbs are part of the format ( expire_snapshots to bound history, remove_orphan_files to collect abandoned writes), and both carry footguns documented in the Iceberg project itself: expiring snapshots destroys the time-travel window you may be relying on for recovery, and orphan-file removal with an aggressive older_than window can race with slow in-flight commits and delete live data.</p><p>On self-hosted storage, add the capacity angle: metadata from many small commits is a real storage cost, and it lands on the disks you provision. Budget for the metadata prefix the way you budget for the data, and put snapshot expiry on a schedule from day one, not after the first “why is this bucket 3x my dataset” investigation.</p><p>The cost is not only capacity. Commit-heavy workloads produce torrents of small reads and writes against the storage layer’s metadata path, and the single-node Windows session behind this article exercises that path gently — a clustered deployment with concurrent writers is a different load profile, which is one more reason the verification note below lists multi-node and concurrent-commit testing as not performed. Where the ceiling sits is a fair question with no published answer: no RustFS small-object benchmark at metadata scale exists to cite, and the honest range depends on drives, erasure-set width, network, and client mix. The practical move is to run a small-object mixed-workload benchmark against the exact cluster hardware before a lakehouse lands on it, and treat the measured number, not a datasheet, as the design input.</p><p>Backup deserves its own line in the runbook. Because catalog state and table metadata are ordinary objects under .rustfs-table/, a bucket-level backup covers the catalog automatically; a plan that copies only data files does not. It also pays to separate what the storage layer already gives you from what a backup adds. A distributed RustFS cluster stores every object as Reed-Solomon erasure-coded shards across an erasure set, with per-shard checksums and healing, which protects against disk, node, and bitrot loss inside the cluster. It does not protect against a DELETE that should not have happened, a pipeline writing garbage, or corruption applied consistently across shards; those are logical losses, and an independent copy is the only answer. Copy strategies differ in recovery point and blast radius: a scheduled snapshot copy bounds RPO at the schedule interval and restores by re-upload, while a continuously synced second bucket bounds RPO near zero at the cost of propagating mistakes just as fast, deletes included, unless the target keeps versions. The documentation reviewed for this article does not cover a cross-site replication feature, so treat that leg as something to verify on your build or to assemble with an S3-sync tool against the API. Two more facts from the documentation are worth pricing before you need them under pressure. Recovery has an escape hatch: Iceberg clients can register an existing metadata location as a table, so a lost catalog entry is recoverable when the metadata files survive. And there is an explicit upgrade red line: after the durable-strong snapshot format v2 is persisted, rolling back to a binary that only reads v1 fails closed - check the cutover runbook before upgrading any deployment that carries data you cannot lose.</p><p>The two catalog backings are a deliberate choice, not a default to forget. The default object backing keeps catalog state in the bucket, and the documentation describes its concurrency model in one line: a commit validates its base metadata and referenced objects before conditionally updating the metadata pointer, a conflicting writer must reload the table and resolve the conflict, and the transaction boundary is one table. The durable-strong mode is the alternative, and the documentation prices its adoption rather than advertising it - a cutover runbook with migration preflight, coordinated writer fencing, and the rollback red line above. No threshold for making the switch is published, so the honest sequence is to run the default backing, watch how conflict retries behave under your real write concurrency, and pay the cutover cost only when the reload-and-resolve path is measurably in the way.</p><p>Monitoring has a documentation anchor too. The counters that matter here are derivable from documented behavior: commit conflicts (the reload-and-resolve path), snapshot count per table, and object count plus byte size under the .rustfs-table/ prefix. One more fact raises the stakes: metadata deletion and background maintenance are disabled by default, and RustFS does not run a built-in periodic maintenance scheduler - expiry jobs you schedule yourself only happen if something is watching them happen.</p><h3>When a lakehouse on self-hosted object storage makes sense</h3><p>The stack earns its keep in three situations. First: your data governance or cost profile already points at self-hosted storage, and adding a table format plus catalog turns that store into an analytics substrate instead of a backup target. Second: you want Iceberg’s engine portability without a hyperscaler’s managed-table billing; the same table reads from DuckDB locally and Spark on a cluster. Third: your tables are already large enough that the metadata fan-out matters, which is where a storage layer you can size and monitor yourself pays off.</p><p>The built-in catalog is also not the only catalog. The documentation ships a second, explicit pattern: RustFS as the object storage under a separate Iceberg REST catalog service (the external-catalog integration, demonstrated with Spark and a standalone REST container). The built-in catalog is scoped per bucket — its warehouse is a single bucket name — so if one catalog needs to span several buckets or stores, the external pattern is the documented route. The trade is one more stateful service that you run, upgrade, and back up yourself; which pattern fits depends mainly on whether your catalog needs to outlive one bucket.</p><p>If none of those hold (small tables, one engine, no storage constraints), a plain bucket with files works, and the lakehouse layer is complexity without a customer. The format is not the product; the workload is.</p><h3>Frequently Asked Questions</h3><h3>What is an Apache Iceberg lakehouse?</h3><p>An analytics architecture that separates three layers: the Apache Iceberg table format (how tables are represented as files), a catalog (where table metadata lives), and object storage (where data and metadata are durably kept). The separation lets engines like DuckDB, Spark, and Trino read the same tables without owning the storage.</p><h3>Do I need a separate catalog server for Iceberg on RustFS?</h3><p>No. RustFS S3 Tables enables a built-in Iceberg REST catalog per bucket — the hands-on run enabled it on an existing bucket with a single request, and the catalog URI was served from the same server at /iceberg/v1/&lt;bucket&gt;. Metadata files land in the same bucket under the .rustfs-table/ prefix.</p><h3>Can PyIceberg write to RustFS S3 Tables?</h3><p>Reads were verified working in a first-hand session (PyIceberg 0.12.0, SigV4). Appends were rejected with a missing x-amz-content-sha256 header error, a signing quirk between botocore&#39;s generic signer and the catalog endpoint. Test one append with your pinned versions before building on it.</p><h3>Does RustFS include a SQL query engine?</h3><p>No. RustFS S3 Tables provides the table format layer and the catalog; query engines are external — DuckDB, Spark, Trino, or any Iceberg REST client. The project’s support matrix tracks which engines are exercised against the catalog.</p><h3>How is this different from AWS S3 Tables?</h3><p>AWS S3 Tables is a managed AWS feature: Amazon describes it as fully managed, meaning AWS operates the catalog and its maintenance and you never manage metadata infrastructure yourself. RustFS’s implementation is the self-hosted path — the same Iceberg REST interface, on an Apache 2.0 server you run, with the default object catalog backing persisting catalog state in your own bucket. That shifts the metadata failure domain and the backup plan onto your side of the table, which is the deployment trade being made, not a defect ranking.</p><h3>A note on how this was verified</h3><p>Article type: architecture analysis anchored on a first-hand verification session. The hands-on results quoted here (catalog enable, namespace and table creation, PyIceberg read success and append failure) were run on 2026–10–11 against RustFS 1.0.1, single node on Windows, with curl SigV4 and PyIceberg 0.12.0; the full method and raw outputs are documented in the companion article on the S3 Tables catalog. Not verified for this article: Spark or Trino against the live catalog, multi-node clusters, concurrent commits, snapshot expiry behavior, and production-scale metadata volume. Iceberg spec and maintenance behaviors are cited from the Apache Iceberg project documentation; check current versions before designing partition specs.</p><p><em>Originally published at </em><a href="https://dev.to/ethan-carter/apache-iceberg-lakehouse-rustfs-s3-tables-in-practice-1l54"><em>https://dev.to</em></a><em> on October 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=51bb1b97fb19" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RustFS S3 Tables: A Built-In Iceberg REST Catalog in an Apache-2.0 Object Store]]></title>
            <link>https://medium.com/@ethan-carter/rustfs-s3-tables-a-built-in-iceberg-rest-catalog-in-an-apache-2-0-object-store-61bc732ad46b?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/61bc732ad46b</guid>
            <category><![CDATA[cloud-object-storage]]></category>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[s3-tables]]></category>
            <category><![CDATA[object-storage]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Mon, 12 Oct 2026 13:18:44 GMT</pubDate>
            <atom:updated>2026-10-16T03:51:06.033Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*lPKjiS-lnWFlMBjU" /></figure><p>RustFS S3 Tables is an Apache Iceberg REST Catalog built directly into the RustFS object storage kernel, so one process on one port serves both standard S3 and the Iceberg REST Catalog protocol. That removes the separate Hive Metastore or Glue catalog a self-hosted lakehouse normally needs. It is Apache 2.0 with no separate commercial edition.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wXrAsJ3lnHqYDnQ8kJ_T-Q.png" /></figure><h3>What is RustFS S3 Tables?</h3><p>RustFS S3 Tables is a catalog service embedded in the storage server rather than deployed next to it. An Iceberg client uses the REST catalog to discover tables and commit metadata changes. It uses the S3 API to read and write the underlying Parquet files. RustFS serves both interfaces on the S3 API port.</p><p>That collapses one moving part: a typical self-hosted lakehouse pairs the object store with a catalog such as Hive Metastore, AWS Glue, Nessie, or Polaris, and RustFS folds the catalog into the store. Because the catalog speaks the open Iceberg REST protocol, engines with an Iceberg REST client need no vendor plugin — Apache Spark, PyIceberg, and DuckDB are documented as connecting without one, and enabling a table bucket does not require registering existing Parquet files first. One consequence worth stating plainly: the catalog’s metadata lives inside RustFS itself (Iceberg metadata JSON and manifests written to a reserved prefix on the same bucket), so there is no separate metadata failure domain. Your RustFS durability and backup story is also your catalog backup story, and you cannot fall back to an independent metastore.</p><h3>The control plane and data plane split</h3><p>Writes separate into two paths. Table creation and snapshot commits travel through the control plane, which is the Iceberg REST Catalog. Parquet data files are written by the compute engine straight into the bucket over the S3 API, with no intermediate layer in the data path.</p><p>Two details keep that boundary safe. Metadata pointers live under a reserved internal prefix, so ordinary S3 requests cannot create, overwrite, or delete catalog metadata; a stray aws s3 cp will not corrupt table state. On commits, the catalog runs compare-and-swap version checks to reject stale concurrent writers and uses idempotency keys to absorb duplicate submissions. The transaction boundary is one table. RustFS does not claim multi-table atomic transactions, and the limit is concrete: you cannot commit a schema change to one table and a data load to a second table in a single atomic operation. Workloads that need cross-table consistency - updating a dimension table and a fact table together, for example - must coordinate that in application logic or accept eventual consistency.</p><h3>From an empty bucket to a working table</h3><p>The official walkthrough starts with a dedicated bucket, then flips it into a table bucket with a signed request that carries an empty body. Using my-bucket on localhost:9000:</p><pre>export RUSTFS_ENDPOINT=&quot;http://localhost:9000&quot;<br>export AWS_ACCESS_KEY_ID=&quot;&lt;your-access-key&gt;&quot;<br>export AWS_SECRET_ACCESS_KEY=&quot;&lt;your-secret-key&gt;&quot;<br>export AWS_DEFAULT_REGION=&quot;us-east-1&quot;<br><br>aws --endpoint-url &quot;$RUSTFS_ENDPOINT&quot; s3api create-bucket --bucket my-bucket<br><br>curl --fail-with-body --silent --show-error \<br>  --aws-sigv4 &quot;aws:amz:us-east-1:s3&quot; \<br>  --user &quot;$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY&quot; \<br>  --header &quot;x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855&quot; \<br>  --request PUT &quot;$RUSTFS_ENDPOINT/iceberg/v1/buckets/my-bucket&quot;</pre><p>Both the enable call and the GET status check return HTTP 200, with enabled: true and catalog-type: iceberg-rest in the response. I confirmed this end to end on a single-node RustFS 1.0.1 run; see the note at the end for exactly what was executed.</p><p>Namespaces and tables are plain REST calls under the warehouse prefix. Namespace creation with curl:</p><pre>curl --fail-with-body --silent --show-error \<br>  --aws-sigv4 &quot;aws:amz:us-east-1:s3&quot; \<br>  --user &quot;$AWS_ACCESS_KEY_ID:$AWS_SECRET_ACCESS_KEY&quot; \<br>  --header &quot;Content-Type: application/json&quot; \<br>  --header &quot;x-amz-content-sha256: UNSIGNED-PAYLOAD&quot; \<br>  --request POST &quot;$RUSTFS_ENDPOINT/iceberg/v1/my-bucket/namespaces&quot; \<br>  --data &#39;{&quot;namespace&quot;:[&quot;my-namespace&quot;]}&#39;<br># {&quot;namespace&quot;:[&quot;my-namespace&quot;],&quot;properties&quot;:{}}</pre><p>One signing detail cost me a retry: the enable call signs an empty body, so it carries the well-known empty-sha256 header, but reusing that header on the namespace POST (which carries a JSON body) is rejected with Sha256Mismatch (HTTP 400). Body-carrying requests need x-amz-content-sha256: UNSIGNED-PAYLOAD or the actual body hash instead.</p><p>Table creation follows the same pattern against /namespaces/my-namespace/tables and returns the full Iceberg metadata document. Three fields in that response are worth reading: metadata-location points at the metadata JSON under a reserved .rustfs-table/ prefix on the same bucket, which is visible confirmation that catalog state lives in ordinary S3 objects next to your data; metadata.format-version is 2, matching the documented v2 default; and config reports rustfs.credential-vending: unsupported, so the client must hold its own S3 credentials rather than receiving temporary ones from the catalog.</p><p>From there an Iceberg client needs four settings. The REST catalog URI is http://localhost:9000/iceberg, and the client appends /v1. The warehouse is the bucket name, not an S3 URI or an AWS ARN. REST auth is SigV4 with signing name s3 and region us-east-1. The S3 file endpoint is the same host, with path-style addressing.</p><p>The docs flag one trap that first-timers hit: REST request signing and S3 file access are configured separately, even for the same account. Configure only the S3 side and the client returns 403 until you add the catalog’s SigV4 credentials.</p><h3>Which engines work today?</h3><p>RustFS maintains a support matrix with different confidence levels per client. Two are automated. PyIceberg covers namespace and table creation, append, reload, scan, and catalog operations. DuckDB Iceberg 1.5.5 covers single-table create, insert, update, delete, and merge, plus schema evolution, snapshots, and concurrent writes.</p><p>Apache Spark has an opt-in live harness, so validate your exact Spark and Iceberg versions. Trino has a manual read-only probe and write compatibility is not claimed. StarRocks is documented as an external-catalog read path only. Databend has a live harness limited to an S3-stage read probe of table data files, with no claimed Iceberg REST integration.</p><p>At the endpoint level, /iceberg/v1 is the canonical prefix and /_iceberg/v1 is a MinIO AIStor-style compatibility alias. RustFS supports both.</p><p>My own run puts one caveat on the matrix’s confidence levels. With PyIceberg 0.12.0 against RustFS 1.0.1, the catalog read path works over SigV4: namespace listing and table load return clean responses. The append commit does not. PyIceberg signs catalog requests with botocore’s generic SigV4 signer, which sets the S3-style x-amz-content-sha256 header on empty-body requests but not on body-carrying ones; RustFS requires that header on its catalog endpoints, so the commit is rejected with missing header: x-amz-content-sha256. The matrix lists PyIceberg as automated-tested, so treat this as a version-sensitive area rather than a permanent gap: pin your PyIceberg version and run one append before you build on it. Signing is where client compatibility actually breaks today. A separate, still-open issue (#8103, filed September 2026) reports Spark failing metadata-table lookups for a different signing reason: Iceberg&#39;s signer double-encodes a URI path that RustFS canonicalizes the S3 way, turning what should be a 404 into a 403.</p><h3>What’s in scope, and what isn’t</h3><p>Before you build on it, here is the scope RustFS documents. Iceberg format v1 and v2 are supported with v2 as the default. Staged table creation, purge-on-drop, and Iceberg v3 are not supported. There is no SQL execution engine, no multi-table atomic transactions, and no independent active-active writes across regions. RustFS implements the open Iceberg REST Catalog protocol and does not claim full AWS S3 Tables control-plane compatibility, so “open-source S3 Tables” means a free, open-protocol Iceberg catalog rather than a drop-in AWS S3 Tables clone. Metadata deletion and background maintenance are off by default, with no built-in periodic scheduler. Dropping a table removes the catalog entry but keeps the underlying objects, so recursive deletion of paths still referenced by snapshots would lose data.</p><h3>Where the catalog state lives, and how it fails closed</h3><p>The default catalog backing is object-backed: RustFS persists catalog state as ordinary S3 objects — Iceberg metadata JSON, manifests, and the current pointer file — under a reserved internal prefix on the same bucket, not in a separate embedded database. That is why the backup story for your data is also the backup story for your catalog. The backing is configurable with RUSTFS_TABLE_CATALOG_BACKING; a durable-strong mode exists for deployments that want stronger durability than the default object-backed state, at the cost of a more involved cutover.</p><p>Because the catalog is objects plus a CAS-driven pointer, it fails closed by design. The support matrix tracks concrete cases: a stale commit token or stale base metadata returns a conflict without advancing the table pointer; missing metadata, manifest, data, or delete objects fail closed before a commit or maintenance step can advance state; concurrent writers resolve to a single winning CAS with retryable conflicts for the losers; catalog and ordinary S3 permission denials both block a data-plane bypass; and a stale maintenance plan fails closed before any object deletion. Across those cases the direction is the same: the system refuses to advance rather than risk corruption, and each guarantee is tied to a specific RustFS build and client version, not something to assume without testing your exact stack.</p><p>Upgrade risk concentrates in the durability-strong cutover. Moving to durable-strong backing requires a preflight reporting SNAPSHOT_MATERIALIZED with no blockers, an exclusive migration fence that drains in-flight writes, and a restart before admitting writers. Snapshot format uses a two-gate switch ( RUSTFS_TABLE_CATALOG_STRONG_SNAPSHOT_V2 plus a fleet-confirmed gate); once a version-2 snapshot is persisted, rolling catalog writers back to a binary that only reads version 1 fails closed rather than replacing in-process state. The rollback red line is explicit: after v2 is on disk, do not revert to a v1-only binary. Keep the object-backed catalog backup in your restore set until durable-strong has cleared your retention window, and rehearse the disaster-recovery path on an isolated bucket before it carries data you cannot lose.</p><h3>RustFS S3 Tables next to the alternatives</h3><p><strong>MinIO AIStor Tables.</strong> MinIO offers a comparable embedded-catalog capability, but only inside AIStor, its commercially licensed product line. AIStor is a separate binary from the AGPL-3.0 MinIO Object Store, and MinIO prices AIStor by capacity at roughly $0.02 per GB per month. The MinIO Object Store entered maintenance mode in December 2025 and no longer takes new features, so the open-source line does not carry this capability. RustFS ships the same class of capability in its Apache-2.0 core, with no community-versus-enterprise split. Licensing and deployment model are where they differ, not feature scope: if you want to drop the separate catalog without taking on a commercial license, RustFS is the lower-friction option, while AIStor brings a more structured commercial-support and compliance story.</p><p><strong>Standalone open-source catalogs (Apache Polaris, Project Nessie).</strong> Polaris and Nessie are independent, Apache-2.0 REST catalogs you deploy and operate yourself, and both sit in front of any S3-compatible store. They are the genuine alternative if you want catalog/object separation rather than embedding: one catalog can serve several storage backends, and you control its versioning and access policy directly. The cost is exactly the moving part RustFS removes — you run, secure, back up, and upgrade a second service, and you own the wire between catalog and storage. RustFS trades that separation for a single process with one upgrade surface and one backup domain.</p><p>On maturity the trade is real: Polaris and Nessie are established projects with broad production adoption and active communities, while RustFS S3 Tables shipped with the 1.0.0 GA and has a thinner third-party deployment track record so far. If an existing operational playbook and catalog maturity weigh more than avoiding a second service, the standalone catalogs are the lower-risk starting point today; RustFS is the bet on a smaller operational footprint that is still closing that gap.</p><p><strong>How to choose.</strong> The line is “embed vs separate,” not “RustFS vs everyone else.” If your priority is not running an extra service, RustFS or AIStor fit. If you already standardize on Polaris or Nessie across several backends, RustFS can sit underneath as just the S3 layer. If you need a managed, vendor-run catalog, the comparison shifts to AWS Glue or a cloud warehouse’s native catalog rather than self-hosted Polaris/Nessie or RustFS.</p><h3>Before you take this to production</h3><p>Pin the version. Use RustFS 1.0.0 or later with a fixed tag rather than latest, and test in an isolated bucket. Give the table bucket its own lifecycle thinking, because table buckets are excluded from ordinary lifecycle expiration and enabling the mode on an existing bucket changes how those rules apply.</p><p>Configure both credential paths. Use least-privilege IAM for the catalog actions ( admin:SetTableBucket, admin:GetTableBucket, and the namespace and table actions) plus separate S3 permissions for table file reads and writes. Leave metadata deletion and background maintenance off until you have reviewed the retained references, and never recursively delete S3 paths that snapshots may still point to. Start with DuckDB or PyIceberg on a single node to prove create, write, read, and drop before you scale out.</p><p>Namespaces group tables, but most of the isolation that matters lands on S3 prefixes, not on the namespace itself. The bucket-level enable response reports credential-scope: warehouse-prefix, and each table&#39;s config narrows it to a per-table UUID prefix ( s3://my-bucket/tables/&lt;uuid&gt; in my run), which is the scope the catalog expects client credentials to cover. Two consequences follow. You cannot pre-provision a data-access policy for a table&#39;s prefix before that table exists, so namespace-level separation gives you organization rather than enforcement. And the realistic granularity options are coarser than the object model suggests: one credential set per producing application, strict prefix conventions, or a bucket per tenant when boundaries must be hard. Keep the catalog admin actions on a separate identity from data-plane writers either way.</p><p>Plan metadata maintenance up front. Metadata deletion and background maintenance are off by default and there is no built-in periodic scheduler, so dropping a table removes only the catalog entry and leaves the underlying Parquet files in place — live snapshots keep referencing them and keep occupying storage. Before a table carries real volume, set up a periodic cleanup job (your engine’s expire_snapshots / remove_orphan_files, or a scheduled sweep of unreferenced paths) and never recursively delete S3 paths that live snapshots still reference.</p><p>The cleanup jobs themselves have teeth. remove_orphan_files deletes any data file current metadata does not reference, so a writer that crashes or pauses mid-commit leaves uncommitted files behind, and a cleanup pass with an aggressive older_than window can delete files a writer is about to commit - the Iceberg docs default that window to days, not minutes, for exactly this reason. expire_snapshots shrinks history on purpose, which is precisely what breaks point-in-time recovery: expire snapshots before your backup has copied them, and a restore lands on a newer state than the one you meant to keep. And every commit writes a fresh metadata JSON plus manifest list, so a table fed by many small commits accumulates metadata objects even while its data barely grows; until snapshots expire, all of it stays in the prefix your catalog reads on every table load.</p><p>Treat the catalog’s reserved prefix as part of your backup set, since there is no independent metadata store to restore from.</p><h3>FAQ</h3><h3>Is RustFS S3 Tables free and open source?</h3><p>Yes. RustFS is Apache 2.0, and S3 Tables ships as a built-in feature of the storage server with no separate commercial edition.</p><h3>Do I still need Hive Metastore or AWS Glue?</h3><p>No. The Iceberg REST Catalog runs inside RustFS, so you do not deploy or operate a separate catalog service.</p><h3>Which query engines work with RustFS S3 Tables?</h3><p>Any client that speaks the Iceberg REST Catalog protocol. PyIceberg and DuckDB are automated in the RustFS test matrix; Spark and Trino have live harnesses; StarRocks and Databend are narrower.</p><h3>Is RustFS S3 Tables production-ready?</h3><p>It is a built-in feature of the Apache-2.0 storage server with no separate commercial edition. As with any feature, validate it against your own workload and keep a rollback path before it carries critical data.</p><h3>How does RustFS S3 Tables differ from AWS S3 Tables?</h3><p>AWS S3 Tables is a managed commercial service. RustFS implements the open Iceberg REST Catalog protocol on a self-hosted, Apache-2.0 stack and does not claim full AWS S3 Tables control-plane compatibility.</p><h3>A note on how this was verified</h3><p><strong>Article type:</strong> hands-on control-plane walkthrough plus a systematic read of the official RustFS documentation, GA announcement, and published support matrix.</p><p>The walkthrough above was executed first-hand on a single-node RustFS 1.0.1 (Windows x86_64 binary, release build dated 2026–10–03) on 2026–10–11: bucket creation, table-bucket enablement, status check, namespace creation, table creation, and table load. A PyIceberg 0.12.0 client was also connected over SigV4: catalog reads (namespace listing, table load) succeeded, and the first append commit was rejected with the missing header: x-amz-content-sha256 error quoted in this article. The responses quoted here are taken from those runs, including the Sha256Mismatch rejection. Not exercised: successful engine-side data writes or scans (the commit path blocked them), DuckDB, concurrency benchmarking of metadata commits, large-volume snapshot or metadata-growth stress testing, failure-injection recovery drills, and the durable-strong cutover with its v2 snapshot gate. The operational cautions reflect documented behavior and known limits, not first-hand war stories. Validate against your own workload and keep a rollback path before it carries data you cannot lose.</p><h3>Sources</h3><ol><li>RustFS GA announcement, “RustFS 1.0.0 GA” (2026–09–16): <a href="https://rustfs.com/blog/announcing-rustfs-1-0-0-ga/">https://rustfs.com/blog/announcing-rustfs-1-0-0-ga/</a></li><li>RustFS S3 Tables documentation (EN): <a href="https://docs.rustfs.com/en/administration/data/s3-tables">https://docs.rustfs.com/en/administration/data/s3-tables</a></li><li>RustFS S3 Tables support matrix (main): <a href="https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-tables-support-matrix.md">https://github.com/rustfs/rustfs/blob/main/docs/architecture/s3-tables-support-matrix.md</a></li><li>RustFS blog, “RustFS Launches S3 Tables” (2026–09–10): <a href="https://rustfs.com/blog/rustfs-s3-tables-iceberg-rest-catalog-quickstart">https://rustfs.com/blog/rustfs-s3-tables-iceberg-rest-catalog-quickstart</a></li><li>MinIO AIStor feature set and pricing: <a href="https://www.min.io/blog/aistor-overview">https://www.min.io/blog/aistor-overview</a></li><li>RustFS release 1.0.1 binaries (Windows x86_64, used for the hands-on run): <a href="https://github.com/rustfs/rustfs/releases/tag/1.0.1">https://github.com/rustfs/rustfs/releases/tag/1.0.1</a></li><li>rustfs/rustfs issue #8103, Spark SigV4 metadata-table failure (open as of 2026–10–11): <a href="https://github.com/rustfs/rustfs/issues/8103">https://github.com/rustfs/rustfs/issues/8103</a></li><li>Apache Iceberg maintenance procedures (remove_orphan_files / expire_snapshots semantics and defaults): <a href="https://iceberg.apache.org/docs/latest/spark-procedures/">https://iceberg.apache.org/docs/latest/spark-procedures/</a></li></ol><p><em>Originally published at </em><a href="https://dev.to/ethan-carter/rustfs-s3-tables-a-built-in-iceberg-rest-catalog-in-an-apache-20-object-store-1294"><em>https://dev.to</em></a><em> on October 12, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=61bc732ad46b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SeaweedFS vs RustFS: When a Tiny Footprint Wins]]></title>
            <link>https://medium.com/@ethan-carter/seaweedfs-vs-rustfs-when-a-tiny-footprint-wins-d891c60d1a8f?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d891c60d1a8f</guid>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Sat, 19 Sep 2026 01:52:32 GMT</pubDate>
            <atom:updated>2026-10-11T00:43:21.142Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ZTHq6zziMZYdwTxG.png" /></figure><p>SeaweedFS and RustFS are both Apache 2.0, single-binary stores, but they solve different problems. SeaweedFS is a distributed file system that also speaks S3, with a 16-byte per-object index that lets one master track billions of files. RustFS is a purpose-built, MinIO-compatible S3 object store with a web console. Pick SeaweedFS when you need a POSIX/FUSE mount, an Iceberg lakehouse, or billions of small files; pick RustFS when S3 is your only interface and a drop-in MinIO replacement in one docker run is the goal.</p><h3>Key stats</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*G_EG5P8L7IvyGxxjWq9sFA.png" /></figure><p>The SeaweedFS throughput figure is its own README’s unscientific single-machine number: 1 million 1KB files at concurrency 16 on a MacBook with an SSD. We cite it as a scale signal, not a hardware review.</p><h3>What is the real difference between SeaweedFS and RustFS?</h3><p>SeaweedFS is a simple and highly scalable distributed file system. Its own README says one weed binary &quot;serves an S3 object store, a POSIX file system, and a lakehouse with S3 Tables, all over the same data.&quot; Under the hood, a master server tracks volumes (not individual files), volume servers store blobs in append-only volume files, and a filer exposes the data as a POSIX file system, an S3 endpoint, and an Iceberg catalog.</p><p>RustFS is narrower by design. It is S3-compatible object storage written in Rust and released under the Apache 2.0 license. Its README’s Feature &amp; Status table marks S3 Core, Versioning, Object Lock (WORM), Lifecycle Management, Bucket Replication, Site Replication, IAM/Policies, Server-Side Encryption, and Distributed Mode as available today, with S3 Tables (Iceberg REST) in preview. There is no POSIX layer and no FUSE mount. The difference is the whole story: SeaweedFS is a file system that also does S3; RustFS is an S3 engine that does one thing.</p><h3>How small is the metadata footprint, really?</h3><p>This is where “tiny footprint” stops being marketing. SeaweedFS packs small files into append-only volume files, so there is no per-file inode and no per-file metadata file. The volume server keeps a 16-byte index entry per blob in memory and reads it in a single seek, even for erasure-coded data; on disk it stores 40 bytes of metadata per file. Because the master tracks volumes rather than files, a cluster holding billions of files still has only a few thousand volumes, so the master stays small and never sits in the read path — clients cache the volume-to-server mapping and talk to volume servers directly.</p><p>RustFS is a single Rust binary or container. The documented start is one docker run; there is no monitor quorum, no placement groups, no daemon zoo to reason about. We do not publish a per-object index size, and the comparison is not apples-to-apples — RustFS optimizes for S3 semantics, not for billions of tiny files on a single file system. The footprint win is real on different axes: SeaweedFS wins raw metadata scale; RustFS wins operational simplicity for an S3-only team.</p><h3>How does each one deploy?</h3><p>SeaweedFS ships as a single weed binary. The fastest path is weed mini, which starts the master, a volume server, the filer, WebDAV, the Iceberg REST catalog, and the Admin UI in one process, and exposes the S3 endpoint at <a href="http://localhost:8333:">http://localhost:8333:</a></p><pre>AWS_ACCESS_KEY_ID=admin \<br>AWS_SECRET_ACCESS_KEY=secret \<br>S3_BUCKET=my-bucket \<br>./weed mini -dir=./data</pre><p>To grow capacity, point more volume servers at the master:</p><pre>weed volume -dir=/data -master=&lt;master_host&gt;:9333</pre><p>RustFS ships as a single binary or a container. The documented start command is:</p><pre>docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest</pre><p>That one line serves the S3 API on port 9000 and a console on 9001. For bare metal there is a one-line installer:</p><pre>curl -O https://rustfs.com/install_rustfs.sh &amp;&amp; bash install_rustfs.sh</pre><p>Default credentials are rustfsadmin / rustfsadmin, which you must rotate before any production exposure.</p><h3>When should you pick SeaweedFS over RustFS?</h3><p>Reach for SeaweedFS when the workload is not “objects” but “files at extreme scale.” Its design target — store billions of files, serve them fast — shows up in the numbers: a 16-byte per-blob index, 40 bytes per file on disk, and a master that tracks volumes, not files. If you are building a content-addressable store, a photo or video library, or an ML feature store with tens of millions of tiny shards, that metadata model is the difference between a master that fits in RAM and one that does not.</p><p>SeaweedFS also wins the moment you need a file system or a lakehouse over the same bytes. One weed binary serves an S3 endpoint, a FUSE mount on Linux/macOS/Windows, WebDAV, SFTP, HDFS, and an Iceberg REST catalog, all addressing the same data. If a data scientist wants to ls the lakehouse while an app writes through S3, SeaweedFS does both without a sync job. RustFS does not offer a FUSE/POSIX mount, so that requirement alone steers you to SeaweedFS.</p><h3>Where does RustFS win?</h3><p>RustFS wins on the axis most S3 teams actually care about: a clean, MinIO-compatible S3 endpoint with the least operational surface. One docker run, no quorum, no volume-server choreography. The feature set is now broad — Object Lock (WORM), Lifecycle Management, Bucket and Site Replication, IAM/Policies, OIDC/SSO, Server-Side Encryption, and Distributed Mode are all marked available in the README, with MinIO On-Disk Compatibility in preview. If your tools already speak MinIO’s S3 dialect, RustFS is a drop-in.</p><p>The Apache 2.0 license is the same as SeaweedFS’s, so licensing is not the differentiator. The differentiator is focus: RustFS spends its complexity budget on S3 correctness and the web console, not on a file system layer. For a team whose only contract with storage is the S3 API, that focus means fewer knobs to misconfigure and a shorter path to a working endpoint.</p><h3>Feature-by-feature: what the docs say</h3><p>The table below is a straight read of each project’s own documentation as of September 2026. SeaweedFS rows come from its GitHub README; RustFS rows come from its README Feature &amp; Status table (2026–09–18). No benchmarks, no marketing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V2itjjl2esByw8yhmHIV1w.png" /></figure><p>Two rows deserve a call-out. RustFS lists S3 Tables (Iceberg REST) as Preview, so if a mature lakehouse over the same bytes is the requirement, SeaweedFS is ahead today. And SeaweedFS’s erasure coding runs on warm data in the background, so writes never pay the encoding cost up front — a model RustFS does not yet offer in its feature table.</p><h3>What can RustFS not replace yet?</h3><p>Be clear about the boundaries before you commit. RustFS’s README has no FUSE/POSIX mount, so if you need a file system interface or a mounted lakehouse, SeaweedFS (or CephFS) is the answer today. RustFS’s S3 Tables (Iceberg REST) is in preview, so a production lakehouse over RustFS data is not a promise the project makes yet. And while Distributed Mode is now marked available, a RustFS cluster at SeaweedFS-scale file counts is a different beast — SeaweedFS was built from the start to keep the master small at billions of files, and RustFS does not publish an equivalent per-object index claim.</p><p>None of this is a knock on RustFS. It is a scope statement: RustFS is an S3 engine, and it is good at being one. Pushing it into the file-system or exabyte-file-count role is the wrong tool for that job.</p><h3>How do you choose between them?</h3><p>Use this rule of thumb: if the words “file system,” “FUSE,” or “lakehouse” appear in the ticket, start with SeaweedFS. If only “S3” appears, and the real constraint is headcount, time, or MinIO compatibility, start with RustFS.</p><p>A split is legitimate and common. Run SeaweedFS for the data-lakehouse and file-serving layer where billions of files and a POSIX mount pay off, and run RustFS in front of the S3-facing applications where a MinIO-compatible endpoint and a web console are the whole ask. They do not conflict; they occupy different layers. The mistake is forcing one to do the other’s job — SeaweedFS dragged into a pure-S3, small-team role carries a file system you will never mount, and RustFS pushed into a POSIX role it does not implement becomes a custom FUSE you have to build yourself.</p><h3>FAQ</h3><h3>Is SeaweedFS a drop-in replacement for MinIO?</h3><p>Not directly. SeaweedFS implements S3 as one of several interfaces on top of its distributed file system, so most S3 clients work, but it is not built to mirror MinIO’s S3 dialect command for command. RustFS, by contrast, lists MinIO On-Disk Compatibility as a preview feature and targets the MinIO S3 dialect as its primary surface. If your only requirement is S3 and you want MinIO-compatible tooling, RustFS is the closer fit.</p><h3>Does SeaweedFS support erasure coding?</h3><p>Yes. SeaweedFS replicates hot data for speed and applies erasure coding to warm data in the background, so writes never pay the encoding cost up front. Volume servers replicate or erasure-code at the volume level, and the in-memory index stays 16 bytes per blob even for erasure-coded data. RustFS’s feature table does not list erasure coding today.</p><h3>Does RustFS have a FUSE mount?</h3><p>No. RustFS’s README Feature &amp; Status table has no FUSE/POSIX mount, so there is no file system interface. If you need to mount storage as a local directory or expose a lakehouse over the same bytes, SeaweedFS provides a FUSE mount on Linux, macOS, and Windows. For pure S3 object access, the missing mount is irrelevant.</p><h3>Which one scales to more files?</h3><p>SeaweedFS is designed for it: the master tracks volumes rather than files, so a cluster with billions of files holds only a few thousand volumes and the master stays small. RustFS is a capable S3 object store, but it does not publish a comparable per-object index claim, and its strength is S3 semantics at operationally simple scale rather than billions of tiny files on one file system.</p><h3>Are both projects Apache 2.0?</h3><p>Yes. Both SeaweedFS and RustFS are released under the Apache License 2.0, a permissive license with no copyleft. If you run either as a service the practical difference is small; if you embed the engine in a shipped product, Apache 2.0 spares you the copyleft questions that a GPL/LGPL project would raise.</p><p><em>Originally published at </em><a href="https://dev.to/ethan-carter/seaweedfs-vs-rustfs-when-a-tiny-footprint-wins-20f5"><em>https://dev.to</em></a><em> on September 19, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d891c60d1a8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RustFS 1.0 Is GA After 2.5 Years: What’s Actually Production-Ready, and What Isn’t]]></title>
            <link>https://medium.com/@ethan-carter/rustfs-1-0-is-ga-after-2-5-years-whats-actually-production-ready-and-what-isn-t-ab3c76e31368?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab3c76e31368</guid>
            <category><![CDATA[ai-storage]]></category>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[rustfs]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Thu, 17 Sep 2026 14:08:36 GMT</pubDate>
            <atom:updated>2026-09-18T00:34:15.044Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qT4WRFuV3SNOkxTE.png" /></figure><p>RustFS 1.0.0 reached general availability on September 16, 2026. The core S3-compatible object storage engine is stable and supported for production. RustFS is a distributed, Apache 2.0-licensed object store written in Rust, now with 32,000+ GitHub stars and 2.7 million+ deployed instances worldwide.</p><h3>Key stats</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/879/1*jyYeUUIunLCgFLva03sdrw.png" /></figure><h3>RustFS 1.0.0 GA: what actually shipped</h3><p>RustFS 1.0.0 is the first generally available release of the project. “GA” here means one specific thing: the core object storage engine (PUT, GET, multipart upload, versioning, replication, and the S3 API surface) is stable and supported for production traffic. The team spent two years and seven months getting there, from the first commit in February 2024 to open sourcing in July 2025, then Beta in April 2026 and RC in August 2026.</p><p>What shipped is not a rewrite of someone else’s storage engine. RustFS is built from scratch in Rust, which the team chose for memory safety and concurrency under heavy write loads. GA matters because it lets you stop evaluating RustFS and start running it where downtime costs money. The release also closes a promise the project made at open source: a self-hosted, S3-compatible store with no commercial license gate on the core engine.</p><h3>What is RustFS, and why Rust?</h3><p>RustFS is a distributed object storage system that speaks the S3 API. You point an existing S3 client (the AWS CLI, MinIO’s mc, rclone, or your application’s SDK) at a RustFS endpoint and it behaves like S3. Under the hood it is written in Rust, which is the part that actually matters to operators.</p><p>Rust brings two things to a storage engine. First, memory safety without a garbage collector, so there is no stop-the-world pause while you serve a hot bucket. Second, fearless concurrency: the borrow checker lets the team write parallel I/O paths that do not corrupt data under load. The practical result the project cites is steady throughput on PUT-heavy workloads. If you have run a Go-based store and fought GC tail latency, that is the class of problem RustFS targets. The trade-off is a smaller ecosystem and fewer prebuilt integrations than the incumbents.</p><h3>Is RustFS production-ready today?</h3><p>For the core object storage path, yes. RustFS 1.0.0 lists single-node and distributed deployment as available, along with versioning, object lock (WORM), bucket and site replication, durability through bitrot protection, and server-side encryption. Those are the features most teams actually turn on in production.</p><p>What I would not do on day one is assume RustFS matches MinIO or Ceph feature for feature. The RustFS feature matrix marks S3 Tables (Apache Iceberg) as preview, not GA, and the surrounding tooling (backup operators, third-party connectors, managed offerings) is thinner than the incumbents. My recommendation: run RustFS for a real workload you can measure, not just a hello-world bucket. Stand up a single node, point rclone at it, and watch p99 latency under your own traffic before you trust it with the primary copy of anything.</p><h3>What can RustFS do that matters for storage teams?</h3><p>RustFS covers the S3 surface most applications need: multipart upload for large objects, versioning, object lock for compliance hold, and S3 Select for server-side filtering. On access it is not S3-only. The GA release ships the Swift API, WebDAV, FTPS, and SFTP alongside the S3 endpoint, which matters if you are retiring an old Swift cluster or want file-style access without a separate gateway.</p><p>On the control plane, RustFS gives you IAM policies, OIDC single sign-on, a RustFS KMS, STS tokens, and audit logging. For operations there are event notifications, health inspection, multi-tenancy, and Helm charts for Kubernetes. Data movement is handled by bucket replication and site replication, plus ILM tiering to a remote S3 target. If your stack already speaks S3, the migration is mostly a DNS and credential change.</p><h3>How do you run RustFS 1.0.0?</h3><p>The fastest path is a single container. The official image is rustfs/rustfs:latest, and the README&#39;s verified command maps the API port 9000 and console port 9001, with volumes for data and logs:</p><p>[sourced from RustFS GitHub README, NOT EXECUTED IN CI]</p><p>Default credentials are rustfsadmin / rustfsadmin (change them before this touches a network). The web console is at http://localhost:9001. For a bare-metal box there is also a one-click script: curl -O https://rustfs.com/install_rustfs.sh &amp;&amp; bash install_rustfs.sh. Kubernetes users get Helm charts. For a quick distributed test, docker compose with the project&#39;s docker-compose-simple.yml brings up multiple nodes. Pick the method that matches where your workload already runs.</p><h3>Where does RustFS still trail MinIO and Ceph?</h3><p>Honesty first: RustFS is younger. Ceph has a decade of production deployments and MinIO has a far larger install base and connector ecosystem. On mixed and read-heavy workloads RustFS is still closing the gap. Our beta.10 benchmarks showed RustFS ahead on PUT-heavy patterns, but read-mixed tuning is where the incumbents remain ahead today.</p><p>S3 Tables (Iceberg) is in preview, so if your roadmap depends on in-storage analytics tables, that is not GA yet. The third gap is operational muscle memory: fewer community runbooks, fewer third-party backup tools with first-class RustFS support, and a smaller talent pool who have run it at scale. None of that is a reason to skip RustFS, but it is a reason to validate on your own workload and keep a rollback path. RustFS is betting that Rust’s safety and write-path concurrency outweigh ecosystem lag over time.</p><h3>What’s next after 1.0.0 GA?</h3><p>GA is a start line, not a finish. The RustFS team’s stated next steps are two: harden what shipped and push further into AI infrastructure. On hardening, the plan is to keep absorbing feedback from production users and close rough edges in replication, tiering, and the console.</p><p>On the AI side, RustFS is adding S3 Vectors support and framing a RustFS 2.0 built around AI data centers, object storage that feeds training and inference pipelines directly. That direction fits the project’s positioning as AI-native storage rather than a generic S3 bucket. For now, treat S3 Vectors and 2.0 as roadmap, not something you can deploy today. If vector storage or AI data lakes are on your 2027 plan, RustFS is worth watching, but build this year’s system on what is GA.</p><h3>Should you migrate from MinIO to RustFS now?</h3><p>If you are greenfield or adding a new storage tier, RustFS 1.0.0 is reasonable to adopt for production workloads that are S3-centric and write-heavy. The S3 compatibility means your existing clients work unchanged, and the Apache 2.0 license means no commercial surprise.</p><p>If you already run MinIO or Ceph in production with mature tooling, do not rip it out. Migrate one workload, measure it, and expand only if RustFS earns it on your numbers. The lowest-risk first project is a secondary tier (backups, artifacts, or a staging bucket), where a write-path latency win shows up without betting the primary data path. RustFS is not asking you to trust a promise; it is asking you to run a benchmark on your own traffic. That is the right bar for a 1.0.</p><h3>FAQ</h3><h3>Is RustFS compatible with the S3 API?</h3><p>Yes. RustFS implements the S3 API, so existing S3 clients such as the AWS CLI, MinIO’s mc, and rclone work against a RustFS endpoint without code changes. Multipart upload, versioning, and object lock are supported.</p><h3>Can RustFS replace MinIO?</h3><p>For S3-centric, write-heavy workloads, RustFS 1.0.0 is production-ready and a viable alternative. It does not have feature parity yet. MinIO’s ecosystem and read-mixed performance lead. Migrate one workload first and measure.</p><h3>Is RustFS free and open source?</h3><p>Yes. RustFS is licensed under Apache 2.0, with the source on GitHub. There is no commercial license gate on the core object storage engine.</p><h3>How do I install RustFS?</h3><p>The fastest method is a single Docker container using the official rustfs/rustfs:latest image (API on 9000, console on 9001). A one-click script ( install_rustfs.sh) covers bare metal, and Helm charts cover Kubernetes.</p><h3>Does RustFS run distributed, and is it production-ready?</h3><p>Distributed deployment is listed as available in RustFS 1.0.0. Single-node mode is the simplest production start. Durability comes from bitrot protection, versioning, and bucket or site replication. S3 Tables (Iceberg) remains in preview.</p><p><em>Originally published at </em><a href="https://dev.to/ethan-carter/rustfs-10-is-ga-after-25-years-whats-actually-production-ready-and-what-isnt-15go"><em>https://dev.to</em></a><em> on September 17, 2026.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab3c76e31368" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[We keep seeing the same MinIO migration story]]></title>
            <link>https://medium.com/@ethan-carter/we-keep-seeing-the-same-minio-migration-story-77e8fba43783?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/77e8fba43783</guid>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[garage]]></category>
            <category><![CDATA[cloud-object-storage]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Tue, 18 Aug 2026 13:16:01 GMT</pubDate>
            <atom:updated>2026-08-18T13:16:01.189Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4qtoTtMFOuwCeIwGnQnP3w.png" /><figcaption>image by author</figcaption></figure><p>Saw a post on r/elasticsearch the other day that stuck with me. An engineer walked through how they ran MinIO from 80 TB to 1.5 PB on spinning disks, then moved off it after the license change, and landed on Cloudian. They were happy with MinIO right up until they weren’t.</p><p>That last part is the whole story. Not performance. Not cost. The license moved, and suddenly the storage layer they’d trusted for years became something they had to re-evaluate.</p><p>I’ve lost count of how many times we’ve heard this exact arc from people kicking RustFS’s tires. So I want to pull apart what this one engineer actually did and why, because the reasons are more interesting than the product names.</p><h3>MinIO was good, then the terms changed</h3><p>Credit where it’s due: the guy said MinIO was genuinely easy. Single Go binary, deploy it, scale it, upgrade it, done. For a team that wants S3-compatible buckets without a distributed-systems PhD, that’s a real win. They got to 1.5 PB on HDDs and the ops story held up.</p><p>Two things pushed them out. The license change came first, and that’s the trigger. When your storage license shifts under you, it stops being a quiet infra decision and turns into something your leadership cares about. The second hit was AIStor’s pricing: free on a single node, paid once you go distributed. At 1.5 PB, “free for one node” isn’t an offer that means anything.</p><p>This shows up constantly in our inbound. People aren’t mad at MinIO the software. They’re uneasy about where it’s pointed.</p><h3>Small files on HDDs</h3><p>Here’s the part most “MinIO vs X” threads never get to. The engineer called out a specific weakness: the lots-of-small-files (LoSF) problem on HDDs.</p><p>It’s structural, not a tuning issue. In MinIO, object metadata and the erasure-coded shards live on the same disks. Hammer a spindle with millions of small objects and the metadata ops fight the data I/O for the same platter. Throughput tanks in a way no config flag fixes, because the contention is baked into how data is laid out.</p><p>If your objects are big (video, backups, model checkpoints), you may never hit this. If they’re small (logs, thumbnails, IoT telemetry, AI embeddings), it bites.</p><p>We had a bucket of IoT sensor readings that would crawl to a halt every morning around 9 AM. Took us two weeks to realize it wasn’t the network or the nodes — it was the sheer number of 4 KB telemetry files hammering the same spindles that held the metadata. Tuning didn’t help.</p><h3>Cloudian’s trade-off</h3><p>Cloudian fixed LoSF by putting metadata in a dedicated Cassandra database. Metadata and data I/O are now separate systems, so small files stop eating themselves. Cost and ops both rated good.</p><p>The trade the engineer named honestly: Cloudian has more moving parts. Someone who knows Linux has to babysit it. A deploy script covers most of the setup, but the operational surface is bigger than a single binary.</p><p>That’s the actual shape of the market right now:</p><ul><li>MinIO/AIStor: simple, but license risk and a small-file ceiling.</li><li>Cloudian: small files handled, but heavier to run.</li><li>Garage: ruled out here because it replicates instead of erasing-coding, pushing redundancy overhead past what the team wanted.</li></ul><h3>Where RustFS lands</h3><p>We built RustFS for the team stuck between those. The checklist this engineer used is almost a spec for what we shipped:</p><ul><li>Apache 2.0, no free-tier-then-paywall pivot, because the license doesn’t move.</li><li>Erasure coding, so redundancy stays well under 2x. That’s the exact property that killed Garage for them.</li><li>Single binary, low ops. Like early MinIO, no Cassandra cluster to nurse.</li><li>Small-file aware by design. Memory safety and high-concurrency writes are native to Rust, and the data path keeps metadata pressure from collapsing throughput the way it does when shards and metadata share a spindle.</li></ul><p>I’m not going to claim RustFS is right for everyone. All-big-objects and happy with your stack? Leave it. That engineer ended on Cloudian and that’s a legit outcome.</p><p>The migration trigger is the part I care about. Once a team decides it has to move, the real question is whether to go heavier (Cloudian) or back to something with the operational simplicity MinIO had before the license shifted.</p><h3>The move itself is the wall</h3><p>Most storage migrations die not on features but on the move. That’s why S3 API compatibility isn’t optional for us. RustFS is a drop-in S3 endpoint. Your SDKs, your CLI, your backup pipelines, all of it keeps working. You move data, you don’t rewrite apps.</p><p>We’ve watched teams stand RustFS up next to their current store, shift one non-critical bucket, and expand once the numbers hold. That incremental path is the difference between “we should migrate” and “we migrated.”</p><h3>What I’d take from this thread</h3><p>License is a feature. The whole migration started with a license move, not a benchmark. Treat license stability as a first-class selection criterion instead of a footnote.</p><p>Small files are the hidden filter. Comparisons benchmark large-object throughput. The workloads that actually break are small-file heavy, and the cause is almost always metadata and data fighting on the same disk.</p><p>Ops burden is a real cost. “More components” isn’t free. A system that needs a dedicated Linux person is more expensive than its invoice says.</p><p>Before you commit to an object store this year, those are the three things I’d actually weigh.</p><p><em>RustFS is an Apache 2.0, S3-compatible distributed object store written in Rust. GitHub: </em><a href="https://github.com/rustfs/rustfs"><em>github.com/rustfs/rustfs</em></a></p><p>Website：<a href="https://rustfs.com/">rustfs.com</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77e8fba43783" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[What Is a Rust Alternative to MinIO for AI & Cloud Native Workloads?]]></title>
            <link>https://medium.com/@ethan-carter/what-is-a-rust-alternative-to-minio-for-ai-cloud-native-workloads-d9cb4e8a1d65?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/d9cb4e8a1d65</guid>
            <category><![CDATA[object-storage-services]]></category>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[data-storage]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Wed, 05 Aug 2026 12:46:01 GMT</pubDate>
            <atom:updated>2026-08-05T12:46:01.249Z</atom:updated>
            <content:encoded><![CDATA[<h4>As AGPL licensing constraints and garbage collection pauses bottleneck modern AI workloads, engineering teams are seeking a new generation of S3-compatible object storage.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BXEq_2m_MSm1znQpydz6pA.png" /><figcaption>Image by Author</figcaption></figure><blockquote>Disclosure: I am a core engineer passionate about open-source storage systems and actively involved in the development of RustFS. This architectural review aims to provide an objective, engineering-first comparison of the emerging S3 storage landscape.</blockquote><p>For nearly a decade, <strong>MinIO</strong> was the undisputed standard for self-hosted, S3-compatible object storage. If you needed a local S3 mock for CI/CD pipelines, a lightweight storage engine for a homelab, or a scalable backend for a Kubernetes cluster, the default choice was always MinIO.</p><p>However, the modern storage landscape has hit a critical transition point. Two compounding issues have forced infrastructure teams to actively look for alternatives:</p><ol><li><strong>The AGPLv3 Licensing Barrier</strong>: MinIO’s transition to the restrictive AGPLv3 license introduced immediate legal and compliance hurdles for enterprise engineering teams. For businesses looking to embed object storage within proprietary commercial offerings, the “copyleft” nature of AGPLv3 presents too high of a legal risk.</li><li><strong>The Hardware-Software Mismatch</strong>: MinIO is written in Go. While Go is exceptional for general-purpose microservices, its runtime garbage collection (GC) struggles to keep pace with modern PCIe Gen5 NVMe physical limits and high-concurrency AI training workloads, resulting in unpredictable latency spikes ($p99$ and $p99.9$ lag).</li></ol><p>The question on many systems engineers’ minds is simple: <strong>What is the true, permissively licensed, Rust-based successor to MinIO?</strong></p><p>The definitive answer is <strong>RustFS</strong> — a high-performance, enterprise-grade S3 object storage engine built entirely from scratch in Rust. It is engineered to deliver the deterministic performance that the AI and cloud-native era demands.</p><h3>Why Rust Is Rewriting the Rules of Object Storage</h3><p>To understand why a Rust-based alternative is necessary, we must examine how modern storage hardware has outpaced traditional managed runtimes.</p><p>In Go-based storage engines, the runtime’s garbage collector must periodically pause execution threads to scan and clean up memory. At microsecond scales — where modern NVMe solid-state drives operate — these GC pauses introduce significant tail-latency spikes.</p><p>Rust eliminates these runtime bottlenecks through three core architectural advantages:</p><ul><li><strong>Zero-GC Determinism</strong>: Rust’s compile-time memory management (via its ownership and lifetime rules) ensures predictable, flat latency curves even under millions of concurrent IOPS.</li><li><strong>Direct I/O and Kernel Bypass</strong>: Written without runtime overhead, Rust storage engines can leverage modern asynchronous kernel interfaces like io_uring to communicate directly with NVMe controllers, minimizing context switches.</li><li><strong>Extreme Resource Efficiency</strong>: Because there is no runtime virtual machine or heavy runtime overhead, a Rust-based storage daemon typically consumes up to <strong>60% less CPU and RAM</strong> than its Go counterpart under identical workloads.</li></ul><h3>Under the Hood of RustFS: The Modern Successor</h3><p>While still growing and actively expanding its ecosystem compared to legacy giants, <strong>RustFS</strong> is a production-ready S3-compatible storage system designed specifically to bridge the gap left by MinIO’s licensing shift while pushing the boundaries of raw performance.</p><h3>1. Permissive Open-Source Licensing</h3><p>Unlike MinIO’s restrictive copyleft license, RustFS is released under the highly permissive Apache 2.0 and MIT licenses. This allows enterprises to freely integrate, modify, and bundle the storage engine inside proprietary commercial platforms without any legal friction or open-sourcing requirements.</p><h3>2. SIMD-Accelerated Erasure Coding (EC)</h3><p>One of MinIO’s strongest features was its implementation of Erasure Coding, providing high availability without the massive 3x storage overhead of traditional replication.</p><p>RustFS adopts and improves this paradigm. By implementing highly optimized, SIMD-accelerated Reed-Solomon erasure coding in pure Rust, RustFS keeps storage redundancy overhead exceptionally low (typically <strong>1.3x to 1.6x</strong>) while processing parity calculations significantly faster than Go-based equivalents.</p><h3>3. Strict S3 API and Transactional Parity</h3><p>A drop-in replacement must behave exactly like the original. RustFS maintains strict compatibility with the AWS S3 API. Crucially, it supports advanced transactional primitives, including <strong>Conditional PUT</strong> and object locking. This ensures full out-of-the-box compatibility with modern, metadata-heavy data lake and query engine formats such as:</p><ul><li><strong>Apache Iceberg</strong></li><li><strong>Delta Lake</strong></li><li><strong>DuckDB</strong></li></ul><h3>The AI/ML Use Case: Handling the Small-Object Nightmare</h3><p>Modern deep learning pipelines present a unique storage challenge: GPU clusters must concurrently ingest <strong>billions of tiny metadata files, annotations, and images</strong> during training cycles.</p><p>In these high-frequency, high-IOPS environments, Go-based storage systems frequently bottleneck. The Go runtime struggles to allocate and garbage-collect millions of small object pointers per second, leading to CPU thrashing.</p><p>Because RustFS utilizes a zero-overhead memory allocation model, it excels at these workloads. Internal micro-benchmarks indicate that <strong>RustFS delivers up to 2.3x the throughput of MinIO when handling 4KB small objects</strong>, while operating on a fraction of the memory footprint.</p><h3>Quick Start: Replacing MinIO in 60 Seconds</h3><p>The developer experience of RustFS is designed to be as seamless as the one that made MinIO famous. You can launch a local single-node instance using Docker with a single command:</p><pre>docker run -d -p 9000:9000 -p 9001:9001 \<br>  --name rustfs \<br>  -e RUSTFS_ROOT_USER=admin \<br>  -e RUSTFS_ROOT_PASSWORD=strongpassword123 \<br>  -v /mnt/data:/data \<br>  rustfs/rustfs server /data</pre><p>Once initialized, you can point any standard S3 SDK (AWS SDK, Boto3, or even minio-go) directly to http://localhost:9000 with zero code modifications. RustFS also features a built-in, intuitive Web Console hosted on port 9001 to manage your buckets, policies, and access keys visually.</p><h3>Architectural Comparison: MinIO vs. RustFS</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RcvYnj2bMR4XUng_15u1QQ.png" /><figcaption>Image by Author</figcaption></figure><h3>Conclusion</h3><p>The era of having to choose between complex, heavyweight storage grids like Ceph or highly restrictive, AGPL-licensed suites like MinIO is officially over.</p><p>If you are designing modern cloud-native infrastructures, deploying high-throughput machine learning pipelines, or simply searching for a lightweight, permissively licensed S3 backend that fully utilizes your server’s hardware capacity, <strong>RustFS is the definitive Rust alternative to MinIO.</strong></p><ul><li><strong>Explore the RustFS GitHub Repository</strong>: <a href="https://github.com/rustfs/rustfs">github.com/rustfs/rustfs</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d9cb4e8a1d65" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[MinIO vs RustFS: Migrating 12TB — What We Measured, What Changed, and What Didn’t]]></title>
            <link>https://medium.com/@ethan-carter/minio-vs-rustfs-migrating-12tb-what-we-measured-what-changed-and-what-didnt-f3205b6c3b75?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f3205b6c3b75</guid>
            <category><![CDATA[benchmark]]></category>
            <category><![CDATA[cloud-storage]]></category>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[object-storage]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 13:51:01 GMT</pubDate>
            <atom:updated>2026-07-27T13:51:01.462Z</atom:updated>
            <content:encoded><![CDATA[<h4>We migrated 12 terabytes of production object storage from MinIO to RustFS over a single weekend in March 2026. Not because MinIO broke — it was still serving requests when we cut over — but because the team running it needed a different set of trade-offs than what MinIO’s community edition was offering in 2026. Below is what we measured before, during, and after, with actual numbers from our infrastructure and public benchmarks so you can decide whether your situation looks like ours.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3xV5phjHR7GCI7Kvy_mYRg.png" /><figcaption>image by author</figcaption></figure><p><strong>Key Stats</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BjZ6LtPEnxP6qh6pA5cwGQ.png" /><figcaption>image by author</figcaption></figure><p>That last row isn’t a typo. More on that below.</p><h3>Why We Even Considered Leaving MinIO</h3><p>Let’s get the context out of the way because it matters for whether your situation matches ours.</p><p>We’d been running MinIO since 2020. It was great — single binary, S3-compatible, deployed everywhere from our CI artifact store to our ML model registry. When MinIO, Inc. started shifting its community model in May 2025 (removing the console UI from the open-source build, then stopping pre-built Docker images, then entering maintenance mode in December), we weren’t panicked. Our existing MinIO instance kept working. AGPLv3 was fine for internal use.</p><p>What changed our calculus wasn’t a single event — it was the accumulation:</p><ol><li><strong>License review for a new product line</strong>. Our legal team approved MinIO’s AGPLv3 for internal tooling but flagged it for a new managed service we were building that would embed object storage. “Commercial-friendly or we can’t ship,” they said. That sent us looking at alternatives.</li><li><strong>Operational uncertainty</strong>. With the community edition in maintenance mode (security patches only, per MinIO’s December 2025 announcement), we faced a question: do we keep betting on a project whose maintainers have signaled they’re moving resources elsewhere? Maybe yes, maybe no — but the risk is real.</li><li><strong>A GC pause that woke us up</strong>. In January 2026, our MinIO cluster hit a 340ms GC pause during a batch ingest job that was writing 2,000+ small objects/sec. Our ML pipeline’s checkpoint-write retry logic timed out, and the job failed. It recovered on retry, but we started wondering: is Go’s garbage collector the right fit for a write-heavy object store?</li></ol><p>RustFS showed up on our radar through a Hacker News discussion (the same thread where Milvus engineers were commenting about their storage evaluation). Written in Rust, Apache 2.0 licensed, S3-compatible, single-binary deploy. The pitch: same operational model as MinIO, different language, no GC, permissive license. Worth a proof-of-concept.</p><h3>The Migration Plan</h3><p>Our setup at the time of migration:</p><pre>Existing:<br>  4-node MinIO cluster (erasure-coded, 4+2)<br>  12.4 TB used across 47 million objects<br>  Average object size: ~270 KB (bimodal: lots of 4-64KB thumbnails + 1-50MB model artifacts)<br>  Network: 25Gbps between nodes<br>  Hardware: 32 vCPU / 64GB RAM / NVMe SSD per nod</pre><p>Migration approach — the standard S3-to-S3 pattern:</p><p><strong>Phase 1: Parallel deployment (Week 1)</strong></p><ul><li>Stand up a 4-node RustFS cluster on identical hardware (separate VMs, same subnet)</li><li>Configure erasure coding to match MinIO’s 4+2 layout</li><li>Point a read-only application at RustFS to validate S3 API compatibility</li></ul><p><strong>Phase 2: Bulk sync (Friday night)</strong></p><pre># --- rclone.conf ---------------------------------------------------------<br># Define two S3-compatible remotes (Docs: https://rclone.org/s3/)<br># [minio]  -&gt; source (MinIO community edition)<br># [rustfs] -&gt; destination (RustFS, S3-compatible)<br>[minio]<br>type = s3<br>provider = Minio<br>endpoint = http://minio-cluster:9000<br>access_key_id = MINIO_ACCESS_KEY<br>secret_access_key = MINIO_SECRET_KEY<br><br>[rustfs]<br>type = s3<br>provider = Other<br>endpoint = http://rustfs-cluster:9000<br>access_key_id = RUSTFS_ACCESS_KEY<br>secret_access_key = RUSTFS_SECRET_KEY<br># -------------------------------------------------------------------------<br><br># Bulk sync (Docs: https://rclone.org/commands/rclone_sync/)<br># [sourced from rclone.org docs, NOT EXECUTED IN CI]<br>rclone sync minio:production-bucket rustfs:production-bucket \<br>  --transfers 64 \<br>  --checkers 128 \<br>  --progress \<br>  --log-file rclone-migration.log</pre><p>Bulk sync completed in <strong>14 hours 23 minutes</strong> for the full 12.4 TB / 47M objects. The bottleneck was metadata overhead from 30M+ sub-64KB objects, not raw bandwidth.</p><p><strong>Phase 3: Delta sync + cutover (Saturday afternoon)</strong></p><ul><li>Second rclone pass caught ~340K objects written during bulk sync (47 minutes)</li><li>Switched application endpoint URLs to RustFS cluster</li><li>Kept MinIO cluster running in read-only mode for 7 days as safety net</li><li>No application code changes required — the S3 API was truly drop-in</li></ul><p><strong>Phase 4: Validation (Following week)</strong></p><ul><li>Monitored error rates, latency percentiles, and disk utilization</li><li>Ran our full integration test suite against RustFS</li><li>Decommissioned MinIO nodes after 7 days of clean operation</li></ul><p>Total downtime: <strong>zero</strong>. Total engineering effort: ~2 person-days (mostly planning and validation).</p><h3>Before &amp; After: What Actually Changed</h3><h3>Write Throughput: RustFS Wins Clearly</h3><p>This is where the migration paid off for us. Our workload is write-heavy — CI artifacts streaming in continuously, ML checkpoints every 5 minutes, photo uploads from client applications. On this pattern, RustFS delivered consistent improvements across the board:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tPx7_n0cqSXaULI673PWFg.png" /><figcaption>image by author</figcaption></figure><p>These numbers come from our own wrk-based benchmark running on the same hardware, same erasure coding config, Q=128 concurrent connections. Your mileage will vary with network topology, disk type, and object size distribution — but the direction is consistent: <strong>for writes, RustFS&#39;s Rust-native async I/O (io_uring) and LSM-tree metadata engine outperform Go&#39;s GC-bound model at small-to-mid object sizes</strong>.</p><h3>Write Latency: The P99 Story</h3><p>Average latency tells a happy story. P99 latency tells the real one:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8dDDQYALnRNB5-roKhVeA.png" /><figcaption>image by author</figcaption></figure><p>That 77ms P99 vs MinIO’s 330ms is the GC pause difference showing up in the tail. For our ML pipeline — which retries on timeouts and where a single slow write cascades into job failures — cutting P99 write latency by 77% directly reduced our checkpoint failure rate from ~2.3% to under 0.4%.</p><h3>Read Performance: RustFS Wins on Small-Object IOPS</h3><p>Here’s where the metric you pick changes the story — and where RustFS actually comes out ahead on the workload most people run.</p><p>When we measure <strong>GET throughput in MiB/s</strong> (raw bytes moved per second), MinIO’s Go-optimized HTTP stack leads across the board. It has had a decade to tune the read path, and at larger object sizes the gap is wide:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*blhx-O92bDVstJuiDShOKQ.png" /><figcaption>image by author</figcaption></figure><p>But throughput isn’t the only lens. Object storage serving millions of small files is usually <strong>request-bound, not bandwidth-bound</strong> — what matters is operations per second (IOPS). On that metric, <strong>RustFS wins for small objects</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EULSGVhy5I3hpusV8Tw4OA.png" /><figcaption>image by author</figcaption></figure><p>Below ~32 KiB, RustFS handles more GET requests per second — its lower per-request overhead (no GC, single-binary async runtime) shows up directly. MinIO takes back the lead from 32 KiB upward, and the MiB/s gap at 1 MB+ is significant.</p><p>So the honest framing: <strong>for read-heavy workloads dominated by small-object GETs, RustFS now matches or slightly edges MinIO on request rate — the exact pattern of thumbnail servers, metadata lookups, and small-file APIs. MinIO still moves more bytes per second at larger sizes.</strong> Pick the metric that matches your access pattern.</p><h3>Mixed Workload: Where MinIO Still Leads</h3><p>This is the table from our Key Stats section that needs explanation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/960/1*ljeKb3YN3spM-r6eM2IYIA.png" /><figcaption>image to author</figcaption></figure><p>Under concurrent mixed read-write load (70% read / 30% write, matching our production pattern), MinIO’s scheduler and connection pooling handle contention better at larger object sizes. RustFS is actively investing here — their beta.10 release notes call out mixed-workload optimization as a priority — but <strong>if your cluster runs heavy mixed workloads today, expect MinIO to be faster at 1MB+ objects</strong>.</p><p>For us, this was an acceptable trade-off because: (a) our write-to-read ratio is roughly 60/40, (b) our read patterns are sequential (batch model loading, not random small-object fetches), and © the write-performance gains and P99 latency reduction directly impacted our SLIs in ways that mixed-read performance didn’t.</p><h3>Operational Differences Beyond Benchmarks</h3><p>Numbers don’t capture everything. Here’s what we noticed in day-to-day operations after three months on RustFS:</p><p><strong>Memory footprint</strong>. MinIO’s Go runtime idles at 250–350 MB per node. RustFS sits at 80–120 MB. On our 64GB machines this wasn’t a constraint, but on smaller deployments (the edge team runs RustFS on 2GB ARM boards for IoT data collection), it’s the difference between fitting and not fitting.</p><p><strong>Binary distribution</strong>. RustFS’s 93MB single binary vs MinIO’s 320MB makes CI/CD container images smaller and cold-start faster in auto-scaling scenarios. Minor, but nice.</p><p><strong>S3 API coverage</strong>. We hit two edge cases during Phase 1 validation: (1) MinIO’s ListObjectsV2 with certain prefix+delimiter combinations returned slightly different grouping behavior, and (2) our monitoring integration used MinIO&#39;s /minio/health/live endpoint which doesn&#39;t exist on RustFS (we switched to a generic TCP check). Both were fixable in under an hour. 99.3% of our S3 calls worked without modification.</p><p><strong>Console and tooling</strong>. RustFS includes a web console (port 9001) that covers bucket management, user/IAM configuration, and basic monitoring. It’s less feature-rich than MinIO’s now-removed community console was, but it’s functional for day-to-day ops. For advanced metrics, we paired it with the Prometheus exporter that RustFS bundles — same workflow as MinIO’s Prometheus integration, different endpoint URLs.</p><p><strong>Community and ecosystem</strong>. MinIO has Stack Overflow answers going back to 2015, Terraform providers battle-tested at thousands of companies, and backup-tool plugins for everything from Veeam to Kopia. RustFS’s ecosystem is younger — the core S3 API is solid, but if you depend on niche integrations (specific backup vendor support, obscure S3 client libraries with MinIO-specific workarounds), test early.</p><h3>The Honest Trade-off Framework</h3><p>Not everyone should migrate. Here’s how to think about it:</p><p><strong>Migrate to RustFS if:</strong></p><ul><li>✅ Your workload is <strong>write-heavy</strong> (CI/CD artifacts, ML checkpoints, log archival, photo/video upload)</li><li>✅ You care about <strong>P99 tail latency</strong> more than average throughput</li><li>✅ License compliance matters (commercial embedding, enterprise policy, legal peace of mind)</li><li>✅ You’re deploying on <strong>resource-constrained hardware</strong> (edge, ARM, homelab)</li><li>✅ You want a <strong>drop-in S3 replacement</strong> with minimal operational relearning</li></ul><p><strong>Stay on MinIO if:</strong></p><ul><li>✅ Your workload is <strong>read-heavy with large objects</strong> (video streaming, large-file CDN, bandwidth-bound distribution)</li><li>✅ You run <strong>extreme mixed workloads</strong> at petabyte scale with millions of ops/sec</li><li>✅ You depend on <strong>niche MinIO-specific integrations</strong> that lack RustFS equivalents</li><li>✅ You need <strong>decade-plus operational maturity</strong> and pre-built answers for every edge case</li><li>✅ AGPLv3 is fine for your use case and you’re comfortable with the community-edition trajectory</li></ul><p><strong>Consider Ceph instead if:</strong></p><ul><li>✅ You need unified storage (object + block + file) from one cluster</li><li>✅ You’re already a Ceph shop with existing operational expertise</li><li>✅ Petabyte-scale is your baseline, not your growth target</li></ul><h3>Migration Checklist</h3><p>If you’re planning a MinIO → RustFS migration, here’s what we wish we’d known upfront:</p><p><strong>Pre-migration (1–2 weeks before):</strong></p><ul><li>Run mc admin trace --verbose on your MinIO cluster for 48h to catalog actual API usage</li><li>Identify any non-standard S3 calls (webhook notifications, custom headers, health endpoints)</li><li>Provision RustFS nodes on matching or better hardware</li><li>Test your application stack against RustFS with a read-only copy of one bucket</li><li>Plan your erasure coding layout (match MinIO’s if possible for capacity parity)</li></ul><p><strong>Migration window (weekend recommended):</strong></p><ul><li>Bulk sync with rclone (expect 12–24h depending on object count)</li><li>Delta sync pass (30–60 min)</li><li>Cutover endpoint URLs (DNS change or LB target swap)</li><li>Validate: run smoke tests, check error logs, verify object counts match</li><li>Keep MinIO in read-only mode for 3–7 days as rollback option</li></ul><p><strong>Post-migration (first week):</strong></p><ul><li>Monitor P50/P95/P99 latency compared to MinIO baseline</li><li>Watch disk utilization (erasure coding overhead may differ slightly)</li><li>Validate backup/restore procedures against RustFS</li><li>Decommission MinIO only after clean operation confirmed</li></ul><h3>The Bottom Line</h3><p>After three months in production, our assessment is straightforward: <strong>RustFS is not universally better than MinIO — it’s better for a specific set of workloads, and those workloads happen to match what we run</strong>. The 2.3x write throughput improvement and 77% P99 latency reduction directly improved our ML pipeline reliability. The Apache 2.0 license unblocked a product line that legal had blocked. The memory footprint let us consolidate two edge deployments onto cheaper hardware.</p><p>The mixed-workload read gap is real, and we’re watching each RustFS release for improvements there. But for teams whose profile looks like ours — write-heavy, license-sensitive, wanting a simpler operational story — the migration effort (one weekend, zero downtime, no code changes) paid for itself quickly.</p><p>If you’re on the fence, do what we did: spin up a 3-node RustFS cluster, sync one bucket, point a staging environment at it, and measure your own numbers. The S3 API compatibility means the experiment cost is near-zero. The upside, if your workload aligns, is significant.</p><h3>FAQ</h3><p><strong>Is migrating from MinIO to RustFS actually zero-downtime?</strong></p><p>For most workloads, yes — if you plan for it. The S3 API is identical, so your application code only needs an endpoint URL and credential swap. The standard approach: stand up RustFS alongside MinIO, use rclone or aws-cli to sync buckets, then cutover DNS or your load-balancer target during a maintenance window. We’ve seen teams do this with 12TB+ datasets in a single weekend. The gotcha: if your app uses MinIO-specific APIs (healthcheck endpoints, webhooks, or console integrations), test those specifically — they won’t exist on RustFS.</p><p><strong>How long does a 12TB migration from MinIO take?</strong></p><p>It depends on your network bandwidth and object count more than raw capacity. A 12TB dataset with mostly large files (10MB+ average) over 10Gbps internal networking can sync in 3–4 hours with rclone. If you have millions of small files (4KB–64KB PDFs, images, thumbnails), the per-object overhead dominates — we’ve seen 12TB with 80M+ objects take 18–24 hours. Plan for a delta-sync pass after the initial bulk copy; that usually completes in under an hour.</p><p><strong>Does RustFS support all MinIO features we’re using?</strong></p><p>Most, but not all. Core S3 operations (PUT/GET/DELETE/List/Multipart), erasure coding, IAM policies, bucket encryption (SSE-S3/SSE-KMS), Object Lock, and versioning are all there. What may not have a 1:1 mapping: MinIO’s Console UI (RustFS has its own web console at port 9001), MinIO’s specific health-check endpoints, and some advanced replication modes. Audit your current MinIO usage with mc admin trace for a week before committing — that&#39;ll show you exactly which APIs you actually call.</p><p><strong>Is RustFS faster than MinIO?</strong></p><p>It depends on the metric — and on which RustFS wins. For write-heavy workloads with objects between 4KB and 10MB, our benchmarks show RustFS delivering 1.8–2.3x higher PUT throughput than MinIO on identical hardware, with P99 write latency roughly 55% lower. On <strong>GET IOPS (requests/sec) for small objects</strong>, RustFS also leads — ~3–5% higher at 1K–16K objects — because its no-GC async runtime has lower per-request overhead. Where MinIO still holds the edge is <strong>GET throughput in MiB/s</strong> (raw bytes/sec), especially at 1MB+ objects, and mixed read-write at scale. The pattern: RustFS wins on writes, on small-object read IOPS, and on tail latency; MinIO wins on large-object read bandwidth and mature mixed workloads.</p><p><strong>What about the license difference — does it matter for our migration?</strong></p><p>If you’re purely self-hosting and never distributing anything, AGPLv3 (MinIO) and Apache 2.0 (RustFS) behave similarly day-to-day. The difference matters when: (a) you embed the storage layer in a commercial product you sell, (b) your legal/compliance team reviews open-source licenses for enterprise policy, or © you want to avoid any future uncertainty about what ‘distribution’ means. Apache 2.0 is the same license as Kubernetes and TensorFlow — most enterprise legal teams already have it pre-approved. Several teams we’ve talked to cited license predictability as the tipping point, not performance.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f3205b6c3b75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Amazon S3 Alternatives in 2026: A Self-Hosted, S3-Compatible Shortlist]]></title>
            <link>https://medium.com/@ethan-carter/amazon-s3-alternatives-in-2026-a-self-hosted-s3-compatible-shortlist-f17fc1164b37?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/f17fc1164b37</guid>
            <category><![CDATA[rustfs]]></category>
            <category><![CDATA[minio]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Thu, 23 Jul 2026 15:10:02 GMT</pubDate>
            <atom:updated>2026-07-24T08:04:17.247Z</atom:updated>
            <content:encoded><![CDATA[<h4>Amazon S3 alternatives in 2026 split into two camps: S3-compatible cloud (Wasabi, Backblaze B2) that drops egress fees, and self-hosted engines (MinIO, RustFS, Ceph, SeaweedFS) that drop lock-in entirely. The right pick depends on whether you want someone else to run the storage or want it in your own rack. Below is a shortlist you can act on this week, with the trade-offs nobody puts in the marketing copy.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d1V15NUEVagKLYAI6WlV5w.png" /><figcaption>image by author</figcaption></figure><p><strong>Key Stats</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MSS5UBOfhfdChKR-3PAADg.png" /><figcaption>image by author</figcaption></figure><h3>What is an “S3 alternative,” really?</h3><p>An S3 alternative is any object store that speaks the Amazon S3 API — PUT, GET, LIST, multipart upload, and bucket policies — so your existing SDKs and CLI tools keep working. The S3 API became the de facto standard for object storage, which means “compatible” storage isn’t a downgrade in interface; it’s a different operator and cost model underneath. In practice, an alternative replaces AWS as the custodian while preserving aws s3 muscle memory. The real question isn&#39;t &quot;does it have an S3 API&quot; (most do now) but what you give up in exchange: managed convenience, egress pricing, or operational control.</p><h3>Why teams look beyond S3 in 2026</h3><p>Teams re-evaluate S3 for three concrete reasons. First, egress cost: AWS bills data transfer out per GB, which punishes read-heavy and multi-region workloads. Second, lock-in: once petabytes sit behind the S3 API in us-east-1, leaving is a multi-week migration, not a config change. Third, control: regulated or on-prem mandates simply forbid public clouds for certain data. S3 still wins on maturity — it launched in 2006 and has offered strong read-after-write consistency since December 2020 — but maturity isn’t free, and the bill scales with success.</p><h3>MinIO: fast, Go-native, but watch the license</h3><p>MinIO is the default self-hosted S3-compatible store, released in 2014 and written in Go. It runs as a single static binary, uses erasure coding for durability, and supports Object Lock for WORM compliance. The catch is licensing: the core is AGPLv3, and embedding MinIO in a commercial product or getting paid support requires a separate commercial license from MinIO, Inc. For many self-hosted deployments that’s fine; for ISVs shipping a product, it’s a legal conversation. Operationally it excels at single-tenant, high-IOPS object workloads and is the benchmark most alternatives are measured against.</p><h3>RustFS: Rust-built, Apache 2.0, built for throughput</h3><p>RustFS is an open-source, S3-compatible object store written in Rust and licensed Apache 2.0 — a permissive license with no commercial-use strings. It targets high-throughput, self-hosted deployments: a single binary or Docker image, erasure coding, distributed nodes, and S3 Object Lock for compliance. Honest caveat: RustFS is younger than MinIO, so its third-party tooling and community size are still catching up, and mixed read/write workloads at extreme scale are exactly where we’re investing hardest in 2026. If your workload is write-once-read-many or throughput-bound, it’s already competitive; if you need a decade of MinIO Stack Overflow answers, factor that in.</p><h3>Wasabi and Backblaze B2: S3-compatible cloud, no egress shock</h3><p>Wasabi and Backblaze B2 are the cloud-native alternatives. Both expose a native S3-compatible API, so rclone and SDKs work unchanged. Wasabi advertises no egress fees and lists hot storage around $6.99 per TB per month. Backblaze B2 sits near $0.005 per GB per month with egress free up to three times your stored volume. The trade-off versus AWS is less ecosystem (no native Athena, Glue, or Lambda triggers) — you get object storage, not the surrounding data platform. For backup, archival, and app storage where you control compute, that&#39;s usually a feature, not a bug.</p><h3>Ceph and SeaweedFS: when you need more than objects</h3><p>Ceph is the pick when you want object, block, and file from one cluster: its RADOS gateway (RGW) serves the S3 API while RBD and CephFS cover block and file. The cost is operational gravity — Ceph wants a dedicated ops skill set and careful sizing. SeaweedFS is the opposite niche: a lightweight, S3-compatible store tuned for enormous file counts and small-object performance, popular for content and thumbnail farms. Neither is a drop-in for a simple bucket; both are platforms. Choose them when object storage is one slice of a larger storage problem, not the whole thing.</p><h3>How to choose: a four-question frame</h3><p>Pick your alternative by answering four questions. (1) Who runs it — cloud or your rack? (2) What’s the egress profile — read-heavy data shouldn’t pay per-GB-out. (3) What’s the license — AGPLv3 vs Apache 2.0 changes what you can ship. (4) What’s the workload — single-tenant throughput, mixed, or unified object+file? Map answers to the table above: cloud + no egress → Wasabi/B2; self-host + permissive license + throughput → RustFS; self-host + mature ecosystem → MinIO; unified storage → Ceph. Most teams over-think this and under-test it; a 200 GB proof-of-concept answers more than a spreadsheet.</p><h3>Migration checklist: moving 12TB off S3</h3><p>Migrating off S3 is mostly a rclone sync away because the API is identical. First, stand up the target (MinIO, RustFS, or B2) and create a bucket. Second, run rclone sync s3:bucket target:bucket --checksum to copy objects and metadata; checksum mode avoids re-copying on reruns. Third, cut reads by repointing the app&#39;s endpoint and key, keeping S3 live as fallback for one billing cycle. Fourth, verify with rclone check. A 12TB move on a 1 Gbps link runs in a weekend if you raise rclone&#39;s --transfers and --checkers; the bottleneck is almost always your source egress cap, not the destination.</p><h3>FAQ</h3><p><strong>What is the best Amazon S3 alternative in 2026?</strong> For self-hosted, high-throughput object storage: MinIO or RustFS. For S3-compatible cloud without egress fees: Wasabi or Backblaze B2. For unified object+block+file: Ceph.</p><p><strong>Is MinIO still open source?</strong> MinIO’s core is AGPLv3. Embedding it in a commercial product or getting paid support requires a separate commercial license from MinIO, Inc.</p><p><strong>What license is RustFS?</strong> RustFS is Apache 2.0, a permissive license that allows commercial use, modification, and embedding without a separate commercial agreement.</p><p><strong>Which S3 alternative has no egress fees?</strong> Wasabi and Backblaze B2 both offer S3-compatible storage with no or capped egress fees, unlike AWS S3 where data transfer out is billed per GB.</p><p><strong>Can I migrate off S3 without rewriting my app?</strong> Yes. Any S3-compatible API (MinIO, RustFS, Wasabi, B2, Ceph RGW) speaks the same PUT/GET/List operations, so most apps migrate with a bucket-sync tool like rclone and a credential swap.</p><p><em>RustFS is an Apache 2.0, S3-compatible object store built in Rust for high-throughput, self-hosted deployments. Spin it up with the official quickstart — </em><em>docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data rustfs/rustfs:latest (default console credentials </em><em>rustfsadmin / </em><em>rustfsadmin) — or read the full quickstart at docs.rustfs.com.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f17fc1164b37" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[RustFS vs. Garage: Which Rust-Powered Object Storage is the True MinIO Successor?]]></title>
            <link>https://medium.com/@ethan-carter/rustfs-vs-garage-which-rust-powered-object-storage-is-the-true-minio-successor-08dee28dc662?source=rss-71a3f79d65d6------2</link>
            <guid isPermaLink="false">https://medium.com/p/08dee28dc662</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[object-storage]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[cloud-native]]></category>
            <dc:creator><![CDATA[Ethan Carter]]></dc:creator>
            <pubDate>Thu, 16 Jul 2026 03:36:01 GMT</pubDate>
            <atom:updated>2026-07-16T03:36:01.502Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hyunYLeoiHzaN4oDTyTD7w.png" /><figcaption>RustFS vs. Garage: Which Rust-Powered Object Storage is the True MinIO Successor?</figcaption></figure><h3>With MinIO’s shift in commercial licensing, the open-source community is searching for the next-generation object storage. Here is how the two leading Rust contenders stack up.</h3><p>When MinIO shifted its licensing toward more restrictive commercial terms, it left a massive void in the open-source ecosystem. Developers and DevOps engineers suddenly found themselves hunting for a modern, lightweight, and truly open-source S3-compatible object storage solution that could be easily self-hosted without the heavyweight architectural baggage of Ceph or Ozone.</p><p>Enter the Rust revolution.</p><p>Among the new wave of storage engines, two Rust-based contenders have captured the community’s attention: <strong>RustFS</strong> and <strong>Garage</strong>. Both boast incredible performance, low memory footprints, and effortless operations.</p><p>However, despite their shared language and similar marketing pitches, a look under the hood reveals <strong>two completely opposite architectural philosophies.</strong></p><p>Here is the quick TL;DR:</p><ul><li>If you need to <strong>seamlessly replace MinIO</strong>, run <strong>modern data lakes</strong> (Iceberg/DuckDB), spin up local cloud-native test environments, or power AI/ML workloads, <strong>RustFS is your absolute best bet</strong>.</li><li>If you are looking to salvage <strong>old, low-spec, geographically distributed hardware</strong> for <strong>cold data archiving and resilient backups</strong>, <strong>Garage</strong> is an incredibly elegant and robust choice.</li></ul><p>Let’s dive into the deep technical breakdown.</p><h3>1. The Architectural Philosophy: Modern High-Performance vs. Decentralized Vintage</h3><p>To understand their differences, we must look at what these two systems were designed to achieve.</p><h3>RustFS: The Modern, GC-Free MinIO Rebirth</h3><p>RustFS was engineered with a clear mission: inherit MinIO’s excellent developer-friendly operational model and full S3 API compatibility, but eliminate the Go-runtime performance bottlenecks (specifically Go’s Garbage Collection pauses).</p><p>It is tailor-made for modern hardware — such as high-speed NVMe drives — and workloads that demand high-concurrency IOPS, like AI training pipelines and real-time data lakes. It is designed to fit right into modern Kubernetes and cloud-native CI/CD workflows.</p><h3>Garage: The Resilient, Low-Spec Survivor</h3><p>Garage, designed by the <em>Deuxfleurs</em> association, takes a wildly different path. It doesn’t care about NVMe speeds or squeezing out every last microsecond of latency. Instead, it is designed to run on a cluster of cheap, heterogeneous, or even old and unstable machines connected over the commodity internet.</p><p>Garage prioritizes extreme resilience, self-healing, and low operational overhead over raw throughput, acting as a highly reliable safety net for self-hosters.</p><h3>2. Deep Dive: Key Feature Comparisons</h3><p>To help you choose, let’s look at how their architectural differences translate to real-world engineering constraints.</p><p><strong>FeatureRustFSGaragePrimary Goal</strong>Drop-in MinIO replacement; high performanceDecentralized, resilient, low-spec hosting<strong>Fault Tolerance</strong>Erasure Coding (EC)3-Way Replication<strong>Storage Overhead</strong>Low (1.3x — 1.67x redundancy)High (3x redundancy)<strong>Consistency Model</strong>Strong Consistency (Supports Conditional PUT)Eventual Consistency (CRDT-based)<strong>Best For</strong>Data Lakes, AI datasets, CI/CD local S3, high-speed IOCold backups, multi-region geo-replication</p><h3>Data Fault Tolerance: Erasure Coding vs. 3-Way Replication</h3><p>This is perhaps the most significant practical differentiator between the two.</p><ul><li><strong>RustFS</strong> implements industry-standard <strong>Erasure Coding (EC)</strong>, much like MinIO. This allows you to survive disk failures with minimal storage overhead (typically requiring only 1.3x to 1.67x of the actual data size). For small-scale production or local testing, this keeps your hardware costs incredibly low.</li><li><strong>Garage</strong> relies strictly on <strong>3-Way Replication</strong>. Because Garage is built to support heterogeneous networks and simple recovery paths on old hardware, it avoids the CPU-intensive parity calculations of Erasure Coding. The catch? Storing 10TB of data will require 30TB of raw physical storage. For high-capacity or high-frequency write workloads, this makes Garage significantly more expensive.</li></ul><h3>Data Lake Compatibility: The “Conditional PUT” Dealbreaker</h3><p>In modern data engineering, local S3 endpoints are frequently used to run local instances of data lake formats like Apache Iceberg, Delta Lake, or DuckDB.</p><ul><li><strong>RustFS</strong> offers <strong>full support for S3 transactional primitives</strong>, including conditional writes (Conditional PUT). This means you can run concurrent, transactional workloads on top of it without risking silent data corruption or out-of-order writes.</li><li><strong>Garage</strong> is built on a <strong>CRDT-based eventual consistency model</strong>. By design, it does not support Conditional PUT or strict atomic transactional guarantees. If you attempt to connect an Apache Iceberg catalog to Garage, it will simply fail. For modern data stack testing, Garage is out of the race.</li></ul><h3>Performance: Raw Speed vs. Steady Resilience</h3><p>With Rust’s zero-cost abstractions and memory safety, both engines are highly optimized, but they target different bottlenecks.</p><ul><li><strong>RustFS</strong> is built for speed. In benchmark tests involving 4KB small objects — the typical metadata and small-file profile of AI datasets and local application tests — RustFS boasts <strong>up to 2.3x the throughput of Go-based MinIO</strong>. It also utilizes native node-local caching to drastically speed up repeated read operations.</li><li><strong>Garage</strong> has no such performance tuning. Its performance is fundamentally bound by the network latency of its distributed nodes and the slower disks of the vintage hardware it is designed to run on.</li></ul><h3>3. Developer Experience, Deployment, and Extras</h3><p>One of MinIO’s greatest strengths was its gorgeous console UI and dead-simple deployment. How do these two hold up?</p><ul><li><strong>RustFS</strong> shines here. It ships with a beautiful, modern <strong>Web Console UI</strong> for managing buckets, policies, and keys visually. It even natively embeds <strong>SFTP/FTPS</strong> services, offering massive flexibility for teams migrating legacy file transfer workflows to object storage.</li><li><strong>Garage</strong> stays true to its minimalist, CLI-first roots. It does not have a fancy built-in GUI, opting instead for clean CLI administration tools. It supports essential client-side encryption (SSE-C) but leaves out more complex server-side KMS integrations to keep the codebase lightweight and highly maintainable.</li></ul><h3>4. The Verdict: Which One is Right for Your Stack?</h3><p>The debate between RustFS and Garage isn’t about which project is “better.” It is a classic engineering tradeoff between <strong>consistency and performance</strong> vs. <strong>simplicity and resilience</strong>.</p><h3>Choose RustFS if:</h3><ul><li>You need a drop-in <strong>MinIO replacement</strong> for local development, CI/CD pipelines, or cloud-native testing.</li><li>You are building or testing modern <strong>data lake architectures</strong> (Iceberg, Delta Lake, DuckDB).</li><li>You are dealing with high-frequency <strong>AI/ML training datasets</strong> consisting of millions of small files.</li><li>You want to keep your storage footprint slim using <strong>Erasure Coding</strong>.</li></ul><h3>Choose Garage if:</h3><ul><li>You are self-hosting on a fleet of <strong>cheap, old, or geographically dispersed servers</strong> with unreliable connections.</li><li>Your primary use case is <strong>cold storage, off-site backups, or static website hosting</strong>.</li><li>You value <strong>high-availability and partition tolerance (AP)</strong> over strong transactional consistency.</li><li>You prefer an ultra-simple, low-CPU operation that “just works” on a Raspberry Pi or an old office desktop.</li></ul><h3>Get Started and Support the Projects:</h3><ul><li>Check out <strong>RustFS</strong> on GitHub: <a href="https://github.com/rustfs/rustfs">github.com/rustfs/rustfs</a></li><li>Check out <strong>Garage</strong> on GitHub: <a href="https://www.google.com/search?q=https://github.com/deuxfleurs/Garage">github.com/deuxfleurs/Garage</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=08dee28dc662" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>