Roadmap¶

leBLOB aims to replace self-hosted MinIO and Garage: single-server installs first, then clusters, including clusters across several locations. It grows in four phases, with native code added alongside only where measurements call for it.

Phase 1: one server, one disk¶

Phase 1 built the S3 core: buckets, objects, listings, multipart uploads, SigV4 with signed, unsigned and chunked payloads, checksums, conditional requests, access keys with per-bucket rights, fsck, and durability tested against kill -9.

Phase 2: one server, several disks¶

Built:

• Volumes, several copies of each blob, read failover in the middle of a response, scrub, repair, draining and forgetting a disk. See Disks and copies.
• Metadata backup and restore, and Prometheus metrics.
• Versioning, lifecycle rules, object tags, UploadPartCopy, CORS, canned ACLs with anonymous reads, browser form uploads, and SSE-C.
• CRC32C and CRC64NVME checksums, which the AWS CLI v2 needs to upload anything.
• Shared commits for writes queued in a worker.
• Tests that cut the power at the block layer and fail disks under a running server, on ext4, XFS and btrfs.

Still to come in this phase:

• Running it for real, as a backup target for rclone for a few weeks, then for something that matters more. The serious bugs found lately all came from running it under load or on failing disks.
• A machine with two real disks, to measure what one drive can't show.
• A published container image and a fuller user's guide.
• Storing small objects inline in the metadata, without a file each. It would remove the two flushes of a blob file that limit small writes. But the metadata lives on one disk, so with two copies an inlined object would have only one, which breaks what copies promise. The choice is open between inlining only in a store keeping one copy, keeping a second copy of the metadata at a cost on every commit, and leaving small objects as they are. The answer may also depend on Phase 3, where the metadata moves to a replicated store and an inlined object would be copied with it.

Phase 3: cluster¶

              S3 clients
                  │
            load balancer
          ┌───────┼────────┐
     ┌────▼───┐┌──▼─────┐┌─▼──────┐
     │gateway ││gateway ││gateway │   stateless; today's code
     └─┬───┬──┘└─┬───┬──┘└─┬───┬──┘
       │   └─────┼───┼─────┘   │
       │  SQL    │   │         │  internal blob API
   ┌───▼─────────▼┐  │         │
   │ metadata     │  │         │
   │ store        │  │         │
   └──────────────┘  │         │
              ┌──────▼─┐ ┌─────▼──┐ ┌────────┐
              │ node A │ │ node B │ │ node C │
              └────────┘ └────────┘ └────────┘

Gateways run today's S3 code and hold no state. Storage nodes store blobs behind a small HTTP API that knows nothing of buckets or keys: put, get with ranges, head and delete a blob, and report capacity. A small deployment runs both roles on each machine.

Writes stream the body to several nodes at once, on separate failure domains, and commit the object with the copies that succeeded, as long as enough did: by default 3 copies and 2 needed. Reads continue from another copy at the same offset on failure. Nodes report a heartbeat to the metadata store, and repair waits a while before copying, so a reboot doesn't start mass copying. Since blobs never change and all commits go through one database, the data tier can't split into two versions of the truth.

An existing single server upgrades by copying its metadata to the shared store. Its disks become the first node's volumes, and stored files stay where they are.

Phase 3 is done when a chaos test under load loses no acknowledged object. The test kills a node and a gateway, cuts a node off and fails over the metadata store. Copy counts must then return to target. A history checker over concurrent clients must also find that read-after-write held.

The metadata store¶

Because all consistency lives in the metadata, the cluster's metadata store is where Phase 3 succeeds or fails. It must offer:

• transactions that run code between statements, since that is how versioning, lifecycle rules, multipart uploads and repair stay readable;
• linearizable commits and reads;
• ordered range scans in byte order;
• commits that stay durable when any one node is lost, and later any one location;
• a way in from SQLite that leaves stored blobs where they are;
• an open license, with nothing an operator needs a commercial license for.

The recommendation is to give meta.py a second implementation for PostgreSQL's dialect and protocol, run on PostgreSQL first, and later unchanged on YugabyteDB, which adds several writers and locations. CockroachDB is technically suitable but no longer open source. Replicated SQLite (rqlite, dqlite, LiteFS) can't offer both interactive transactions and commits that a failover can't lose. Writing its own consensus or CRDTs is ruled out.

Before building it, a spike of about a week would port meta.py behind an interface, run the integration tests against PostgreSQL and YugabyteDB, measure a small PUT's commit over a network, and settle isolation under concurrent gateways.

Several locations¶

In a cluster across locations, an acknowledged write either has already reached another location or hasn't. Waiting for it keeps the write through the loss of a location, but costs each write at least one round trip between locations. Copying in the background stays local, but loses what wasn't copied yet when a location is lost, as S3's cross-region replication does. A third way lets each bucket choose.

Open decisions¶

Three decisions are open:

1. The metadata store: PostgreSQL's dialect, on PostgreSQL first and YugabyteDB later, or another option.
2. Several locations: synchronous writes across locations, asynchronous copying, or a choice per bucket.
3. Failover: running Patroni and etcd next to PostgreSQL for the first clusters, or YugabyteDB from the start, which fails over by itself but is heavier to run.

The rest of Phase 3 (the storage node and its API, failure detection, placement across failure domains) can be built against any answer.

Phase 4: cluster at scale¶

Most of this phase is metadata updates and background copying, because the metadata records where every blob is:

• Rebalancing: copy a blob, add the new location, remove the old one, and queue the old bytes for collection. Reads in flight are unaffected.
• Draining a node or a disk.
• Erasure coding, with a redundancy policy per bucket, such as 8+3. Blobs never change, so stripes are never rewritten. The coding itself uses a native library.
• Rolling upgrades, with a versioned internal API and schema changes that only add.

Acceleration¶

Native code replaces a Python hot path only when a profile shows it matters. So far none has:

• Hashing uses hashlib, zlib and awscrt, which are native already.
• HTTP parsing uses uvicorn's native parser.
• Flushes to disk limit small durable PUTs, and native code wouldn't change them.

Compared with MinIO and Garage on the same machine, large GETs and listings are level or ahead, and small GETs are about half MinIO's rate, with no single hot spot to rewrite. Candidates, should profiles point there, include aws-chunked decoding and XML rendering as extensions, sendfile for GETs through an ASGI server that supports it, and a storage node rewritten in Rust behind the same HTTP API.