Architecture¶

The design bet¶

Blobs are immutable, and all consistency lives in the metadata.

An object is an ordered list of blobs, recorded in a metadata database. A blob has a random name and is written once. Copies of a blob therefore can't disagree: a copy is either present and correct, or missing or damaged, and a checksum tells which. The storage layer needs no quorums, version vectors or conflict resolution.

The hard problem of a distributed store then becomes "have a consistent database". Each phase solves that with the simplest tool that fits. On one server, that is SQLite. For clusters, it is still to be chosen: see the roadmap.

The price is that the metadata database limits how far a store scales and how available it is. That is accepted, because the scale leBLOB aims at fits within it.

Invariants¶

These hold in every phase, and later phases depend on them, so breaking one is a change of architecture.

1. The metadata commit is the only point of visibility. A blob is flushed and renamed into place before the metadata commits, so a failure before the commit changes nothing visible. Readers see the old state or the new one.
2. Blobs are immutable and randomly named. User keys never become file paths, which removes path traversal and every key-encoding problem from the storage layer.
3. An object is an ordered list of blobs. A PUT makes one blob, and a multipart upload one per part. Completing an upload is a metadata operation and copies no data.
4. The metadata records where blobs are. Placement is looked up in a table. Moving a blob means copying it, then updating a row.
5. Workers hold no authoritative state. Several worker processes share the database and the files, and background work runs in every worker with no leader, so each step must be safe to run concurrently. Several gateways on several machines will be the same case.
6. Deleting is deferred. A transaction removes the reference and queues the blob. A collector removes the bytes after a grace period of an hour, so reads in flight can finish.
7. No transaction spans I/O. Each metadata function is one complete transaction, run on a worker thread, and callers can't hold one open across an await. A slow client can't block writers.

Some follow from these, and are just as easy to break:

• A volume only counts if its volume.json matches the metadata, because an unmounted disk looks like an empty directory.
• A write commits with the copies that succeeded. A read moves to another copy at the same offset, which immutability makes safe. Repair records a copy only after it is durable.
• A worker never refuses to start, because the server would restart it for ever with nothing answering clients. It starts with no volumes, answers with errors, and retries. leblob serve is where a wrong store is refused.
• The current version of an object stays in objects, and older versions and delete markers in object_versions, so moving between them touches no bytes. A blob is queued for collection only when no version refers to it any more.
• A lifecycle rule deletes data, so the parser for rules refuses anything it doesn't understand: skipping a condition could widen what a rule applies to.

Functional core, imperative shell¶

Logic and I/O are kept apart. The pure modules take data and return data, and are tested without a server or a disk.

src/leblob/
  cli.py          serve, key, volume, fsck, scrub, backup, restore
  asgi.py         the HTTP boundary; the only place an S3Error becomes a response
  handlers.py     one async function per S3 operation
  streams.py      an object's bytes in and out: decoding, checking, encrypting, storing
  meta.py         every SQL statement; one function per transaction
  blobstore.py    blob files and volumes
  maintenance.py  collector, repair, scrub, lifecycle passes, fsck
  metrics.py      counters for /-/metrics, added up across workers
  model.py        plain data shared by every layer                  [pure]
  s3/
    routing.py    request → operation, bucket, key; refusals         [pure]
    access.py     what a scoped key may do                          [pure]
    sigv4.py      signatures, presigned URLs, form policies         [pure]
    chunked.py    aws-chunked decoding, trailers, chunk signatures  [pure]
    payload.py    what headers promise about a body, and checking it [pure]
    conditions.py conditional requests and byte ranges              [pure]
    listing.py    prefixes, delimiters, continuation tokens         [pure]
    multipart.py  assembling parts, multipart checksums             [pure]
    lifecycle.py  judging keys against lifecycle rules              [pure]
    cors.py       matching CORS rules                               [pure]
    sse.py        SSE-C encryption format                           [pure]
    tagging.py    object tag rules                                  [pure]
    post_form.py  parsing browser form uploads                      [pure]
    xml.py, xml_config.py, xml_base.py   XML in and out             [pure]
    errors.py     S3Error and the table of S3 error codes           [pure]

listing.scan takes a fetch function, so the scan runs without a database.

meta.py and blobstore.py have no interface in front of them, because each has one implementation. An interface comes with the second implementation, planned for clusters. Until then, anything specific to SQLite stays inside meta.py.

There is no web framework. Litestar was measured to rewrite keys: foo/ became /foo and a//c became /a/c, which would corrupt real data without an error. leBLOB is a bare ASGI application on uvicorn, and routing and signatures read the raw path as the client sent it. Its runtime dependencies are uvicorn, httptools, uvloop, Cyclopts, defusedxml, cryptography and awscrt.

Writing an object¶

1. Route the request and check the signature, before reading any body.
2. Read the body in 1 MiB buffers: decode aws-chunked if used, and on a
   worker thread write each buffer to tmp/ while updating the hashes.
3. Compare size, hashes and chunk signatures with what the client declared.
4. Flush the file, rename it into blobs/, flush the directory.
5. In one transaction: check the bucket and any condition, insert the rows,
   queue the replaced blobs for collection.
6. Answer with the ETag.

Any failure before step 5 leaves the store unchanged: a bad signature on a late chunk, a client that goes away, a checksum that doesn't match, or a crash. After a crash, a file left in tmp/ is swept once it is a day old. fsck reports a file in blobs/ with no row as an orphan.

The metadata store¶

SQLite in WAL mode with synchronous=FULL, since NORMAL can lose acknowledged commits on power loss. Keys sort in UTF-8 byte order, which SQLite's BINARY collation gives, so a listing is a range scan of the primary key. With a delimiter, the scan skips over each common prefix, so a page costs what it shows.

Writers queue on a kernel lock. SQLite's own busy handler sleeps up to 100 ms between tries, which left writers asleep with the lock free and gave a p99 above a second under contention. Writers take a flock on meta.db.writer-lock instead: the kernel wakes the next one on release, and releases the lock if its process dies.

Writes in a worker share a commit. A commit takes as long as the disk needs to flush. Writes queued in a worker run in one transaction, up to 32 at a time, each in a savepoint of its own, so one that fails takes none with the others. No caller hears of success before that commit is durable, and if it fails, all of them failed. So each public function in meta.py still behaves as its own transaction. At 32 clients and 4 workers, this took small PUTs from about 155/s to about 320/s on the Linux test machine (btrfs on a consumer NVMe).

Background work¶

Every worker runs the same low-frequency loops, with no leader: the collector, the temp sweep, repair, scrub and lifecycle passes. Each step is safe to run concurrently. Where a store-wide pace matters, as for scrub and lifecycle rules, the cursor and pace live in the metadata, and the writer lock takes workers one at a time.

Phases¶

Alongside them, native code replaces a hot path only when a profile shows it matters. See the roadmap.