Skip to content

leBLOB, S3 storage for the masses

leBLOB: S3 storage for the masses leBLOB: S3 storage for the masses

leBLOB is an S3-compatible object store written in Python. One command starts it on one machine, for any S3 client to use. It is designed to grow into a redundant cluster across several locations. It is meant to replace self-hosted MinIO and Garage, starting with single-server installs.

Development status

leBLOB is alpha software. It runs on one server, with one or several disks, and clusters aren't built yet. Run it next to your existing backups before you depend on it.

In six slides

leBLOB, in six slides (PDF)

Why leBLOB?

Tested to keep data safe

An object is acknowledged only once it is on disk and committed, so a power cut can't lose it. A client never sees half an object. This is tested by replaying power cuts at the block layer on ext4, XFS and btrfs, next to a build that skips one fsync, which the test has to catch. See Testing.

One machine first, a cluster later

Blobs never change once written, and all consistency lives in the metadata. Copies of a blob can't disagree, so the storage needs no quorums. A cluster keeps the same model and changes only the metadata store. See Architecture.

Real clients, unmodified

boto3, the AWS CLI v1 and v2, rclone and s5cmd are tested end to end over HTTP and HTTPS, and 349 tests of Ceph's s3-tests pass. A request leBLOB can't honour gets an error, so a client is never told a promise was kept when it wasn't.

Small enough to read

leBLOB is a Python program with no web framework. It keeps metadata in SQLite and data in files. One person can read the codebase and keep it in their head.

Features

  • Buckets, objects, ranges, conditional requests and listings
  • Multipart uploads, CopyObject and UploadPartCopy
  • Versioning, lifecycle rules and object tags
  • CORS, public-read buckets and objects, and form uploads
  • SigV4 in headers and presigned URLs, aws-chunked bodies and checksums from MD5 to CRC64NVME
  • Encryption with the client's own key (SSE-C)
  • Several disks, with a number of copies of each blob
  • A read moves to another copy if a disk fails, even mid-response
  • Background scrub and repair
  • Draining a disk before removing it
  • Access keys with full access, or scoped per bucket at read, write or admin level
  • Metadata backup while the server runs, restore and fsck
  • Prometheus metrics and structured access logs
  • TLS, a Dockerfile, and upgrades tested from each release's stored data

See Features for the full list, and Known Limits for what isn't there yet.

Getting started

Run it from a checkout, with uv:

git clone https://git.sr.ht/~sfermigier/leblob
cd leblob
uv run leblob serve --data-dir ./data

The first start creates an access key and prints it once. Then follow the Quick Start.

Source and license

The code is at git.sr.ht/~sfermigier/leblob, under the Apache 2.0 license.