Known limits¶
leBLOB is early software. This page lists what it doesn't do yet, and where it does something with a known ceiling. Each limit comes with the reason for it and the way out, should it start to matter. Figures were measured on one Linux machine with btrfs on a consumer NVMe drive, shared with other work, unless said otherwise. Treat them as orders of magnitude.
Scope¶
- One server. Volumes and copies spread a store over several disks of one machine. Clusters, with several servers and several locations, are the next phase: see the roadmap.
- One owner. All keys belong to one account. Scoped keys limit a key to some buckets, but there are no bucket policies, no IAM, and no grants to other accounts.
Speed¶
| Limit | Why | Way out |
|---|---|---|
| About 320 small PUTs/s at 32 clients and 4 workers | Every write is flushed to disk before it is acknowledged, and SQLite has one writer at a time. Writes queued in a worker share a commit, but workers take turns | Sharing commits across workers; the shared metadata store of the cluster phase |
| A PUT can stall for seconds under heavy sync load: p99 of 2 to 4 s, 15 s at most | Measured below leBLOB, in the filesystem or the drive, on a shared host | Not yet measured on an idle host, ext4 or XFS, or a drive with power-loss protection |
| One file per blob is costly for many tiny objects | Inode and block overhead, and 4 flushes per PUT | Storing small objects inline in the metadata, planned |
| CopyObject takes time in proportion to the size | The copy gets new bytes; blobs aren't shared | Reference counts |
| GET passes through user space | uvicorn has no sendfile support |
An ASGI server that has it |
| Repair restores about 92 blobs/s of empty files: a day and more for 10M blobs | Each copy is committed separately | Record a batch of copies in one transaction |
| Lifecycle rules walk 2,000 versions/s and remove about 200 keys/s, so 10M keys take about 80 min | Slow enough to leave the writer to clients | Settings, if a store needs another rate |
fsck loads every blob ID into memory: 4.51 GiB and 548 s at 10M keys, from a cold cache |
A simple comparison | A sorted merge |
Copies and disks¶
| Limit | Why | Way out |
|---|---|---|
| A PUT is acknowledged with fewer copies than asked for, even one, until repair catches up | Availability chosen over refusing the write | A minimum number of copies, if someone needs it |
A failed disk needs a person to run volume forget --disk-lost before its copies are replaced |
On one server, a disk that is away may well come back | Failure detection, in the cluster phase |
| A server with one copy won't start with a volume missing | Those objects couldn't be read | Decide per object, if someone needs it |
| With one copy, a PUT that started on a disk that then fails, fails | Its bytes are gone and can't be replayed | Use two copies |
A disk that hangs rather than fails stalls requests that use it, and /-/metrics too |
No timeouts on disk I/O | Timeouts per volume |
| A bad copy that reads without error is served until scrub reaches it, unless the client checks a full-object checksum | Reads don't check a blob's MD5 | Verify whole-blob reads, if the cost is worth it |
| Scrub runs at one fixed rate, 32 MiB/s across the store, and reads a single large blob at full speed | No setting until someone needs one | A setting, and pacing within a blob |
On btrfs, leblob_volume_up can stay 1 for a dead device |
It answers from cache | Alert on leblob_volume_failures_total |
| Copies beyond the number wanted are never removed after lowering it | Harmless | Trim them in repair |
| A volume can't move to another path | Not needed yet | A command to move a volume |
| Two volumes on one filesystem get only a warning | Useful to try things out | Refuse it, if someone is caught out |
fsck reports a missing blob by ID, so finding the affected keys takes a lookup |
It compares blobs only | Look up the keys, if restores need it |
| On a full disk, new workers may be able to answer only with errors, even to reads, until there is room | SQLite must write to recover its log | Keep space in reserve |
| A worker can place a blob on a just-added volume up to 2 min before the others can read it there | Workers follow changes each maintenance round | Follow at once on an unknown volume |
S3 behaviour¶
| Limit | Why | Way out |
|---|---|---|
Creating an existing bucket again is always 409 |
AWS answers 200 in us-east-1, for historical reasons |
Match it, if a client needs it |
Retrying CompleteMultipartUpload more than a day later gets NoSuchUpload |
Completed uploads are remembered for a day | Keep them longer |
| Non-ASCII metadata comes back as raw bytes | S3 encodes it as RFC 2047 | Encode it, if a client needs it |
ListMultipartUploads ignores delimiter |
No client seen needing it | Reuse the object listing's scan |
A conditional write or delete is checked against the current version only, and If-Match on a delete by versionId is ignored |
Rarely used | Refuse or honour it, if a client sends it |
| CopyObject doesn't keep the source's checksum | The copy is written as a new blob | |
| Encrypted parts must reach the 5 MiB minimum by their stored size, a few hundred bytes more than what was sent | The check doesn't know about encryption | Pass it the plain size |
Tag keys starting with aws:, which AWS reserves, are accepted |
Nothing here gives them a meaning | Refuse them, if portability matters |
| A form upload can't be encrypted | Not built | |
Default boto3 presigned URLs get a misleading "Anonymous access" 403 |
They use SigV2, which leBLOB refuses | Name SigV2 in the error |
Lifecycle rules¶
| Limit | Why | Way out |
|---|---|---|
| A rule can take a day to start acting; an expiry can run up to a day late | A pass runs once a day | Start a pass when rules change |
Days count exactly from the object's time, while AWS rounds up to the next midnight UTC. Both wait the full number of days |
A pass runs at no set hour anyway | Round up, if a client relies on the date |
| A key with a great many versions, or many uploads between two keys, makes one long batch | A batch keeps a key whole | Walk them in stretches |
Transitions are refused. x-amz-expiration isn't sent |
There is one storage class. The header would cost a query on each read | Wait for a client that uses it |
| A key with write access can tag the current version, so that a rule on tags later erases it once replaced | Tagging an older version takes admin, but a current version becomes an older one | Don't rely on a rule on tags to protect against a stolen write key |
Security¶
| Limit | Why | Way out |
|---|---|---|
Access keys' secrets are stored in plaintext in meta.db, and in its backups |
SigV4 needs the secret itself | Encrypt them with a master key from the environment |
/-/health and /-/metrics answer without a signature |
Prometheus can't sign | Block them at the proxy on a public store |
| A public-read object or bucket can't be forbidden store-wide | There is no "block public access" switch yet |
Platforms¶
| Limit | Why | Way out |
|---|---|---|
On macOS, fsync doesn't flush the drive's cache, so a power cut can lose acknowledged writes |
Linux is the platform leBLOB is built and tested for | F_FULLFSYNC, if it is ever deployed on macOS |
| Power-loss tests don't model blocks written in part, or drives that acknowledge a flush they haven't done | The test replays whole block writes | A drive with power-loss protection removes the second risk |