Disks and copies¶
A store starts with one volume, the data directory. Add volumes on other disks to grow it, and raise the number of copies so that losing a disk loses no object. All of this happens on one server, and none of it needs a restart: a running server follows changes within a couple of minutes.
Volumes¶
A volume is a directory, usually the mount point of a disk, holding a tmp/, a blobs/ and a volume.json that names it. Each blob is stored whole on one or more volumes, and the metadata records which.
volume add takes an existing empty directory. New blobs go to it from then on. Blobs already stored stay where they are, unless more copies are wanted.
volume list shows how full each volume is. It also lists any that can't be used.
The data directory is always volume 1, and always stays a volume.
One copy
With one copy of each blob, which is the default, losing a disk loses every object with a blob on it. Adding a second disk without raising the number of copies spreads that risk over two disks. Raise it as below.
Where new blobs go¶
Each new blob goes to volumes chosen at random, weighted by their free space, so a fuller disk receives less. A volume that fails while a blob is being started is passed over for the next one. leBLOB warns if two volumes are on the same filesystem, which is useful to try things out but protects nothing.
How a volume is recognised¶
volume.json and the metadata both hold the volume's identity. A volume counts only if the two match. An unmounted disk leaves an empty directory at its mount point, and writing there would fill the disk underneath and hide those blobs once the disk is mounted again.
So a missing or wrong volume.json makes the volume unusable:
- With one copy of each blob,
leblob serverefuses to start, since some objects couldn't be read. - With two or more copies, the server starts without that volume and logs a warning.
leblob fsckrefuses either way, because it can't compare a volume it can't read.
A volume can't move to another path yet.
Copies¶
sets how many volumes hold each blob, at most the number of volumes. Each copy needs a separate disk to protect against losing one. The number is kept in the metadata, so a restart can't lower it by mistake.
A running server then adds the missing copies of blobs already stored, in the background. Copies beyond the number are left in place if it is lowered later.
Writes¶
A blob is hashed once and then written to the volumes chosen. A PUT waits until the copies are durable before the metadata commits, so it waits for the slowest disk. A copy that fails is dropped. The blob commits with the copies that succeeded: refusing the write wouldn't make the remaining copies any safer. The PUT fails only if no copy is left, and repair later brings the blob back to the number of copies wanted.
So an acknowledged PUT can have fewer copies than asked for, even one, until repair catches up. fsck counts blobs with too few copies, without calling that damage, and leblob_repairs_queued in the metrics shows how many are waiting.
Reads¶
Each read starts on a random copy. On an I/O error or a short read, it moves to the next copy at the same offset, in the middle of the response if need be. That is safe because blobs never change, so copies are identical. A volume that failed is tried last for 30 s by that worker, so a dead disk doesn't keep costing requests.
Scrub and repair¶
Scrub reads the copies from the disk and checks it against the MD5 recorded when the blob was written. It drops the page cache for the file first, so it reads what is really on the disk. A running server scrubs all the time at a store-wide rate of 32 MiB/s, shared by the workers, and rests for a day between passes on a store that hasn't changed. Progress is kept in the metadata, so a pass resumes after a restart.
A bad or missing copy is removed, and repair replaces it. The only copy of a blob is never removed, however bad: it is logged as an error and counted in leblob_only_copies_bad_total. An unmounted volume is left alone, since the copies on it would all look missing.
runs a full pass now, at full speed. It exits non-zero if a blob's only copy is bad. A running server replaces the bad copies it removes.
Repair restores copies: for a blob written with too few, after volume copies is raised, after scrub removes a bad copy, and when a volume is drained or forgotten. Workers look for work every 5 s. A new copy is read from a good one, checked against the MD5, made durable, and only then recorded. It restores about 92 blobs/s of empty files, measured on the Linux test machine (btrfs on a consumer NVMe), because each copy is committed separately: a day or more for 10M blobs.
A bad copy that reads without an I/O error is served until scrub reaches it, unless the client checks a full-object checksum. leBLOB doesn't check a blob's MD5 when reading it.
Retiring a disk¶
To take a disk out of service while it still works:
leblob volume drain 2 --data-dir /srv/leblob
leblob volume list --data-dir /srv/leblob # wait until volume 2 is empty
leblob volume forget 2 --data-dir /srv/leblob
drain stops new blobs going to the volume and has a running server move what it holds elsewhere. A copy on a draining volume doesn't count towards the number of copies, so repair makes a new one before removing it. forget then removes the empty volume from the store.
When a disk dies¶
With two or more copies, objects stay readable from the other copies, and writes go on to the remaining volumes. Then, once the disk is known to be gone for good:
forgets the copies on it and queues their blobs, so a running server makes new copies on the remaining volumes. leBLOB doesn't decide that a disk is gone by itself, because on one server a disk that is away may well come back: a person has to run this.
If some blobs had their only copy on that disk, forget refuses, unless told --accept-loss. Their objects are then lost, and fsck reports those blobs.
With one copy, a dead disk means the server won't start until the disk returns or is forgotten with --disk-lost --accept-loss.
What happens when disks die, fill up or rot under a running server is tested on real block devices: see Testing.
Watching the disks¶
Monitoring lists the metrics: for each volume whether it can be used, whether it is draining, free bytes and I/O errors.
btrfs
On btrfs, leblob_volume_up can stay at 1 for a dead device, which still answers from cache. The counter leblob_volume_failures_total still counts its errors, so alert on it.