Testing¶

Most of the defects that mattered in leBLOB were found by real clients, real disks and a second platform, not by unit tests. So the tests go all the way down: from pure functions, through a real server driven by real S3 clients, to power cut at the block layer.

Commands¶

To run one test, or one layer:

uv run pytest tests/a_unit/test_routing.py::test_root_lists_buckets
uv run pytest -m unit          # or -m integration, -m e2e
uv run pytest -m "not slow"    # skip the kill -9 crash loop

The layers¶

Markers come from the directory. Tests use no mocks, and async tests use anyio.

The end-to-end tests run each client over both schemes because clients send different requests over each. boto3 over HTTPS sends aws-chunked with a trailing CRC32 and no Content-Length. The AWS CLI v2 uses aws-chunked for parts even over plain HTTP. A server that only handled plain bodies would pass the tests over HTTP and fail as soon as TLS was on.

The rclone, s5cmd and AWS CLI v2 tests are skipped if those tools aren't installed. The v2 tests look for aws outside the virtual environment, whose own aws is v1.

The tests ignore the developer's ~/.aws/config, which once hid that boto3 presigns with SigV2 by default. They also turn off botocore's CRT signer, which most boto3 users don't run, so boto3 in the tests can't send crc32c or crc64nvme: the AWS CLI v2 and make compat cover those.

tests/c_e2e/test_durability.py kills the server with kill -9 in a loop under PUT load, then checks that every acknowledged object reads back with the right hash and no partial object is visible. A build that acknowledges before committing loses objects and fails it.

tests/upgrade/stores/ holds a sample store written by each release, and the tests open each with the current code.

S3 compatibility¶

make compat downloads ceph/s3-tests and runs it against a real server, with lifecycle days shortened to 2 s so the suite's own timing tests can run. 349 tests pass. The run fails if any of those named in tests/compat/s3-tests-passing.txt stops passing.

When a change is meant to alter that list, update it with uv run python tests/compat/run_s3tests.py --update, which names any test it drops. A rising or falling count isn't enough to judge a change: some encryption tests once passed only because the server ignored the encryption headers.

Power cuts¶

make power runs a real server on loop devices in a privileged container on Linux, through dm-log-writes, which records the block writes and flushes the server causes. The log is then replayed up to a chosen point onto a copy of the filesystem, which gives what a disk would hold had the power failed there. Half of the cuts fall right after a flush. The others fall inside the writes before one, keeping a random half of them, as a drive's cache may.

Four writers run PUTs, overwrites, deletes and multipart uploads, with the collector running as if the grace period were over. Each operation is marked in the log when acknowledged. At each cut:

• acknowledged operations must all be there, plus at most the one in flight;
• objects must hold exactly the right bytes, with no unknown object among them;
• leblob fsck must accept the store;
• a new server must start on it and take a write.

The harness refuses to judge unless replaying the whole log reproduces the device byte for byte. POWER_ARGS=--two-copies runs the same on three volumes, two copies of each blob, one volume draining throughout and versioning on.

A test that can't fail proves nothing. POWER_ARGS=--control runs a build that never flushes a blob's file, and passes only if every filesystem then has failed cuts. Earlier runs of builds with one protection removed showed that each of them is needed on all three filesystems:

The real build failed no cut in 160 on each filesystem, on Linux 7.2.9 with the store on loop devices over tmpfs.

It can't see a block written in part, a drive that acknowledges a flush it hasn't done, or a promise broken by only milliseconds. A build that acknowledged writes just before committing them passed 80 cuts, because acknowledging and marking the log takes longer than the commit. Such an ordering needs a test of its own, as test_no_one_is_told_a_write_is_done_before_the_commit is for shared commits.

Run it from a Mac on a Linux machine with Docker, through DOCKER_HOST=ssh://host make power or a Docker context. OrbStack's VM lacks ext4 and XFS. tests/power/README.md says how to run and read these tests, and when to.

Failing disks¶

make disks runs a real server on ext4, XFS and btrfs over device-mapper, and uses dm-error to make all I/O to a mounted volume fail with EIO, as a dying disk does. Testing with chmod instead would give EACCES, and wouldn't stop root. It checks that:

• with one copy, objects on the other volume still read, and reads on the dead one fail cleanly, with no wrong bytes;
• a full volume sends new blobs elsewhere. When the only disk is full, PUTs fail until there is room and leave nothing behind;
• with two copies, a volume dying or a disk pulled under load causes no client error, and volume forget --disk-lost brings the blobs back to two copies with a clean fsck;
• a block corrupted under the filesystem is found by scrub and repaired.

Continuous integration¶

A push runs three jobs on sourcehut, with manifests in .builds/:

• alpine.yml and ubuntu-2404.yml: the tests, the linters, then nox across Python 3.12 to 3.14. Alpine's aws-cli package is v2, so the v2 tests run there.
• disks.yml: on an Ubuntu 24.04 VM, which gives root, loop devices and device-mapper with no container, make disks, then the power test with 20 cuts, with two copies, and as the control.

Measuring¶

Keep measurement runs to minutes. uv run python -m tests.bench.run_scale fill --dir D --keys 1000000 fills a store with 1M keys in under a minute. Its other subcommands probe, read, run a sustained load and check it. tests/bench/baseline.md holds the results so far, with what each was measured on.

Don't judge write speed on macOS, whose fsync doesn't flush the drive's cache: a 1 KiB PUT looks 10 to 30 times faster there than on Linux.