Skip to content

Contributing

Setting up

leBLOB needs Python 3.12 or later, and uv.

git clone https://git.sr.ht/~sfermigier/leblob
cd leblob
uv sync
uv run leblob serve --data-dir ./data    # the first start prints an access key once
make test
make lint

Development happens on the devel branch.

The end-to-end tests use rclone, s5cmd and the AWS CLI v2 if they are installed, and skip those tests otherwise. make power and make disks need a Linux machine with Docker that can run a privileged container: see Testing.

Before changing storage or consistency

Read the architecture, and keep its invariants. In particular:

  • The metadata commit is the only point of visibility. Bytes are durable before it.
  • No transaction spans I/O. Each public function in meta.py is one transaction, written as synchronous code and decorated with @_read or @_write.
  • User keys never become file paths.
  • Background work runs in every worker with no leader, so each step must be safe to run concurrently.
  • Logging and similar setup happens in asgi.create_app_from_env, which runs in each worker, not in cli.serve, which runs in the parent.

After changing what makes data durable, run make power, and check that it still fails on a build missing an fsync, with make power POWER_ARGS=--control.

S3 behaviour that is easy to break

  • Routing and signatures read scope["raw_path"], never scope["path"], which uvicorn has already decoded.
  • Refuse what can't be honoured, with a 501. Before refusing a header, check what real clients send: rclone sends x-amz-bucket-object-lock-enabled: false on every CreateBucket.
  • Every x-amz-* header must be signed, including on presigned URLs.
  • A scoped key is authorized once, before any handler looks at a bucket, so being refused says nothing about what exists. An operation missing from the table in s3/access.py is closed to scoped keys. A handler that reads a second bucket must check it too.
  • Validate untrusted values at the boundary, before they reach hmac.compare_digest (which raises on non-ASCII) or a response header.
  • Test client-facing behaviour over HTTP and HTTPS. The server fixture in tests/c_e2e/conftest.py runs a test over both.

Conventions

  • Functional core, imperative shell. Logic goes in pure modules, s3/ and model.py, which do no I/O. asgi.py is the only place an S3Error becomes a response.
  • No interface with one implementation. meta.py and blobstore.py have none in front of them until a second implementation exists.
  • Never silence a linter. src has no # noqa or # type: ignore comments. When ruff reports too many arguments, look for a missing noun: UploadRef and ListQuery came from that.
  • Mark shortcuts. A shortcut with a known ceiling gets a ponytail: comment naming the ceiling and how to lift it, and a line in Known limits.
  • No mocks. Tests use real SQLite, real files and real servers. Async tests use anyio.
  • Keep the documentation in step. The pages under docs/src describe what is built, and change with it. Markdown uses soft wraps: one paragraph per line.
  • Commit messages follow Conventional Commits: a short first line, then if needed one paragraph wrapped at 72 characters. A message must make sense without the files it touches.
  • Measure before optimising. Python is a requirement of the project. A hot path moves to native code only when a benchmark justifies it. Keep measurement runs to minutes, and judge write speed on Linux only.

Compatibility tests

make compat fails if any test listed in tests/compat/s3-tests-passing.txt stops passing. When a change is meant to alter the list, update it with uv run python tests/compat/run_s3tests.py --update, which names any test it drops, and say why in the commit.