Using S3 clients¶

leBLOB is tested end to end with boto3, the AWS CLI v1 and v2, rclone and s5cmd, over HTTP and HTTPS. Other S3 clients should work if they sign with SigV4.

What every client needs¶

The endpoint, such as http://127.0.0.1:9000, or the https:// address of your server.
An access key and its secret, from the first start or from leblob key create. See Access keys.
A region. Any name works, since a single store has one location. The examples use us-east-1.
SigV4. SigV2 and SigV4A are refused.

Addressing¶

S3 names the bucket in one of two ways:

Path-style: https://s3.example.org/my-bucket/my-key. This always works. boto3 and the AWS CLI use it with a custom endpoint.
Virtual-hosted: https://my-bucket.s3.example.org/my-key. Start the server with --domain s3.example.org to accept it. Every bucket name then needs to resolve to the server, usually through a wildcard DNS record, and HTTPS needs a certificate that covers *.s3.example.org. Path-style requests keep working with --domain set.

HTTPS¶

Start the server with --ssl-certfile and --ssl-keyfile to serve HTTPS itself, or put it behind a reverse proxy that terminates TLS; see Running in production. With a self-signed certificate, clients need to be told: --no-verify-ssl for the AWS CLI and s5cmd, --no-check-certificate for rclone, and verify=False or a path to the certificate for boto3.

Clients send different requests over HTTPS. boto3, for one, sends a plain body with a Content-Length over HTTP, but over HTTPS it streams the body as aws-chunked, without a Content-Length, and sends the checksum in a trailer. leBLOB accepts both. It matters if a proxy sits in front of it: the proxy has to pass streamed bodies through.

boto3¶

import boto3
from botocore.config import Config

s3 = boto3.client(
    "s3",
    endpoint_url="https://s3.example.org",
    aws_access_key_id="...",
    aws_secret_access_key="...",
    region_name="us-east-1",
    config=Config(signature_version="s3v4"),
)

signature_version="s3v4" matters for presigned URLs only. Without it, boto3 presigns with SigV2, and leBLOB refuses those URLs with 403 AccessDenied and the message "Anonymous access is not allowed". Ordinary requests are SigV4 either way. A ~/.aws/config can set this for you on one machine and not on the next, so set it in code.

The transfer manager (upload_file, download_file, copy) works, including multipart copies with UploadPartCopy. Form uploads can be signed with generate_presigned_post.

boto3 computes a CRC32 for each upload by default. For CRC32C or CRC64NVME, boto3 needs awscrt installed on the client side.

AWS CLI¶

Environment~/.aws/config

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=us-east-1
aws --endpoint-url https://s3.example.org s3 ls

[profile leblob]
region = us-east-1
endpoint_url = https://s3.example.org

With the keys in ~/.aws/credentials under [leblob], then:

aws --profile leblob s3 ls

Both versions work: version 1 is what pip install awscli gives, and version 2 is the one AWS distributes. Version 2 sends a CRC64NVME checksum on every upload by default, and its CRT transfer client streams parts as aws-chunked even over HTTP. leBLOB accepts all of it, so there is nothing to set.

rclone¶

[leblob]
type = s3
provider = Other
endpoint = https://s3.example.org
access_key_id = ...
secret_access_key = ...
force_path_style = true

provider = Other and force_path_style = true are what the tests use. rclone sends x-amz-bucket-object-lock-enabled: false when it creates a bucket, which leBLOB accepts, since it asks for nothing. Large files go up in multipart uploads, and rclone copyto between two objects on the server copies in parts on the server once the file is over --s3-copy-cutoff.

With --metadata, rclone copies an object's headers and x-amz-meta-* and adds mtime and btime. See Moving from MinIO or Garage.

s5cmd¶

export AWS_ACCESS_KEY_ID=...
export AWS_SECRET_ACCESS_KEY=...
export AWS_REGION=us-east-1
s5cmd --endpoint-url https://s3.example.org cp ./photos/ s3://my-bucket/photos/
s5cmd --endpoint-url https://s3.example.org ls s3://my-bucket/

Checksums¶

leBLOB verifies, stores and returns Content-MD5, x-amz-content-sha256, and x-amz-checksum-* for CRC32, CRC32C, CRC64NVME, SHA-1 and SHA-256. A request whose body doesn't match its checksum is refused before anything is stored. A client that asks with x-amz-checksum-mode: ENABLED gets the stored checksum back on a read.

A multipart object gets an object-level checksum worked out from the parts. A composite one ends in -N, and clients don't check it against the bytes they read. A full-object one is checked by clients that read the whole object.

What leBLOB refuses¶

A request asking for something leBLOB can't do gets 501 NotImplemented. That includes encryption other than SSE-C, ACL grants other than private and public-read, Object Lock, storage classes other than STANDARD, and bucket policies. A client that sets one of these by default needs it turned off. The S3 compatibility reference has the full list.