Monitoring¶

leBLOB answers two paths that no bucket name can shadow: /-/health for liveness and /-/metrics for Prometheus. Both answer without a signature, so on a store reachable from the internet, block them at the proxy in front of it. Each worker also logs one line per request.

Health¶

curl http://127.0.0.1:9000/-/health

answers ok with status 200 once the worker can read the metadata. A load balancer or Docker's HEALTHCHECK can use it. The Dockerfile in the repository does.

Metrics¶

Metrics are served at /-/metrics in the Prometheus text format. A scrape configuration:

scrape_configs:
  - job_name: leblob
    static_configs:
      - targets: ["127.0.0.1:9000"]
    metrics_path: /-/metrics

A scrape reaches one worker, chosen by the kernel, so the counters are added up across workers. Each worker writes its counts to a file under data/metrics/ every 5 s and on each scrape. The worker answering adds up the files. Totals can therefore lag by up to 5 s. They never go down until the server restarts, which Prometheus reads as a reset. State, such as free space, is read from the metadata and the disks on each scrape.

The metrics name operations, statuses and volume numbers, never a bucket, a key or a path. They do reveal how busy a store is and how full its disks are.

Requests¶

operation is the S3 operation's name in capitals, such as PUT_OBJECT or LIST_OBJECTS_V2, and unknown when a request couldn't be routed.

Background work¶

State¶

Failures¶

What to alert on¶

• leblob_only_copies_bad_total rising: a blob with one copy is damaged. Its object can't be repaired from the store.
• leblob_volume_failures_total rising, or leblob_volume_up at 0: a disk is failing. On btrfs, leblob_volume_up can stay at 1 for a dead device that answers from cache, so the counter is the one to rely on.
• leblob_repairs_queued staying high: blobs have fewer copies than wanted. That is normal for a while after a write that lost a copy, a raised number of copies or a forgotten disk.
• leblob_volume_free_bytes running low. On a full disk, new workers may only be able to answer with errors until there is room, because SQLite needs to write to recover its log.
• A rising share of leblob_http_requests_total with a status of 500 or above.

A hung disk stalls the requests that use it, and /-/metrics too, since leBLOB has no timeouts on disk I/O yet.

Access logs¶

Each worker writes one JSON line per request to standard error:

{"request_id": "3f9c0e5b7a1d2c44", "method": "GET", "path": "/my-bucket/hello.txt", "operation": "GET_OBJECT", "status": 200, "seconds": 0.0021}

The path holds bucket and key names, so the logs are as private as the store's listings.

Server errors, status 500 and above, are logged too, with their traceback under the same request ID. A response cut off by the client is logged with how many bytes had gone out.