S3 destination

safekeep can push every project to one S3-compatible bucket (S3 = Simple Storage Service) alongside — or instead of — the filesystem destinations. The client is botocore, pulled in by an optional extra, so the core package stays pure standard library.

Install the extra

pipx install "safekeep[s3]"     # or: pip install "safekeep[s3]"

Without the extra a config containing s3_* still parses but fails fast with the install hint: run, sync-once and status exit non-zero, while doctor reports it as a non-fatal warning. Without any s3_* key nothing changes — no botocore, no import, no cost.

Configuration

The s3_* keys are valid only in ~/.safekeep: a .sync can't add them (they're classified as invalid lines, so a cloned repo can't redirect your backup). Keep the file at chmod 600.

s3_endpoint: https://s3.example.com:443   # http:// → cleartext warning
s3_bucket: my-backup-bucket               # at least one dest: OR s3_bucket:
s3_prefix: safekeep                       # optional, no '..'
s3_region: us-east-1                      # default
s3_access_key: REPLACEME_ACCESS_KEY       # required when s3_bucket is set
s3_secret_key: REPLACEME_SECRET_KEY       # chmod 600 — or use s3_secret_key_env:
# s3_secret_key_env: SAFEKEEP_S3_SECRET   # secret from env, never in the file
s3_addressing: path                       # path (default) | virtual
s3_tls_check: true                        # false = self-signed certificates

With s3_secret_key_env: the variable must be in the agent's environment (launchd EnvironmentVariables, systemd Environment=) — a shell export alone never reaches a daemon started at boot. Both unit templates ship the snippet commented out: uncomment, set it and reload — see Agent.

Validation is fail-fast at parse time with a line number: bucket without access key, both secret forms set, both missing, an env var that doesn't exist, an unknown s3_* key, .. in the prefix, an invalid enum. A non-HTTPS endpoint is a warning, not an error.

Compatible services

Service	Example s3_endpoint	s3_addressing	s3_region	TLS
MinIO (self-hosted)	https://minio.example.com:9000	path	any (us-east-1 default)	https + s3_tls_check: false if self-signed
Silo (fork of MinIO)	https://s3.example.com:443	path	any (us-east-1 default)	as the endpoint's own certificate
Ceph RGW	https://rgw.example.com	path	any, must match the RGW zone (us-east-1 default)	as configured on the RGW
Wasabi	https://s3.wasabisys.com	path	required, must match the bucket (us-east-1 on that endpoint)	https
Backblaze B2	https://s3.<region>.backblazeb2.com	virtual or path	required, must match the bucket region	https only — cleartext rejected
AWS S3	(omit s3_endpoint)	virtual	must match the bucket (us-east-1 default)	https

All of them speak SigV4 (Signature Version 4) through the same client — there is no service-specific code path. B2 wants an Application Key (not the Master Key) as s3_access_key / s3_secret_key.

What it does (v1)

• Upload only, idempotent. needs_copy reads the object's metadata: x-amz-meta-mtime (nanoseconds, 1s tolerance — same as filesystem destinations), x-amz-meta-mode, size. Second reconcile → zero copies.
• Symlinks are zero-byte objects with x-amz-meta-symlink = target; the target is never followed.
• Nothing is ever deleted. No DeleteObject, no list-then-prune — --prune only touches filesystem destinations. The prefix is exclusively safekeep's, and no bucket versioning is required or managed.
• Network errors → pending with the usual exponential backoff, the same path as an unmounted volume; the destination comes back on the next retry.
• One PUT per file, streamed (no full file in RAM). The atomic temp-file dance isn't needed: a PUT is atomic by itself.
• Also a restore source. The bucket doubles as a destination you can read back from: safekeep restore --from s3 pulls files out of it (0.8.0) — see commands.

Accepted limits

• Single process: there is no upload thread — a slow PUT or HeadObject blocks the daemon loop until it times out (connect 10s / read 60s, limited retries). Accepted for v1.
• No multipart: one put_object per file, so a single object is capped by the service's limit for a plain PUT (~5 GB). Bigger files log a non-fatal error; multipart would mean adding s3transfer on top of botocore.
• One S3 destination per config.

Checks

safekeep status    # s3://bucket/prefix [ok] | [offline] — cached HeadBucket (30s)
safekeep doctor    # credentials origin, chmod 600, cleartext endpoint, HeadBucket

Every S3 check in doctor is non-fatal: a warning you can act on, never a failed run. The runtime also logs a one-time warning when an inline s3_secret_key sits in a file readable by others.

Testing against a real bucket

tests/test_s3_itest.py talks to a real endpoint and is skipped unless every variable is set — no credential is hard-coded anywhere:

export SAFEKEEP_S3_TEST_ENDPOINT=https://s3.example.com:443 \
       SAFEKEEP_S3_TEST_BUCKET=my-backup-bucket \
       SAFEKEEP_S3_TEST_ACCESS_KEY=REPLACEME_ACCESS_KEY \
       SAFEKEEP_S3_TEST_SECRET_KEY=REPLACEME_SECRET_KEY \
       SAFEKEEP_S3_TEST_REGION=us-east-1        # optional, default us-east-1
       # SAFEKEEP_S3_TEST_PREFIX=safekeep       # optional, default safekeep-itest
python3 -m unittest tests.test_s3_itest -v

The test uploads under a throwaway timestamped prefix and performs noDeleteObject — whatever it writes stays in the bucket under that prefix.