SeaweedFS vs Garage for a lot of small files

Five thousand objects, each 4 KiB, uploaded over S3 with eight parallel workers on one lab host. SeaweedFS 3.97 on 127.0.0.1:8333 finished in 10.5 seconds (475 objects/s). Garage v2.5.0 on 127.0.0.1:3900 finished the same corpus in 21.7 seconds (230 objects/s). Both returned matching sha256 on a sample get. After the load, SeaweedFS sat at about 372 MiB in docker stats; Garage sat at about 29 MiB. SeaweedFS packed the payload into 15 volume .dat files (21 MiB under sw-data). Garage held 5001 files under its data directory (59 MiB data + 28 MiB meta).

SeaweedFS is built to pack many small payloads into large volume files. Garage is the light multi-site S3 already measured in the Garage on a VPS post. If you arrived here because MinIO’s Hub path died, these two are the usual shortlist for self-hosted S3 on a single box.

Why “a lot of small files” is its own problem

A 4 KiB object on a normal filesystem still costs an inode, directory entries, and usually a full filesystem block. Multiply by millions and you drown in metadata long before you fill the disk with payload. Backups that walk every file, du across deep trees, and per-object S3 ceremony all pay a fixed cost that barely shrinks with object size.

SeaweedFS follows the Haystack idea: append many small blobs (“needles”) into a few large volume files, keep an in-memory map from file id to offset, and serve a read with one seek into a known volume. The filer (and the S3 gateway in front of it) adds path names on top of those ids.

Garage stores objects through its own block and metadata engines. It is a solid S3 for moderate object counts and geo-ish layouts. Its own performance writing notes that S3 was never designed as a database of tiny rows. On this lab, the data directory grew one file per object for the 5000 puts, which is fine at five thousand and painful at five million.

How do you run SeaweedFS with S3 on a VPS?

Pin chrislusf/seaweedfs:3.97 (digest sha256:bb05d66d2963b1cc48073190781c3dd29e2c4c88a2ae2986bf58e38c86d89c6e on this lab). Override the image entrypoint so you control volume preallocation and size limits. Keep S3 on loopback.

services:
  seaweed:
    image: chrislusf/seaweedfs:3.97
    container_name: seaweed
    restart: unless-stopped
    entrypoint: ["/usr/bin/weed", "-logtostderr=true"]
    command:
      - server
      - -dir=/data
      - -s3
      - -ip.bind=0.0.0.0
      - -master.volumeSizeLimitMB=256
      - -volume.max=40
      - -s3.port=8333
    ports:
      - "127.0.0.1:8333:8333"
      - "127.0.0.1:9333:9333"
    volumes:
      - ./sw-data:/data

docker compose up -d
curl -s 'http://127.0.0.1:9333/vol/grow?count=8&replication=000'
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8333/
# 200

This lab used anonymous-ish S3 keys (any/any) because no IAM config was mounted. For anything beyond a closed loopback proof, mount an IAM identity file and rotate keys. Put TLS on Caddy if the API must leave the box. Leave 8333 closed on the panel firewall.

How does Garage sit on the same host?

Garage stayed on the existing lab Compose from the Garage S3 post: dxflrs/garage:v2.5.0, S3 on 127.0.0.1:3900, --single-node --default-bucket. Create a fresh bucket for the benchmark so you are not mixing with other lab objects.

export AWS_ENDPOINT_URL=http://127.0.0.1:3900
export AWS_DEFAULT_REGION=garage
export AWS_ACCESS_KEY_ID='GK…'          # from compose .env
export AWS_SECRET_ACCESS_KEY='…'
aws s3 mb s3://small-garage

Full install, layout, and Caddy notes live in that sibling post. This comparison reuses the running instance so both stores share the same CPU and disk.

What did the small-file put bench do?

Corpus: 5000 files named obj-00000.bin … obj-04999.bin, each exactly 4096 bytes (about 19.5 MiB total). Uploader: a short Python script using boto3 put_object with eight worker threads, five retries, then a list count and one get for sha256. Same script, same workers, against each endpoint.

# SeaweedFS
python3 put_bench.py --endpoint http://127.0.0.1:8333 \
  --bucket small-sw --workers 8 --access-key any --secret-key any

# Garage
python3 put_bench.py --endpoint http://127.0.0.1:3900 \
  --bucket small-garage --workers 8 --region garage \
  --access-key "$AWS_ACCESS_KEY_ID" --secret-key "$AWS_SECRET_ACCESS_KEY"

Results on this host:

SeaweedFS was about 2.1× faster on this put-heavy small-object run. Treat it as a lab number for this disk and default filer store. The needle path is doing real work when object count dominates.

Disk layout after the load

SeaweedFS stayed close to the raw corpus size while collapsing thousands of logical objects into a handful of volume files. Garage paid more bytes and one filesystem file per object. At five thousand that is a curiosity. At five million it is an ops problem for anything that walks the data directory.

After the load, SeaweedFS held about 372 MiB (docker stats); Garage about 29 MiB. Before the bulk put, a fresh SeaweedFS process was closer to 19 MiB. The needle index and filer state grow with object count. Garage’s idle story from the sibling post (~7 MiB on an empty-ish single node) still holds as the “small S3” default when object counts stay moderate.

Which one should you ship?

Choose SeaweedFS when the product is many small objects: thumbnails, tiles, mail fragments, build-cache shards, millions of KiB-scale blobs. You accept a multi-component mental model (master, volume, filer, S3) and higher RAM once the needle map is warm. Put production IAM on the S3 gateway before you leave loopback.

Choose Garage when you want a small, maintained S3 API, geo-friendly replication later, and a quiet idle footprint on a 2 GB Compute Instance. Keep object counts in the range where one-file-per-object and metadata size stay comfortable, or plan capacity before you import a photo-site’s worth of thumbs.

Design intent: Haystack volumes vs geo-simple S3

SeaweedFS packs many small objects into append-only volume files with a tiny in-memory index entry per file, which is why “billions of small files” guides keep naming it. Garage aims at geo-distributed, low-power clusters as a single binary with replication across zones; Deuxfleurs has said large numbers of tiny objects were not the original optimisation target and has funded follow-up work there. Feature surface differs too: Seaweed’s S3 gateway covers a wider IAM/versioning/lock set; Garage publishes what it does not implement (versioning, object lock, ACLs as S3 knows them) and uses its own per-key grants. Licence is Apache 2.0 for SeaweedFS and AGPLv3 for Garage, which matters if AGPL was why you left MinIO. Use this lab’s small-file put timings for the head-to-head on one VPS; pick Seaweed when object count and LIST behaviour dominate, Garage when multi-site simplicity dominates.

FAQ

Is 5000 objects enough to prove “a lot”? It is enough to show packing and put rate on one VPS without filling the disk. Millions would widen the gap on file-count and metadata pressure; this post stays honest about the scale it measured.

Does SeaweedFS need a separate filer store? This lab used the default leveldb2 under /data/filerldb2. Large deployments often move filer metadata to something faster; that is a second project after S3 works.

Can Garage inline tiny objects? Garage can keep very small objects inline in metadata in some configurations. The lab still saw one data file per object for these 4 KiB puts. Read current Garage docs before assuming inlining saves you at thumbnail scale.

What about MinIO? Community Hub images and dl.min.io binaries are gone; see MinIO after AGPL. This comparison starts from stores you can still pull and run.

Weed volumePreallocate? The stock image command enables preallocate and a large volume size limit. Override the entrypoint (as above) on a small VPS or you will burn disk accounting on empty volumes.

What to run on the VPS

A 2 GB Compute Instance can host Garage for everyday S3 or SeaweedFS for a small-file heavy app, with room for the client that talks S3. Pin the image, keep the API on loopback until Caddy is in front, run one put/get proof, then decide from object count and RAM. The README logo can wait.