The same guest did 248,000 4k random read IOPS on LVM-thin, 186,000 on ZFS, and 72,000 on Ceph. Writes were 132,000, 54,000, and 19,400. A 1M sequential read was 3,480 MB/s, 3,120 MB/s, and 2,150 MB/s.
One shape of guest for all three. Ubuntu, 4 vCPU, 8 GB of RAM, a 32 GB data disk, virtio-scsi, iothread=1, cache=none, discard=on. fio used that second disk. Queue depth 32 and four jobs for the 4k runs, one job for the 1M read, 60 seconds, direct I/O. Each figure is the median of three runs. The spread on every pattern stayed inside 15 percent.
LVM-thin and ZFS were one 64 GB node and one enterprise NVMe. Ceph was three of those nodes, one OSD each, pool size 3, a 25 GbE link, ping 0.21 ms. Same disk. The extra cost on Ceph is the copies and the cable, not a second model of drive.
A current Proxmox install still caps the ZFS ARC at 10 percent of the host. On this 64 GB node that is 6.4 GiB. The one Ceph OSD was planned at 8 GiB. LVM-thin took no cache of either size.
4k IOPS on the same guest
| Pattern | LVM-thin | ZFS | Ceph |
|---|---|---|---|
| 4k random read, IOPS | 248,000 | 186,000 | 72,000 |
| 4k random write, IOPS | 132,000 | 54,000 | 19,400 |
| 4k read, 99th percentile | 0.48 ms | 0.74 ms | 2.1 ms |
| 4k write, 99th percentile | 0.96 ms | 2.4 ms | 5.8 ms |
| 1M sequential read | 3,480 MB/s | 3,120 MB/s | 2,150 MB/s |
The read command, against the data disk:
fio --name=randread --filename=/dev/sdb --ioengine=libaio --direct=1 --bs=4k --rw=randread --iodepth=32 --numjobs=4 --runtime=60 --time_based --group_reportingWrites were the same line with --rw=randwrite. The 1M read was --bs=1M --rw=read --numjobs=1.
ZFS gave up about a quarter of the read IOPS and about 60 percent of the write IOPS. The read loss is the checksum and the volblocksize. The write loss is the intent log on the same NVMe. At queue depth 1 those writes fall off a cliff. Depth 32 is what a busy guest actually offers.
Ceph's read is about 30 percent of LVM-thin. A read can come back from one OSD, and 0.21 ms each way is still in the latency. The write has to land three copies, so 19,400 IOPS is the number that matches the pool. The 99th percentile on those writes was 5.8 ms. LVM-thin's was 0.96 ms.
How much RAM ZFS keeps
The ZFS page says a current install writes the ARC cap into /etc/modprobe.d/zfs.conf. Ten percent of host memory, and 16 GiB if ten percent is larger. On a 64 GB node the cap is 6.4 GiB. On 128 GB it is 12.8 GiB. On 256 GB the clamp is the whole cap, 16 GiB, because ten percent would be 25.6.
Before Proxmox VE 8.1 the installer did not write that file. ZFS used its own default: 50 percent on older ZFS, 62.5 percent from ZFS 2.3.0. On this same 64 GB node that is 32 GiB, or 40 GiB. An install from that era keeps the old default until someone sets zfs_arc_max and reboots. If the root pool is ZFS, the page also says to rebuild the initramfs before that reboot.
The page's floor, if you are about to shrink the cache, is 2 GiB plus 1 GiB for each TiB in the pool. An 8 TiB pool wants 10 GiB. ZFS will not go below 64 MiB. Cutting the 6.4 GiB cap to sell two more gigabytes also cuts the cache that was answering reads. On this guest the ARC did not hide the disk. The fio file was the 32 GB data disk, larger than 6.4 GiB, and direct I/O was on.
186,000 read IOPS, divided by that 6.4 GiB, is about 29,000 IOPS for each GiB the cap holds. That is the trade on this box.
How much RAM one Ceph OSD keeps
The Ceph deployment page sets the default OSD at 4 GiB. It says to configure at least 8 GiB per OSD for good performance, with ceph config set osd osd_memory_target 8G. This run used the 8 GiB line. Recovery, rebalance, and backfill use more than the quiet number.
72,000 read IOPS divided by 8 GiB is 9,000 IOPS per GiB planned for that OSD. Four OSDs on one node, the even split of a 12-OSD cluster, is 32 GiB, and the guest shares the drive with three other OSDs. Here the node had one disk. The gap against LVM-thin is the three copies and the 0.21 ms ping.
The page asks for 10 Gbps reserved for Ceph, and 25 Gbps when one NVMe can fill a 10 Gbps link. The 2,150 MB/s sequential read is that 25 GbE path, not the drive's ceiling. The local guest was already at 3,480 MB/s on the same model of disk.
| What is held back | On this 64 GB node | 4k read IOPS per GiB held |
|---|---|---|
| LVM-thin | No cache target | 248,000 IOPS, none held |
| ZFS ARC, current install | 6.4 GiB | about 29,000 |
| Ceph, one OSD at 8 GiB | 8 GiB | 9,000 |
| Ceph, four OSDs at 8 GiB | 32 GiB | a 12-OSD cluster |
| ZFS before VE 8.1, 50 percent | 32 GiB | an older install |
LVM-thin allocates a block when the guest writes it. Snapshots and clones work. Another node cannot see the pool. Fill it and the guests get I/O errors. Proxmox staff put HA startup at 2 or 3 minutes whether the disk is ZFS or Ceph. ZFS replication leaves the other node with the disk as of the last run. Ceph already has the copies. The ping on this cluster was 0.21 ms. Proxmox staff want Ceph under 1 ms.
What you can still sell
Take the 64 GB auction node from what one Proxmox node costs per sold gigabyte. At 70 percent sold, each gigabyte that has a customer has to earn €1.61. The ARC cap is 6.4 GB of that box, €10.30 of the monthly rent, and the guest still did 186,000 read IOPS. The old 32 GB default would be €51.52 of a €72 invoice, held as cache. One Ceph OSD at 8 GiB is €12.88, and the guest did 72,000.
LVM-thin keeps the 248,000 and the gigabytes. The disk stays on the node. ZFS keeps a local disk, a replication lag, 6.4 GiB, and most of the read speed. Ceph keeps the disk on the other two nodes, at about 30 percent of the read IOPS and about 15 percent of the write IOPS, on a 0.21 ms link.
An empty node is the same invoice one step up. How the node is attached is the platforms and nodes guide.
Questions that change the node
Is LVM-thin faster than ZFS?
On this guest, yes. 248,000 against 186,000 on 4k reads, 132,000 against 54,000 on 4k writes, 3,480 MB/s against 3,120 MB/s on a 1M read. The read gap is the small one. The write gap is the intent log.
Does ZFS use half the RAM?
On an install from Proxmox VE 8.1 on, the installer writes a 10 percent cap, and 16 GiB is as high as that cap goes. This 64 GB node is 6.4 GiB. Half the RAM is the older behavior, from before 8.1, when the installer left ZFS at 50 percent, or at 62.5 percent once ZFS 2.3.0 was the default and nobody had set zfs_arc_max.
How much RAM does one Ceph OSD need?
4 GiB is the default target. The current Proxmox page says at least 8 GiB for good performance. This guest's OSD was set to 8 GiB. Four OSDs at that line is 32 GiB, on a node with four disks.
Why is the Ceph write so much slower than the read?
The pool keeps three copies. A read can return from one OSD. A write waits until the copies are on the other nodes. At 0.21 ms that wait is 19,400 IOPS and a 5.8 ms 99th percentile, against 132,000 IOPS and 0.96 ms on LVM-thin.