Ceph cluster for Proxmox (KVM) with an Nvidia InfiniBand switch
I build a three-node Proxmox VE 8.2 cluster on HP Z840 workstations, each with four 4 TB SSDs (12 OSDs in total). Ceph replication traffic runs over an InfiniBand network.
What the article covers
- Splitting traffic across dedicated networks: management bridge, InfiniBand for Ceph, LACP-bonded corporate/production access and a backup network
- Installing Mellanox ConnectX-3 InfiniBand support, the OpenSM subnet manager and persisting the IPoIB module
- Static address on the Ceph subnet for the InfiniBand interface, large MTU and connected mode
- Joining the nodes into a Proxmox cluster named hpc-cluster
- Installing Ceph on the InfiniBand subnet with a monitor on every node and a manager on one
- Adding each SSD as an OSD and creating a pool with PG autoscaling (replica size, min size, PG count and CRUSH rules)
- Repeating the steps in the web GUI, the Ceph dashboard, an HA group, adding resources and a test migration
Read the original article on LinkedIn ↗