Hi all,
we're about to deploy a new 3-node Proxmox VE 9 cluster with Ceph (NVMe OSDs) and are deciding how to connect it to a pair of MikroTik CRS520-4XS-16XQ-RM switches.
Planned setup:
Each node: 2 × 100G (Ceph public/cluster VLANs, MTU 9000) and 2 × 25G (VM traffic), one port per switch
Option A: MLAG between the two CRS520, LACP bonds on the nodes (one link per switch), LACP uplink to the core across both switches
Option B: no MLAG, active-backup bonds on the nodes, plain inter-switch link with RSTP
Corosync runs separately over two links on our existing switches
Why we're asking: We've read the forum reports about MLAG on 7.20–7.23 (FDB/host table filling up, "no buffer space available for fdb notify", peers falling out of sync, heartbeat misses) and the many MLAG fixes in 7.24/7.24.1 (MAC aging/flushing, host table aging, STP sync). We've also seen mixed reports on the 7.25 betas. What we couldn't find is anyone running Ceph on MikroTik MLAG in production.
Questions:
Is anyone running MLAG on 7.24.x (ideally 7.24.5) with CRS518/CRS520 or similar, especially with Ceph or other latency-sensitive storage traffic? How long, and how stable?
Did you test failover under load (rebooting one switch, disabling a port, pulling cables)? How long was the interruption, and did Ceph stay healthy?
Any issues with the FDB/host table or MLAG heartbeat on 7.24.x?
Has anyone tried the new mlag-init-delay / mlag-lacp-fallback options in 7.25rc yet?
If you decided against MLAG for Ceph: did you go active-backup or something else, and why?
Any first-hand experience (good or bad) is appreciated. Thanks!