UCX + OpenMPI + Slurm on FreeBSD¶
Note
This guide is a work in progress. Sections marked [TODO] are placeholders to be filled in as the port work matures.
This guide covers wiring the full HPC runtime stack on FreeBSD:
UCX (Unified Communication X) — high-performance transport layer
OpenMPI — MPI implementation built on top of UCX
Slurm — workload manager and job scheduler
Overview¶
The three components form a layered stack:
┌─────────────────────────────────┐
│ User application (MPI program) │
├─────────────────────────────────┤
│ Slurm (job scheduling / PMI) │
├─────────────────────────────────┤
│ OpenMPI (MPI runtime / PRRTE) │
├─────────────────────────────────┤
│ UCX (transport / RDMA / SM) │
├─────────────────────────────────┤
│ FreeBSD kernel / hardware │
└─────────────────────────────────┘
Slurm launches jobs and provides the process management interface (PMI2/PMIx). OpenMPI uses PRRTE as its runtime environment and UCX as its transport layer. UCX speaks directly to the network hardware — shared memory, InfiniBand, TCP.
Prerequisites¶
The following ports must be installed:
pkg install \
net/ucx \
mpi/openmpi \
mpi/pmix \
mpi/prrte \
sysutils/slurm-wlm
Verify versions after install:
ucx_info -v
ompi_info | grep -E "Open MPI:|UCX"
sinfo --version
UCX Configuration¶
[TODO] Transport selection, memory registration, shared memory configuration.
# List available UCX transports
ucx_info -d
OpenMPI + UCX Wiring¶
[TODO] How OpenMPI discovers and uses UCX, relevant mca parameters.
# Run with UCX transport explicitly selected
mpirun --mca pml ucx -np 4 ./your_mpi_program
OpenMPI + Slurm Integration¶
[TODO] PMIx handshake, srun vs mpirun, process binding.
# Submit an MPI job via Slurm
sbatch --ntasks=16 --mpi=pmix ./job.sh
Troubleshooting¶
[TODO] Common failure modes — UCX transport fallback warnings, PMIx version mismatches, Slurm cgroup issues on FreeBSD.
See also
UCX documentation: https://openucx.readthedocs.io
OpenMPI FAQ: https://www.open-mpi.org/faq/
FreeBSD Bugzilla: https://bugs.freebsd.org