UCX + OpenMPI + Slurm on FreeBSD

Note

This guide is a work in progress. Sections marked [TODO] are placeholders to be filled in as the port work matures.

This guide covers wiring the full HPC runtime stack on FreeBSD:

  • UCX (Unified Communication X) — high-performance transport layer

  • OpenMPI — MPI implementation built on top of UCX

  • Slurm — workload manager and job scheduler


Overview

The three components form a layered stack:

┌─────────────────────────────────┐
│  User application (MPI program) │
├─────────────────────────────────┤
│  Slurm  (job scheduling / PMI)  │
├─────────────────────────────────┤
│  OpenMPI  (MPI runtime / PRRTE) │
├─────────────────────────────────┤
│  UCX  (transport / RDMA / SM)   │
├─────────────────────────────────┤
│  FreeBSD kernel / hardware      │
└─────────────────────────────────┘

Slurm launches jobs and provides the process management interface (PMI2/PMIx). OpenMPI uses PRRTE as its runtime environment and UCX as its transport layer. UCX speaks directly to the network hardware — shared memory, InfiniBand, TCP.

Prerequisites

The following ports must be installed:

pkg install \
    net/ucx \
    mpi/openmpi \
    mpi/pmix \
    mpi/prrte \
    sysutils/slurm-wlm

Verify versions after install:

ucx_info -v
ompi_info | grep -E "Open MPI:|UCX"
sinfo --version

UCX Configuration

[TODO] Transport selection, memory registration, shared memory configuration.

# List available UCX transports
ucx_info -d

OpenMPI + UCX Wiring

[TODO] How OpenMPI discovers and uses UCX, relevant mca parameters.

# Run with UCX transport explicitly selected
mpirun --mca pml ucx -np 4 ./your_mpi_program

OpenMPI + Slurm Integration

[TODO] PMIx handshake, srun vs mpirun, process binding.

# Submit an MPI job via Slurm
sbatch --ntasks=16 --mpi=pmix ./job.sh

Troubleshooting

[TODO] Common failure modes — UCX transport fallback warnings, PMIx version mismatches, Slurm cgroup issues on FreeBSD.

See also