How to Configure Multi-Path TCP and Advanced Routing Tables on FreeBSD

FreeBSD’s approach to multi-path networking is built around multiple, independent routing tables (fibs — forwarding information bases) rather than a single table with complex per-route policy attached. Understanding this architecture first makes the actual configuration much more straightforward than treating it as one big feature to enable.

Enabling multiple routing tables

The number of independent fibs the kernel supports is a boot-time tunable, not something changeable at runtime — it has to be set before the routing subsystem initializes:

echo 'net.fibs="4"' >> /boot/loader.conf

This needs a reboot to take effect. Once active, the system has fibs numbered 0 through 3, with fib 0 being the default table every process uses unless explicitly told otherwise.

Assigning routes to specific fibs

Each fib maintains its own completely independent set of routes — a default route in fib 0 pointing at one uplink doesn’t exist in fib 1 at all unless you explicitly add it there:

setfib 1 route add default 203.0.113.1
setfib 2 route add default 198.51.100.1

setfib N prefixes a command to run it against fib N instead of the default. Adding routes this way builds up fib 1 with its own default gateway pointing at one ISP uplink, and fib 2 with a different default gateway pointing at a second uplink — two completely independent routing views coexisting on the same machine.

Making processes use a specific fib

setfib also runs arbitrary commands against a specified fib, which is how you actually direct traffic through one uplink or another:

setfib 1 curl https://example.com/

This single command resolves DNS and makes its outbound connection entirely through fib 1’s routing table — whatever default gateway and routes exist there, independent of what fib 0 (the system default) would have chosen. For a persistent service that should always use a specific uplink, wrap its rc.d startup or systemd-equivalent invocation with setfib N, or, for services that support it directly, configure the fib in the service’s own configuration if it has native fib-awareness (some daemons, including parts of the base system, are fib-aware and can bind sockets to a specific fib without needing an external wrapper).

Binding a jail to a specific fib

A particularly clean use of multi-fib routing is giving a jail its own exclusive routing view — a jail assigned to fib 1 sees only fib 1’s routes, with no visibility into or interference from the host’s default routing table at all:

jail -c name=webjail path=/jails/webjail ip4.addr=10.0.0.5 vnet.set_fib=1 persist

This is a genuinely useful pattern for scenarios where different jails on the same host need to egress through different uplinks — a jail handling one customer’s traffic routed out one ISP connection, another jail’s traffic routed out a completely separate connection, with the fib assignment providing hard isolation between the two rather than relying on more fragile policy-routing rules layered onto a single shared table.

Policy routing based on source address or firewall marking

For routing decisions based on something other than “which fib was this command/jail assigned to” — for instance, routing based on source IP address for outbound traffic from a multi-homed host — pf’s route-to and reply-to directives operate at the packet-filter level, which is often a more practical mechanism than fib assignment alone for fine-grained, condition-based routing decisions:

pass out on $ext_if from 10.0.0.5 route-to (em1 203.0.113.1)

This routes traffic specifically sourced from 10.0.0.5 out through em1 toward a specific next-hop, regardless of what the system’s normal routing table would otherwise choose — useful when the routing decision needs to be based on packet contents (source address, port, protocol) rather than simply “which fib is this process using.”

MPTCP: a genuinely different mechanism

It’s worth being precise about what Multipath TCP actually is, since it solves a different problem than multi-fib routing. MPTCP lets a single TCP connection use multiple network paths simultaneously, for the same connection — not “route different connections over different links,” but “split one connection’s data across several physical paths at once,” with the protocol handling reordering and path-failure recovery transparently to the application. This requires MPTCP support in both the FreeBSD kernel’s TCP stack and, critically, cooperation from the remote endpoint — it’s not something one side can unilaterally enable and have it silently improve an ordinary TCP connection to a server that doesn’t support MPTCP itself; unlike fib-based traffic separation (which requires no cooperation from the remote endpoint since it’s purely a local routing decision) MPTCP requires both ends to negotiate its use during the TCP handshake, so a comprehensive MPTCP deployment is a considerably larger undertaking than fib-based routing, and generally only makes sense when both communicating endpoints are under your own control (or when connecting to a service that has deliberately implemented MPTCP support).

For the overwhelmingly common case — “I have two ISP uplinks and want to distribute or fail over traffic between them” — multi-fib routing with setfib and, where finer policy control is needed, pf’s route-to, is the practical, well-supported path on FreeBSD, and doesn’t require anything from the remote side of any connection at all.