How to Set Up High Availability with CARP on FreeBSD

CARP (Common Address Redundancy Protocol) lets two or more FreeBSD hosts share a virtual IP address, with automatic failover to a backup host if the primary stops responding — no special switch configuration, no external load balancer, just a protocol running directly between the participating hosts over the network they’re already on.

The core mechanism: virtual hosts and advertisements

Each CARP-participating host runs a virtual CARP interface bound to a physical interface, and the hosts continuously exchange CARP advertisements to agree on which one is currently MASTER (actively holding the shared IP) versus BACKUP (standing by, ready to take over). The host configured with the lowest advskew value wins master status under normal conditions — this is the mechanism you use to designate a preferred primary rather than leaving master selection arbitrary.

Configuring the primary host

ifconfig igb0 inet 203.0.113.10/24
ifconfig igb0 vhid 1 pass mysecret advskew 0

This creates CARP virtual host ID 1 on igb0, with a shared secret (pass) that both participating hosts must agree on (CARP advertisements are authenticated with this shared value specifically to prevent a rogue host on the same network segment from claiming the virtual IP), and an advskew of 0 — the lowest possible value, making this host the preferred master.

Configuring the secondary host

ifconfig igb0 inet 203.0.113.11/24
ifconfig igb0 vhid 1 pass mysecret advskew 100

Same vhid (this is what ties the two hosts together as participants in the same CARP group) and the same shared secret, but a higher advskew — this host will remain in BACKUP state as long as the primary is sending healthy advertisements, and will only assume MASTER (and start responding on the shared virtual IP) if the primary stops advertising.

Persisting the configuration across reboots

Add the CARP configuration to /etc/rc.conf on each host so it survives a reboot rather than needing to be re-run manually:

# On the primary
ifconfig_igb0="inet 203.0.113.10/24"
cloned_interfaces="carp0"
ifconfig_carp0="vhid 1 pass mysecret advskew 0 alias 203.0.113.100/24"

Note the pattern here differs slightly from the ad-hoc ifconfig example above: in /etc/rc.conf, CARP is commonly configured as a cloned pseudo-interface (carp0) with the shared virtual IP as an alias, rather than attaching CARP parameters directly to the physical interface’s own address — this is the more standard persistent configuration pattern and keeps the physical interface’s own real address cleanly separate from the shared virtual address the CARP group actually manages.

Confirming CARP state

ifconfig carp0

The output includes a carp: line reporting the interface’s current state — MASTER or BACKUP — directly. Watching this on both hosts while deliberately triggering a failover (temporarily disconnecting the primary’s network interface, or stopping its netif/CARP service) is the correct way to validate the setup actually works before trusting it in production, rather than assuming a syntactically correct configuration behaves correctly under an actual failure.

What CARP alone does not do

This is the part that trips up people setting up their first CARP-based HA pair: CARP moves which host answers on the shared IP address. It does absolutely nothing about the state of whatever service is running behind that IP. If the service is a stateless web server serving static content, this is fine — the backup host serving the same static files after a failover is indistinguishable from the primary having served them. If the service maintains any session state, in-memory data, or an active database connection, CARP failover alone will drop all of that state instantly, because the backup host’s copy of that service starts from whatever state it was already in, not a live replica of the primary’s current state at the moment of failover.

Real HA deployments pair CARP with something else entirely for state synchronization: HAST (for block-level storage replication, covered in a companion post) if the service’s persistent state lives on disk and needs to be identical on both hosts; database-native replication if the service is backed by a database with its own replication mechanism; or, for services with meaningful in-memory session state, application-level session replication or an external, shared session store (a separate Redis instance both hosts talk to, for instance) that doesn’t depend on either host’s local memory at all.

A common, correct pairing: CARP plus pfsync

For firewall/router HA specifically — a very common CARP use case — pfsync handles the missing piece: it replicates pf’s active connection-state table between the two hosts in real time, so that when CARP fails the shared IP over, the newly-active firewall already has the previous primary’s connection state table and doesn’t drop every in-progress connection through the firewall at the moment of failover:

ifconfig pfsync0 syncdev igb1 syncpeer 203.0.113.11

This example synchronizes pf state specifically over a dedicated interface (igb1, ideally a private, direct link between the two hosts rather than the same network the CARP virtual IP lives on) to the peer’s address — keeping state synchronization traffic off the production network segment is a reasonable default, since it’s both bandwidth that shouldn’t compete with production traffic and state data you generally don’t want visible on the same broadcast domain as regular client traffic.

The pattern generalizes: CARP handles “who answers this IP right now,” and whatever service sits behind that IP needs its own, separate answer to “how does the backup host have the state it needs to actually take over correctly” — treating CARP as a complete HA solution on its own, without addressing that second question for whatever specific service is involved, is the most common way a CARP-based HA setup looks correct in testing and then loses data or drops connections during an actual production failover.