Edited on 2023-08-25 to add how to start the jail on each boot.

The latest iteration of my home firewall has a spare interface. Many switches these days have a dedicated management interface, and I wanted something similar for my firewall. I want an interface that I can plug in, get an IP via DHCP, and I can ssh into and it’ll “just work” even if there’s a misconfiguration, or an IP conflict.

On FreeBSD, vnet jails are a way to isolate an interface such that it won’t interfere (or receive interference) from other interfaces. Some may have just used epair and assigned IPs, but this could cause conflict, while using a unix domain socket keeps things isolated, and FreeBSD ships w/ everything needed to make this work. A package like socat isn’t needed.

Configuring the host was straight forward, make a directory for the socket:

mkdir /var/mgmt

And then add a line to /etc/inetd.conf to listen to incoming connections on the socket and launch sshd:

/var/mgmt/mgmt.ssh.sock stream  unix    nowait  root    /usr/sbin/sshd  sshd -i

The -i option to sshd tells it to run in “inetd” mode, which means that the stdin and stdout of the process is the socket to use for communication.

The next harder part was getting a jail configured that would accept incoming connections and forward them to the unix domain socket.

First the jail configuration, which goes in /etc/jail.conf or similar location:

mgmt {
        host.hostname = mgmt;                           # Hostname

        vnet ="new";
        vnet.interface="mgmt0";

        path = "/usr/jails/mgmt/root";                   # Path to the jail
        mount.fstab="/usr/jails/mgmt/fstab";             # mount spec

        mount.devfs;                                    # Mount devfs inside the jail
        devfs_ruleset = "101";

        exec.start = "/bin/sh /etc/rc";                 # Start command
        exec.stop = "/bin/sh /etc/rc.shutdown";         # Stop command
}

There isn’t anything special in this. It’s pretty standard jail configuration, the differences are the vnet configuration, and the devfs_ruleset.

The devfs_ruleset is necessary in order to expose the bpf interface used by dhclient. This required the following lines in /etc/devfs.rules:

[mydevfsrules_jail=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login

[devfsrules_jail_dhcp=101]
add include $mydevfsrules_jail
add path 'bpf*' unhide

Note that after adding the above lines, you need to run:

service devfs start

to load the rules (per devfs(8) man page).

Note: I learned my mistake not to number my blocks immediately after the standard defaults (from /etc/defaults/devfs.rules). I had done that befure but there’s now a conflict, so I skip ahead a bit to get a unique range.

Now I needed to make a number of directories for the jail:

mkdir -p /usr/jails/mgmt/root
mkdir -p /usr/jails/mgmt/etc
mkdir -p /usr/jails/mgmt/var/mgmt
mkdir -p /usr/jails/mgmt/tmp

I needed to setup the fstab for the jail:

# Device                Mountpoint      FStype  Options         Dump    Pass#
/                       /usr/jail/mgmt/root     nullfs  ro      0       0
/usr/jail/mgmt/etc      /usr/jail/mgmt/root/etc nullfs  rw      0       0
/usr/jail/mgmt/var      /usr/jail/mgmt/root/var nullfs  rw      0       0
/var/mgmt               /usr/jail/mgmt/root/var/mgmt    nullfs  ro      0       0
/usr/jail/mgmt/tmp      /usr/jail/mgmt/root/tmp nullfs  rw      0       0

This is a little bit more tricky, It first nullfs mounts the root system. I’m using ZFS boot environments, so this is a pretty clean FreeBSD install without much host specific data. It then mounts some jail specific directories for etc, tmp and var and finally mounts the shared directory w/ the unix domain socket to the host system. Also note that a couple of the mounts are read-only to prevent the jail from modifying the system.

The etc directory was populated from the system etc via:

tar -cf - -C /etc . | tar -xf - -C /usr/jails/mgmt/etc

Then the jail was configured, first /usr/jails/mgmt/etc/rc.conf:

hostname="mgmt.funkthat.com"

sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"

# Management port
ifconfig_mgmt0="DHCP"

# necessary as devd can't be run in a jail
synchronous_dhclient="YES"

inetd_enable="YES"              # Run the network daemon dispatcher (YES/NO).

The key part of this configuration that took me a while to figure out was the synchronous_dhclient line. It used to be that netif would start dhclient, but in order to better handle USB ethernet devices and other removable interfaces, it was moved to devd. The only problem is that devd hasn’t been jail’ified, and you can’t run it to get things like link notifications that would normally launch dhclient. Setting this to yes, makes sure it gets launched when the jail starts.

And then the following line was added to /usr/jails/mgmt/etc/inetd.conf:

ssh     stream  tcp     nowait  root    /usr/bin/nc     nc -N -U /var/mgmt/mgmt.ssh.sock

This is the part that will forward incoming connections to the ssh port on to the unix domain socket.

Now that everything is configured, a simple jail -c mgmt will get the jail running and accepting connections.

To get the jail to start every boot, add the following to the host’s /etc/rc.conf:

jail_enable="YES"       # Set to NO to disable starting of any jails
jail_list="mgmt"

This was testing and deployed on a FreeBSD 14-CURRENT build as of August 8th, 2023, or more specifically, from main-n264621-09c20a29328, but it should work on all currently supported FreeBSD releases.