Edited on 2023-08-25 to add how to start the jail on each boot.
The latest iteration of my home firewall has a spare interface. Many
switches these days have a dedicated management interface, and I wanted
something similar for my firewall. I want an interface that I can plug in,
get an IP via DHCP, and I can ssh into and it’ll “just work” even if
there’s a misconfiguration, or an IP conflict.
On FreeBSD,
vnet jails
are a way to isolate an interface such that it won’t interfere (or receive
interference) from other interfaces. Some may have just used epair and
assigned IPs, but this could cause conflict, while using a unix domain
socket keeps things isolated, and FreeBSD ships w/ everything needed to
make this work. A package like socat isn’t needed.
Configuring the host was straight forward, make a directory for the socket:
mkdir /var/mgmt
And then add a line to /etc/inetd.conf to listen to incoming connections
on the socket and launch sshd:
/var/mgmt/mgmt.ssh.sock stream unix nowait root /usr/sbin/sshd sshd -i
The -i option to sshd tells it to run in “inetd” mode, which means that
the stdin and stdout of the process is the socket to use for communication.
The next harder part was getting a jail configured that would accept
incoming connections and forward them to the unix domain socket.
First the jail configuration, which goes in /etc/jail.conf or similar location:
mgmt {
host.hostname = mgmt; # Hostname
vnet ="new";
vnet.interface="mgmt0";
path = "/usr/jails/mgmt/root"; # Path to the jail
mount.fstab="/usr/jails/mgmt/fstab"; # mount spec
mount.devfs; # Mount devfs inside the jail
devfs_ruleset = "101";
exec.start = "/bin/sh /etc/rc"; # Start command
exec.stop = "/bin/sh /etc/rc.shutdown"; # Stop command
}
There isn’t anything special in this. It’s pretty standard jail
configuration, the differences are the vnet configuration, and the
devfs_ruleset.
The devfs_ruleset is necessary in order to expose the bpf interface
used by dhclient. This required the following lines in /etc/devfs.rules:
[mydevfsrules_jail=100]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
[devfsrules_jail_dhcp=101]
add include $mydevfsrules_jail
add path 'bpf*' unhide
Note that after adding the above lines, you need to run:
service devfs start
to load the rules (per
devfs(8)
man page).
Note: I learned my mistake not to number my blocks immediately after
the standard defaults (from /etc/defaults/devfs.rules). I had done that
befure but there’s now a conflict, so I skip ahead a bit to get a unique range.
Now I needed to make a number of directories for the jail:
mkdir -p /usr/jails/mgmt/root
mkdir -p /usr/jails/mgmt/etc
mkdir -p /usr/jails/mgmt/var/mgmt
mkdir -p /usr/jails/mgmt/tmp
I needed to setup the fstab for the jail:
# Device Mountpoint FStype Options Dump Pass#
/ /usr/jail/mgmt/root nullfs ro 0 0
/usr/jail/mgmt/etc /usr/jail/mgmt/root/etc nullfs rw 0 0
/usr/jail/mgmt/var /usr/jail/mgmt/root/var nullfs rw 0 0
/var/mgmt /usr/jail/mgmt/root/var/mgmt nullfs ro 0 0
/usr/jail/mgmt/tmp /usr/jail/mgmt/root/tmp nullfs rw 0 0
This is a little bit more tricky, It first nullfs mounts the root system.
I’m using ZFS boot environments, so this is a pretty clean FreeBSD install
without much host specific data. It then mounts some jail specific directories
for etc, tmp and var and finally mounts the shared directory w/ the
unix domain socket to the host system. Also note that a couple of the mounts
are read-only to prevent the jail from modifying the system.
The etc directory was populated from the system etc via:
tar -cf - -C /etc . | tar -xf - -C /usr/jails/mgmt/etc
Then the jail was configured, first /usr/jails/mgmt/etc/rc.conf:
hostname="mgmt.funkthat.com"
sendmail_enable="NONE"
sendmail_submit_enable="NO"
sendmail_outbound_enable="NO"
sendmail_msp_queue_enable="NO"
# Management port
ifconfig_mgmt0="DHCP"
# necessary as devd can't be run in a jail
synchronous_dhclient="YES"
inetd_enable="YES" # Run the network daemon dispatcher (YES/NO).
The key part of this configuration that took me a while to figure out was
the synchronous_dhclient line. It used to be that netif would start
dhclient, but in order to better handle USB ethernet devices and other
removable interfaces, it was moved to devd. The only problem is that
devd hasn’t been jail’ified, and you can’t run it to get things like
link notifications that would normally launch dhclient. Setting this to
yes, makes sure it gets launched when the jail starts.
And then the following line was added to /usr/jails/mgmt/etc/inetd.conf:
ssh stream tcp nowait root /usr/bin/nc nc -N -U /var/mgmt/mgmt.ssh.sock
This is the part that will forward incoming connections to the ssh port
on to the unix domain socket.
Now that everything is configured, a simple jail -c mgmt will get
the jail running and accepting connections.
To get the jail to start every boot, add the following to the host’s
/etc/rc.conf:
jail_enable="YES" # Set to NO to disable starting of any jails
jail_list="mgmt"
This was testing and deployed on a FreeBSD 14-CURRENT build as of August
8th, 2023, or more specifically, from main-n264621-09c20a29328, but it
should work on all currently supported FreeBSD releases.