FreeBSD 13.5 → 15.1 | Gossamer Web Design

FreeBSD 13.5 → 15.1 Upgrade Runbook (with poudriere package rebuilds)

Scope: Upgrade a production FreeBSD 13.5-RELEASE host to 15.1-RELEASE, rebuilding all third-party packages with poudriere at each ABI break.

Important: FreeBSD does not support skipping a major version. You cannot go directly from 13.5 to 15.x — the path is 13.5 → 14.4 → 15.1. 15.0 reaches EOL on 2026-09-30, so 15.1 is the correct target, not 15.0.

Do a dry run on a clone or VM before touching the production box. The 14→15 ABI jump has broken poudriere builder jails in the field (jail-newer-than-host errors), so it’s worth knowing what you’re dealing with first.

Stage -1 — Dry Run on a Clone (do this first)

Test the entire procedure below on a clone before touching production. Three options, depending on what hardware you have available.

Option 1: Local clone + bhyve (no second machine needed)

If you don’t have a separate physical or virtual box to test on, you can still get a genuinely isolated dry run on the same hardware, using a local ZFS clone and bhyve (built into the FreeBSD base system).

Create the clone. Which command depends on your pool layout:

zfs snapshot -r zroot@dryrun-clone

# If the target filesystem (e.g. /zuser) is a separate pool:
zfs send -R zroot@dryrun-clone | zfs receive -F zuser/dryrun-clone

# If it's a dataset within the same zroot pool, use zfs clone instead —
# instant, copy-on-write, costs ~zero space up front:
zfs clone zroot@dryrun-clone zroot/zuser/dryrun-clone

Boot it under bhyve:

pkg install vm-bhyve            # wraps raw bhyve(8), much less fiddly

kldload vmm
sysrc vmm_load="YES"            # persist across reboots

zfs create -V 20G zuser/dryrun-vmdisk     # size to match your usage
vm create -t freebsd -s /dev/zvol/zuser/dryrun-vmdisk dryrun-test
vm console dryrun-test

Run the full upgrade procedure (Stage 0 through Stage 2) inside this VM before touching production — real isolation, zero extra hardware.

If you’d rather not set up bhyve, the clone is still worth keeping for two narrower purposes even without booting it:

Rollback insurance — an independent full copy of pre-upgrade state, separate from your bectl/snapshot rollback path, to restore individual files from if production goes sideways mid-upgrade.
Reference for diffing — zfs mount -o ro the clone after the real upgrade and diff configs/package lists against the live post-upgrade system to spot anything that changed unexpectedly.

Option 2: Exact clone to a separate machine (best fidelity, needs a second box)

On production:

zfs snapshot -r zroot@dryrun-clone
zfs send -R zroot@dryrun-clone | ssh testvm "zfs receive -F testpool"
# or, to a file:
zfs send -R zroot@dryrun-clone > /path/to/external/zroot-dryrun.zfs

On the test VM:

zfs receive -F zroot < zroot-dryrun.zfs        # if using the file method
zpool set bootfs=zroot/ROOT/default zroot
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 vtbd0   # adjust device name
rm /etc/hostid                                  # regenerate so it won't collide with prod

Give the VM an isolated/host-only NIC and a different hostname/IP before booting it, so it can’t collide with the real box (duplicate IP, mail loops, cron jobs hitting real endpoints).

Option 3: Config-replica fallback (faster, lower fidelity)

If wiring zfs send into a hypervisor is more hassle than it’s worth, build a fresh 13.5-RELEASE VM and replicate just what matters for testing the upgrade mechanics:

pkg query -e '%a=0' '%n' | sort > /root/explicit-pkgs-13.5.txt
tar czf /root/etc-backup.tgz /etc
cp /etc/sysctl.conf /boot/loader.conf /root/

Copy those to the VM, install the package list, drop in the config files, merge /etc by hand. You lose real jail/DB contents, but you get an accurate test of whether the upgrade + poudriere rebuild sequence works cleanly with this package set.

What to watch for on the clone

Run Stage 0 through Stage 2 in full and specifically check:

Any port that fails to build in the poudriere bulk run
Any unexpected config merge prompt during freebsd-update install
The sysctl verification steps reporting all four values live and correct

Once clean end-to-end on the clone, proceed to production with confidence.

Stage 0 — Preparation

Fully patch the current system before upgrading:

freebsd-update fetch install
reboot
freebsd-update fetch install   # repeat until "no updates needed"

Snapshot or create a boot environment:

zfs snapshot -r zroot@pre-14.4-upgrade
# or, if using bectl:
bectl create pre-14.4-upgrade

Record explicitly installed packages:

pkg query -e '%a=0' '%n' | sort > /root/explicit-pkgs-13.5.txt

Update poudriere and its ports tree:

pkg upgrade poudriere
poudriere ports -u

Optional efficiency tip: point poudriere at your host’s existing distfiles cache so source tarballs aren’t downloaded twice (once for the jail’s bulk build, again later when you make install clean something by hand):

# in /usr/local/etc/poudriere.conf
DISTFILES_CACHE=/usr/ports/distfiles

Back up your current pkg repo configuration before it gets overridden for the upgrade cutover — you’ll restore this once the poudriere package repos are torn down at the end of Stage 2:

cp -R /usr/local/etc/pkg/repos /root/pkg-repos.pre-upgrade

Back up custom kernel tunables (/etc/sysctl.conf merges automatically across upgrades since the stock file rarely changes, but keep a copy to diff against):

cp /etc/sysctl.conf /root/sysctl.conf.pre-upgrade
cp /boot/loader.conf /root/loader.conf.pre-upgrade

Custom buffer/queue tunables to track (adjust to your actual values):

kern.ipc.somaxconn=2048
kern.ipc.soacceptqueue=2048
net.inet.tcp.syncookies=1
net.inet.tcp.keepinit=5000

Stage 1 — 13.5 → 14.4

1.1 Base system

freebsd-update -r 14.4-RELEASE upgrade
freebsd-update install        # new kernel only
reboot
freebsd-update install        # new userland, merges configs

Do not run the third freebsd-update install yet. It deletes the old shared libraries that your current (13.x-built) packages still depend on.

1.2 Rebuild packages with poudriere

If you normally build ports by hand with custom OPTIONS selections or a customized /etc/make.conf, carry those into poudriere first — a default bulk build won’t know about them:

# Copy your host's saved OPTIONS selections into a poudriere options tree
cp -R /var/db/ports /usr/local/etc/poudriere.d/options

# Merge any custom settings from your host's make.conf into the jail/portset-specific
# make.conf, e.g.:
#   /usr/local/etc/poudriere.d/144amd64-<ports>-make.conf
cat /etc/make.conf

If you’ve never customized OPTIONS beyond port defaults, skip this — the procedure below is unchanged.

Tuning for long builds (llvm/rust, multi-hour)

Worth setting before kicking off a build you expect to run 12+ hours.

Check your core count and cap ZFS ARC first — MAX_MEMORY below is a per-builder limit, not global, so what actually matters is PARALLEL_JOBS × MAX_MEMORY versus total RAM, with headroom left for the host and ARC:

sysctl -n hw.ncpu
sysctl vfs.zfs.arc_max=4G        # or persist via /boot/loader.conf: vfs.zfs.arc_max="4G"

Reserve ~2-4GB for the host/ARC, then divide the rest across builders. On a 32GB box with 8 cores, for example, 24GB ÷ 8 is too tight for llvm/rust (they can spike several GB per translation unit) — throttle concurrency instead of starving memory:

# in /usr/local/etc/poudriere.conf
PARALLEL_JOBS=4        # fewer simultaneous builders rather than 8
MAX_MEMORY=6            # per-builder safety ceiling, in GB

# Kill a port only if it produces no output for this long (default 2h) —
# bump this if hardware is slower or under memory pressure
NOHANG_TIME=14400          # 4 hours

# Absolute cap per port (default 24h)
MAX_EXECUTION_TIME=172800  # 48 hours

# llvm/rust compile largely the same source in both Stage 1 and Stage 2 —
# ccache meaningfully cuts the second pass
USE_CCACHE=yes
CCACHE_DIR=/var/cache/ccache

Keep CCACHE_DIR outside the jail/package-repo directories you plan to delete during cleanup, so the cache survives from Stage 1 into Stage 2. Adjust PARALLEL_JOBS/MAX_MEMORY to your own core count and available RAM — the numbers above are just an 8-core/32GB example.

Build the heavy ports (llvm/rust) serially, in their own pass, rather than letting them land on top of each other in the middle of the full run — PARALLEL_JOBS alone only controls total concurrency, not which specific ports run together:

# List your llvm/rust variants explicitly
echo "lang/rust
devel/llvm18
devel/llvm19" > /root/heavy-ports.txt

# Pass 1: heavy ports, one at a time regardless of PARALLEL_JOBS
poudriere bulk -j 144amd64 -p <your-ports-tree> -o /usr/local/etc/poudriere.d/options -J 1 -f /root/heavy-ports.txt

# Pass 2: everything else, normal concurrency — already-built packages from
# Pass 1 are reused, not rebuilt
poudriere bulk -j 144amd64 -p <your-ports-tree> -o /usr/local/etc/poudriere.d/options -f /root/explicit-pkgs-13.5.txt

Also worth setting, so heavy ports get scheduled early rather than clustering near the end of a run when builders are idle:

# in /usr/local/etc/poudriere.conf
PRIORITY_BOOST="llvm* rust chromium*"

Run it in a detachable session

poudriere bulk normally ties to your terminal. Over a multi-day upgrade window, a dropped SSH connection shouldn’t kill a build 10 hours into compiling llvm:

tmux new -s bulk-upgrade

Run the jail creation and both bulk passes (above/below) inside this session, then Ctrl-b d to detach — tmux attach -t bulk-upgrade to check back in later. poudriere status from any other session works regardless, so you’re not locked to this terminal just to monitor progress.

# Fresh builder jail matching the new host ABI — don't update the old jail in place
poudriere jail -c -j 144amd64 -v 14.4-RELEASE

Then run the two poudriere bulk passes shown above (heavy ports first with -J 1, then the full package list at normal concurrency).

What you’ll see during the run

Live console output shows a per-port status line as things build, then a summary table at the end:

[144amd64-mytree] [2026-09-10_14h32m] Queued: 412
[144amd64-mytree] Building 412 packages using 4 builders
...
[144amd64-mytree] [00:47:12] Finished perl5.40-5.40.0
[144amd64-mytree] [00:47:33] Failed: www/php83-extensions: build failed
...
====>> Log directory: /usr/local/poudriere/data/logs/bulk/144amd64-mytree/2026-09-10_14h32m
[144amd64-mytree] Packages built: 398
[144amd64-mytree] Packages failed: 6
[144amd64-mytree] Packages skipped: 4
[144amd64-mytree] Packages ignored: 2
[144amd64-mytree] Packages fetched: 0

Handling bulk build failures

Every port lands in one of four buckets:

Built — succeeded, package written to the repo. No action needed.
Failed — the actual compile/install step errored. Needs investigation.
Skipped — depends on a failed port; clears automatically once the dependency is fixed and rebuilt.
Ignored — the port declared itself unbuildable on this platform/ABI (IGNORE=/BROKEN/DEPRECATED/EXPIRED in the Makefile). Retrying won’t help — needs an alternative port, a patch, or dropping it from the install set.

Per-port build logs:

/usr/local/poudriere/data/logs/bulk/144amd64-mytree/latest/logs/<category>/<origin>.log

Quick list of everything that failed:

cat /usr/local/poudriere/data/logs/bulk/144amd64-mytree/latest/list.failed

Browsable HTML report (if you have a web server pointed at the logs directory):

/usr/local/poudriere/data/logs/bulk/144amd64-mytree/latest/index.html

Optional: monitoring the bulk build in a browser

poudriere bulk shows live status in the console, and poudriere status from another SSH session gives the same built/failed/skipped/ignored counts without any web server at all. If you’d rather watch it in a browser, poudriere already generates a self-refreshing index.html + build.json in the logs directory — you just need something to serve those static files, temporarily:

# Quick disposable server, bound to localhost only
cd /usr/local/poudriere/data/logs
python3 -m http.server 8080 --bind 127.0.0.1

# From your workstation, tunnel in rather than exposing the port
ssh -L 8080:127.0.0.1:8080 youruser@productionhost

Browse to http://127.0.0.1:8080/bulk/144amd64-mytree/latest/. Kill the python3 process when done — nothing persists, nothing gets exposed beyond localhost.

If you’d rather reuse Apache instead of a second process, add a temporary alias restricted to your own IP and remove it after the upgrade:

# /usr/local/etc/apache24/Includes/poudriere-monitor.conf
Alias /poudriere-logs /usr/local/poudriere/data/logs
<Directory /usr/local/poudriere/data/logs>
    Options Indexes FollowSymLinks
    Require ip 203.0.113.0/24   # replace with your own IP/range
</Directory>

apachectl configtest && apachectl graceful

Delete the config file and reload Apache once done — don’t leave build logs reachable indefinitely, even behind an IP restriction.

Common causes on a major-version jump:

Missing/dead distfiles or license-acceptance prompts — fetch errors in the log.
Patches that don’t apply against an updated upstream tarball — run poudriere ports -u to pick up port fixes, or patch files/ locally.
Resource limits on large C++ ports (llvm, rust, webkit) — signal 9/Killed in the log means bump MAX_MEMORY/MAX_MEMORY_BYTES in poudriere.conf.
Language ABI conflicts (e.g. a port hardcoding an old Perl version instead of using the tree default) — shows as a build error referencing the wrong path.

Once fixed, retry just the failed set rather than rebuilding everything:

poudriere bulk -j 144amd64 -p <your-ports-tree> -o /usr/local/etc/poudriere.d/options -f /usr/local/poudriere/data/logs/bulk/144amd64-mytree/latest/list.failed

Skipped ports cascade back in automatically on the next full run against explicit-pkgs-13.5.txt.

1.3 Cut the host over to the new packages

# Point pkg repo config (e.g. /usr/local/etc/pkg/repos/local.conf) at:
#   /usr/local/poudriere/data/packages/144amd64-<ports>/
pkg update -f
pkg upgrade -f

pkg upgrade -f only replaces files on disk — any daemon already running keeps executing the old binary/libraries until it’s restarted. Apache, nginx, PHP-FPM, Postgres, etc. will keep serving on the pre-upgrade build until you bounce them.

1.3b Restart exposed services

Restart anything user-facing now, rather than waiting for the final reboot at the end of the stage — minimizes the window where external clients see the old build:

apachectl configtest && apachectl graceful     # or: service apache24 graceful
service nginx reload
service php-fpm restart
service postgresql restart

Verify each one actually picked up the new build:

curl -sI http://localhost/ | grep -i server

1.4 Finish the base upgrade

freebsd-update install        # third pass — removes obsolete 13.x libs
reboot
freebsd-version -kru          # confirm 14.4-RELEASE / 14.4-RELEASE / 14.4-RELEASE

1.5 Verify custom sysctls survived

diff /root/sysctl.conf.pre-upgrade /etc/sysctl.conf
sysctl kern.ipc.somaxconn kern.ipc.soacceptqueue net.inet.tcp.syncookies net.inet.tcp.keepinit

Confirm the live values match what’s in the file — if any oid errors with “unknown oid”, that tunable was renamed or removed in 14.x and needs research before continuing.

1.6 Clean up the builder jail (once validated)

Only after you’re confident 14.4 is stable and fully tested — the jail is your quickest path back to a working build environment if something surfaces later:

poudriere jail -d -j 144amd64

Leaving it in place costs disk space but nothing else; there’s no rush to delete it.

Stage 2 — 14.4 → 15.1

2.1 Snapshot

zfs snapshot -r zroot@pre-15.1-upgrade

2.2 Base system

freebsd-update -r 15.1-RELEASE upgrade
freebsd-update install
reboot
freebsd-update install

2.3 Rebuild packages with poudriere

The ports tree stays exactly as it was from Stage 0’s sync — nothing changes it on its own. Only re-sync if real calendar time has passed since Stage 1 (you paused to validate 14.4, waited for a maintenance window, etc.) and you want to pick up any port fixes or 15.x-specific patches that landed since. If Stage 1 and Stage 2 happen back-to-back, skip this and reuse the same tree/options as Stage 1.

# Only if time has passed and you want newer ports-tree fixes:
poudriere ports -u
cp -R /var/db/ports /usr/local/etc/poudriere.d/options

Reuse the same PARALLEL_JOBS/MAX_MEMORY/NOHANG_TIME/MAX_EXECUTION_TIME/USE_CCACHE/PRIORITY_BOOST tuning from Stage 1.2 — poudriere.conf is shared across jails, so nothing to redo here. Run this stage’s build in a fresh detachable session too:

tmux new -s bulk-upgrade-stage2

poudriere jail -c -j 151amd64 -v 15.1-RELEASE

# Pass 1: heavy ports serially again (llvm/rust rebuilt fresh against 15.1 ABI)
poudriere bulk -j 151amd64 -p <your-ports-tree> -o /usr/local/etc/poudriere.d/options -J 1 -f /root/heavy-ports.txt

# Pass 2: everything else, normal concurrency
poudriere bulk -j 151amd64 -p <your-ports-tree> -o /usr/local/etc/poudriere.d/options -f /root/explicit-pkgs-13.5.txt

See “Handling bulk build failures” under Stage 1.2 for what to expect and how to retry — same categories (Built/Failed/Skipped/Ignored), same log paths, just under .../logs/bulk/151amd64-mytree/latest/.

2.4 Cut over and finish

# Repoint pkg repo config at the 151amd64 package set
pkg update -f
pkg upgrade -f

2.4b Restart exposed services

apachectl configtest && apachectl graceful     # or: service apache24 graceful
service nginx reload
service php-fpm restart
service postgresql restart

curl -sI http://localhost/ | grep -i server

2.5 Finish the base upgrade

freebsd-update install        # removes obsolete 14.x libs
reboot
freebsd-version -kru          # confirm 15.1-RELEASE across the board

2.6 Verify custom sysctls survived

diff /root/sysctl.conf.pre-upgrade /etc/sysctl.conf
sysctl kern.ipc.somaxconn kern.ipc.soacceptqueue net.inet.tcp.syncookies net.inet.tcp.keepinit

Same check as Stage 1 — confirm all four are present and live, with no “unknown oid” errors.

2.7 Clean up the builder jail and revert pkg repo config

poudriere jail -d -j 151amd64

Also a good point to remove the Stage 1 (144amd64) jail and its package repo if you haven’t already, once you’re satisfied 15.1 is solid.

Once the poudriere package repos are gone, restore your original pkg repo config so nothing points at a directory that no longer exists:

rm -rf /usr/local/etc/pkg/repos
cp -R /root/pkg-repos.pre-upgrade /usr/local/etc/pkg/repos
pkg update -f

This matters even if you build via make install clean day-to-day: any ad hoc pkg audit, pkg update, or pkg upgrade run later would otherwise fail trying to reach a repo directory that’s been deleted.

Gotchas checklist

ABI changes every major version (FreeBSD:14:amd64 → FreeBSD:15:amd64). Every package must be rebuilt — no partial reuse. Never run poudriere jail -u across a major version; always create a fresh jail with -c.
Config merges: 15.x renames UseBlacklist to UseBlocklist in sshd_config, and OpenSSH 10’s defaults changed. Review merge diffs instead of accepting them blindly.
Language runtime default-version bumps (Perl, PHP, Python): the ports tree’s Mk/bsd.default-versions.mk can set a new default (e.g. Perl 5.38 → 5.40). This only matters if you choose to re-sync the tree (poudriere ports -u) between stages — the tree doesn’t change on its own. If you do re-sync and a package you rely on doesn’t yet support the new default, pin it explicitly in /usr/local/etc/poudriere.d/<jail>-<portset>-make.conf: DEFAULT_VERSIONS+= perl5=5.38
Keep old jails/package sets until the new version is validated — that’s your fast rollback path (repoint pkg repo config back, or bectl activate / zfs rollback the old boot environment).
Base rollback: freebsd-update rollback only works before the third install pass. After that, restoring means falling back to the snapshot/boot environment.
Set PAGER=cat before running freebsd-update interactively over SSH, so it doesn’t try to page every merge diff.

Runbook generated for a 13.5-RELEASE → 15.1-RELEASE upgrade using poudriere for package rebuilds. Verify release numbers are still current before running in production — check freebsd.org/releases for the latest supported versions on each branch.