rpi4 zfs-on-root boot-to-usb3

Discussion:

tech-lists

2021-05-06 11:12:48 UTC

Hi,

How can zfs-on-root boot-to-usb3 on rpi4 be accomplished?

I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to be
problems with it that I can't work around. What's really needed is an
installer, but these aren't made for arm64.aarch64 rpi4 from what I can
see (I'm no expert though, it's entirely feasible i've missed
something).

Maybe one way of doing it would be to have a usb key (as ufs2) for the
system to boot on, then have /home /usr/obj and other larger dirs on the
usb3-zfs disk.

--
J.

Mark Millard via freebsd-arm

2021-05-06 12:55:21 UTC

Permalink

Post by tech-lists
How can zfs-on-root boot-to-usb3 on rpi4 be accomplished?
I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to be
problems with it that I can't work around. What's really needed is an
installer, but these aren't made for arm64.aarch64 rpi4 from what I can
see (I'm no expert though, it's entirely feasible i've missed
something).
Maybe one way of doing it would be to have a usb key (as ufs2) for the
system to boot on, then have /home /usr/obj and other larger dirs on the
usb3-zfs disk.

I used bsdinstall from booting a releng/13.0's release/13.0.0.0
microsd card in a 8 GiBYte RPi4B to produce the:

# gpart show -pl da0
=> 40 468862048 da0 GPT (224G)
40 532480 da0p1 efiboot0 (260M)
532520 2008 - free - (1.0M)
534528 25165824 da0p2 swp12a (12G)
25700352 25165824 da0p4 swp12b (12G)
50866176 417994752 da0p3 zfs0 (199G)
468860928 1160 - free - (580K)

# gpart show -p da0
=> 40 468862048 da0 GPT (224G)
40 532480 da0p1 efi (260M)
532520 2008 - free - (1.0M)
534528 25165824 da0p2 freebsd-swap (12G)
25700352 25165824 da0p4 freebsd-swap (12G)
50866176 417994752 da0p3 freebsd-zfs (199G)
468860928 1160 - free - (580K)

that I'm using to experiment with bectl and the like.
It boots the RPi4B 8 or 4 GiByte, the OverDrive 1000,
and the MACCHIATObin Double Shot. (I had to separately
add the RPi firmware and U-Boot to the efi file
system efiboot0 in order to enable the RPi4B booting.
Later I tried booting the other two.)

But until I go back and do it again to another USB3
SSD, recording steps as I go, I can not supply a
step-by-step sequence to go through. I might have to
again figure things out that I figured out the first
time. I just do not remember all the detail. It had
been years since I'd used bsdinstall.

It will be a while before I can try that but I
expect that I can and I then I should be able to
report an example sequence.

(I split the original swap space in two later.
I use only 1 of them as swap for 4 GiByte RPi4B's.
Part of my experiments have been using tmpfs in
poudriere and the large swap total for 8 GiBYte+
machines is tied to that activity for how I have
poudriere configured and building things like
llvm*, possibly multiple of them in parallel.)

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-arm

2021-05-06 19:12:43 UTC

Permalink

Post by Mark Millard via freebsd-arm

I used bsdinstall from booting a releng/13.0's release/13.0.0.0
. . .

Various details shown will just be my specific
choices. (The RPi4B's that I have access to have
the 2021-Apr-29 default(/critical) EEPROM image.)

Taking notes as I go (and readjusting as I
progress and figure things out, eliminating
failing attempts as well) . . .

Booting based on a microsd card with releng/13.0 's
release/13.0.0 as its basis. The context has a
working network with internet access.

# uname -apKU
FreeBSD generic 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 06:06:55 UTC 2021 ***@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch64 1300139 1300139

Plug in USB3 SSD. Ends up as da0.

# /bin/sh # Just for my familiarity
# set -o vi

# mkdir -p /usr/freebsd-dist
# cd /usr/freebsd-dist
# fetch http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST
MANIFEST 782 B 6147 kBps 00s
# cd ~

# bsdinstall

Continue with default keymap : Select

Enter hostname as ZFStest : OK

[*] base-dbg
[*] kernel-dbg
[ ] ports
[ ] src
[*] tests
then: OK

(Note: I use git for src and ports.)

Main Site : OK

Auto (ZFS) : OK

Pool Name : Select

Enter name for zpool ztstp : OK

Swap Size : Select

Enter swap size 24g : OK

Proceed with Installation : Select

Stripe - No Redundancy : OK

[*] da0 : OK

Last Chance for da0 : YES

Downloads . . .
Extracts . . .

New Password: . . .
Retype New Password: . . .

genet0 : OK

configure IPv4 : YES
configure DHCP : YES
configure IPv6 : YES
try SLAAC : YES
Resolver Configuration : OK

time is UTC? : YES

America : OK
United States of America : OK
Pacific : OK
Does PDT look reasonable? : Yes
May 2021 6 : Set Date
11 07 00 : Set Time

[ ] local_unbound
[*] sshd
[ ] moused
[ ] ntpdate
[*] ntpd
[*] powerd
[*] dumpdev
Then : OK

No hardening options enabled : OK

Add uses? : Yes
. . . details omitted . . .
OK ? yes
Add another user? no

Handbook : OK
[*] en : OK

Apply configuration and exit installer : OK
open a shell : No

# shutdown -p now

At this point it still can not boot an RPi4B
for lack of rpi firmware and U-Boot.

I have such available on other machine based
on the latest ports instead of quarterly. There
are RPi4B's in the world that need the more
modern U-Boot compared to the quarterly that
releng/13.0 is tied to by default. But you
likely could install rpi-firmware and
u-boot-rpi-arm64 on the microsd card and
then copy over materials from there.

In my context . . .

# gpart show -p da1
=> 40 468862048 da1 GPT (224G)
40 532480 da1p1 efi (260M)
532520 2008 - free - (1.0M)
534528 50331648 da1p2 freebsd-swap (24G)
50866176 417994752 da1p3 freebsd-zfs (199G)
468860928 1160 - free - (580K)

# mount -onoatime -tmsdosfs /dev/da1p1 /mnt
# cp -aRx /usr/local/share/rpi-firmware/* /mnt/
# cp -aRx /mnt/config_arm64.txt /mnt/config.txt
# cp -aRx /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin /mnt/
# umount /mnt

Back to the RPi4B, no microsd card but plugging in the
USB3 SSD and booting and logging in:

Dec 31 16:00:48 ZFStest login[1351]: ROOT LOGIN (root) ON ttyu0
FreeBSD 13.0-RELEASE (GENERIC) #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021

# gpart show -p
=> 40 468862048 da0 GPT (224G)
40 532480 da0p1 efi (260M)
532520 2008 - free - (1.0M)
534528 50331648 da0p2 freebsd-swap (24G)
50866176 417994752 da0p3 freebsd-zfs (199G)
468860928 1160 - free - (580K)

# uname -apKU
FreeBSD ZFStest 13.0-RELEASE FreeBSD 13.0-RELEASE #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021 ***@releng1.nyi.freebsd.org:/usr/obj/usr/src/arm64.aarch64/sys/GENERIC arm64 aarch64 1300139 1300139

I end up adding to /etc/rc.conf:

#
ntpd_sync_on_start="YES"
ntpd_user="root"

The first boot's time will be messed up for
lack of the ntpd_sync_on_start="YES" .

# shutdown -r now

After login:

# ls -Tld /etc/rc.conf
-rw-r--r-- 1 root wheel 279 Dec 31 16:12:37 1969 /etc/rc.conf
# touch /etc/rc.conf

There are other files around with such an odd timestamp.

# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ztstp 199G 1.09G 198G - - 0% 0% 1.00x ONLINE -

# zfs list
NAME USED AVAIL REFER MOUNTPOINT
ztstp 1.09G 192G 96K /ztstp
ztstp/ROOT 1.08G 192G 96K none
ztstp/ROOT/default 1.08G 192G 1.08G /
ztstp/tmp 96K 192G 96K /tmp
ztstp/usr 416K 192G 96K /usr
ztstp/usr/home 128K 192G 128K /usr/home
ztstp/usr/ports 96K 192G 96K /usr/ports
ztstp/usr/src 96K 192G 96K /usr/src
ztstp/var 680K 192G 96K /var
ztstp/var/audit 96K 192G 96K /var/audit
ztstp/var/crash 96K 192G 96K /var/crash
ztstp/var/log 200K 192G 200K /var/log
ztstp/var/mail 96K 192G 96K /var/mail
ztstp/var/tmp 96K 192G 96K /var/tmp

# more /etc/sysctl.conf
# $FreeBSD$
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#

# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12

I'll note that in:

# more /boot/efi/config.txt
[all]
arm_64bit=1
dtparam=audio=on,i2c_arm=on,spi=on
dtoverlay=mmc
dtoverlay=disable-bt
device_tree_address=0x4000
kernel=u-boot.bin

[pi4]
hdmi_safe=1
armstub=armstub8-gic.bin

The hdmi)safe=1 line restricts the HDMI display
resolution/scaling. Any of the following
replacements for that line will avoid that but
in some contexts one could end up in a "blind
display" context instead, which is why hdmi_safe
is enabled by default.

hdmi_safe=0
or:
#hdmi_safe=1
or just delete the line.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-arm

2021-05-06 20:09:31 UTC

Permalink

Post by Mark Millard via freebsd-arm

I used bsdinstall from booting a releng/13.0's release/13.0.0.0
. . .

I suggested an inappropriate later stage if one
wants to try the same vintage of rpi-firmware
and u-boot as releng/13.0 itself uses. One
can copy over materials before the shutdown
by getting them from /boot/msdos/ .

Note that you might not want to copy over
/boot/msdos/EFI/... as the install will
already have such.

So, something like:

# mount -onoatime -tmsdosfs /dev/da0p1 /mnt
# cp -aRx /boot/msdos/[LRa-z]* /mnt/
# umount /mnt

then do the shutdown and remove the microsd card.

Post by Mark Millard via freebsd-arm
# shutdown -p now

The following is only if one did not copy from
/boot/msdosfs/ or one needs more recent
materials, such as U-Boot for *C0T revision
processors.

Post by Mark Millard via freebsd-arm
At this point it still can not boot an RPi4B
for lack of rpi firmware and U-Boot.
I have such available on other machine based
on the latest ports instead of quarterly. There
are RPi4B's in the world that need the more
modern U-Boot compared to the quarterly that
releng/13.0 is tied to by default. But you
likely could install rpi-firmware and
u-boot-rpi-arm64 on the microsd card and
then copy over materials from there.

The above is dumb suggestion unless newer
material is needed. See before the shutdown
above.

The below is me getting more recent materials
(well, U-Boot) from another context. You might
not do similarly.

Post by Mark Millard via freebsd-arm
In my context . . .
# gpart show -p da1
=> 40 468862048 da1 GPT (224G)
40 532480 da1p1 efi (260M)
532520 2008 - free - (1.0M)
534528 50331648 da1p2 freebsd-swap (24G)
50866176 417994752 da1p3 freebsd-zfs (199G)
468860928 1160 - free - (580K)
# mount -onoatime -tmsdosfs /dev/da1p1 /mnt
# cp -aRx /usr/local/share/rpi-firmware/* /mnt/
# cp -aRx /mnt/config_arm64.txt /mnt/config.txt
# cp -aRx /usr/local/share/u-boot/u-boot-rpi-arm64/u-boot.bin /mnt/
# umount /mnt

(Note: It is possible to be more selective in
what to copy over. I did not complicate the
sequence with such handling.)

Back to the normal flow on the RPi4B given
appropriate RPi* materials copied over to the
msdos file system . . .

Post by Mark Millard via freebsd-arm
Back to the RPi4B, no microsd card but plugging in the
Dec 31 16:00:48 ZFStest login[1351]: ROOT LOGIN (root) ON ttyu0
FreeBSD 13.0-RELEASE (GENERIC) #0 releng/13.0-n244733-ea31abc261f: Fri Apr 9 03:54:53 UTC 2021
# gpart show -p
=> 40 468862048 da0 GPT (224G)
40 532480 da0p1 efi (260M)
532520 2008 - free - (1.0M)
534528 50331648 da0p2 freebsd-swap (24G)
50866176 417994752 da0p3 freebsd-zfs (199G)
468860928 1160 - free - (580K)
# uname -apKU
#
ntpd_sync_on_start="YES"
ntpd_user="root"
The first boot's time will be messed up for
lack of the ntpd_sync_on_start="YES" .
# shutdown -r now
# ls -Tld /etc/rc.conf
-rw-r--r-- 1 root wheel 279 Dec 31 16:12:37 1969 /etc/rc.conf
# touch /etc/rc.conf
There are other files around with such an odd timestamp.
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
ztstp 199G 1.09G 198G - - 0% 0% 1.00x ONLINE -
# zfs list
NAME USED AVAIL REFER MOUNTPOINT
ztstp 1.09G 192G 96K /ztstp
ztstp/ROOT 1.08G 192G 96K none
ztstp/ROOT/default 1.08G 192G 1.08G /
ztstp/tmp 96K 192G 96K /tmp
ztstp/usr 416K 192G 96K /usr
ztstp/usr/home 128K 192G 128K /usr/home
ztstp/usr/ports 96K 192G 96K /usr/ports
ztstp/usr/src 96K 192G 96K /usr/src
ztstp/var 680K 192G 96K /var
ztstp/var/audit 96K 192G 96K /var/audit
ztstp/var/crash 96K 192G 96K /var/crash
ztstp/var/log 200K 192G 200K /var/log
ztstp/var/mail 96K 192G 96K /var/mail
ztstp/var/tmp 96K 192G 96K /var/tmp
# more /etc/sysctl.conf
# $FreeBSD$
#
# This file is read when going to multi-user and its contents piped thru
# ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details.
#
# Uncomment this to prevent users from seeing information about processes that
# are being run under another UID.
#security.bsd.see_other_uids=0
vfs.zfs.min_auto_ashift=12
# more /boot/efi/config.txt
[all]
arm_64bit=1
dtparam=audio=on,i2c_arm=on,spi=on
dtoverlay=mmc
dtoverlay=disable-bt
device_tree_address=0x4000
kernel=u-boot.bin
[pi4]
hdmi_safe=1
armstub=armstub8-gic.bin
The hdmi)safe=1 line restricts the HDMI display
resolution/scaling. Any of the following
replacements for that line will avoid that but
in some contexts one could end up in a "blind
display" context instead, which is why hdmi_safe
is enabled by default.
hdmi_safe=0
#hdmi_safe=1
or just delete the line.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-arm

2021-05-06 20:42:50 UTC

Permalink

Post by Mark Millard via freebsd-arm

I used bsdinstall from booting a releng/13.0's release/13.0.0.0
. . .

I'll note that setting up the microsd card context to
have the correct timezone and time is something I
presumed was already in place. But such is not the
case for an RPI image from the servers.

So I effectively presumed booting with /etc/rc.conf
having, say,

#
ntpd_enable="YES"
ntpd_sync_on_start="YES"
ntpd_user="root"

already added (or a manual time set)
and having done a:

# tzsetup

sequence with appropriate selections.

Post by Mark Millard via freebsd-arm

Post by Mark Millard via freebsd-arm
# uname -apKU
Plug in USB3 SSD. Ends up as da0.
# /bin/sh # Just for my familiarity
# set -o vi
# mkdir -p /usr/freebsd-dist
# cd /usr/freebsd-dist
# fetch http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST
MANIFEST 782 B 6147 kBps 00s
# cd ~
# bsdinstall
Continue with default keymap : Select
Enter hostname as ZFStest : OK
[*] base-dbg
[*] kernel-dbg
[ ] ports
[ ] src
[*] tests
then: OK
(Note: I use git for src and ports.)
Main Site : OK
Auto (ZFS) : OK
Pool Name : Select
Enter name for zpool ztstp : OK
Swap Size : Select
Enter swap size 24g : OK
Proceed with Installation : Select
Stripe - No Redundancy : OK
[*] da0 : OK
Last Chance for da0 : YES
Downloads . . .
Extracts . . .
New Password: . . .
Retype New Password: . . .
genet0 : OK
configure IPv4 : YES
configure DHCP : YES
configure IPv6 : YES
try SLAAC : YES
Resolver Configuration : OK
time is UTC? : YES
America : OK
United States of America : OK
Pacific : OK
Does PDT look reasonable? : Yes
May 2021 6 : Set Date
11 07 00 : Set Time
[ ] local_unbound
[*] sshd
[ ] moused
[ ] ntpdate
[*] ntpd
[*] powerd
[*] dumpdev
Then : OK
No hardening options enabled : OK
Add uses? : Yes
. . . details omitted . . .
OK ? yes
Add another user? no
Handbook : OK
[*] en : OK
Apply configuration and exit installer : OK
open a shell : No

I suggested an inappropriate later stage if one
wants to try the same vintage of rpi-firmware
and u-boot as releng/13.0 itself uses. One
can copy over materials before the shutdown
by getting them from /boot/msdos/ .
Note that you might not want to copy over
/boot/msdos/EFI/... as the install will
already have such.
# mount -onoatime -tmsdosfs /dev/da0p1 /mnt
# cp -aRx /boot/msdos/[LRa-z]* /mnt/
# umount /mnt
then do the shutdown and remove the microsd card.

Post by Mark Millard via freebsd-arm
# shutdown -p now

The following is only if one did not copy from
/boot/msdosfs/ or one needs more recent
materials, such as U-Boot for *C0T revision
processors.

The above is dumb suggestion unless newer
material is needed. See before the shutdown
above.
The below is me getting more recent materials
(well, U-Boot) from another context. You might
not do similarly.

(Note: It is possible to be more selective in
what to copy over. I did not complicate the
sequence with such handling.)
Back to the normal flow on the RPi4B given
appropriate RPi* materials copied over to the
msdos file system . . .

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-arm

2021-05-06 21:54:53 UTC

Permalink

I see that:

https://download.freebsd.org/ftp/snapshots/arm64/aarch64/13.0-STABLE/

is now populated with a MANIFEST and *.txz files and such
for 11af9a9cf93 ("cap_sysctl.3: Fix bugs in the example"
commited 2021-05-05 15:01:56 +0000).

So, no longer must one go into:

https://artifact.ci.freebsd.org/snapshot/stable-13/?C=M&O=D

to have examples of stable/13 for such materials.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

Mark Millard via freebsd-arm

2021-05-06 22:56:18 UTC

Permalink

Post by Mark Millard via freebsd-arm

I used bsdinstall from booting a releng/13.0's release/13.0.0.0
. . .

I'll note that setting up the microsd card context to
have the correct timezone and time is something I
presumed was already in place. But such is not the
case for an RPI image from the servers.
So I effectively presumed booting with /etc/rc.conf
having, say,
#
ntpd_enable="YES"
ntpd_sync_on_start="YES"
ntpd_user="root"
already added (or a manual time set)
# tzsetup
sequence with appropriate selections.

Post by Mark Millard via freebsd-arm

I suggested an inappropriate later stage if one
wants to try the same vintage of rpi-firmware
and u-boot as releng/13.0 itself uses. One
can copy over materials before the shutdown
by getting them from /boot/msdos/ .
Note that you might not want to copy over
/boot/msdos/EFI/... as the install will
already have such.
# mount -onoatime -tmsdosfs /dev/da0p1 /mnt
# cp -aRx /boot/msdos/[LRa-z]* /mnt/
# umount /mnt
then do the shutdown and remove the microsd card.

Post by Mark Millard via freebsd-arm
# shutdown -p now

The following is only if one did not copy from
/boot/msdosfs/ or one needs more recent
materials, such as U-Boot for *C0T revision
processors.

The above is dumb suggestion unless newer
material is needed. See before the shutdown
above.
The below is me getting more recent materials
(well, U-Boot) from another context. You might
not do similarly.

(Note: It is possible to be more selective in
what to copy over. I did not complicate the
sequence with such handling.)
Back to the normal flow on the RPi4B given
appropriate RPi* materials copied over to the
msdos file system . . .

By warned that things are not set up with /etc/fstab being
based on identification via unique labels:

# more /etc/fstab
# Device Mountpoint FStype Options Dump Pass#
/dev/da0p1 /boot/efi msdosfs rw 2 2
/dev/da0p2 none swap sw 0 0

This can be an issue when multiple USB devices are
present: which will end up as /dev/da0 ? But default
label names would also tend to not be unique as well.

Setting and using unique labels for such can be
appropriate, sort of like unique zpool naming
on various media can be appropriate.

===
Mark Millard
marklmi at yahoo.com
( dsl-only.net went
away in early 2018-Mar)

tech-lists

2021-05-07 07:33:13 UTC

Permalink

On Thu, May 06, 2021 at 12:12:43PM -0700, Mark Millard wrote:

[loads of stuff]

Thank you very much for this Mark, for explaining it. I now have it
booting without an sdcard to usb3 zfs.

I did this:

# mkdir -p /usr/freebsd-dist
# cd /usr/freebsd-dist
# fetch
# http://ftp3.freebsd.org/pub/FreeBSD/releases/arm64/13.0-RELEASE/MANIFEST
# cd
# bsdinstall

The pi4 is one previous to *C0T processor. Then copied over
/boot/msdos/* to /dev/da0p1, powered down the pi, removed the mmcsd and
powered on.

--
J.

Carl Johnson

2021-05-06 16:01:59 UTC

Permalink

Post by tech-lists
Hi,
How can zfs-on-root boot-to-usb3 on rpi4 be accomplished?
I've tried bsdinstall from a mmcsd-booted rpi4 but there seems to be
problems with it that I can't work around. What's really needed is an
installer, but these aren't made for arm64.aarch64 rpi4 from what I can
see (I'm no expert though, it's entirely feasible i've missed
something).
Maybe one way of doing it would be to have a usb key (as ufs2) for the
system to boot on, then have /home /usr/obj and other larger dirs on the
usb3-zfs disk.

I am currently running a system that way (zfs-on-root and boot from
USB3), but I had problems doing it. I used bsdinstall, but it
complained that it couldn't find a MANIFEST file and I couldn't find a
way around that. It had already partitioned the disk and set up zfs, so
I just downloaded the base.txz and kernel.txz files from the FreeBSD
website and extracted them to the filesystem. I then copied over the
u-boot and rpi-firmware files from the mmcsd to the appropriate
directories on the SSD, and booted it up. I think the default on the
RPI4 is to boot from USB if there isn't a mmcsd card, or you can use the
RPi OS to reverse that if you want to.

Sorry if this doesn't sound as easy as it really should be. This was
only a few weeks ago, but I didn't keep detailed notes of the exact
steps.

--
Carl Johnson ***@peak.org