Re: Booting off raidz *mostly* work

Ollivier Robert wrote:
> According to Ollivier Robert:
>> By popular demand:
>> http://www.keltia.net/howtos/zfsboot
> 
> I've succeeded in booting off raidz!
> 
> There is a remaining issue with /boot/loader getting zfs errors at boot time
> resulting in loader.conf not being read. After setting the various variables
> manually (vm.kmem_size, vfs.root.mountfrom) and loading zfs, it works...
> 
> http://static.keltia.net/download/raidz-boot.png
> 
> I've updated the howto to reflect that.
> 
> Any help in tackling the last issue is welcome.
> 

I don't have help for raid-z, but I can contribute some parts of my own 
procedure that have been helpful:

--------------------------------------------------------------------------

- If you would prefer to SSH into the livefs shell (have network access 
but hate typing out the commands manually, or you have to put up with a 
painful console terminal such as serial, not to mention that ctrl-c 
makes the livefs shell abort and try to go back to sysinstall):

ifconfig foo0 inet 10.0.0.2/24
route add default 10.0.0.1
echo "nameserver 10.0.0.1" >> /etc/resolv.conf
echo "domain example.com" >> /etc/resolv.conf
mkdir /etc/ssh
ssh-keygen -t rsa1 -b 1024 -f /etc/ssh/ssh_host_key -N ""
ssh-keygen -t dsa -f /etc/ssh/ssh_host_dsa_key -N ""
ssh-keygen -t rsa -f /etc/ssh/ssh_host_rsa_key -N ""
cp /dist/etc/ssh/sshd_config /tmp/
echo "PermitRootLogin yes" >> /tmp/sshd_config
env > /tmp/env
ln -s /mnt2/bin/csh /bin/csh
/mnt2/usr/sbin/sshd -Ddf /tmp/sshd_config

 From the network:

ssh root_at_10.0.0.2
sh
. /tmp/env
. /mnt2/.profile

(I used /tmp/env to get the PATH settings from the sysinstall shell 
propagated to the ssh shell, and I specifically used -Dd so you can only 
login once since root has no password!)

----------------------------------------------------------------------------

- Thinking ahead to when I need to reinstall a system in the future 
(happens many times during testing), I need to remember to wipe out
existing gmirror or partitions/partition tables:

gmirror clear /dev/da0p2 /dev/da1p2
gpart delete -i 3, 2, 1, gpart destroy da0 etc

I strongly suggest putting swap on a gmirror if it isn't going to be 
protected by another form of raid, even though it contains no valuable 
data, I don't want my system to go down because of a failure on a 
non-redundant swap volume.

Something like:
gmirror label -v -b prefer swap /dev/da0p2 /dev/da1p2 && \
echo '/dev/mirror/swap none swap sw 0 0' >> /rpool/ROOT/freebsd/etc/fstab

It would also be handy to enable gmirror status checks in periodic.conf 
although I used a cronned script so I get paged more than once a day if 
there is a problem.

---------------------------------------------------------------------

- You could use jot to duplicate various gpart commands to multiple 
drives, examples:

Clear 2 drives with up to 3 partitions each:
jot -w "gpart delete -i 3 da" -s \; 2 0 1 | sh
jot -w "gpart delete -i 2 da" -s \; 2 0 1 | sh
jot -w "gpart delete -i 1 da" -s \; 2 0 1 | sh
jot -w "gpart destroy da" -s \; 2 0 1 | sh

Clear 8 drives with up to 3 partitions each:
jot -w "gpart delete -i 3 da" -s \; 8 0 7 | sh
jot -w "gpart delete -i 2 da" -s \; 8 0 7 | sh
jot -w "gpart delete -i 1 da" -s \; 8 0 7 | sh
jot -w "gpart destroy da" -s \; 8 0 7 | sh

-----------------------------------------------------------------------

- In 8.0-BETA2 or above, you can use some new features in gpart for gpt 
auto sizing to automatically calculate the starting sector or even let 
your last partition use all remaining space: (helpful so you don't need 
to do as much math and less typing)

Create gpt, set partition active, create 3 partitions on 2 drives using 
all remaining space after the boot and swap:
jot -w "gpart create -s GPT da" -s \; 2 0 1 | sh
jot -w "echo 'a 1' | fdisk -f - -q /dev/da" -s \; 2 0 1 | sh
jot -w "gpart add -s 128 -t freebsd-boot da" -s \; 2 0 1 | sh
jot -w "gpart add -s 4194304 -t freebsd-swap da" -s \; 2 0 1 | sh
jot -w "gpart add -t freebsd-zfs da" -s \; 2 0 1 | sh

Similar for installing boot code for 2 drives:
jot -w "gpart bootcode -b /rpool/ROOT/freebsd/boot/pmbr -p \
/rpool/ROOT/freebsd/boot/gptzfsboot -i 1 da" -s \; 2 0 1 | sh

------------------------------------------------------------------------

- Instead of extracting dists from the install media, I chose to build 
tar files of a system image using procedures gleaned from the jail 
manpage (its pretty easy), that way I can add loader support and plenty 
of customizations to my image, use an up to date build, even include the 
source and ports trees for convenience into my tar, and the tar tends to 
extract in just a couple minutes on the destination system (70-180 
seconds depending on the system and what is in the tar).  This way I can 
also use the 8.0 livefs to install either 7.x or 8.x just based on which 
tar I download.  Obviously this has more benefit if you have more than a 
small number of systems to install; if its just one or two then 
pre-customizing isn't worth it.  I stopped including the portsnap files 
dir in favor of a post-install fresh 'portsnap fetch' though, because 
after just a couple weeks, it seemed like my 'portsnap fetch' and 
'update' took longer than getting a fresh copy.  Since I track -stable 
on my servers, its nice to keep my tar updated every month or few weeks 
and then my new installs don't each require a source update since they 
start life being fairly up to date already.  Also a very nice result is 
by adding NO_FOO options to the image build, the resulting image never 
has parts of the base that you never wanted in the first place, for 
example if you leave out sendmail or lpr, so you don't have to worry 
about 'make check-old' being complete enough.