[SOLVED]Re: How to create ZFS on Root using MBR slices?

I was able to re-solve this problem with creating a ZFS root using a
partition on a slice.  The hint came from a message sent by Doug
Rabson in response to the 'ZFS boot on zfs mirror' discussion on the
FreeBSD-Stable list.

On Tue, May 26, 2009 at 12:06 PM, Doug Rabson<dfr_at_rabson.org> wrote:
>
> On Tue, 26 May 2009 19:57:03 +0300, Andriy Gapon <avg_at_icyb.net.ua> wrote:
>> on 26/05/2009 19:42 George Hartzell said the following:
>>> I'm still confused about the two parts of zfsboot and what's magical
>>> about seeking to 1024.
>>
>> Can't help with answer to this, but cc-ing the one who can (I think).
>> I am interested too :-)
>
> This is due to the primitive DOS boot sequence. Basically the BIOS loads
> the first sector of the partition and executes it. For zfsboot, that is the
> first 512 bytes of /boot/zfsboot. The next stage of the bootstrap is tucked
> away in a convenient hole in the ZFS on-disk formwat which is located just
> after the ZFS metadata - this is the seek=1024 part. The first 512 byte
> part is a tiny assembler program that loads the rest into memory and
> executes it. The second part is large enough and smart enough to understand
> the ZFS filesystem format directly and it loads /boot/loader directly from
> the filesystem and transfers control to that. The third stage
> (/boot/loader) is what puts up the boot menu and loads the kernel etc.

Thanks Doug for a great explanation.

Below is the revised instructions for creating a ZFS Root w/MBR,
Slices, and Partitions

Scot

---------------------------------------------------------------------------------------------------------------------------

Creating ZFS Root ZFS Root w/MBR, Slices, and Partitions

1. Boot FreeBSD install CD/DVD

2. Choose Fixit option in sysinstall

3. Create Slice using gpart

Fixit# gpart show ad0
=>       63  625142385  ad0  MBR  (289G)
         63  209712447    1  !7  (100G)
  209712510     417690    2  !136  (204M)
  210130200  415012248       - free -  (198G)


Fixit# gpart add -b 210130200  -s 415012248 -t freebsd ad0
ad0s3 added

Fixit# gpart show ad0
=>       63  625142385  ad0  MBR  (289G)
         63  209712447    1  !7  (100G)
  209712510     417690    2  !136  (204M)
  210130200  415012248     3  freebsd  (198G)

4. Create partitions (ad0s3a and ad0s3b)

Fixit# gpart show ad0s3
=>        0  415012248  ad0s3  BSD  (198G)
          0  415012248  ad0s3  - free -  (198G)

Fixit# gpart add -i 1 -b 0 -s 40663421 -t freebsd-zfs ad0s3
ad0s3a added

Fixit# gpart show ad0s3
=>        0  415012248  ad0s3  BSD  (198G)
  406631421      1  freebsd-zfs  (194G)
  406631421    8380827         - free -  (4.0G)

Fixit# gpart add -i 2 -b 406626334 -s 8380811 -t freebsd=swap ad0s3
ad0s3b added

Fixit# gpart show ad0s3
=>        0  415012248  ad0s3  BSD  (198G)
  406631421      1  freebsd-zfs  (194G)
  406631421    8380827      2  freebsd-swap  (4.0G)

5. load zfs kernel module

kldload /mnt2/boot/kernel/opensolaris.ko
kldload /mnt2/boot/kernel/zfs.ko

6. Create Zpool

zpool create zroot /dev/ad0s3a
zpool set bootfs=zroot zroot

7. Create filesystem hierarchy

zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/tmp

zfs create							zroot/usr
zfs create -o compression=on			-o setuid=off	zroot/usr/ports
zfs create -o compression=off	-o exec=off	-o
setuid=off	zroot/usr/ports/distfiles
zfs create -o compression=off	-o exec=off	-o setuid=off	zroot/usr/ports/packages
zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/usr/src

zfs create							zroot/var
zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/var/crash
zfs create			-o exec=off	-o setuid=off	zroot/var/db
zfs create -o compression=on	-o exec=on	-o setuid=off	zroot/var/db/pkg
zfs create			-o exec=off	-o setuid=off	zroot/var/empty
zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/var/log
zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/var/mail
zfs create			-o exec=off	-o setuid=off	zroot/var/run
zfs create -o compression=on	-o exec=off	-o setuid=off	zroot/var/tmp

8. install FreeBSD to zroot

cd /dist/8.0-20090628-SNAP
export DESTDIR=/zroot
for dir in base catpages dict doc games info lib32 manpages ports; do
(cd $dir ; ./install.sh) ; done
cd src ; ./install.sh all
cd ./kernel ; ./install.sh dv8135nr ; ./install.sh generic
cd /zroot/boot ; cp -rp GENERIC/* ../kernel

9. create rc.conf, loader.conf, src.conf

echo 'zfs_enable="YES"' > /zroot/etc/rc.conf

echo 'LOADER_ZFS_SUPPORT=YES' >> /zroot/etc/src.conf

echo 'zfs_load="YES"' >> /zroot/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /zroot/boot/loader.conf

10. change root password

chroot /zroot
passwd
exit

11. create zpool.cache

mkdir /boot/zfs
zpool export zroot && zpool import zroot
cp /boot/zfs/zpool.cache /zroot/boot/zfs/zpool.cache

12. export LD_LIBRARY_PATH

export LD_LIBRARY_PATH=/mnt2/lib

13. Change mount points for zroot pool

zfs set mountpoint=legacy zroot
zfs set mountpoint=/tmp zroot/tmp
zfs set mountpoint=/usr zroot/usr
zfs set mountpoint=/var zroot/var

14. Install boot Manager

fdisk -B -b /boot/boot0 ad0

15. Install ZFS boot

- Install the boot1 stage on the drive:

dd if=/mnt2/boot/zfsboot of=/dev/ad0s3 count=1

Install the boot2 zfs stage into the convienient hole in the ZFS
filesystem on-disk format which is located just after the ZFS metadata
(this is the seek=1024).

dd if=/mnt2/boot/zfsboot of=/dev/ad0s3a skip=1 seek=1024