Re: ZFS: amd64, devd, root file system.

Pawel Jakub Dawidek wrote:
> On Mon, Apr 09, 2007 at 03:17:23AM +0200, Pawel Jakub Dawidek wrote:

Hi Pawel,

great to see ZFS support committed to -current!

It is amazingly simple to get a test setup going and it worked fine
in my initial simple test cases. But now I've run into problems that
probably are not technical but caused by a lack of understanding ...

>> 3. It is now possible to have root file system on ZFS. You would still
>>    need UFS for your /boot/ file system.

Hmmm, I yesterday bought a new disk for playing with ZFS and while
it worked fine in most of my tests, I could not get the kernel to
mount the ZFS based root partition. Maybe there is something wrong
in the way I tried it (I did not strictly follow the method you
suggested, because of special requirements).

> Let me explain how this suppose to work.
> 
> You have ad0 disk. Create one slice covering entire disk:
> 
> 	# fdisk -BI /dev/ad0

In my case I have a ad0s1a (boot file system) and ad0s1b (swap for
core dumps) and planned to have ad0s2 for ZFS. DOS and BSD partitions
were created from, within sysinstall (and I have verified with CLI
tools that the result looks OK).

> Initialize BSDlabel:
> 
> 	# bsdlabel -wB /dev/ad0s1
> 
> Edit your label and create small (like 256MB-512MB) 'a' partition and
> use the rest for 'd' partition:
> 
> 	# bsdlabel -e /dev/ad0s1
> 
> 'd' partition will be used for ZFS:

In my case 1GB for ad0s1a (256MB) and ad0s1b (768MB) and 300GB for
ad0s2.

> 	# zpool create tank ad0s1d

zpool create test ad0s2

> Create UFS file system on /dev/ad0s1a and copy /boot/ directory in
> there:
> 
> 	# newfs /dev/ad0s1a
> 	# mount /dev/ad0s1a /mnt/tmp
> 	# cp -Rp /boot/* /mnt/tmp/

I have copied /boot and /rescue to ad0s1a (and later, to simplify
further testing, also /etc, /bin, /sbin, /lib, /libexec, since I
wanted to be able to run recovery tools from ad0s1a).

> Note that there is no /boot/ directory on ad0s1a yet. This is one of the
> two possibilities. You now need to create symlink:
> 
> 	# cd /mnt/tmp
> 	# ln -s . boot
> 
> From what I checked our loader should handle symlinks just fine.
> This will allow us to mount /dev/ad0s1a on /boot directory and use it as
> usual.
> 
> Another option is to:
> 
> 	# cp -Rp /boot /mnt/tmp/
>
> and in the future mount /dev/ad0s1a on eg. /bootdisk and create symlink:
> 
> 	# ln -s bootdisk/boot /boot

I decided to habe a /.ufs file system and mount ad0s1a there, with
symlinks from /boot and /rescue to /.ufs/boot and /.ufs/rescue ...

> All in all, you should see your kernel when you do:
> 
> 	# ls -l /mnt/tmp/boot/kernel
> 
> Now don't forget to add zfs_load="YES" to /mnt/tmp/boot/loader.conf.
> 
> Ok, you also need to tell your loader where your root file system is.
> You can do it by adding:
> 
> 	vfs.root.mountfrom="zfs:tank"
> 
> to /mnt/tmp/boot/loader.conf or you can create /mnt/tmp/etc/fstab file
> with one entry only:
> 
> 	tank	/	zfs	rw	0	0

Neither of these make the kernel accept the ZFS root file system in
my case. (I currently have both, but even when I enter zfs:test at
the prompt displayed by the kernel after failure to mount a root fs,
it is not accepted).

I do then enter ufs:ad0s1a to boot my UFS file system for recovery
(with /bin, /sbin, /lib, and /etc besides /boot and /rescue). Then
I have access to "zfs" and "zpool" commands, which let me mount the
ZFS file system, even overloading the UFS root. I only have to make
sure, that I manually mount DEVFS into the ZFS root.

But in most of my tests, I need to "export" and "import -f" the
ZFS pool, since it is displayed as "FAULTED" at first. This is true
even if I manage to mount the UFS partition in such a way, that
/boot/zfs/zpool.cache is correctly updated. I assume that file is
used to control the automatic ZFS root mounting?

Do I need to set the "alternate root" flag on the pool during import
to have it accepted as a root FS during boot??? (This will be my next
test, I think ...)

> On your ZFS file system, your /etc/fstab should contains the line above
> and:
> 
> 	/dev/ad0s1a	/boot	ufs	rw	0	0

I use:

/dev/ad0s1a /.ufs ufs rw 1 2

> (and everything else, ie. your swap and other file systems)

Hmmm, there are a few points that I do not fully understand:

It seems that ZFS "legacy" mounts are not supported under FreeBSD,
is this correct? (E.g. if I enter "zfs set mountpoint=legacy test"
then "test" can not be mounted with "zfs mount test" and there is
no other way to mount it since we do not have a "mount_zfs", yet?)

I tried to set the mountpoint of my to-be root file system to "/"
with "zfs set mountpoint=/ test" but I'm under the impression that
this does not really work. Setting it to "//" does appear to have
the desired effect, though, but may lead to a panic during shutdown.
(Sorry, I've got no core-dumps but could try producing one later
if there is interest. The panic is because of a ref count becoming
negative but I did not write down the message.)

I decided to have multiple zfs file systems (test/var, test/usr ...)
and can see them with zfs list. What is the correct way to get them
mounted automatically? (Assuming I get the problem to have the kernel
automatically mount the ZFS root solved ...)

Do I need fstab entries for for ZFS file systems (e.g. "test/usr")
or does ZFS mount them automatically when the pool "test" is mounted?

Or do I need a fstab line for each of them?
What's supposed to go into /etc/zfs, besides the ZFS exports file?

Regards, STefan