Re: rc.d/zpool runs before ada(4) attaches

On Tue, 2020-12-01 at 16:22 +0100, Ronald Klop wrote:
>  
> Van: Harry Schmalzbauer <freebsd_at_omnilan.de>
> Datum: dinsdag, 1 december 2020 12:51
> Aan: Ronald Klop <ronald-lists_at_klop.ws>, FreeBSD Current <
> freebsd-current_at_freebsd.org>
> Onderwerp: Re: rc.d/zpool runs before ada(4) attaches
> > 
> > Am 01.12.2020 um 10:33 schrieb Ronald Klop:
> > :
> > :
> > :
> > > > One machine fails importing zpool because the correponding
> > > > vdevs >> (ada0-ada2)
> > > > are not available at the time rc.d/zpool runs.
> > > > 
> > > > 
> > > > Adhoc  I'm not aware of any rc(8) vs. driver awareness.
> > > > Is there any?
> > > > 
> > > > Suggestions how to fix else than 'sleep 1'?
> > > > 
> > > > Thanks,
> > > > 
> > > > -harry
> > > > 
> > > > P.S.: ahci(4) is compiled into kernel, machine is a HPE U48
> > > > (Gen 10 >> plus MicroServer), zfsloader loads root_MFS kernel
> > > > module
> > > > 
> > > 
> > > 
> > > There have been some changes to etc/rc.d/zpool in September.
> > > Do you have the latest version? Compare with:
> > > 
https://github.com/freebsd/freebsd/blob/master/libexec/rc/rc.d/zpool
> > > or
> > > 
https://svnweb.freebsd.org/base/head/libexec/rc/rc.d/zpool?revision=365354&view=markup
> > >  >
> > > 
> > > Otherwise it would be helpful for readers if you could post some
> > > logs > which indicate what is happening.
> > > /var/run/dmesg.boot or the output of "dmesg"
> > > Part of /var/log/messages
> > > Part of /var/log/console.log if it exists.
> > > 
> > 
> > Thanks, I'm on -current from view days ago.
> > The problem is that cam(4) is still probing devices, when
> > rc.d/zpool runs, since mount_root_from succeeded, because it is a
> > RAM disk, so succeeds independent of any real drive/controller
> > probing.
> > I can imagine of other seldom edgecases hitting the issue too.
> > 
> > So my proposed patch, working for me, looks like this:
> > Index: libexec/rc/rc.d/zpool
> > ===================================================================
> > --- libexec/rc/rc.d/zpool       (revision 368202)
> > +++ libexec/rc/rc.d/zpool       (working copy)
> > _at__at_ -18,8 +18,16 _at__at_
> > 
> >   zpool_start()
> >   {
> > -       local cachefile
> > +        local cachefile n=0 camlist=`camcontrol devlist -v`
> > 
> > +       # Wait for cam(4) devices attaching, 4 times at max by
> > increasing
> > +       # 1s each (10s max in total)
> > +        while [ X"${camlist#*target*lun*probe}" != X"${camlist}"
> > ]; do
> > +               [ $n -lt 4 ] || break
> > +               sleep $((n+=1))
> > +               camlist=`camcontrol devlist -v`
> > +       done
> > +
> >          for cachefile in /etc/zfs/zpool.cache
> > /boot/zfs/zpool.cache; do
> >                  if [ -r $cachefile ]; then
> >                          zpool import -c $cachefile -a -N && break
> > 
> > best,
> > -harry
> > 
> > 
> > 
> 
> You can define these in /boot/loader.conf:
> #kern.cam.boot_delay="10000" # Delay (in ms) of root mount for CAM
> bus
> #kern.cam.scsi_delay="2000" # Delay (in ms) before probing SCSI
> 
> Maybe that helps.
> 
> Ronald.
> 

Those settings control waiting before mounting root.  Harry's problem
is that root is mounted quickly, before other drives are ready for zfs.
 
The zpool script waits for 'disks'.  It would be nice if the cam
subsystem had something like a sysctl it set to indicate when initial
probing for disks was done, then there could be an rc.d/camprobe script
with 'PROVIDE: disks' which waits for the probing to complete.

-- Ian