cam subsystem vs usb

Hi,

I use a 5-hdd bunch connected via USB3 in JBOD mode; that 5 hdd builds a 
zpool and are listed during boot process as da0 to da4.

Here is my /boot/loader.conf file:

.../...

hw.snd.default_device=2
dev.pcm.1.play.vchans=2
fuse_load="YES"
kern.cam.boot_delay=12000
kern.vty=vt
sysctl net.local.stream.recvspace=65536
cpu_microcode_load="YES"
cpu_microcode_name="/boot/firmware/intel-ucode.bin"
sysctl net.local.stream.recvspace=65536
sysctl net.local.stream.sendspace=65536

.../...

Under FBSD12.0, the boot process got fine.

Since recent upgrade to 12.1, the first "cold" boot shows:

.../...

Dec  7 10:17:25 H97M-G43 kernel: Trying to mount root from 
ufs:/dev/ada1p2 [rw]...
Dec  7 10:17:25 H97M-G43 kernel: da0 at umass-sim0 bus 0 scbus3 target 0 
lun 0
Dec  7 10:17:25 H97M-G43 kernel: da0: <External USB 3.0 5203> Fixed 
Direct Access SPC-4 SCSI device
Dec  7 10:17:25 H97M-G43 kernel: da0: Serial Number 2019083100443
Dec  7 10:17:25 H97M-G43 kernel: da0: 400.000MB/s transfers
Dec  7 10:17:25 H97M-G43 kernel: da0: 953869MB (1953525168 512 byte sectors)
Dec  7 10:17:25 H97M-G43 kernel: da0: quirks=0x2<NO_6_BYTE>
Dec  7 10:17:25 H97M-G43 kernel: da1 at umass-sim0 bus 0 scbus3 target 0 
lun 1
Dec  7 10:17:25 H97M-G43 kernel: da1: <External USB 3.0 5203> Fixed 
Direct Access SPC-4 SCSI device
Dec  7 10:17:25 H97M-G43 kernel: da1: Serial Number 2019083100443
Dec  7 10:17:25 H97M-G43 kernel: da1: 400.000MB/s transfers
Dec  7 10:17:25 H97M-G43 kernel: da1: 953869MB (1953525168 512 byte sectors)
Dec  7 10:17:25 H97M-G43 kernel: da1: quirks=0x2<NO_6_BYTE>
Dec  7 10:17:25 H97M-G43 kernel: da2 at umass-sim0 bus 0 scbus3 target 0 
lun 2
Dec  7 10:17:25 H97M-G43 kernel: da2: <External USB 3.0 5203> Fixed 
Direct Access SPC-4 SCSI device
Dec  7 10:17:25 H97M-G43 kernel: da2: Serial Number 2019083100443
Dec  7 10:17:25 H97M-G43 kernel: da2: 400.000MB/s transfers
Dec  7 10:17:25 H97M-G43 kernel: da2: 953869MB (1953525168 512 byte sectors)
Dec  7 10:17:25 H97M-G43 kernel: da2: quirks=0x2<NO_6_BYTE>
Dec  7 10:17:25 H97M-G43 kernel: da3 at umass-sim0 bus 0 scbus3 target 0 
lun 3
Dec  7 10:17:25 H97M-G43 kernel: da3: <External USB 3.0 5203> Fixed 
Direct Access SPC-4 SCSI device
Dec  7 10:17:25 H97M-G43 kernel: da3: Serial Number 2019083100443
Dec  7 10:17:25 H97M-G43 kernel: da3: 400.000MB/s transfers
Dec  7 10:17:25 H97M-G43 kernel: da3: 953869MB (1953525168 512 byte sectors)
Dec  7 10:17:25 H97M-G43 kernel: da3: quirks=0x2<NO_6_BYTE>
Dec  7 10:17:25 H97M-G43 kernel: da4 at umass-sim0 bus 0 scbus3 target 0 
lun 4
Dec  7 10:17:25 H97M-G43 kernel: da4: <External USB 3.0 5203> Fixed 
Direct Access SPC-4 SCSI device
Dec  7 10:17:25 H97M-G43 kernel: da4: Serial Number 2019083100443
Dec  7 10:17:25 H97M-G43 kernel: da4: 400.000MB/s transfers
Dec  7 10:17:25 H97M-G43 kernel: da4: 953869MB (1953525168 512 byte sectors)
Dec  7 10:17:25 H97M-G43 kernel: da4: quirks=0x2<NO_6_BYTE>
Dec  7 10:17:25 H97M-G43 kernel: (da0:umass-sim0:0:0:0): READ(10). CDB: 
28 00 74 70 6d 71 00 00 04 00
Dec  7 10:17:25 H97M-G43 kernel: (da0:umass-sim0:0:0:0): CAM status: 
SCSI Status Error
Dec  7 10:17:25 H97M-G43 kernel: (da0:umass-sim0:0:0:0): SCSI status: 
Check Condition
Dec  7 10:17:25 H97M-G43 kernel: (da0:umass-sim0:0:0:0): SCSI sense: NOT 
READY asc:4,1(Logical unit is in process of becoming ready)
Dec  7 10:17:25 H97M-G43 kernel: (da0:umass-sim0:0:0:0): Polling device 
for readiness
Dec  7 10:17:25 H97M-G43 kernel: ZFS filesystem version: 5Dec  7 
10:17:25 H97M-G43 kernel: ZFS storage pool version: features support (5000)

.../...

The readiness time takes about 10 seconds.

Curiously, further "warm" reboots are going fine, without  any readiness 
sequence.

Could this behaviour be related to the cam_system/usb_system problem 
leading to hang the boot process, recently reported?

Hope this be usefull for debugging the cam subsystem.

Roger Genre