Re: Resizing a zpool as a VMware ESXi guest ...

On 11/27/2015 12:56 PM, Matthew Grooms wrote:
> I thought it would be useful to get more output from the geom layer, 
> similar to the camcontrol debug output ...
>
> [root_at_iscsi-i /home/mgrooms]# sysctl kern.geom.debugflags=0x81
>
> When I resize the iSCSI LUN and run the 'camcontrol readcap da2 -h', I 
> see this in the log output ...
>
> [root_at_iscsi-i /home/mgrooms]# tail -f /var/log/messages
> ...
> Nov 27 12:20:07 iscsi-i kernel: (pass3:iscsi1:0:0:0): Capacity data 
> has changed
> Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff80973850, 
> 0xfffff80003f4e000, 1, 0)
> Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff8097a6b0, 
> 0xfffff80003f42b60, 2, 0)
> Nov 27 12:20:07 iscsi-i kernel: 
> g_resize_provider_event(0xfffff800038c6700)
> Nov 27 12:20:07 iscsi-i kernel: g_part_resize(da2)
> Nov 27 12:20:07 iscsi-i kernel: GEOM_PART: da2 was automatically resized.
> Nov 27 12:20:07 iscsi-i kernel: Use `gpart commit da2` to save changes 
> or `gpart undo da2` to revert them.
> Nov 27 12:20:07 iscsi-i kernel: g_raid_taste(RAID, da2)
> Nov 27 12:20:07 iscsi-i kernel: g_attach(0xfffff80003e64780, 
> 0xfffff800038c6700)
> Nov 27 12:20:07 iscsi-i kernel: g_detach(0xfffff80003e64780)
> Nov 27 12:20:07 iscsi-i kernel: g_destroy_consumer(0xfffff80003e64780)
> Nov 27 12:20:07 iscsi-i kernel: 
> g_destroy_geom(0xfffff800038c9c00(raid:taste))
> Nov 27 12:20:07 iscsi-i kernel: g_label_taste(LABEL, da2)
>
> However, when I resize the VMDK disk and run the 'camcontrol readcap 
> da1 -h' command, I see nothing in the log output. So it would appear 
> that even though cam is reporting the new capacity in the command line 
> output, the this info is not being forwarded to geom in this case. 
> Maybe the VMware virtual SAS device ( mpt0 ) isn't reporting some 
> special capability bit to cam which causes it to squelch the info?
>
> I'm not sure if this is useful but here is what the device info looks 
> like at boot time ...
>
> mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 
> 0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3
> mpt0: MPI Version=1.5.0.0
> ...
> da0 at mpt0 bus 0 scbus2 target 0 lun 0
> da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
> da0: 300.000MB/s transfers
> da0: Command Queueing enabled
> da0: 20480MB (41943040 512 byte sectors)
> da0: quirks=0x40<RETRY_BUSY>
> da1 at mpt0 bus 0 scbus2 target 1 lun 0
> da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
> da1: 300.000MB/s transfers
> da1: Command Queueing enabled
> da1: 20480MB (41943040 512 byte sectors)
> da1: quirks=0x40<RETRY_BUSY>
> ...
> da2 at iscsi1 bus 0 scbus3 target 0 lun 0
> da2: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
> da2: Serial Number MYSERIAL   0
> da2: 150.000MB/s transfers
> da2: Command Queueing enabled
> da2: 16384MB (33554432 512 byte sectors)
>

I spent the day looking over the FreeBSD cam and scsi_da source code. 
After sprinkling a bunch of printf's around to see what code paths were 
being called, It's obvious that Edward was correct in assuming that ESXi 
doesn't return any 'Unit Attention' sense information in response to a 
'Read Capacity' request. This kinda makes sense as ESXi emulates SCSI-2 
disk devices and, as far as I can tell, the 0x2A/0x09 ASC/ASCQ sense 
code that denotes 'Capacity Data Has Changed' wasn't defined until the 
SCSCI-3 spec. It's frustrating that the only way to get the scsci_da 
code to call reprobe() is by receiving a command from the device. Would 
something like this work? ...

1) Register a callback using xpt_register_async( daasync, 
AC_REPROBE_DEVICE, path ) that calls reprobe()
2) Implement a new IOCTL in cam_xpt that camcontrol can call with the 
bus:target:lun as the argument
3) have cam_xpt capture the IOCTL request and call xpt_async( 
AC_REPROBE_DEVICE, path ) as a result

This way users would have the option of manually asking cam to 
communicate the new size to geom. The only option now is one or more 
reboots to gain access to the increased disk capacity. If this sounds 
like a reasonable approach, I'll take a stab at implementing it.

Thanks,

-Matthew