Re: Resizing a zpool as a VMware ESXi guest ...

On 11/28/2015 6:10 PM, Matthew Grooms wrote:
> On 11/27/2015 7:44 PM, Matthew Grooms wrote:
>> I spent the day looking over the FreeBSD cam and scsi_da source code. 
>> After sprinkling a bunch of printf's around to see what code paths 
>> were being called, It's obvious that Edward was correct in assuming 
>> that ESXi doesn't return any 'Unit Attention' sense information in 
>> response to a 'Read Capacity' request. This kinda makes sense as ESXi 
>> emulates SCSI-2 disk devices and, as far as I can tell, the 0x2A/0x09 
>> ASC/ASCQ sense code that denotes 'Capacity Data Has Changed' wasn't 
>> defined until the SCSCI-3 spec. It's frustrating that the only way to 
>> get the scsci_da code to call reprobe() is by receiving a command 
>> from the device. Would something like this work? ...
>>
>> 1) Register a callback using xpt_register_async( daasync, 
>> AC_REPROBE_DEVICE, path ) that calls reprobe()
>> 2) Implement a new IOCTL in cam_xpt that camcontrol can call with the 
>> bus:target:lun as the argument
>> 3) have cam_xpt capture the IOCTL request and call xpt_async( 
>> AC_REPROBE_DEVICE, path ) as a result
>>
>> This way users would have the option of manually asking cam to 
>> communicate the new size to geom. The only option now is one or more 
>> reboots to gain access to the increased disk capacity. If this sounds 
>> like a reasonable approach, I'll take a stab at implementing it.
>>
>
> Here is a proof of concept patch. I'm a complete noob when it comes to 
> cam, scsi or freebsd kernel development for that matter, so I'm sure 
> it could have been done a better way. In any case, I added a new 
> command to camcontrol that allows you to specify a bus, target and lun 
> as an argument. For example ...
>
> # camcontrol readcap da1 -h
> Device Size: 32 G, Block Length: 512 bytes
>
> # gpart show da1
> =>      40  58720176  da1  GPT  (28G)
>         40  58720176    1  freebsd-ufs  (28G)
>
> Note, I resized the VMDK disk in ESXi. The camcontrol output shows the 
> size as 32G but geom thinks its 28G.
>
> # camcontrol devlist
> <NECVMWar VMware IDE CDR10 1.00>   at scbus1 target 0 lun 0 (cd0,pass0)
> <VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (pass1,da0)
> <VMware Virtual disk 1.0>          at scbus2 target 1 lun 0 (pass2,da1)
> <FREEBSD CTLDISK 0001>             at scbus3 target 0 lun 0 (da2,pass3)
>
> # camcontrol reprobe 2:1:0
>
> This generates an event that is captured by the scsci da device to 
> forces a reprobe. The kernel output looks almost identical to when the 
> 'Unit Attention' sense data is received ...
>
> Nov 28 17:46:13 iscsi-i kernel: (da1:mpt0:0:1:0): Re-probe requested
> Nov 28 17:46:13 iscsi-i kernel: GEOM_PART: da1 was automatically resized.
> Nov 28 17:46:13 iscsi-i kernel: Use `gpart commit da1` to save changes 
> or `gpart undo da1` to revert them.
>
> Now that geom knows about the increased disk capacity, I can increase 
> the partition size and grow the fs ...
>
> [root_at_iscsi-i /home/mgrooms]# gpart show da1
> =>      40  67108784  da1  GPT  (32G)
>         40  58720176    1  freebsd-ufs  (28G)
>   58720216   8388608       - free -  (4.0G)
>
> # gpart resize -i 1 da1
> da1p1 resized
>
> # growfs da1p1
> Device is mounted read-write; resizing will result in temporary write 
> suspension for /var/data1.
> It's strongly recommended to make a backup before growing the file 
> system.
> OK to grow filesystem on /dev/da1p1, mounted on /var/data1, from 28GB 
> to 32GB? [Yes/No] Yes
> super-block backups (for fsck_ffs -b #) at:
>  58983232, 60265472, 61547712, 62829952, 64112192, 65394432, 66676672
>
> # df -h
> Filesystem    Size    Used   Avail Capacity  Mounted on
> /dev/da0p3     18G    5.3G     12G    31%    /
> devfs         1.0K    1.0K      0B   100%    /dev
> /dev/da1p1     31G     32M     28G     0%    /var/data1
> /dev/da2p1     15G     32M     14G     0%    /var/data2
>
> Sure would be nice to have something like this in the tree. It's 
> really a drag to have to reboot production VMs to increase disk 
> capacity when it could be easily avoided. I'm not sure what the 
> correct IOCTL should look like. Maybe CAMIOCOMMAND is a better way to 
> go? If someone with some experience with the cam/scsi subsystems was 
> willing to give me some direction I'd be willing to try and rewrite 
> the patch in a way that would be commit worthy. I just need some 
> direction.
>

Ok, last post until I get some feedback. Here's a new version of the 
patch complete with man page updates. It communicates via CAMIOCOMMAND 
instead of introducing a new ioctl value. I tried to model it after the 
device reset option, hopefully with some degree of success. Functionally 
it should be the same as the first patch.

Thanks,

-Matthew