Re: Resizing a zpool as a VMware ESXi guest ...

From: Matthew Grooms <mgrooms_at_shrew.net>
Date: Fri, 27 Nov 2015 12:56:36 -0600
On 11/27/2015 12:08 PM, Matthew Grooms wrote:
> On 11/27/2015 3:16 AM, Willem Jan Withagen wrote:
>> On 27-11-2015 06:59, Matthew Grooms wrote:
>>> All,
>>>
>>> I know this is a very late follow up, but spent some more time looking
>>> at this today and found some additional information that I found quite
>>> interesting. I setup two VMs, one that acts as an iSCSI initiator (
>>> CURRENT ) and another that acts as a target ( 10.2-RELEASE ). Both are
>>> running under ESXi v5.5. There are two block devices on the initiator,
>>> da1 and da2, that I used for resize testing ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# camcontrol devlist
>>> <NECVMWar VMware IDE CDR10 1.00>   at scbus1 target 0 lun 0 (cd0,pass0)
>>> <VMware Virtual disk 1.0>          at scbus2 target 0 lun 0 (pass1,da0)
>>> <VMware Virtual disk 1.0>          at scbus2 target 1 lun 0 (pass2,da1)
>>> <FREEBSD CTLDISK 0001>             at scbus3 target 0 lun 0 (da2,pass3)
>>>
>>> The da1 device is a virtual disk hanging off of a VMware virtual SAS
>>> controller ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# pciconf
>>> ...
>>> mpt0_at_pci0:3:0:0:        class=0x010700 card=0x197615ad chip=0x00541000
>>> rev=0x01 hdr=0x00
>>>      vendor     = 'LSI Logic / Symbios Logic'
>>>      device     = 'SAS1068 PCI-X Fusion-MPT SAS'
>>>      class      = mass storage
>>>      subclass   = SAS
>>>
>>> [root_at_iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
>>> Device Size: 10 G, Block Length: 512 bytes
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart show da1
>>> =>      40  20971440  da1  GPT  (10G)
>>>          40  20971440    1  freebsd-ufs  (10G)
>>>
>>> The da2 device is an iSCSI LUN mounted from my FreeBSD 10.2 VM running
>>> ctld ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# iscsictl
>>> Target name                          Target portal    State
>>> iqn.2015-01.lab.shrew:target0        iscsi-t.shrew.lab Connected: da2
>>>
>>> [root_at_iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
>>> Device Size: 10 G, Block Length: 512 bytes
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart show da2
>>> =>      40  20971440  da2  GPT  (10G)
>>>          40        24       - free -  (12K)
>>>          64  20971392    1  freebsd-ufs  (10G)
>>>    20971456        24       - free -  (12K)
>>>
>>> When I increased the size of da1 ( the VMDK ) and then re-ran
>>> 'camcontrol readcap' without a reboot, it clearly showed that the disk
>>> size had increased. However, geom failed to recognize the additional
>>> capacity ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# camcontrol readcap da1 -h
>>> Device Size: 16 G, Block Length: 512 bytes
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart show da1
>>> =>      40  20971440  da1  GPT  (10G)
>>>          40  20971440    1  freebsd-ufs  (10G)
>>>
>>> Here is the interesting bit. I increased the size of da2 by modifying
>>> the lun size in ctld.conf on the target and then issued a 
>>> /etc/rd.d/ctld
>>> reload. When I re-ran 'camcontrol readcap' on the initiator without a
>>> reboot, it also showed that the disk size had increased, but this time
>>> geom recognized the additional capacity as well ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# camcontrol readcap da2 -h
>>> Device Size: 16 G, Block Length: 512 bytes
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart show da2
>>> =>      40  33554352  da2  GPT  (16G)
>>>          40        24       - free -  (12K)
>>>          64  20971392    1  freebsd-ufs  (10G)
>>>    20971456  12582936       - free -  (6.0G)
>>>
>>> I was then able to resize the partition and then grow the UFS
>>> filesystem, all without rebooting the VM ...
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart resize -i 1 da2
>>> da2p1 resized
>>>
>>> [root_at_iscsi-i /home/mgrooms]# gpart show da2
>>> =>      40  33554352  da2  GPT  (16G)
>>>          40        24       - free -  (12K)
>>>          64  33554304    1  freebsd-ufs  (16G)
>>>    33554368        24       - free -  (12K)
>>>
>>> [root_at_iscsi-i /home/mgrooms]# growfs da2p1
>>> Device is mounted read-write; resizing will result in temporary write
>>> suspension for /var/data2.
>>> It's strongly recommended to make a backup before growing the file 
>>> system.
>>> OK to grow filesystem on /dev/da2p1, mounted on /var/data2, from 
>>> 10GB to
>>> 16GB? [Yes/No] Yes
>>> super-block backups (for fsck_ffs -b #) at:
>>>   21798272, 23080512, 24362752, 25644992, 26927232, 28209472, 29491712,
>>> 30773952, 32056192, 33338432
>>>
>>> [root_at_iscsi-i /home/mgrooms]# df -h
>>> Filesystem    Size    Used   Avail Capacity  Mounted on
>>> /dev/da0p3     15G    1.2G     12G     9%    /
>>> devfs         1.0K    1.0K      0B   100%    /dev
>>> /dev/da1p1    9.7G     32M    8.9G     0%    /var/data1
>>> /dev/da2p1     15G     32M     14G     0%    /var/data2
>>>
>>> It's also worth noting that the additional space was not recognized by
>>> gpart/geom on the initiator until after the 'camcontrol readcap da2'
>>> command was run. In other words, I'm skeptical that it was a Unit
>>> Attention notification that made the right thing happen since it still
>>> took manual prodding of cam to get the new disk geometry up into the
>>> geom layer.
>>
>> I remember doing this for a bhyve VM, and had the type same problem.
>> Getting gpart in the VM to actually pickup the new size required some
>> extra prodding (I like that word) or rebooting the VM.
>> I can remember reporting this:
>>     tpoic: "resampeling of a ZVOL that has been resized"
>> and getting a fix from Andrey V. Elsukov...
>>
>> Index: head/sys/geom/part/g_part_gpt.c
>> ===================================================================
>> --- head/sys/geom/part/g_part_gpt.c    (revision 282044)
>> +++ head/sys/geom/part/g_part_gpt.c    (working copy)
>> _at__at_ -760,7 +760,7 _at__at_ g_part_gpt_resize(struct g_part_table *basetable,
>>      struct g_part_gpt_entry *entry;
>>
>>      if (baseentry == NULL)
>> -        return (EOPNOTSUPP);
>> +        return (g_part_gpt_recover(basetable));
>>
>>      entry = (struct g_part_gpt_entry *)baseentry;
>>      baseentry->gpe_end = baseentry->gpe_start + gpp->gpp_size - 1;
>>
>> Which went into the tree, but perhaps only in HEAD.
>> And that helped me getting the correct retasting of the GPART 
>> partitions.
>>
>> Not sure if this snippet would help you to get around GEOM tasting the
>> new size.
>>
>
> Thanks for the response. I read the commit logs daily but I don't 
> recall that one. However, it looks like that change went in over five 
> months ago ( r284151 ) and I'm running these tests on a CURRENT ...
>
> [mgrooms_at_iscsi-i ~]$ uname -a
> FreeBSD iscsi-i.shrew.lab 11.0-CURRENT FreeBSD 11.0-CURRENT #0 
> r291085: Thu Nov 19 21:48:13 UTC 2015 
> root_at_releng2.nyi.freebsd.org:/usr/obj/usr/src/sys/GENERIC  amd64
>
> Something else is amiss here and I don't have the chops to fix it 
> myself. I tried compiling a custom kernel with CAMDEBUG and twiddling 
> with the camcontrol debug options to compare the output during the 
> VMDK vs iSCSI disk resize process. Nothing was obvious to my untrained 
> eye. I'd be more than happy to test patches or provide additional 
> information to anyone willing to help.
>

I thought it would be useful to get more output from the geom layer, 
similar to the camcontrol debug output ...

[root_at_iscsi-i /home/mgrooms]# sysctl kern.geom.debugflags=0x81

When I resize the iSCSI LUN and run the 'camcontrol readcap da2 -h', I 
see this in the log output ...

[root_at_iscsi-i /home/mgrooms]# tail -f /var/log/messages
...
Nov 27 12:20:07 iscsi-i kernel: (pass3:iscsi1:0:0:0): Capacity data has 
changed
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff80973850, 
0xfffff80003f4e000, 1, 0)
Nov 27 12:20:07 iscsi-i kernel: g_post_event_x(0xffffffff8097a6b0, 
0xfffff80003f42b60, 2, 0)
Nov 27 12:20:07 iscsi-i kernel: g_resize_provider_event(0xfffff800038c6700)
Nov 27 12:20:07 iscsi-i kernel: g_part_resize(da2)
Nov 27 12:20:07 iscsi-i kernel: GEOM_PART: da2 was automatically resized.
Nov 27 12:20:07 iscsi-i kernel: Use `gpart commit da2` to save changes 
or `gpart undo da2` to revert them.
Nov 27 12:20:07 iscsi-i kernel: g_raid_taste(RAID, da2)
Nov 27 12:20:07 iscsi-i kernel: g_attach(0xfffff80003e64780, 
0xfffff800038c6700)
Nov 27 12:20:07 iscsi-i kernel: g_detach(0xfffff80003e64780)
Nov 27 12:20:07 iscsi-i kernel: g_destroy_consumer(0xfffff80003e64780)
Nov 27 12:20:07 iscsi-i kernel: 
g_destroy_geom(0xfffff800038c9c00(raid:taste))
Nov 27 12:20:07 iscsi-i kernel: g_label_taste(LABEL, da2)

However, when I resize the VMDK disk and run the 'camcontrol readcap da1 
-h' command, I see nothing in the log output. So it would appear that 
even though cam is reporting the new capacity in the command line 
output, the this info is not being forwarded to geom in this case. Maybe 
the VMware virtual SAS device ( mpt0 ) isn't reporting some special 
capability bit to cam which causes it to squelch the info?

I'm not sure if this is useful but here is what the device info looks 
like at boot time ...

mpt0: <LSILogic SAS/SATA Adapter> port 0x4000-0x40ff mem 
0xfd4ec000-0xfd4effff,0xfd4f0000-0xfd4fffff irq 18 at device 0.0 on pci3
mpt0: MPI Version=1.5.0.0
...
da0 at mpt0 bus 0 scbus2 target 0 lun 0
da0: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da0: 300.000MB/s transfers
da0: Command Queueing enabled
da0: 20480MB (41943040 512 byte sectors)
da0: quirks=0x40<RETRY_BUSY>
da1 at mpt0 bus 0 scbus2 target 1 lun 0
da1: <VMware Virtual disk 1.0> Fixed Direct Access SCSI-2 device
da1: 300.000MB/s transfers
da1: Command Queueing enabled
da1: 20480MB (41943040 512 byte sectors)
da1: quirks=0x40<RETRY_BUSY>
...
da2 at iscsi1 bus 0 scbus3 target 0 lun 0
da2: <FREEBSD CTLDISK 0001> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number MYSERIAL   0
da2: 150.000MB/s transfers
da2: Command Queueing enabled
da2: 16384MB (33554432 512 byte sectors)

Any thoughts?

Thanks,

-Matthew
Received on Fri Nov 27 2015 - 17:56:36 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:01 UTC