Re: [RFC][CFT] GEOM direct dispatch and fine-grained CAM locking

On 02.10.2013 20:30, John Baldwin wrote:
> On Saturday, September 07, 2013 2:32:45 am Alexander Motin wrote:
>> On 07.09.2013 02:02, Jeremie Le Hen wrote:
>>> On Fri, Sep 06, 2013 at 11:29:11AM +0300, Alexander Motin wrote:
>>>> On 06.09.2013 11:06, Jeremie Le Hen wrote:
>>>>> On Fri, Sep 06, 2013 at 12:46:27AM +0200, Olivier Cochard-Labbé wrote:
>>>>>> On Thu, Sep 5, 2013 at 11:38 PM, Alexander Motin <mav_at_freebsd.org>
> wrote:
>>>>>>> I've found and fixed possible double request completion, that could
> cause
>>>>>>> such symptoms if happened. Updated patch located as usual:
>>>>>>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130905.patch
>>>>>>>
>>>>> With this new one I cannot boot any more (I also updated the source
>>>>> tree).  This is a hand transcripted version:
>>>>>
>>>>> Trying to mount root from zfs:zroot/root []...
>>>>> panic: Batch flag already set
>>>>> cpuid = 1
>>>>> KDB: stack backtrace:
>>>>> db_trace_self_wrapper()
>>>>> kdb_backtrace()
>>>>> vpanic()
>>>>> kassert_panic()
>>>>> xpt_batch_start()
>>>>> ata_interrupt()
>>>>> softclock_call_cc()
>>>>> softclock()
>>>>> ithread_loop()
>>>>> fork_exit()
>>>>> fork_trampoline()
>>>>
>>>> Thank you for the report. I see my fault. It is probably specific to
>>>> ata(4) driver only. I've workarounded that in new patch version, but
>>>> probably that area needs some rethinking.
>>>>
>>>> http://people.freebsd.org/~mav/camlock_patches/camlock_20130906.patch
>>>
>>> I'm not sure you needed a confirmation, but it boots.  Thanks :).
>>>
>>> I didn't quite understand the thread; is direct dispatch enabled for
>>> amd64?  ISTR you said only i386 but someone else posted the macro for
>>> amd64.
>>
>> Yes, it is enabled for amd64. I've said x86, meaning both i386 and amd64.
>
> FYI, I tested mfi with this patch set and mfid worked fine for handling g_up
> directly:
>
> Index: dev/mfi/mfi_disk.c
> ===================================================================
> --- dev/mfi/mfi_disk.c  (revision 257407)
> +++ dev/mfi/mfi_disk.c  (working copy)
> _at__at_ -162,6 +162,7 _at__at_
>          sc->ld_disk->d_unit = sc->ld_unit;
>          sc->ld_disk->d_sectorsize = secsize;
>          sc->ld_disk->d_mediasize = sectors * secsize;
> +       sc->ld_disk->d_flags = DISKFLAG_DIRECT_COMPLETION;
>          if (sc->ld_disk->d_mediasize >= (1 * 1024 * 1024)) {
>                  sc->ld_disk->d_fwheads = 255;
>                  sc->ld_disk->d_fwsectors = 63;
>

Thank you for the feedback. But looking on mfi driver sources I would 
say that it calls biodone() from mfi_disk_complete() from cm_complete() 
method, which is called while holding mfi_io_lock mutex. I guess that if 
on top of mfi device would be some GEOM class, supporting direct 
dispatch and sending new requests down on previous request completion 
(or retrying requests), that could cause recursive mfi_io_lock 
acquisition. That is exactly the cause why I've added this flag. May be 
it is a bit paranoid, but it is better to be safe then sorry.

Another good reason to drop the lock before calling biodone() would be 
reducing the lock hold time. Otherwise it may just increase lock 
congestion there and destroy all benefits of the direct dispatch.

-- 
Alexander Motin