Re: Increasing MAXPHYS

On Mon, Mar 22, 2010 at 2:45 PM, M. Warner Losh <imp_at_bsdimp.com> wrote:
> In message: <D9D66012-16FD-4FB6-AB6A-9A8D17727901_at_samsco.org>
>            Scott Long <scottl_at_samsco.org> writes:
> : I'd like to go in the opposite direction.  The queue-dispatch-queue
> : model of GEOM is elegant and easy to extend, but very wasteful for
> : the simple case, where the simple case is one or two simple
> : partition transforms (mbr, bsdlabel) and/or a simple stripe/mirror
> : transform.  None of these need a dedicated dispatch context in order
> : to operate.  What I'd like to explore is compiling the GEOM stack at
> : creation time into a linear array of operations that happen without
> : a g_down/g_up context switch.  As providers and consumers taste each
> : other and build a stack, that stack gets compiled into a graph, and
> : that graph gets executed directly from the calling context, both
> : from the dev_strategy() side on the top and the bio_done() on the
> : bottom.  GEOM classes that need a detached context can mark
> : themselves as such, doing so will prevent a graph from being
> : created, and the current dispatch model will be retained.
>
> I have a few things to say on this.
>
> First, I've done similar things at past companies for systems that are
> similar to geom's queueing environment.  It is possible to convert the
> queueing nodes in the graph to filtering nodes in the graph.  Another
> way to look at this is to say you're implementing direct dispatch into
> geom's stack.  This can be both good and bad, but should reduce
> latency a lot.
>
> One problem that I see is that you are calling into the driver from a
> different set of contexts.  The queueing stuff was there to protect
> the driver from LoRs due to its routines being called from many
> different contexts, sometimes with other locks held (fact of life
> often in the kernel).
>
> So this certainly is something worth exploring, especially if we have
> optimized paths for up/down for certain geom classes while still
> allowing the current robust, but slow, paths for the more complicated
> nodes in the tree.  It remains to be see if there's going to be issues
> around locking order, but we've hit that with both geom and ifnet in
> the past, so caution (eg, running with WITNESS turned on early and
> often) is advised.

Am I going crazy or does this sound a lot like Sun/SVR's stream based
network stack?

 (design and problems, stream stack locking was notoriously tricky for
the exact issue mentioned above, different running contexts with
different locking granularity/requirements).

-aps