Re: RFC: GEOM MULTIPATH rewrite

On Mon, Oct 31, 2011 at 10:10:14PM +0200, Alexander Motin wrote:
> Hi.
> 
> Attempt to fix some GEOM MULTIPATH issues made me almost rewrite it. So
> I would like to present my results and request for testing and feedback.
> 
> The main changes:
>  - Improved locking and destruction process to fix crashes in many cases.
>  - Improved "automatic" configuration method to make it safe by reading
> metadata back from all specified paths after writing to one.
>  - Added provider size check to reduce chance of conflict with other
> GEOM classes.
>  - Added "manual" configuration method without using on-disk metadata.
>  - Added "add" and "remove" commands to manage paths manually.
>  - Failed paths no longer dropped from GEOM, but only marked as FAIL and
> excluded from I/O operations.
>  - Automatically restore failed paths when all others paths are marked
> as failed, for example, because of device-caused (not transport) errors.
>  - Added "fail" and "restore" commands to manually control FAIL flag.
>  - GEOM is now destroyed on last provider disconnection. IMHO it is
> right to do if device was completely removed.
>  - Added optional Active/Active mode support. Unlike Active/Passive
> mode, load evenly distributed between all working paths. If supported by
> device, it allows to significantly improve performance, utilizing
> bandwidth of all paths. It is controlled by -A option during creation.
> Disabled by default now.
>  - Improved `status` and `list` commands output.
> 
> Latest patch can be found here:
> http://people.freebsd.org/~mav/gmultipath4.patch
> 
> Feedbacks are welcome!
> 
> Sponsored by: iXsystems, Inc.

There are two possible issues that comes to my mind, not sure if you
address them.

1. When configuration is based on on-disk metadata, GEOM spoil/taste is
   not fully helpful - if you have two paths: da0 and da1 and I write
   to da0, gmultipath won't be informed by GEOM that da1 changed as well.
   One solution is to basically keep all paths open exclusively all the
   time, even if gmultipath provider is not open or emulate spoil/taste
   for other paths if any path was modified.

2. In active/active mode do you do anything to handle possible
   reordering? Ie. if you have overlapping writes and send both of them
   using different paths, you cannot be sure that order will be
   preserved. Most of the time that's not a problem, as file systems
   rarely if at all send overlapping writes to device, but this is weak
   assumption.

-- 
Pawel Jakub Dawidek                       http://www.wheelsystems.com
FreeBSD committer                         http://www.FreeBSD.org
Am I Evil? Yes, I Am!                     http://yomoli.com