Re: access to hard drives is "blocked" by writes to a flash drive

On Sun, 2013-03-03 at 15:20 +1100, Peter Jeremy wrote:
> On 2013-Mar-02 18:29:54 +0100, deeptech71 <deeptech71_at_gmail.com> wrote:
> 
> >When one of my flash drives is being heavily written to; typically by
> >``svn update'' on /usr/src, located on the flash drive; the following
> >can be said about filesystem behavior:
> >
> >- ``svn update'' seems to be able to quickly update a bunch of files,
> >   but is then unable to continue for a period of time. This behavior
> >   is cyclical, and cycles several times, depending on the amount of
> >   updating work to be done for a particular run of ``svn update''.
> 
> This sounds like normal flash behaviour:  You can only write to erased
> blocks.  The SSD firmware attempts to keep a free pool of erased blocks
> but if you write too fast, you empty the free pool and need to wait for
> the wear-levelling algorithm to move blocks around and erase them.
> 
> Enabling TRIM (the '-t' flag on tunefs) will help if the drive supports
> TRIM (if it doesn't, it'll probably just lockup).  Otherwise, you need
> to either put up with it or upgrade to a better SSD.
> 
> I run into this regularly with the low-end SuperTalent drive in my
> Netbook but have never seen it with the OCZ Agility4 that I use for
> L2ARC in my fileserver.
> 

I run into this behavior all the time too, mostly on arm systems that
have an sd card or usb thumb driver as their main/only drive.  It's
especially annoying when you try to launch a program that isn't in the
filesystem cache already, and you have to wait 10-20 seconds for it to
be read in.  

I stop just short of calling that "normal" because it seems like we
should be able to do better somehow... I don't think the problem is on
the flash device itself (like waiting for some big internal cache to
drain), it acts more like there aren't enough buffers on the OS side
(they're all tied up waiting to be written), or it's a bio-queue sort
problem, or maybe it's somehow intentional that writes have precedence
over reads (that might make more sense on a system without a big
disparity between read and write speed, especially given that writing
for the purpose of swapping is important to free up memory).

The fact that a huge backlog of writes to a slow flash-based device can
block reads from faster devices makes me think this is somehow related
to available buffers in the OS.  I wonder if this could be one of those
situations where reducing the amount of buffers improves responsiveness
(if not necessarily overall throughput).

-- Ian