Re: access to hard drives is "blocked" by writes to a flash drive

On Tue, Mar 05, 2013 at 09:01:36AM -0700, Ian Lepore wrote:
> On Mon, 2013-03-04 at 21:27 -0800, Don Lewis wrote:
> > On  4 Mar, Ian Lepore wrote:
> > > On Sun, 2013-03-03 at 19:01 -0800, Don Lewis wrote:
> > >> On  3 Mar, Poul-Henning Kamp wrote:
> > >> 
> > >> > For various reasons (see: Lemming-syncer) FreeBSD will block all I/O
> > >> > traffic to other disks too, when these pileups gets too bad.
> > >> 
> > >> The Lemming-syncer problem should have mostly been fixed by 231160 in
> > >> head (231952 in stable/9 and 231967 in stable/8) a little over a year
> > >> ago. The exceptions are atime updates, mmaped files with dirty pages,
> > >> and quotas. Under certain workloads I still notice periodic bursts of
> > >> seek noise. After thinking about it for a bit, I suspect that it could
> > >> be atime updates, but I haven't tried to confirm that.
> > >> 
> > >> When using TCQ or NCQ, perhaps we should limit the number of outstanding
> > >> writes per device to leave some slots open for reads.  We should
> > >> probably also prioritize reads over writes unless we are under memory
> > >> pressure.
> > >>  
> > > 
> > > Then either those changes didn't have the intended effect, or the
> > > problem we're seeing with lack of system responsiveness when there's a
> > > large backlog of writes to a slow device is not the lemming-syncer
> > > problem.  It's also not a lack of TCQ/NCQ slots, given that no such
> > > thing exists with SD card IO.
> > > 
> > > When this is going on, the process driving the massive output spends
> > > almost all its time in a wdrain wait, and if you try to launch an app
> > > that isn't already in cache, a siginfo generally shows it to be in a
> > > getblk wait.
> > 
> > If your only drive is a single SD card, then you're pretty much hosed
> > when I/O is blocked because the SD card is doing an erase.  It can only
> > handle one command at a time, and if a write blocks, there's nothing
> > that we can do to get it to execute a read until it is done with the
> > write command that it is hung up on.  I'm not familiar with the lower
> > layers, but things might be less bad if read ops can jump ahead and get
> > sent to the drive before any queued writes.
> > 
> 
> Yes, but an erase-block operation on a nand flash takes on the order of
> 500uS, not 8-10 seconds, which is the kind of interactive
> non-responsiveness you experience in these situations.  The very nature
> of SD cards is one operation at a time, so no internal operation
> queueing is in play to explain the long (apparent) hangs.

Values from datasheet I recently read (for an older 1G NAND chip) was
500uS for writing and 3ms for erasing - though erasing happens on larger
granularities than writing.
Of course a write request may include more than one flash write for
bookkeeping.
Nevertheless the long read starvation comes from writing multiple requests
before reading.
I often saw with gstat that a slowly decreasing L(q) was the situation
when I was waiting for a read to complete.
Since r/s never showed any other value than 0 and w/s showed small
numbers during the starvation it is a clear sign that a bunch of write
requests were queued handled before any read attempt.
This usually happens in waves - L(q) pops high and decreases slowly to
zero, while ms/w clims to multiple seconds then directly after another
bunch of writes pop up.

> I've debated playing with the bio work loop in mmcsd to see if moving
> reads ahead of writes was helpful, but that seems like a dangerous path
> to go down without some mitigation strategy to ensure that writes go
> through eventually.  That seems especially important when you consider
> that writes may be necessary to free up memory to un-wedge other things
> that are waiting.  (Yeah, people don't often use sd cards as swap
> storage, but I've done so in a pinch.)  All in all, I've never pursued
> it because it feels like the wrong layer to address the problem at.

Sometimes there is no choice and you have to setup swap on SD, but there
are other reasons to write on your SD card.
But honestly - most of the time you get crazy when you are maintaining
a system - e.g. install a package, which can't get faster anyway, but
very often you want to edit configfiles at the same time.
During production this rarely was an issue for me.

-- 
B.Walter <bernd_at_bwct.de> http://www.bwct.de
Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.