Re: access to hard drives is "blocked" by writes to a flash drive

On Mon, 2013-03-04 at 21:27 -0800, Don Lewis wrote:
> On  4 Mar, Ian Lepore wrote:
> > On Sun, 2013-03-03 at 19:01 -0800, Don Lewis wrote:
> >> On  3 Mar, Poul-Henning Kamp wrote:
> >> 
> >> > For various reasons (see: Lemming-syncer) FreeBSD will block all I/O
> >> > traffic to other disks too, when these pileups gets too bad.
> >> 
> >> The Lemming-syncer problem should have mostly been fixed by 231160 in
> >> head (231952 in stable/9 and 231967 in stable/8) a little over a year
> >> ago. The exceptions are atime updates, mmaped files with dirty pages,
> >> and quotas. Under certain workloads I still notice periodic bursts of
> >> seek noise. After thinking about it for a bit, I suspect that it could
> >> be atime updates, but I haven't tried to confirm that.
> >> 
> >> When using TCQ or NCQ, perhaps we should limit the number of outstanding
> >> writes per device to leave some slots open for reads.  We should
> >> probably also prioritize reads over writes unless we are under memory
> >> pressure.
> >>  
> > 
> > Then either those changes didn't have the intended effect, or the
> > problem we're seeing with lack of system responsiveness when there's a
> > large backlog of writes to a slow device is not the lemming-syncer
> > problem.  It's also not a lack of TCQ/NCQ slots, given that no such
> > thing exists with SD card IO.
> > 
> > When this is going on, the process driving the massive output spends
> > almost all its time in a wdrain wait, and if you try to launch an app
> > that isn't already in cache, a siginfo generally shows it to be in a
> > getblk wait.
> 
> If your only drive is a single SD card, then you're pretty much hosed
> when I/O is blocked because the SD card is doing an erase.  It can only
> handle one command at a time, and if a write blocks, there's nothing
> that we can do to get it to execute a read until it is done with the
> write command that it is hung up on.  I'm not familiar with the lower
> layers, but things might be less bad if read ops can jump ahead and get
> sent to the drive before any queued writes.
> 

Yes, but an erase-block operation on a nand flash takes on the order of
500uS, not 8-10 seconds, which is the kind of interactive
non-responsiveness you experience in these situations.  The very nature
of SD cards is one operation at a time, so no internal operation
queueing is in play to explain the long (apparent) hangs.

I've debated playing with the bio work loop in mmcsd to see if moving
reads ahead of writes was helpful, but that seems like a dangerous path
to go down without some mitigation strategy to ensure that writes go
through eventually.  That seems especially important when you consider
that writes may be necessary to free up memory to un-wedge other things
that are waiting.  (Yeah, people don't often use sd cards as swap
storage, but I've done so in a pinch.)  All in all, I've never pursued
it because it feels like the wrong layer to address the problem at.

-- Ian