Re: access to hard drives is "blocked" by writes to a flash drive

On  4 Mar, Peter Jeremy wrote:
> On 2013-Mar-03 23:12:40 -0800, Don Lewis <truckman_at_freebsd.org> wrote:
>>On  4 Mar, Konstantin Belousov wrote:
>>> It could be argued that the current typical value of 16MB for the
>>> hirunningbufspace is too low, but experiments with increasing it did
>>> not provided any measureable change in the throughput or latency for
>>> some loads.
>>
>>The correct value is probably proportional to the write bandwidth
>>available.
> 
> The problem is that write bandwidth varies widely depending on the
> workload.  For spinning rust, this will vary between maybe 64KBps
> (512B random writes) and 100-150MBps (single-theaded large sequential
> writes).  The (low-end) SSD in my Netbook also has about 100:1 variance
> due to erase blocking.  How do you tune hirunningbufspace in the face
> of 2 or 3 orders of magnitude variance in throughput?  Especially since
> SSDs don't gradually degrade - they hit a brick wall.

I think a latency target would be desirable.  As a first cut, maybe a
limit on the number of write ops per device instead of a limit on the
amount of data. That should more or less do the correct thing for small
random I/O vs. large sequential writes.  It could be somewhat self
tuning if we kept track of the average service time, which should
probably be measured with write caching off.  There might still need to
be a higher size limit to avoid excessive memory consumption.

If you only have a single drive and it is an SSD, then there's probably
not much you can do about the erase blocking problem.  About the best
you could do is if it supports NCQ is to turn off write caching so that
you can keep track of how many writes are outstanding so that you can
limit the write load to something less than the maximum number of
outstanding commands that it supports, and then hope that when you run
into the erase blocking situation, that *maybe* it will still process
reads.

If you have multiple drives, then you don't want one backlogged drive to
block writes to your other drive(s), which would require something other
than a global hirunningbufspace limit.