Re: 4BSD process starvation during I/O

On Thu, Nov 24, 2005 at 11:54:05AM +1100, Sam Lawrance wrote:
> 
> On 24/11/2005, at 7:18 AM, Kris Kennaway wrote:
> 
> >I have noticed that when multiple identical processes (e.g. gtar, or
> >dd) are run on 4BSD, when there are N CPUs on a machine there will be
> >N processes that run with a higher CPU share than all the others.  As
> >a result, these N processes finish first, then another N, and so on.
> >
> >This is true under both 4.11 and 6.0 (so in that sense it's not so
> >surprising), but the effect is much more pronounced on 6.0 (which may
> >be possible to fix).
> >
> >Here are the exit times for 6 identical gtar processes (and same 4.11
> >gtar binary) started together on a 2-CPU machine:
> >
> >6.0:
> >
> >1132776233
> >1132776235
> >1132776264
> >1132776265
> >1132776279
> >1132776279
> >      238.86 real        10.87 user       166.00 sys
> >
> >You can see they finish in pairs, and there's a spread of 46 seconds
> >from first to last.
> >
> >On 4.11:
> >
> >1132775426
> >1132775429
> >1132775431
> >1132775432
> >1132775448
> >1132775449
> >      275.56 real         0.43 user       336.26 sys
> >
> >They also finish in pairs, but the spread is half, at 23 seconds.
> >
> >This seems to be correlated to the rate at which the processes perform
> >I/O.  On a quad amd64 machine running 6.0 when I run multiple dd
> >processes at different offsets in a md device:
> >
> >268435456 bytes transferred in 1.734285 secs (154781618 bytes/sec)
> >268435456 bytes transferred in 1.737857 secs (154463501 bytes/sec)
> >268435456 bytes transferred in 1.751760 secs (153237575 bytes/sec)
> >268435456 bytes transferred in 3.263460 secs (82254865 bytes/sec)
> >268435456 bytes transferred in 3.295294 secs (81460244 bytes/sec)
> >268435456 bytes transferred in 3.349770 secs (80135487 bytes/sec)
> >268435456 bytes transferred in 4.716637 secs (56912467 bytes/sec)
> >268435456 bytes transferred in 4.850927 secs (55336941 bytes/sec)
> >268435456 bytes transferred in 4.953528 secs (54190760 bytes/sec)
> >
> >They finish in groups of 3 here since the 4th CPU is being used to
> >drive the md worker thread (which takes up most of the CPU).  In this
> >case the first 3 dd processes get essentially 100% of the CPU, and the
> >rest get close to 0% until those 3 processes finish.
> >
> >Perhaps this can be tweaked.
> >
> 
> I tried this on a dual Xeon, with 12 processes like
> 
> 	mdconfig -a -t swap -s 320m
> 	dd if=/dev/md0 of=1 bs=1m skip=0 count=40 &
> 	dd if=/dev/md0 of=2 bs=1m skip=40 count=40 &

You're reading from the md, not writing to it.  Sorry if that wasn't
clear.

My test is:

#!/bin/sh

mdconfig -d -u 0
mdconfig -a -t swap -s 16g
for i in `jot $1 1`; do
        dd if=/dev/zero of=/dev/md0 seek=$(($i*16384)) count=16384 bs=16k > /dev/null &
done
wait

Kris