Re: 4BSD process starvation during I/O

On 24/11/2005, at 7:18 AM, Kris Kennaway wrote:

> I have noticed that when multiple identical processes (e.g. gtar, or
> dd) are run on 4BSD, when there are N CPUs on a machine there will be
> N processes that run with a higher CPU share than all the others.  As
> a result, these N processes finish first, then another N, and so on.
>
> This is true under both 4.11 and 6.0 (so in that sense it's not so
> surprising), but the effect is much more pronounced on 6.0 (which may
> be possible to fix).
>
> Here are the exit times for 6 identical gtar processes (and same 4.11
> gtar binary) started together on a 2-CPU machine:
>
> 6.0:
>
> 1132776233
> 1132776235
> 1132776264
> 1132776265
> 1132776279
> 1132776279
>       238.86 real        10.87 user       166.00 sys
>
> You can see they finish in pairs, and there's a spread of 46 seconds
> from first to last.
>
> On 4.11:
>
> 1132775426
> 1132775429
> 1132775431
> 1132775432
> 1132775448
> 1132775449
>       275.56 real         0.43 user       336.26 sys
>
> They also finish in pairs, but the spread is half, at 23 seconds.
>
> This seems to be correlated to the rate at which the processes perform
> I/O.  On a quad amd64 machine running 6.0 when I run multiple dd
> processes at different offsets in a md device:
>
> 268435456 bytes transferred in 1.734285 secs (154781618 bytes/sec)
> 268435456 bytes transferred in 1.737857 secs (154463501 bytes/sec)
> 268435456 bytes transferred in 1.751760 secs (153237575 bytes/sec)
> 268435456 bytes transferred in 3.263460 secs (82254865 bytes/sec)
> 268435456 bytes transferred in 3.295294 secs (81460244 bytes/sec)
> 268435456 bytes transferred in 3.349770 secs (80135487 bytes/sec)
> 268435456 bytes transferred in 4.716637 secs (56912467 bytes/sec)
> 268435456 bytes transferred in 4.850927 secs (55336941 bytes/sec)
> 268435456 bytes transferred in 4.953528 secs (54190760 bytes/sec)
>
> They finish in groups of 3 here since the 4th CPU is being used to
> drive the md worker thread (which takes up most of the CPU).  In this
> case the first 3 dd processes get essentially 100% of the CPU, and the
> rest get close to 0% until those 3 processes finish.
>
> Perhaps this can be tweaked.
>

I tried this on a dual Xeon, with 12 processes like

	mdconfig -a -t swap -s 320m
	dd if=/dev/md0 of=1 bs=1m skip=0 count=40 &
	dd if=/dev/md0 of=2 bs=1m skip=40 count=40 &
	...

and all processes finished within a frame of about 1 second.  No  
interesting difference with HTT on or off.  Generic SMP RELENG_6 from  
8 November.

Is your dd test different somehow?