Re: nice handling in ULE (was: Re: SCHEDULE and high load situations)

On 13 Aug, To: rwatson_at_freebsd.org wrote:
> On 12 Aug, Don Lewis wrote:
> 
>> I did some experimentation, and the problem I'm seeing appears to just
>> be related to how nice values are handled by ULE.  I'm running two
>> copies of the following program, one at nice +15, and the other not
>> niced:
>> 
>> hairball:~ 102>cat sponge.c
>> int
>> main(int argc, char **argv)
>> {
>>         while (1)
>>                 ;
>> }
>> 
>> The niced process was started second, but it has accumulated more CPU
>> time and is getting a larger percentage of the CPU time according to
>> top.
>> 
>> last pid:   662;  load averages:  2.00,  1.95,  1.45    up 0+00:22:35  15:14:27
>> 31 processes:  3 running, 28 sleeping
>> CPU states: 45.3% user, 53.1% nice,  1.2% system,  0.4% interrupt,  0.0% idle
>> Mem: 22M Active, 19M Inact, 44M Wired, 28K Cache, 28M Buf, 408M Free
>> Swap: 1024M Total, 1024M Free
>> Seconds to delay: 
>>   PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
>>   599 dl       139   15  1180K   448K RUN      8:34 53.91% 53.91% sponge
>>   598 dl       139    0  1180K   448K RUN      7:22 42.97% 42.97% sponge
>>   587 dl        76    0  2288K  1580K RUN      0:03  0.00%  0.00% top
>>   462 root      76    0 56656K 46200K select   0:02  0.00%  0.00% Xorg
>>   519 gdm       76    0 11252K  8564K select   0:01  0.00%  0.00% gdmlogin
>>   579 dl        76    0  6088K  2968K select   0:00  0.00%  0.00% sshd
>> 
>> 
>> 
>> I thought it might have something to do with grouping by niceness, which
>> would group the un-niced process with a bunch of other processes that
>> wake up every now and then for a little bit if CPU time, so I tried the
>> experiment again with nice +1 and nice +15.  This gave a rather
>> interesting result.  Top reports the nice +15 process as getting a
>> higher %CPU, but the nice +1 process has slowly accumulated a bit more
>> total CPU time.  The difference in total CPU time was initially seven
>> seconds or less.
>> 
>> last pid:   745;  load averages:  2.00,  1.99,  1.84    up 0+00:43:30  15:35:22
>> 31 processes:  3 running, 28 sleeping
>> CPU states:  0.0% user, 99.6% nice,  0.4% system,  0.0% interrupt,  0.0% idle
>> Mem: 22M Active, 19M Inact, 44M Wired, 28K Cache, 28M Buf, 408M Free
>> Swap: 1024M Total, 1024M Free
>> Seconds to delay: 
>>   PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU    CPU COMMAND
>>   675 dl       139   15  1180K   448K RUN      9:48 52.34% 52.34% sponge
>>   674 dl       139    1  1180K   448K RUN     10:03 44.53% 44.53% sponge
>>   587 dl        76    0  2288K  1580K RUN      0:06  0.00%  0.00% top
>>   462 root      76    0 56656K 46200K select   0:03  0.00%  0.00% Xorg
>>   519 gdm       76    0 11252K  8564K select   0:02  0.00%  0.00% gdmlogin
>>   579 dl        76    0  6088K  2968K select   0:00  0.00%  0.00% sshd
> 
> 
> I compiled a kernel with the KTR stuff and ran this last experiment
> again.  It looks like the two niced processes get the appropriate slice
> values assigned by ULE, and they both have the same priority.  Where
> things seem to be going wrong is that the two processes are being run in
> a round robin fashion, alternating execution once every tick or two. The
> less-nice process gets prempted multiple times by the more-nice process
> before the less-nice process has exhausted its slice.

I managed to figure out a bit more of what is going on.

The following code in sched_choose() pulls the selected kse off the run
queue.

        ke = kseq_choose(kseq);
        if (ke) {
#ifdef SMP
                if (ke->ke_ksegrp->kg_pri_class == PRI_IDLE)
                        if (kseq_idled(kseq) == 0)
                                goto restart;
#endif
                kseq_runq_rem(kseq, ke);
                ke->ke_state = KES_THREAD;

                if (ke->ke_ksegrp->kg_pri_class == PRI_TIMESHARE) {
                        CTR4(KTR_ULE, "Run kse %p from %p (slice: %d, pri: %d)",
                            ke, ke->ke_runq, ke->ke_slice,
                            ke->ke_thread->td_priority);
                }
                return (ke);
        }


At some later time, setrunqueue() gets called for this thread, which
calls sched_add(), which calls sched_add_internal(), which executes the
following code frament.

        class = PRI_BASE(kg->kg_pri_class);
        switch (class) {
        case PRI_ITHD:
        case PRI_REALTIME:
                ke->ke_runq = kseq->ksq_curr;
                ke->ke_slice = SCHED_SLICE_MAX; 
                ke->ke_cpu = PCPU_GET(cpuid);
                break;
        case PRI_TIMESHARE:
                if (SCHED_CURR(kg, ke)) 
                        ke->ke_runq = kseq->ksq_curr;
                else
                        ke->ke_runq = kseq->ksq_next;
                break;
[snip]
        kseq_runq_add(kseq, ke);
        kseq_load_add(kseq, ke);


Because the thread is a CPU hog, it gets put on the next run queue, even
though it hasn't exhausted its current slice, which means that it has to
wait for all the other CPU hogs to get a turn at the CPU before it can
execute again.

I don't know how to fix this problem.  I think the desired behaviour
would be for the kse to be restored to its previous location on the run
queue.