Re: SCHED_ULE makes 256Mbyte i386 unusable

On Sat, Apr 21, 2018 at 07:21:58PM +0000, Rick Macklem wrote:
> I decided to start a new thread on current related to SCHED_ULE, since I see
> more than just performance degradation and on a recent current kernel.
> (I cc'd a couple of the people discussing performance problems in freebsd-stable
>  recently under a subject line of "Re: kern.sched.quantum: Creepy, sadistic scheduler".
> 
> When testing a pNFS server on a single core i386 with 256Mbytes using a Dec. 2017
> current/head kernel, I would see about a 30% performance degradation (elapsed
> run time for a kernel build over NFSv4.1) when the server kernel was built with
> options SCHED_ULE
> instead of
> options SCHED_4BSD
> 
> Now, with a kernel from a couple of days ago, the
> options SCHED_ULE
> kernel becomes unusable shortly after starting testing.
> I have seen two variants of this:
> - Became essentially hung. All I could do was ping the machine from the network.
> - Reported "vm_thread_new: kstack allocation failed
>   and then any attempt to do anything gets "No more processes".
This is strange.  It usually means that you get KVA either exhausted or
severly fragmented.

Enter ddb, it should be operational since pings are replied.  Try to see
where the threads are stuck.

> with the only difference being a kernel built with
> options SCHED_4BSD
> everything works and performs the same as the Dec 2017 kernel.
> 
> I can try rolling back through the revisions, but it would be nice if someone
> could suggest where to start, because it takes a couple of hours to build a
> kernel on this system.
> 
> So, something has made things worse for a head/current kernel this winter, rick

There are at least two potentially relevant changes.

First is r326758 Dec 11 which bumped KSTACK_PAGES on i386 to 4.
Second is r332489 Apr 13, which introduced 4/4G KVA/UVA split.

Consequences of the first one are obvious, it is much harder to find
the place to map the stack.  Second change, on the other hand, provides
almost full 4G for KVA and should have mostly compensate for the negative
effects of the first.

And, I cannot see how changing the scheduler would fix or even affect that
behaviour.