Re: sbrk(2) broken

On 08/01/2008, Poul-Henning Kamp <phk_at_phk.freebsd.dk> wrote:
> In message <a2b6592c0801071606g4c0dcb9ap117e345fda5e7e5f_at_mail.gmail.com>, "Igor
>  Mozolevsky" writes:
> >On 07/01/2008, Andrew Reilly <andrew-freebsd_at_areilly.bpc-users.org> wrote:
> >> On Mon, 07 Jan 2008 13:18:47 +0000
> >> "Poul-Henning Kamp" <phk_at_phk.freebsd.dk> wrote:
> >>
> >> > Yes, but you will not see this complication, it will be hidden
> >> > in the implementation of malloc(3).
> >>
> >> How could you hide it inside malloc?  Would malloc start
> >> returning 0 after receiving the "less mem than desirable"
> >> signal?  Would it ever go back to returning non-zero?
> >
> >I'm with Andrew on this one. The only (sensible) way I could see it
> >being hidden behind malloc() is if malloc() blocks until sufficient
> >memory becomes available.
>
> You should read some recent literature on malloc(3), my own and
> Jasons papers are good places to start.

Can you provide some refs/links, unfortunately googling for
PKH+malloc() is pretty futile!


> For performance reasons, malloc(3) will hold on to a number of pages
> that theoretically could be given back to the kernel, simply because
> it expects to need them shortly.
>
> Such parameters and many others of the malloc implementation can
> be tweaked to "waste" more or less memory, in response to a sensibly
> granular indication from the kernel about how bad things are.
>
> Also, many subsystems in the kernel could adjust their memory use
> in response to a "memory pressure" indication, if memory is tight,
> we could cache vnodes and inodes less agressively, if things are
> going truly bad, we can even ditch all non-active entries from
> these caches.

I don't think it's the kernel that is being ill-mannered (unless, of
course, it's running ZFS ;-)) by eating up the memory, it's the user
processes that consume most resources.

> Nothing prevents an intelligent process from listening in and
> doing sensible things, firefox could ditch the memory cache of
> pages for instance.

How do you propose they 'eavesdrop' on the kernel? Baring in mind that
most apps nowadays are written for Linux and are hacked to be portable
afterwards (just look at the number of patches in the ports tree),
it's much simpler to write a signal handler than FreeBSD-kernel
specific listening code...

> But we can't get anywhere until some VM wizard produces the
> three "lamps" for us to look at in the first place, that's where
> we have been stuck for the last 10 years.

I think the problem is not in providing the lamps to indicate the
state, but figuring out an algorithm for judging green->yellow and
yellow->green transitions...

Igor