Re: sbrk(2) broken

In message <a2b6592c0801071606g4c0dcb9ap117e345fda5e7e5f_at_mail.gmail.com>, "Igor
 Mozolevsky" writes:
>On 07/01/2008, Andrew Reilly <andrew-freebsd_at_areilly.bpc-users.org> wrote:
>> On Mon, 07 Jan 2008 13:18:47 +0000
>> "Poul-Henning Kamp" <phk_at_phk.freebsd.dk> wrote:
>>
>> > Yes, but you will not see this complication, it will be hidden
>> > in the implementation of malloc(3).
>>
>> How could you hide it inside malloc?  Would malloc start
>> returning 0 after receiving the "less mem than desirable"
>> signal?  Would it ever go back to returning non-zero?
>
>I'm with Andrew on this one. The only (sensible) way I could see it
>being hidden behind malloc() is if malloc() blocks until sufficient
>memory becomes available.

You should read some recent literature on malloc(3), my own and
Jasons papers are good places to start.

For performance reasons, malloc(3) will hold on to a number of pages
that theoretically could be given back to the kernel, simply because
it expects to need them shortly.

Such parameters and many others of the malloc implementation can
be tweaked to "waste" more or less memory, in response to a sensibly
granular indication from the kernel about how bad things are.

Also, many subsystems in the kernel could adjust their memory use
in response to a "memory pressure" indication, if memory is tight,
we could cache vnodes and inodes less agressively, if things are
going truly bad, we can even ditch all non-active entries from
these caches.

If one implements this with three states:

Green - "all clear"

Yellow - "tight" - free one before you allocate one if you can.

Red - "all out" - free all that you sensibly can.

And implemented strategies like I propose above (and have proposed
for the last 10 years), then it is very unlikely that the system
would ever get into the red state, because the yellow state will
mitigate and reduce the memory pressure.

Nothing prevents an intelligent process from listening in and
doing sensible things, firefox could ditch the memory cache of
pages for instance.

But we can't get anywhere until some VM wizard produces the
three "lamps" for us to look at in the first place, that's where
we have been stuck for the last 10 years.

-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk_at_FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.