Re: sbrk(2) broken

> For performance reasons, malloc(3) will hold on to a number of pages
> that theoretically could be given back to the kernel, simply because
> it expects to need them shortly.

Although the primary concern is malloc(), I would like to point out that 
various programs implementing copying garbage collection could more 
efficiently give memory back to the system than malloc(), and could therefor 
benefit more than malloc() from some kind of feedback from the kernel.

There was concern over the complexity involved with intelligently doing 
something about the memory pressure hints in userspace, but this does not 
apply here since the allocator/garbage collection would be the equivalent of 
malloc() and complexity there would not affect application code.

The problem with malloc() being that, unless I am missing something, malloc 
will never be able to give back memory to the kernel except insofar as the 
memory mapped is continuously unused between some location and the break (in 
the case of sbrk()) or over the entire range (mmap()). malloc() cannot force 
this to be the case, since pointers must remain valid. The possibility of 
reclamation is then often going to be limited to completely unused space 
being held by malloc() for future use, rather than also applying to areas 
already used for allocation.

Programs implementing copying GC, or able to for some other reason to move 
allocated memory around, could compact the heap and give back left-over 
memory. In some cases this would only entail a temporary improvement due to 
defragmentation, but in others (such as a long-running program spiking in 
memory use, only then to drop a lot of that memory) it could have a pretty 
massive effect on memory use.

Where a malloc() using program might be unable to sbrk() or munmap() because 
there happens to be some left-over non-free piece of memory at the top of the 
mapped range, a GC could use indications from the system to ensure this is 
not the case (depending on details of the implementation; for example, 
compactation of tenured generations could be forced early, etc).

(This is not to say I am aware of any implementation that actually supports 
this, but on the other hand perhaps that is due to the lack of operating 
systems that provide the required feedback.)

-- 
/ Peter Schuller

PGP userID: 0xE9758B7D or 'Peter Schuller <peter.schuller_at_infidyne.com>'
Key retrieval: Send an E-Mail to getpgpkey_at_scode.org
E-Mail: peter.schuller_at_infidyne.com Web: http://www.scode.org