Re: sbrk(2) broken

On 04/01/2008, Dag-Erling Smørgrav <des_at_des.no> wrote:

> For the same reason as it has for the last 20 years or so: memory
> overcommit, which means that malloc() allocates address space, not
> memory.  Actual memory is allocated on-demand when the address space is
> used (read from or written to).  If there is no RAM left and none can be
> freed by swapping out, the process gets killed.  The process that gets
> killed is not necessarily the memory hog, it is merely the process that
> is unlucky enough to touch a new page at the wrong moment, i.e. when all
> RAM and swap is exhausted *or* everything in RAM is wired down and
> unswappable.

Broadcasting SIGDANGER would be a much better option; followed by
SIGTERM to the memory hogger (to allow for graceful termination) and
only then SIGKILL. I can imagine a few (legitimate) scenarios when a
user process would want to hog as much RAM as possible...

> Of course, if you're afraid of memory overcommit and you know in advance
> how much memory you need, you can simply allocate a sufficient amount of
> address space at startup and touch it all.  This way, you will either be
> killed right away, or be guaranteed to have sufficient memory for the
> rest of your (process) lifetime.  Alternatively, do what Varnish does:
> create a large file, mmap it, and allocate everything you need from that
> area, so you have your own private swap space.  Just make sure to
> actually allocate the disk space you need (by filling the file with
> zeroes, or at the minimum writing a zero to the file every sb.st_blksize
> bytes, preferably sequentially to avoid excessive fragmentation)

Surely you can just fseek() on the file at the correct lenght?

> The ability to specify a backing file to use instead of anonymous
> mappings would be a cool addition to jemalloc.

That would be really cool and even better if it allocated it in a
contiguous chunk.


Igor :-)