Re: sbrk(2) broken

"Igor Mozolevsky" <igor_at_hybrid-lab.co.uk> writes:
> Robert Watson <rwatson_at_freebsd.org> writes:
> > To be clear, in the new world order, instead of getting NULL
> > back from malloc(3), SIGKILL is delivered to large processes.
> Huh??? Again, huh???

For the same reason as it has for the last 20 years or so: memory
overcommit, which means that malloc() allocates address space, not
memory.  Actual memory is allocated on-demand when the address space is
used (read from or written to).  If there is no RAM left and none can be
freed by swapping out, the process gets killed.  The process that gets
killed is not necessarily the memory hog, it is merely the process that
is unlucky enough to touch a new page at the wrong moment, i.e. when all
RAM and swap is exhausted *or* everything in RAM is wired down and
unswappable.

Of course, if you're afraid of memory overcommit and you know in advance
how much memory you need, you can simply allocate a sufficient amount of
address space at startup and touch it all.  This way, you will either be
killed right away, or be guaranteed to have sufficient memory for the
rest of your (process) lifetime.  Alternatively, do what Varnish does:
create a large file, mmap it, and allocate everything you need from that
area, so you have your own private swap space.  Just make sure to
actually allocate the disk space you need (by filling the file with
zeroes, or at the minimum writing a zero to the file every sb.st_blksize
bytes, preferably sequentially to avoid excessive fragmentation) or you
may run into the same problem as with malloc() if the disk fills up
while your backing file is still sparse.

The ability to specify a backing file to use instead of anonymous
mappings would be a cool addition to jemalloc.

DES
-- 
Dag-Erling Smørgrav - des_at_des.no