Re: RLIMIT_DATA and malloc(3) use of mmap(2)

On Tue, Nov 22, 2011 at 07:43:57PM +0400, Maxim Dounin wrote:
> Hello!
> 
> On Tue, Nov 22, 2011 at 02:44:10PM +0200, Kostik Belousov wrote:
> 
> > I was reminded about the patch I wrote for Igor Sysoev some time ago.
> > The issue the patch tries to handle is that jemalloc uses mmap() instead
> > of sbrk() for pages allocation, and thus RLIMIT_DATA limit is no longer
> > effective to put the bounds on the process heap. Since reverting to sbrk
> > for such purpose is worse then the issue itself, I proposed a solution
> > of 'self-restricting malloc'.
> 
> Just a little clarification for others: currently, there is no way 
> to *safely* limit memory usage of a process while using jemalloc 
> with mmap().
> 
> The only thing available is RLIMIT_VMEM, but it's not safe as it 
> may be reached on stack grow (leaving no possibility for an 
> application to handle this).
> 
> > The patch adds a flag MAP_DATALIMIT to the flags argument of mmap(2),
> > which instructs the system to account the mapping in the RLIMIT_DATA
> > resource count. The malloc(3) also gets new option 'L' to enable
> > passing MAP_DATALIMIT to mmap() when allocating pages. By default,
> > the 'L' option is not activated.
> > 
> > Now, if user wants to ensure that process heap is restricted by the
> > ulimit -d and still use mmap() for jemalloc, he supplies the option
> > using any mechanism. The behaviour is voluntaristic, to prevent the
> > trashing use RLIMIT_SWAP.
> > 
> > Do people consider the facility useful ?
> 
> Yes, at least some way to safely limit process memory usage is 
> certainly needed.
> 
> It's a bit sad this isn't enabled by default, but it's probably 
> too late for this.  RLIMIT_DATA was (almost) a noop for too long 
> and making it work again to limit all memory allocations will 
> break POLA.
> 
> > Any comments for the patch itself ?
> > 
> > http://people.freebsd.org/~kib/misc/map_datalimit.1.patch
> 
> Patch looks ok for me (though I'm not a VM expert).
> 
> Another possible aproach would be to introduce separate anonymous 
> (private?) mmap limit, this will allow to do essentially the same 
> thing in a bit more consistent (IMHO) manner.

This is already done in some form as the per-user swap limit. Converting
swap limit to the per-process limit raises the architectural questions,
due to shadow chains of the backing vm objects. In particular, we have
to account the anonymous memory used for backing the changed pages
from the writeable private mapping of the files. Similar issues appear
due to fork().

Anyway, the patch needs testers before I will push it forward.