Re: very high memory usage in -current

Stefan Ehmann wrote:
> [huge jpeg/png images cause memory bloat with konqueror and gtk apps, on -CURRENT.]
> 
> I don't really know what's causing this problem. Maybe it's related to 
> jemalloc, but I'd be surprised if no one else has noticed this before.
> 
> size/res values reported by top
> 
> 6.2 desktop pc with nvidia binary driver
> konqueror 40M/30M
> 
> CURRENT notebook with the i810 driver
> konqueror 290M/250M
> 
> Any clues or more information needed?

If you think jemalloc is involved, the easiest way to check is by
reverting src/lib/libc/stdlib/malloc.c to revision 1.92, which is
phkmalloc.  If this substantially changes memory usage, then there are
further diagnostics that can be used to help understand the issue.

-----

If jemalloc is involved, here's what could cause such behavior.  First,
you would have to be running on a 32-bit platform, so that sbrk() is in
use (rather than pure mmap() as for the 64-bit platforms).

1) The application would allocate some large chunks of memory for image
processing.  These data would be allocated in the data segment.

2) The application would allocate some additional long-lived object for
which no existing memory region could be recycled.  This would cause the
object to be allocated in the data segment, at the end of the sbrk()ed
memory.

3) The application would free most/all of the memory that was used for
image processing.  Unfortunately, due to the long-lived object from step
(2), this would be in the middle of the sbrk()ed memory, and so malloc
would have to hold onto the memory for possible future use, unable to
return it to the kernel.

The right solution to the problem is to get rid of the hard line between
data segment and heap, and use the address space that is currently
reserved for sbrk() as the last resort for mmap() requests, thus
allowing the data segment and heap to gracefully grow toward each other
until memory is exhausted.  We really don't need the data segment
anymore except for legacy support of applications that do their own
memory management via sbrk().  However, when I talked with the VM
wizards, I was sufficiently intimidated by the difficulty of removing
the hard-coded data segment size that I did not pursue it any further.

Personally, in the absence of a dynamic boundary between the data
segment and the heap, I would be quite happy to completely disable
sbrk() support in jemalloc, and let those who really need that last 512
MB of address space adjust resource limits for their applications as
necessary.  In practice, I expect this would cause people far less
trouble than does the current state of affairs.

Jason