Re: Memory problem with latest malloc.c

On Wed, 2008-01-02 at 18:26 -0800, Jason Evans wrote:
> Joe Marcus Clarke wrote:
> > GNOME applications use a Python driven XML parser to generate help
> > document translations.  The engine takes the English XML document and
> > applies translations to it.  The tool, xml2po, is installed as part of
> > textproc/gnome-doc-utils.  I'm currently working on porting GNOME 2.21,
> > and one of the Evolution help doc changes triggered a memory problem on
> > my test machine.  Basically, with up to and including rev 1.154 of
> > malloc.c, I am able to generate the help file with no errors.  With all
> > later revs including 1.160, the Python process balloons up to about 512
> > MB of memory, then dies.
> > 
> > The only malloc config I've done is symlink Aj to /etc/malloc.conf.  I'd
> > be happy to provide the file an exact command used, but it might be
> > easier to let me know if there's any debugging I could provide that
> > would help here.
> 
> Revision 1.160 of malloc.c uses sbrk instead of mmap by default.  The 
> intent is to make data segment resource limits useful.  So, there are 
> two possible explanations I can think of for your problems.  The first 
> possibility is that there is still a bug in malloc that you're hitting. 
>   However, the other possibility is that the python program you're 
> running really needs more than 512 MB of memory, and you're hitting the 
> resource limit.
> 
> It would be really helpful to me if you run your program with 
> MALLOC_OPTIONS=dM and monitor memory usage.  These flags cause mmap to 
> be used instead of sbrk, and we can find out from that how much memory 
> you really need.  If peak memory usage is substantially different when 
> using mmap versus sbrk, there's probably a malloc bug.

Memory climbed up to 976 MB SZ, 974 MB RSS MB with dM
-> /etc/malloc.conf.  The file was eventually generated without error.
Again, with Aj -> /etc/malloc.conf, the python2.5 process operating on
the same file planed out at 504 MB SZ, 501 MB RSS.

> 
> This latest round of malloc changes hasn't been much fun.  Thanks for 
> your help and patience.

No problem.  Thanks for getting back to me.

Joe

-- 
Joe Marcus Clarke
FreeBSD GNOME Team      ::      gnome_at_FreeBSD.org
FreeNode / #freebsd-gnome
http://www.FreeBSD.org/gnome