Re: net/asterisk13: memory leak under 12-CURRENT?

On Tue, 26 Sep 2017 15:06:23 +0200
Guido Falsi <madpilot_at_FreeBSD.org> wrote:

> On 09/26/2017 14:45, O. Hartmann wrote:
> > Befor starting a PR I'd liek to ask for some advice to document a supposedly
> > existent memory leak in net/asterisk13 and 12-CURRENT.
> > 
> > Background:
> > 
> > Running recent 12-CURRENT (FreeBSD 12.0-CURRENT #58 r323999: Tue Sep 26
> > 06:18:27 CEST 2017 amd64) on a PCengine APU 2C4, equipted with 4GB of RAM
> > and 4080 KB free after the most recent SeaBIOS has been started, I try to
> > utilise a net/asterisk13 as PBX on that platform. Asterisk13 has been build
> > via poudriere and is compiled with CLANG/LLVM, not GCC!
> >   
> 
> I'm running asterisk on similar hardware and also building with clang, 
> and have it going for months without any problems. I have disabled many 
> modules in that build though. Problem could be caused by some ancillary 
> library pulled in by some module.

Since I run net/asterisk with automatic module loading (I'm new to asterisk),
this is very likely and might cause the problem somehow.

> 
> > When FreeBSD (NanoBSD) has been started, depending on the recent revision
> > of the OS/Kernel, top shows ~3680 - 3705 MBytes of free memory. Starting
> > net/asterisk13 via "service asterisk onestart" indicates an overall usage
> > of up to 100 MBytes of memory. After having now run the Asterisk 13.17.2
> > daemon for two days resulted in ~ 3100 MBytes of free memory, while the
> > asterisk daemon was still running and doing nothing. But performing several
> > restarts on a freshly started box gives the same result after ~ 10 restarts
> > - and even after stopping asterisk and let the system run for a couple of
> > days doesn't free the memory. I stopped after 15 restarts or so watching
> > the loss of memory after each restart of asterisk, so i came to the
> > conclusion that there must be a memory leak somewhere. now i try to provide
> > more informations as needed for a PR.
> > 
> > Can someone help?  
> 
> Not sure, restarting the daemon should free any leaked memory the daemon 
> has. If a killed process leaves memory locked at the system level there 
> should be some other cause.

Even with no runnidng asterisk, memory level drops after the last shutdown of
asterisk and keeps that low. Even for weeks! My router never shows that high
memory consumption, even under load.

> 
> Have you tried diagnosing this memory with more tools?

top so far ...

> 
> 
> Depending on how deep you want to investigate ps, top and vmstat are 
> simple to use and give you some indication. To go deeper you'll need to 
> use more complicated tools. Others are better suited than me for 
> suggesting those. DTrace could be a powerful but not easy to use instrument.

Well, DTrace would be an overkill for me. I'd like to provide more informations
in case any FreeBSD developer wants to look at this - in case there is a real
bug in either FreeBSD CURRENT - or net/asterisk.

> 
> Looking just at free memory (which under a UNIX OS can have various 
> valid definitions, depending on how you account for inactive memory, 
> buffers and laundry RAM) is definitely not enough. Have you seen 
> asterisk process allocate more and more memory?
> 
> You should check also what other processes in the system are doing. A 
> computer software when running never really does "nothing"(except, 
> maybe, when swapped out) and some memory usage can accumulate in many 
> parts of the system.

Well, since it is a well defined NanoBSD system which has run for weeks with
the same memory consumptions ebven under load, it seems obvious that there must
be a black hole for memory when it somes to loading/unloading asterisk. The
memory drain can be reproduced.

>

The question would be: how to use vmstat to give hints for those familiar with
memory subsystems to indicate a real bug?

I tried to find some advices, but maybe my English isn't good enough to make
google help.

Kind regards,

Oliver