Re: standard error handling for malloc() broken for user root and group wheel

On Wed, 18 Feb 2004, Poul-Henning Kamp wrote:

> In message <20040218140756.S20006_at_gamplex.bde.org>, Bruce Evans writes:
> >User root and group wheel cannot get standard error handling for malloc()
> >even if they specifically asked for it using MALLOC_OPTIONS=a or
> >equivalent.  This was broken in rev.1.73 of malloc.c.  Fix:
>
> The situations which can result in the 'a' vs 'A' flag making a
> difference in malloc(3) behavior are all violations of the malloc(3)
> API as defined by ISO C and as such the standard defines the behaviour
> as "undefined".

False.

> Talking about a "standard error handling" in this
> context is therefore at best confused, at worst misleading.

The 'A' flag causes malloc() to abort() where the standard requires it
to return NULL.  This breaks all programs and libraries that handle
malloc() allocation failures gracefully.

There is a general problem with memory overcommit that may result in
memory returned by malloc() not actually being available.  This is
also a standards violation, but it need not be a problem in all cases.
E.g., the sysadmin, the user or the program may have limited the
program's memory usage using getrlimit().

> Given that we have run with this change for a year now, and given that
> there still have not been shown one single real-world application which
> gets into trouble by this, I am not inclined to loosen up the current
> behaviour

This bug report was the result of of investigating why a real-world
program's graceful handling of malloc() failures was broken:

%%%
$ MALLOC_OPTIONS=A grep foo /dev/zero
grep in realloc(): error: allocation failed

/: write failed, filesystem is full
Abort trap
$ # before the fix, with user bde and group wheel (no change):
$ MALLOC_OPTIONS=a grep foo /dev/zero
grep in realloc(): error: allocation failed

/: write failed, filesystem is full
Abort trap
$ # after the fix, with user bde and group wheel:
$ MALLOC_OPTIONS=a grep foo /dev/zero
grep: memory exhausted
%%%

Grepping /dev/zero is not very useful, but it is real-world.  The
real-world generates lots of input that can cause problems.

> Seen in hindsight, I should have made 'A' the overall default and
> 'a' the optional mode, because the reason for the existence of the
> choice in the first place was to give users a hand when they try
> to survive with badly written third-party software.
>
> In fact, given the almost universal absense of third-party binary
> software, and the hostility and sofistication of current attacks
> on security, I am almost convinced that the correct thing to do is
> to discontinue 'a' entirely and make 'A' mandatory for all programs.
>
> Any objections to that ?

Of course.  There must be a way to get standard, useful behaviour.

Bruce