Re: crash when bpf is used heavily

On Fri, 28 May 2004, Sergey Lyubka wrote:

> > bump up your kernel virtual memory - see the handbook looking for the
> > "KVA" setting.
> 
> Yeah, did that.  I had crashes when 20 simultaneous BPF apps where
> running.  Now it crashes at 30. 
> 
> Guys, these are not the solutions. vm allocator should check for memory
> availability, imho, of course. I see it as a seriuos bug in -current. 
> Such unstable kernel no way can be put on production environment. 

Does seem like a non-ideal failure mode, doesn't it...

It looks like the BPF code is written to handle the case where allocation
fails, but that it passes flags to the memory allocator that prevent the
memory allocator from returning a failure.  Specifically,
src/sys/net/bpf.c:bpf_allocbufs() passes M_WAITOK into malloc().  Try
changing that flag (in both instances) to M_NOWAIT.  This will still
permit BPF to consume large quantities of memory if the maximum buffer
size is set so large, but it will cause BPF itself not to cause a direct
panic if address space is exhausted.  I'm a little surprised M_NOWAIT
isn't already the setting there, actually. 

The system will still be running in a low address space scenario which
might cause other parts of the system to bump into the failure, however.
Unfortunately, balancing multiple consumers of address space is a "hard
problem".  With the mbuf allocator, we make use of a separate address
space map of bounded size to prevent the total address space consumed by
packet buffers exceeding a certain size.  It might be interesting to
experiment with allocating BPF space from the same map, as it would change
the trade-off from "panic if there's no room"  to "stall the network stack
if there's no room".  The other common solution is to use smaller buffers,
making the trade-off become "If the packets come too fast, we drop them".
I realize that is the problem you're trying to solve... :-)  On systems
I've worked with that need to do processing of many high speed packet
streams, we've generally tried to combine all the processing into modules
in a single process, as this has a number of benefits:

(1) It avoids storing the same packet many times in many buffers for
    different consumers.

(2) It reduces the over-all memory overhead of buffering in the kernel.

(3) It reduces the number of memory copies by avoiding copying the same
    packet many times (in particular, between user and kernel space)

(4) It avoids performing additional context switches during high speed
    tracing, which can substantially impact available CPU resources for
    packet copying and monitoring.

I realize, of course, that that approach cannot apply in all environments.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research