Re: FreeBSD 7-STABLE mbuf corruption

Hi,

[added -current_at_ to the CC list, as the issue is still present in 9.0-BETA2]

On Wed, Sep 7, 2011 at 7:19 PM, Arnaud Lacombe <lacombar_at_gmail.com> wrote:
> Hi,
>
> On Mon, Sep 5, 2011 at 2:59 AM, Arnaud Lacombe <lacombar_at_gmail.com> wrote:
>> Hi folks,
>>
>> We have been trying to track down a bad mbuf management for about two
>> weeks on a customized 7.1 base. I have finally been able to reproduce
>> it with a stock FreeBSD 7-STABLE (kernel from r225276, userland from
>> 7.4).
>>
>> With the help of the attached patches, I have just been able to
>> trigger the following panic:
>>
>> panic: Corrupted unused flags, expected 0xffffffff00000000, got 0x0, flags 0x3
>> cpuid = 1
>> Uptime: 3d10h5m3s
>> Cannot dump. No dump device defined
>>
> General form of the crash is:
>
> panic: Corrupted unused flags, expected 0xffffffff00000000, got
> 0xbabe0000000000, flags 0xbabe0000babe00
> cpuid = 0
> KDB: stack backtrace:
> db_trace_self_wrapper(c0874e29,0,c0835757,f4574c48,0,...) at
> db_trace_self_wrapper+0x26
> panic(c0835757,0,ffffffff,0,babe00,...) at panic+0x10b
> igb_txeof(c6a25008,0,c0837083,5ea,17c,...) at igb_txeof+0x399
> igb_msix_que(c6a2b800,0,c084d367,4b6,c69dd068,...) at igb_msix_que+0x7b
> ithread_loop(c6a29090,f4574d38,c084d0db,31c,c6a16828,...) at ithread_loop+0xc3
> fork_exit(c061d520,c6a29090,f4574d38) at fork_exit+0xa6
> fork_trampoline() at fork_trampoline+0x8
> --- trap 0, eip = 0, esp = 0xf4574d70, ebp = 0 ---
> Uptime: 1m42s
>
I converted igb(4) to use the legacy if_start() logic and triggered
the following panic on the latest FreeBSD 9.0-BETA2:

panic: Corrupted mbuf tainting, expected 0xffff, got 0xaabb, taint 0xaabb
cpuid = 6
KDB: enter: panic
[ thread pid 0 tid 100045 ]
Stopped at      kdb_enter+0x3b: movl    $0,kdb_why
db> bt
Tracing pid 0 tid 100045 td 0xc6bd52e0
kdb_enter(c081831c,c081831c,c08026c1,c673ec28,6,...) at kdb_enter+0x3b
panic(c08026c1,ffff,aabb,aabb,c6bd1400,...) at panic+0x103
igb_txeof(c6bd1408,0,c080411c,558,c6bd1408,...) at igb_txeof+0x318
igb_handle_que(c6bac400,1,c081e508,130,c673ecb0,...) at igb_handle_que+0xae
taskqueue_run_locked(c6bdc400,c6bdc418,0,c080a966,0,...) at
taskqueue_run_locked+0xa3
taskqueue_thread_loop(c6bac430,c673ed28,c0812d90,3f9,0,...) at
taskqueue_thread_loop+0x4d
fork_exit(c063ea10,c6bac430,c673ed28) at fork_exit+0xa4
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xc673ed60, ebp = 0 ---

for those who have not followed the thread on -net, the same mbuf is
queued twice in the interface queue, transmitted twice... and freed
twice. Of course, after having been released first, it ends up
eventually in a socket buffer, and when it gets released the second
time, it triggers all kind of funny panic() and crashes.

The 0xaabb pattern comes from memory tainting with INVARIANTS at the
ends of m_free().

I can provide the patches I am testing with.

 - Arnaud

> It happens particularly easily when the box receives wall of SYN
> (about 1000 cnx attempts at once) every 5s or so.
>
>  - Arnaud
>
>>
>> [cut stuff no one cares about...]
>