Re: Panic on head (r255759) [_callout_stop_safe()-> panic: Lock lle not exclusively locked @ /usr/src/sys/kern/kern_rwlock.c:140]

On Sat, 21 Sep 2013 09:34:59 -0700
Davide Italiano <davide_at_freebsd.org> wrote:

> On Sat, Sep 21, 2013 at 9:31 AM, Bryan Drewery <bdrewery_at_freebsd.org>
> wrote:
> > On 9/21/2013 11:18 AM, Adam McDougall wrote:
> >> On 09/21/13 09:41, Davide Italiano wrote:
> >>> On Sat, Sep 21, 2013 at 2:51 PM, O. Hartmann
> >>> <ohartman_at_zedat.fu-berlin.de> wrote:
> >>>> On Sat, 21 Sep 2013 07:08:25 -0500
> >>>> Bryan Drewery <bdrewery_at_FreeBSD.org> wrote:
> >>>>
> >>>>> On 9/21/2013 7:06 AM, Bjoern A. Zeeb wrote:
> >>>>>> On Sat, 21 Sep 2013, Bryan Drewery wrote:
> >>>>>>
> >>>>>>>> Unread portion of the kernel message buffer:
> >>>>>>>> panic: Lock lle not exclusively locked _at_
> >>>>>>>> /usr/src/sys/kern/kern_rwlock.c:140
> >>>>>>>>
> >>>>>>>> cpuid = 0
> >>>>>>>> KDB: stack backtrace:
> >>>>>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> >>>>>>>> 0xfffffe118aeef820
> >>>>>>>> kdb_backtrace() at kdb_backtrace+0x39/frame
> >>>>>>>> 0xfffffe118aeef8d0 vpanic() at vpanic+0x126/frame
> >>>>>>>> 0xfffffe118aeef910 panic() at panic+0x43/frame
> >>>>>>>> 0xfffffe118aeef970 __rw_assert() at __rw_assert+0xa3/frame
> >>>>>>>> 0xfffffe118aeef980 _callout_stop_safe() at
> >>>>>>>> _callout_stop_safe+0x54/frame 0xfffffe118aeef9f0 arptimer()
> >>>>>>>> at arptimer+0x14e/frame 0xfffffe118aeefa30
> >>>>>>>> softclock_call_cc() at softclock_call_cc+0x188/frame
> >>>>>>>> 0xfffffe118aeefb10 softclock() at softclock+0x47/frame
> >>>>>>>> 0xfffffe118aeefb30 intr_event_execute_handlers() at
> >>>>>>>> intr_event_execute_handlers+0x93/frame 0xfffffe118aeefb70
> >>>>>>>> ithread_loop() at ithread_loop+0xa6/frame 0xfffffe118aeefbb0
> >>>>>>>> fork_exit() at fork_exit+0x84/frame 0xfffffe118aeefbf0
> >>>>>>>> fork_trampoline() at fork_trampoline+0xe/frame
> >>>>>>>> 0xfffffe118aeefbf0 --- trap 0, rip = 0, rsp =
> >>>>>>>> 0xfffffe118aeefcb0, rbp = 0 ---
> >>>>>>
> >>>>>> +1 from me;  I  guess introduced somwhere between 255569 and
> >>>>>> 255758, as these are my edges of kernel.old and kernel.
> >>>>>>
> >>>>> r255726 was stable for me. r255759 is not.
> >>>>>
> >>>>> r255755 converted ipfilter to callout, but I am unsure if that
> >>>>> is the problem.
> >>>>>
> >>>> r255729 is also stable for me - I'm with r255729 again, since
> >>>> r255757 crashed.
> >>> Let me know if this fixes the problem for you:
> >>> http://people.freebsd.org/~davide/review/lc_calloutfix.diff
> >>>
> >>> Thanks,
> >>>
> >> Worked for me so far. I generally couldn't stay up more than 30
> >> minutes before the patch and now my uptime is 90 minutes. Thanks!
> >
> > Same here.
> >
> >
> > --
> > Regards,
> > Bryan Drewery
> >
> 
> I would wait another couple of hours before the commit, but still I'm
> confident this fixed the problem.
> 

I hadn't enough time to let the systems in question run overnight, but
from this morning, one of the boxes has been patched and is now under
heavy load (buildworld and some other nasty stuff I artificially put
onto the box, i.e. some numerical calculations).

At least, the system lasted the buildworl for now over 35 minutes and
it crashed before the patch after a minute or so under load.

I put the patch now onto the second system in row and check whether the
stability is the same on another CPU generation as well.

Thanks for the fast response.

Regards,
Oliver