Re: Panic on head (r255759) [_callout_stop_safe()-> panic: Lock lle not exclusively locked @ /usr/src/sys/kern/kern_rwlock.c:140]

On Sat, 21 Sep 2013 09:34:59 -0700
Davide Italiano <davide_at_freebsd.org> wrote:

> On Sat, Sep 21, 2013 at 9:31 AM, Bryan Drewery <bdrewery_at_freebsd.org>
> wrote:
> > On 9/21/2013 11:18 AM, Adam McDougall wrote:
> >> On 09/21/13 09:41, Davide Italiano wrote:
> >>> On Sat, Sep 21, 2013 at 2:51 PM, O. Hartmann
> >>> <ohartman_at_zedat.fu-berlin.de> wrote:
> >>>> On Sat, 21 Sep 2013 07:08:25 -0500
> >>>> Bryan Drewery <bdrewery_at_FreeBSD.org> wrote:
> >>>>
> >>>>> On 9/21/2013 7:06 AM, Bjoern A. Zeeb wrote:
> >>>>>> On Sat, 21 Sep 2013, Bryan Drewery wrote:
> >>>>>>
> >>>>>>>> Unread portion of the kernel message buffer:
> >>>>>>>> panic: Lock lle not exclusively locked _at_
> >>>>>>>> /usr/src/sys/kern/kern_rwlock.c:140
> >>>>>>>>
> >>>>>>>> cpuid = 0
> >>>>>>>> KDB: stack backtrace:
> >>>>>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> >>>>>>>> 0xfffffe118aeef820
> >>>>>>>> kdb_backtrace() at kdb_backtrace+0x39/frame
> >>>>>>>> 0xfffffe118aeef8d0 vpanic() at vpanic+0x126/frame
> >>>>>>>> 0xfffffe118aeef910 panic() at panic+0x43/frame
> >>>>>>>> 0xfffffe118aeef970 __rw_assert() at __rw_assert+0xa3/frame
> >>>>>>>> 0xfffffe118aeef980 _callout_stop_safe() at
> >>>>>>>> _callout_stop_safe+0x54/frame 0xfffffe118aeef9f0 arptimer()
> >>>>>>>> at arptimer+0x14e/frame 0xfffffe118aeefa30
> >>>>>>>> softclock_call_cc() at softclock_call_cc+0x188/frame
> >>>>>>>> 0xfffffe118aeefb10 softclock() at softclock+0x47/frame
> >>>>>>>> 0xfffffe118aeefb30 intr_event_execute_handlers() at
> >>>>>>>> intr_event_execute_handlers+0x93/frame 0xfffffe118aeefb70
> >>>>>>>> ithread_loop() at ithread_loop+0xa6/frame 0xfffffe118aeefbb0
> >>>>>>>> fork_exit() at fork_exit+0x84/frame 0xfffffe118aeefbf0
> >>>>>>>> fork_trampoline() at fork_trampoline+0xe/frame
> >>>>>>>> 0xfffffe118aeefbf0 --- trap 0, rip = 0, rsp =
> >>>>>>>> 0xfffffe118aeefcb0, rbp = 0 ---
> >>>>>>
> >>>>>> +1 from me;  I  guess introduced somwhere between 255569 and
> >>>>>> 255758, as these are my edges of kernel.old and kernel.
> >>>>>>
> >>>>> r255726 was stable for me. r255759 is not.
> >>>>>
> >>>>> r255755 converted ipfilter to callout, but I am unsure if that
> >>>>> is the problem.
> >>>>>
> >>>> r255729 is also stable for me - I'm with r255729 again, since
> >>>> r255757 crashed.
> >>> Let me know if this fixes the problem for you:
> >>> http://people.freebsd.org/~davide/review/lc_calloutfix.diff
> >>>
> >>> Thanks,
> >>>
> >> Worked for me so far. I generally couldn't stay up more than 30
> >> minutes before the patch and now my uptime is 90 minutes. Thanks!
> >
> > Same here.
> >
> >
> > --
> > Regards,
> > Bryan Drewery
> >
> 
> I would wait another couple of hours before the commit, but still I'm
> confident this fixed the problem.
> 

At the moment all systems are under load and duty, I can patch earliest
tonight and report tomorrow after the boxes stayed up overnight.