Re: panic: sleeping thread owns a non-sleepable lock

On Wednesday 15 August 2007, Rong-en Fan wrote:
> On 8/12/07, Max Laier <max_at_love2party.net> wrote:
> > On Saturday 11 August 2007, Kris Kennaway wrote:
> > > On Sun, Aug 12, 2007 at 02:22:35AM +0800, Rong-en Fan wrote:
> > > > I'm running 7.0-CURRENT as of  yesterday, and it's very easy
> > > > to make it panic:
> > > >
> > > > Sleeping thread (tid 100065, pid 1066) owns a non-sleepable lock
> > > > sched_switch(c50a1600,0,1,1c7a7e4,4217e5,...) at
> > > > sched_switch+0x190 mi_switch(1,0) at mi_switch+0x13f
> > > > sleepq_switch(c50a1600,0,c078a4e2,21b,c07e3820,...) at
> > > > sleepq_switch+0x87 sleepq_wait(c07e3820,0,c0770b7e,3,0,...) at
> > > > sleepq_wait+0x36 _sx_xlock_hard(c07e3820,c50a1600,0,0,0,...) at
> > > > _sx_xlock_hard+0x21d
> > > > fr_checknatout(f9c7a8d0,f9c7a8cc,64,c57ad900,c4de7400,...) at
> > > > fr_checknatout+0x29d
> > > > fr_check(c8cc4644,14,c4de7400,1,f9c7a9b4,...) at fr_check+0x9b1
> > > > fr_check_wrapper(0,f9c7a9b4,c4de7400,2,c54dab28,...) at
> > > > fr_check_wrapper+0x3f
> > > > pfil_run_hooks(c08057c0,f9c7aa4c,c4de7400,2,c54dab28,...) at
> > > > pfil_run_hooks+0x74 ip_output(c8cc4600,0,f9c7aa10,0,0,...) at
> > > > ip_output+0x913
> > > > tcp_output(cae322d0,cb277200,0,0,0,...) at tcp_output+0x1106
> > > > tcp_usr_send(c51e7318,0,cb277200,0,0,...) at tcp_usr_send+0x240
> > > > kern_sendfile(c50a1600,f9c7acfc,0,0,0,...) at
> > > > kern_sendfile+0x1037
> > > > sendfile(c50a1600,f9c7acfc,20,16,f9c7ad2c,...) at sendfile+0xa8
> > > > syscall(f9c7ad38) at syscall+0x315
> > > > Xint0x80_syscall() at Xint0x80_syscall+0x20
> > > > --- syscall (393, FreeBSD ELF32, sendfile), eip = 0x28290bff, esp
> > > > = 0xbfbfc6ac, ebp = 0xbfbfe718 ---
> > >
> > > What is the lock it holds, and where is it acquired?
> >
> > My bet is on the pfil rwlock - accquired in pfil_run_hooks and
> > tcbinfo / inp mtxs from tcp_output.  Nothing in the transmission path
> > must use sx locks.  I keep on telling that.
>
> I compiled kernel with WITNESS and INVARIANTS as in GENERIC.
> After boot, without doing anything (I even set ipnat_enable="NO", i.e.
> only ipfw is running, but ipf/ipnat is compiled in). I got panic.
> I don't have serial console attached, but the console screen is at
>
> http://www.rafan.org/tmp/pfil_panic.jpg
>
> You can find 'show locks', 'show allpcpu', and 'show lock' for each
> lock. Also two trace for thread that are found in 'show allpcpu'.
>
> If necessary, I can setup serial console, but it will take few days.

The problem is understood and has been reported to the ipfilter 
maintainer.  One sollution might be to use rwlocks instead, but if there 
is a path that sleeps with the lock held another sollution might be 
required.  Most likely the ioctl path (on copyin/copyout) exercises that 
problem.

-- 
/"\  Best regards,                      | mlaier_at_freebsd.org
\ /  Max Laier                          | ICQ #67774661
 X   http://pf4freebsd.love2party.net/  | mlaier_at_EFnet
/ \  ASCII Ribbon Campaign              | Against HTML Mail and News