Re: lots of "exclusive sleep mutex"

On 2003/10/31 10:09:04, YONETANI Tomokazu wrote:
> On 2003/10/30 00:16:47, Clive Lin wrote:
> > On Sat, Oct 04, 2003 at 02:00:33AM +0800, Clive Lin wrote:
> > > Hi,
> > > 
> > >     I've seen lots of messages on rescent -CURRENT
> > > 
> > > malloc() of "16" with the following non-sleepable locks held:
> > > exclusive sleep mutex g_xdown r = 0 (0xe044eca8) locked _at_ /usr/src/sys/geom/geom_io.c:351
> > > malloc() of "16" with the following non-sleepable locks held:
> > > exclusive sleep mutex g_xdown r = 0 (0xe044eca8) locked _at_ /usr/src/sys/geom/geom_io.c:351
> > 
> > Many of above are still seen on the latest current.
> > malloc() of "16" with the following non-sleepable locks held:
> > exclusive sleep mutex g_xdown r = 0 (0xe044eca8) locked _at_ /usr/src/sys/geom/geom_io.c:355
> > malloc() of "16" with the following non-sleepable locks held:
> > exclusive sleep mutex g_xdown r = 0 (0xe044eca8) locked _at_ /usr/src/sys/geom/geom_io.c:355
> > 
> > Perhaps it's a ServeRAID specific glitch?
> > > dmesg|grep ips
> > ips0: <IBM ServeRAID Adapter> mem 0xf0000000-0xf3ffffff irq 16 at device 1.0 on pci4
> > ips0: logical drives: 1
> > ipsd0: <Logical Drive> on ips0
> > GEOM: create disk ipsd0 dp=0xc6b25310
> > ipsd0: Logical Drive  (69430MB)
> 
> Does this fix?
> 
> Index: ips.c
[snipped]

Of course it doesn't. Hmm, I didn't realize that I've removed WITNESS* from
the kernel config file months ago. Anyway, this is the backtrace of
the above warnings from witness code(by manually setting a breakpoint
inside witness_warning and issuing panic command from DDB):

Script started on Fri Oct 31 12:47:51 2003
# gdb -k /usr/obj/home/source/freebsd/current/src/sys/MAIL/kernel.debug vmcore.0
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: from debugger
panic messages:
---
---
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/usb.ko...done.
Loaded symbols for /boot/kernel/usb.ko
Reading symbols from /boot/kernel/nfsclient.ko...done.
Loaded symbols for /boot/kernel/nfsclient.ko
#0  doadump () at /home/source/freebsd/current/src/sys/kern/kern_shutdown.c:240
240		dumping++;
(kgdb) bt
#0  doadump () at /home/source/freebsd/current/src/sys/kern/kern_shutdown.c:240
#1  0xc0576346 in boot (howto=260)
    at /home/source/freebsd/current/src/sys/kern/kern_shutdown.c:372
#2  0xc0576677 in panic ()
    at /home/source/freebsd/current/src/sys/kern/kern_shutdown.c:550
#3  0xc045d1c2 in db_panic ()
    at /home/source/freebsd/current/src/sys/ddb/db_command.c:450
#4  0xc045d122 in db_command (last_cmdp=0xc073e460, cmd_table=0x0,
    aux_cmd_tablep=0xc0709cec, aux_cmd_tablep_end=0xc0709d04)
    at /home/source/freebsd/current/src/sys/ddb/db_command.c:346
#5  0xc045d265 in db_command_loop ()
    at /home/source/freebsd/current/src/sys/ddb/db_command.c:472
#6  0xc0460265 in db_trap (type=3, code=0)
    at /home/source/freebsd/current/src/sys/ddb/db_trap.c:73
#7  0xc069765c in kdb_trap (type=3, code=0, regs=0xd77e5b3c)
    at /home/source/freebsd/current/src/sys/i386/i386/db_interface.c:171
#8  0xc06a7fe8 in trap (frame=
      {tf_fs = -1065877480, tf_es = -1065877488, tf_ds = -679608304, tf_edi = -1065836412, tf_esi = -1065836416, tf_ebp = -679584872, tf_isp = -679584920, tf_ebx = 0, tf_edx = 0, tf_ecx = -1056882688, tf_eax = 19, tf_trapno = 3, tf_err = 0, tf_eip = -1067865189, tf_cs = 8, tf_eflags = 514, tf_esp = -1066450495, tf_ss = -679584852}) at /home/source/freebsd/current/src/sys/i386/i386/trap.c:582
#9  0xc0699048 in calltrap () at {standard input}:102
#10 0xc0674b34 in uma_zalloc_arg (zone=0xc078a080, udata=0x0, flags=0)
    at /home/source/freebsd/current/src/sys/vm/uma_core.c:1385
#11 0xc056bd05 in malloc (size=0, type=0xc07235c0, flags=0)
    at /home/source/freebsd/current/src/sys/vm/uma.h:234
#12 0xc04e5d44 in ips_add_waiting_command (sc=0xc479f000,
    callback=0xc04e8050 <ips_send_io_request>, data=0xc7ea9a18, flags=0)
    at /home/source/freebsd/current/src/sys/dev/ips/ips.c:147
#13 0xc04e5f1e in ips_get_free_cmd (sc=0x0,
    callback=0xc04e8050 <ips_send_io_request>, data=0xc7ea9a18, flags=0)
    at /home/source/freebsd/current/src/sys/dev/ips/ips.c:222
#14 0xc04e813e in ips_start_io_request (sc=0xc7ea9a18, iobuf=0x0)
    at /home/source/freebsd/current/src/sys/dev/ips/ips_commands.c:168
#15 0xc04e7b62 in ipsd_strategy (iobuf=0x0)
    at /home/source/freebsd/current/src/sys/dev/ips/ips_disk.c:100
#16 0xc0546036 in g_disk_start (bp=0xc8798110)
    at /home/source/freebsd/current/src/sys/geom/geom_disk.c:272
---Type <return> to continue, or q <return> to quit---
#17 0xc0547fcc in g_io_schedule_down (tp=0xc1d17000)
    at /home/source/freebsd/current/src/sys/geom/geom_io.c:356
#18 0xc05482b8 in g_down_procbody ()
    at /home/source/freebsd/current/src/sys/geom/geom_kern.c:113
#19 0xc0562a8e in fork_exit (callout=0xc0548290 <g_down_procbody>, arg=0x0,
    frame=0x0) at /home/source/freebsd/current/src/sys/kern/kern_fork.c:793


> By the way, does this panic your machine?
> 
> $ exec sh
> $ mkdir foo
> $ i=0; while :; do echo $i > foo/$i; i=$(($i+1)); done

and I found that this doesn't trigger the panic if I have WITNESS and
WITNESS_SKIPSPIN in my kenrel config file.
-- 
YONETANI Tomokazu / Ergo-Brains Inc.