Re: r248583 Kernel panic: negative refcount 0xfffffe0031b59168

I'm seeing what I believe is related panic, reliably being generated by
the Python regression test suite on a newly created FreeBSD 10-CURRENT
buildbot.

Symptoms first seen in an freebsd.org FTP snapshot dated "Thu May 30
20:01:46 UTC 2013" and also reproducible on a freshly updated r252400

It is additionally reproducible after checking out pure upstream python
sources, using the following steps:

hg clone http://hg.python.org/cpython
cd cpython && configure && make buildbottest

An interesting possible correlation is that it seems to drop out
during/around "test_socket"

Backtrace below:

> koobs_at_10-CURRENT-amd64:~ % sudo kgdb /boot/kernel/kernel /var/crash/vmcore.1
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and you are
> welcome to change it and/or distribute copies of it under certain conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for details.
> This GDB was configured as "amd64-marcel-freebsd"...
> 
> Unread portion of the kernel message buffer:
> panic: negative refcount 0xfffffe0009316de8
> cpuid = 0
> KDB: enter: panic
> 
> Reading symbols from /boot/kernel/zfs.ko.symbols...done.
> Loaded symbols for /boot/kernel/zfs.ko.symbols
> Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
> Loaded symbols for /boot/kernel/opensolaris.ko.symbols
> #0  doadump (textdump=114425856) at pcpu.h:236
> 236     pcpu.h: No such file or directory.
>         in pcpu.h
> (kgdb) bt
> #0  doadump (textdump=114425856) at pcpu.h:236
> #1  0xffffffff80338a75 in db_fncall (dummy1=<value optimized out>, dummy2=<value optimized out>, dummy3=<value optimized out>, dummy4=<value optimized out>)
>     at /usr/src/sys/ddb/db_command.c:578
> #2  0xffffffff8033875d in db_command (cmd_table=<value optimized out>) at /usr/src/sys/ddb/db_command.c:449
> #3  0xffffffff803384d4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:502
> #4  0xffffffff8033ae80 in db_trap (type=<value optimized out>, code=0) at /usr/src/sys/ddb/db_main.c:231
> #5  0xffffffff808ec1b3 in kdb_trap (type=3, code=0, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
> #6  0xffffffff80c4cadb in trap (frame=0xffffff80978a7880) at /usr/src/sys/amd64/amd64/trap.c:579
> #7  0xffffffff80c35ca2 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
> #8  0xffffffff808eb98e in kdb_enter (why=0xffffffff80f10487 "panic", msg=<value optimized out>) at cpufunc.h:63
> #9  0xffffffff808b5a76 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:747
> #10 0xffffffff808b5926 in kassert_panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:642
> #11 0xffffffff80875b4f in closef (fp=<value optimized out>, td=<value optimized out>) at refcount.h:66
> #12 0xffffffff80873890 in closefp (fdp=0xfffffe0009652000, fd=<value optimized out>, fp=0xfffffe0009316dc0, td=0xfffffe0009337490, holdleaders=<value optimized out>)
>     at /usr/src/sys/kern/kern_descrip.c:1140
> #13 0xffffffff80c4d725 in amd64_syscall (td=0xfffffe0009337490, traced=0) at subr_syscall.c:134
> #14 0xffffffff80c35f8b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:391
> #15 0x000000080119b00a in ?? ()
> Previous frame inner to this frame (corrupt stack?)
> Current language:  auto; currently minimal
> (kgdb)

--
koobs