panic: bremfree: removing a buffer not on a queue

My bitty box just crashed out while doing some light desktop work and a
small amount of NFS server stuff.

The source was cvsup'd a few minutes before the kernel was compiled last
Sunday.  The contents of dmesg.boot are include below.

If it's of any use, the machine crashed badly the day before yesterday
during the middle of a background fsck, and a whole pile of unexpected
soft update errors popped up.  Most of the files reappeared in
lost+found, but a few went to swim with the fishes.

Nick

pancake# uname -a
FreeBSD pancake.netability.ie 5.0-CURRENT FreeBSD 5.0-CURRENT #0: Sun Mar 23 11:50:37 GMT 2003     nick_at_pancake.netability.ie:/opt/usr.obj.pancake/usr/src/sys/PANCAKE  i386
pancake# gdb -k kernel.debug.0 vmcore.0
GNU gdb 5.2.1 (FreeBSD)
Copyright 2002 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-undermydesk-freebsd"...
panic: bremfree: removing a buffer not on a queue
panic messages:
---
panic: softdep_disk_io_initiation: read
cpuid = 1; lapic.id = 01000000
boot() called on cpu#1

syncing disks, buffers remaining... panic: bremfree: removing a buffer not on a queue
cpuid = 1; lapic.id = 01000000
boot() called on cpu#1
Uptime: 1d22h51m8s
Dumping 255 MB
ata0: resetting devices ..
done
 16 32 48 64 80 96 112 128 144 160 176 192 208 224 240
---
Reading symbols from /boot/kernel/snd_pcm.ko...done.
Loaded symbols for /boot/kernel/snd_pcm.ko
Reading symbols from /boot/kernel/snd_ich.ko...done.
Loaded symbols for /boot/kernel/snd_ich.ko
Reading symbols from /boot/kernel/nvidia.ko...done.
Loaded symbols for /boot/kernel/nvidia.ko
Reading symbols from /boot/kernel/linux.ko...done.
Loaded symbols for /boot/kernel/linux.ko
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
Reading symbols from /boot/kernel/usb.ko...done.
Loaded symbols for /boot/kernel/usb.ko
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:239
239             dumping++;
(kgdb) bt full
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:239
No locals.
#1  0xc01d0c22 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:371
No locals.
#2  0xc01d0f70 in poweroff_wait (junk=0xc035da3f, howto=1)
    at /usr/src/sys/kern/kern_shutdown.c:542
        td = (struct thread *) 0xc0eca1e0
        bootopt = 260
        buf = "bremfree: removing a buffer not on a queue", '\0' <repeats 213 times>
#3  0xc0215800 in bremfreel (bp=0xc035da3f) at /usr/src/sys/kern/vfs_bio.c:636
        old_qindex = 260
#4  0xc0215713 in bremfree (bp=0x0) at /usr/src/sys/kern/vfs_bio.c:618
No locals.
#5  0xc02177cd in vfs_bio_awrite (bp=0x104) at /usr/src/sys/kern/vfs_bio.c:1687
        i = 1
        j = 0
        lblkno = 11
        vp = (struct vnode *) 0xc0eca1e0
        ncl = -1070212545
        nwritten = -1070212545
        size = 8192
        maxcl = 16
#6  0xc02c6ef2 in ffs_fsync (ap=0xcd2079cc) at /usr/src/sys/ufs/ffs/ffs_vnops.c:255
---Type <return> to continue, or q <return> to quit---
        vp = (struct vnode *) 0xc3458490
        ip = (struct inode *) 0xc7729440
        bp = (struct buf *) 0xc7729440
        nbp = (struct buf *) 0xc7749470
        error = 0
        wait = 0
        passes = 4
        skipmeta = 0
        lbn = 14
#7  0xc02c5ff9 in ffs_sync (mp=0xc2657c00, waitfor=2, cred=0xc0eb5e80, td=0xc037e5c0)
    at vnode_if.h:612
        nvp = (struct vnode *) 0xc31fea44
        vp = (struct vnode *) 0xc3458490
        devvp = (struct vnode *) 0xc3458490
        ip = (struct inode *) 0x0
        ump = (struct ufsmount *) 0xc2752a00
        fs = (struct fs *) 0xc25e6800
        error = 0
        count = -853509684
        wait = 0
        lockreq = 18
        allerror = 0
#8  0xc022c83b in sync (td=0xc037e5c0, uap=0x0) at /usr/src/sys/kern/vfs_syscalls.c:138
---Type <return> to continue, or q <return> to quit---
        mp = (struct mount *) 0xc2657c00
        nmp = (struct mount *) 0x0
        asyncflag = 0
#9  0xc01d0747 in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:280
        bp = (struct buf *) 0xc0ec7080
        iter = 4
        nbusy = 0
        pbusy = 1
        subiter = 0
#10 0xc01d0f70 in poweroff_wait (junk=0xc0366096, howto=-1071556845)
    at /usr/src/sys/kern/kern_shutdown.c:542
        td = (struct thread *) 0xc0eca1e0
        bootopt = 256
        buf = "bremfree: removing a buffer not on a queue", '\0' <repeats 213 times>
#11 0xc02bf38c in softdep_disk_io_initiation (bp=0xc0eca1e0)
    at /usr/src/sys/ufs/ffs/ffs_softdep.c:3465
        wk = (struct worklist *) 0xc0366096
        nextwk = (struct worklist *) 0xc775fba8
        indirdep = (struct indirdep *) 0x100
        inodedep = (struct inodedep *) 0x0
#12 0xc021e43d in cluster_wbuild (vp=0xc3458490, size=8192, start_lbn=11, len=2)
    at buf.h:422
        bp = (struct buf *) 0xc76b2fe8
---Type <return> to continue, or q <return> to quit---
        tbp = (struct buf *) 0xc7729440
        i = 1
        j = 2
        totalwritten = 0
        dbsize = 16
#13 0xc02177c3 in vfs_bio_awrite (bp=0xc771adf0) at /usr/src/sys/kern/vfs_bio.c:1681
        i = 2
        j = 0
        lblkno = 10
        vp = (struct vnode *) 0xc3458490
        ncl = 2
        nwritten = 2
        size = 8192
        maxcl = 16
#14 0xc02c6ef2 in ffs_fsync (ap=0xcd207c18) at /usr/src/sys/ufs/ffs/ffs_vnops.c:255
        vp = (struct vnode *) 0xc3458490
        ip = (struct inode *) 0xc771adf0
        bp = (struct buf *) 0xc771adf0
        nbp = (struct buf *) 0xc7729440
        error = 0
        wait = 0
        passes = 4
        skipmeta = 0
---Type <return> to continue, or q <return> to quit---
        lbn = 14
#15 0xc02c5ff9 in ffs_sync (mp=0xc2657c00, waitfor=3, cred=0xc0eb5e80, td=0xc0eca1e0)
    at vnode_if.h:612
        nvp = (struct vnode *) 0xc31fea44
        vp = (struct vnode *) 0xc3458490
        devvp = (struct vnode *) 0xc3458490
        ip = (struct inode *) 0x0
        ump = (struct ufsmount *) 0xc2752a00
        fs = (struct fs *) 0xc25e6800
        error = 0
        count = -1071894485
        wait = 0
        lockreq = 18
        allerror = 0
#16 0xc022bfda in sync_fsync (ap=0xcd207cc4) at /usr/src/sys/kern/vfs_subr.c:3493
        syncvp = (struct vnode *) 0x0
        mp = (struct mount *) 0xc2657c00
        td = (struct thread *) 0xc0eca1e0
        error = -853508924
        asyncflag = 0
#17 0xc022829b in sched_sync () at vnode_if.h:612
        slp = (struct synclist *) 0xcd207cc4
        vp = (struct vnode *) 0x0
---Type <return> to continue, or q <return> to quit---
        mp = (struct mount *) 0x0
        starttime = 1048769029
        td = (struct thread *) 0xc0eca1e0
#18 0xc01bb7e3 in fork_exit (callout=0xc02280c0 <sched_sync>, arg=0x0, frame=0x0)
    at /usr/src/sys/kern/kern_fork.c:875
        td = (struct thread *) 0x0
        p = (struct proc *) 0xc25e7200
(kgdb) 

pancake# cat /var/run/dmesg.boot
Copyright (c) 1992-2003 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD 5.0-CURRENT #0: Sun Mar 23 11:50:37 GMT 2003
    nick_at_pancake.netability.ie:/opt/usr.obj.pancake/usr/src/sys/PANCAKE
Preloaded elf kernel "/boot/kernel/kernel" at 0xc0605000.
Preloaded elf module "/boot/kernel/snd_pcm.ko" at 0xc06050a8.
Preloaded elf module "/boot/kernel/snd_ich.ko" at 0xc0605154.
Preloaded elf module "/boot/kernel/nvidia.ko" at 0xc0605200.
Preloaded elf module "/boot/kernel/linux.ko" at 0xc06052ac.
Preloaded elf module "/boot/kernel/acpi.ko" at 0xc0605358.
Timecounter "i8254"  frequency 1193182 Hz
CPU: Intel Pentium III (728.44-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x683  Stepping = 3
  Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 268034048 (255 MB)
avail memory = 253800448 (242 MB)
Changing APIC ID for IO APIC #0 from 0 to 2 on chip
Programming 24 pins in IOAPIC #0
IOAPIC #0 intpin 2 -> irq 0
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
 cpu0 (BSP): apic id:  0, version: 0x00040011, at 0xfee00000
 cpu1 (AP):  apic id:  1, version: 0x00040011, at 0xfee00000
 io0 (APIC): apic id:  2, version: 0x00170020, at 0xfec00000
Allocating major#253 to "net"
Allocating major#252 to "pci"
Pentium Pro MTRR support enabled
npx0: <math processor> on motherboard
npx0: INT 16 interface
acpi0: <DELL   WS 220 > on motherboard
    ACPI-0625: *** Info: GPE Block0 defined as GPE0 to GPE15
pcibios: BIOS version 2.10
Using $PIR table, 9 entries at 0xc00fbbc0
acpi0: power button is handled as a fixed feature programming model.
Timecounter "ACPI-fast"  frequency 3579545 Hz
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
IOAPIC #0 intpin 19 -> irq 2
IOAPIC #0 intpin 17 -> irq 13
agp0: <Intel 82820 host to AGP bridge> mem 0xf0000000-0xf3ffffff at device 0.0 on pci0
pcib1: <PCIBIOS PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
nvidia0: <RIVA TNT2 Model 64> mem 0xf4000000-0xf5ffffff,0xfc000000-0xfcffffff irq 16 at device 0.0 on pci1
pcib2: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci2: <ACPI PCI bus> on pcib2
IOAPIC #0 intpin 18 -> irq 17
ahc0: <Adaptec 2940 SCSI adapter> port 0xec00-0xecff mem 0xfafff000-0xfaffffff irq 16 at device 11.0 on pci2
aic7870: Single Channel A, SCSI Id=7, 16/253 SCBs
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xe880-0xe8ff mem 0xfaffec00-0xfaffec7f irq 17 at device 12.0 on pci2
xl0: Ethernet address: 00:b0:d0:a1:12:07
miibus0: <MII bus> on xl0
xlphy0: <3c905C 10/100 internal PHY> on miibus0
xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH UDMA66 controller> port 0xffa0-0xffaf at device 31.1 on pci0
ata0: at 0x1f0 irq 14 on atapci0
ata1: at 0x170 irq 15 on atapci0
pci0: <serial bus, USB> at device 31.2 (no driver attached)
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pcm0: <Intel 82801AA (ICH)> port 0xdc80-0xdcbf,0xd800-0xd8ff irq 13 at device 31.5 on pci0
pcm0: <Analog Devices AD1881 AC97 Codec>
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
Allocating major#251 to "devstat"
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
kbd0 at atkbd0
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: model IntelliMouse, device ID 3
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
ppc0 port 0x778-0x77f,0x378-0x37f irq 7 on acpi0
ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
ppc0: FIFO with 16/16/8 bytes threshold
pmtimer0 on isa0
orm0: <Option ROMs> at iomem 0xcb800-0xcbfff,0xc9000-0xcb7ff,0xc0000-0xc8fff on isa0
sc0: <System console> on isa0
sc0: VGA <16 virtual consoles, flags=0x200>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
APIC_IO: Testing 8254 interrupt delivery
APIC_IO: routing 8254 via IOAPIC #0 intpin 2
Timecounters tick every 4.000 msec
ad0: 39082MB <Maxtor 54098H8> [79406/16/63] at ata0-master UDMA66
acd0: CD-RW <PLEXTOR CD-R PX-W4824A> at ata1-master PIO4
Waiting 2 seconds for SCSI devices to settle
pcm0: measured ac97 link rate at 682666666 Hz
SMP: AP CPU #1 Launched!
da0 at ahc0 bus 0 target 0 lun 0
da0: <SEAGATE ST32155N 0532> Fixed Direct Access SCSI-2 device
da0: 10.000MB/s transfers (10.000MHz, offset 15), Tagged Queueing Enabled
da0: 2049MB (4197405 512 byte sectors: 255H 63S/T 261C)
Mounting root from ufs:/dev/ad0s1a