Re: PANIC: vrele: negative ref cnt

On Fri, 13 Feb 2004, John Baldwin wrote:

> Yesterday I got the above panic and I think I have a guess about what prompts 
> the bug.  First:
> 
> ---
> panic: vrele: negative ref cnt
> at line 2015 in file ../../../kern/vfs_subr.c
> Debugger("panic")
> Uptime: 16h36m17s
> Dumping 1023 MB
>  16 32 48 64 80 96 112 128 144 160 176 192 208 224 240 256 272 288 304 320 336 
> 352 368 384 400 416 432 448 464 480 496 512 528 544 560 576 592 608 624 640 
> 656 672 688 704 720 736 752 768 784 800 816 832 848 864 880 896 912 928 944 
> 960 976 992 1008
> ---
> (kgdb) where
> #0  doadump () at ../../../kern/kern_shutdown.c:240
> #1  0xc0581e1c in boot (howto=16640) at ../../../kern/kern_shutdown.c:374
> #2  0xc0582186 in poweroff_wait (junk=0xc074b9b9, howto=2015)
>     at ../../../kern/kern_shutdown.c:552
> #3  0xc05d70a6 in vput (vp=0xdfcc3be8) at ../../../kern/vfs_subr.c:2015
> #4  0xc068712f in ffs_unmount (mp=0xc6332800, mntflags=524288, td=0xc2282540)
>     at ../../../ufs/ffs/ffs_vfsops.c:1002
> #5  0xc05d2bfe in dounmount (mp=0xc6332800, flags=524288, td=0xc2282540)
>     at ../../../kern/vfs_mount.c:1155
> #6  0xc05d89df in vfs_unmountall () at ../../../kern/vfs_subr.c:3002
> #7  0xc0581cc0 in boot (howto=16384) at ../../../kern/kern_shutdown.c:359
> #8  0xc05815ed in reboot (td=0x0, uap=0x0) 
> at ../../../kern/kern_shutdown.c:178
> 
> Now, just before I rebooted, I had a single ata I/O error during a package 
> install:
> 
> ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=40<UNCORRECTABLE> 
> LBA=15887823
> 
> Which resulted in an EIO out in userland. When I was working on the
> splitfs floppy stuff and was running into the 64k boundary bug which
> gave me truncated mfsroots, I would get this same panic when the kernel
> tried to mount the memory filesystem root.  I'm guessing the kernel had
> the equivalent of an EIO during that mount operation.  Robert Watson
> (cc'd) also just had this panic and I know he's been having problems
> with ata(4) and I/O errors, so I think this panic is the result of some
> bogus handling of I/O errors where the vnode is being double vrele()'d
> somewhere. 

The vrele() panic is indeed the same as the one I just ran into on my Dell
C600 notebook, which has been having persisting ATA problems for the last
six months.  I was running from battery with X Windows, and doing a fairly
intensive set of builds, and got the following console messages:

ad0: TIMEOUT - READ_DMA retrying (2 retries left) LBA=25072055
ad0: TIMEOUT - WRITE_DMA retrying (2 retries left) LBA=2842751
pid 16552 (ld), uid 1000 inumber 931101 on /usr: filesystem full
ad0: FAILURE - WRITE_DMA status=51<READY,DSC,ERROR> error=0 LBA=9035815
initate_write_filepage: already started
ad0: FAILURE - READ_DMA status=51<READY,DSC,ERROR> error=0 LBA=20793151

On shutdown, I then got the same vrele() panic you did.  Unfortunately,
the crash dump I took appears to be corrupted, so I don't have much
debugging information about the crash itself.   I was able to extract the
following from running strings on the vmcore, however:

Waiting (max 60 seconds) for system process `vnlru' to stop...stopped
Waiting (max 60 seconds) for the system process `bufdaemon' to
stop...stopped
Waiting (max 60 seconds) for the system process `syncer' to stop...stopped
syncing disks, buffers remaining... 13 13 2 2
done
panic: vrele: negative ref cnt
at line 2015 in file /usr/src/sys/kern/vfs_subr.c
Debugger("panic")
panic: from debugger
at line 450 in file /usr/src/sys/ddb/db_command.cUptime: 1h33m152
Dumping 511 MB

FYI, I've replaced several drives, and run full disk diagnostics using
Dell's diagnostic tool, and not managed to identify any problems with this
drive, so it looks like there is some poor interaction between FreeBSD and
this system/controller/disk.  The vrele problem is almost certainly what
you suggest -- a failure to handle an I/O error properly somewhere,
perhaps in a situation where we have an extra vrele() in the error path
where the caller doesn't expect it. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research