The "unmount of /dev failed (BUSY)" message, explained

Folks,

In a show of way too much free time, I set off on a quest to figure out 
what was up with the "unmount of /dev failed (BUSY)" message that is seen 
on reboot. I am pleased to announce that I found what I believe to be the 
reason behind the message:

vfs_unmountall: Processing /dev/amrd1a:
vfs_unmountall: Processing /dev/amrd0s1d:
vfs_unmountall: Processing /dev/amrd0s1f:
vfs_unmountall: Processing /dev/amrd0s1e:
vfs_unmountall: Processing devfs:
0xc53efcc0: tag devfs, type VCHR
     usecount 1, writecount 0, refcount 22 mountedhere 0xc53d4c00
     flags ()
     v_object 0xc10608c4 ref 0 pages 72

         dev amrd0s1a
0xc5442550: tag devfs, type VCHR
     usecount 1, writecount 0, refcount 1 mountedhere 0xc53d4b00
     flags ()

         dev amrd0s1b
unmount of /dev failed (BUSY)
vfs_unmountall: Processing /dev/amrd0s1a:
0xc53efaa0: tag ufs, type VDIR
     usecount 1, writecount 0, refcount 1 mountedhere 0xc5396800
     flags ()

         ino 32896, on dev amrd0s1a
0xc53ef880: tag ufs, type VREG
     usecount 3, writecount 0, refcount 5 mountedhere 0
     flags (VV_TEXT)
     v_object 0xc1060738 ref 2 pages 89

         ino 16527, on dev amrd0s1a
Uptime: 12m8s

The patch that produces this output is available from: 
http://bling.properkernel.com/freebsd/vfs_unmountalldiag.diff

As you can see from the output, the unmount of /dev fails because the 
/dev/amrd0s1a (Which is the root mountpoint on this system) and 
/dev/amrd0s1b (Primary swap) device nodes still exist. The logic that we 
have that uses reverse TAILQ unmounting is flawed because the root 
filesystem is mounted read-only before devfs.

I will be cleaning up my patch a bit before I submit it to be committed. I 
will also be producing a patch that will make vfs_unmountall do the right 
thing.

I haven't found a way to make it vacuum my apartment yet but I will submit 
that functionality to ACME Labs when it's available... ;-)

Andy

PS: If you want to cross-reference the filesystems, the list is here:
http://bling.properkernel.com/freebsd/fstab

/*  Andre Guibert de Bruet  * 6f43 6564 7020 656f 2e74 4220 7469 6a20 */
/*   Code poet / Sysadmin   * 636f 656b 2e79 5320 7379 6461 696d 2e6e */
/*   GSM: +1 734 846 8758   * 5520 494e 2058 6c73 7565 6874 002e 0000 */
/* WWW: siliconlandmark.com *      Tormenting bytes since 1980.       */