Panic on shutdown _at_r328436: "Unholding 6 with cnt = -559038242"

This is on my "build machine" (laptop is still building updated ports
for today, so I don't know yet whether or not it encounters this.) 

I had performed a source-based update from r328393 to r328436,
rebooted, performed "make delete-old-libs", and all seemed well.

I then issued "sudo shutdown -p now", and serial console shows:

FreeBSD/amd64 (freebeast.catwhisker.org) (ttyu0)

login: Jan 26 12:11:03 Stopping sshd.
Waiting for PIDS: 681.
Stopping rsyncd.
Waiting for PIDS: 652.
Stopping powerd.
Waiting for PIDS: 636.
Stopping ntpd.
Waiting for PIDS: 633, 633.
Stopping lpd.
Waiting for PIDS: 610.
Stopping lockd.
WaitingWARNING: autofs_unmount: vflush failed with error 16
 for PIDS: 592.
Stopping statd.
Waiting for PIDS: 589.
Stopping nfsd.
WaitinlJan 26 12:11:05 ock ordefreebeast syslogr reversal:
 1st 0xfffff800692cd490 filed: exiting on sidesc structure (filedesc structure) _at_ /usr/src/sys/kern/sys_generic.c:1567
 2nd 0xfffff8006973f9a0 devfs (devfs) _at_ /usr/src/gnal 15
sys/kern/vfs_vnops.c:1526
stack backtrace:
#0 0xffffffff80b2c2e3 at witness_debugger+0x73
#1 0xffffffff80b2c164 at witness_checkorder+0xe34
#2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1
#3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80ba7746 at _vn_lock+0x66
#5 0xffffffff80ba654b at vn_poll+0x3b
#6 0xffffffff80992f4d at devfs_poll_f+0xcd
#7 0xffffffff80b33255 at kern_poll+0x385
#8 0xffffffff80b32ec0 at sys_poll+0x50
#9 0xffffffff80f7a37b at amd64_syscall+0x79b
#10 0xffffffff80f569c8 at fast_syscall_common+0xfc
Waiting (max 60 seconds) for system process `vnlru' to stop... done
Waiting (max 60 seconds) for system process `bufdaemon' to stop... done
Waiting (max 60 seconds) for system process `syncer' to stop... 
Syncing disks, vnodes remaining... 5 5 lock order reversal:
 1st 0xfffff80007927240 syncer (syncer) _at_ /usr/src/sys/kern/vfs_subr.c:2157
 2nd 0xfffff80007570d50 devfs (devfs) _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1583
stack backtrace:
#0 0xffffffff80b2c2e3 at witness_debugger+0x73
#1 0xffffffff80b2c164 at witness_checkorder+0xe34
#2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1
#3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80ba7746 at _vn_lock+0x66
#5 0xffffffff80dc1dac at ffs_sync+0x2cc
#6 0xffffffff80b9ce8f at sync_fsync+0xff
#7 0xffffffff810f7c19 at VOP_FSYNC_APV+0xd9
#8 0xffffffff80b9acc4 at sched_sync+0x284
#9 0xffffffff80a88414 at fork_exit+0x84
#10 0xffffffff80f56e1e at fork_trampoline+0xe
5 3 2 2 1 1 1 1 1 0 0 0 0 0 done
All buffers synced.
lock order reversal:
 1st 0xfffff800079c57c8 ufs (ufs) _at_ /usr/src/sys/kern/vfs_mount.c:1280
 2nd 0xfffff8000795e418 devfs (devfs) _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1371
stack backtrace:
#0 0xffffffff80b2c2e3 at witness_debugger+0x73
#1 0xffffffff80b2c164 at witness_checkorder+0xe34
#2 0xffffffff80a9ca21 at lockmgr_lock_fast_path+0x1b1
#3 0xffffffff810f8ca9 at VOP_LOCK1_APV+0xd9
#4 0xffffffff80ba7746 at _vn_lock+0x66
#5 0xffffffff80dbef63 at ffs_flushfiles+0x93
#6 0xffffffff80da24f2 at softdep_flushfiles+0x82
#7 0xffffffff80dc15f7 at ffs_unmount+0x77
#8 0xffffffff80b8e2c9 at dounmount+0x519
#9 0xffffffff80b9800b at vfs_unmountall+0x6b
#10 0xffffffff80b73d25 at bufshutdown+0x3a5
#11 0xffffffff80ac754a at kern_reboot+0x1da
#12 0xffffffff80ac7312 at sys_reboot+0x3c2
#13 0xffffffff80f7a37b at amd64_syscall+0x79b
#14 0xffffffff80f569c8 at fast_syscall_common+0xfc
Swap device [file] removed.
Uptime: 1m44s
(ada0:ahcich0:0:0:0): spin-down
(ada1:ahcich2:0:0:0): spin-down
(ada2:ahcich3:0:0:0): spin-down
(ada3:ahcich4:0:0:0): spin-down
panic: Unholding 6 with cnt = -559038242
cpuid = 3
time = 1516968697
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00004288c0
vpanic() at vpanic+0x18d/frame 0xfffffe0000428920
panic() at panic+0x43/frame 0xfffffe0000428980
dadiskgonecb() at dadiskgonecb+0x42/frame 0xfffffe00004289a0
g_disk_providergone() at g_disk_providergone+0x25/frame 0xfffffe00004289d0
g_destroy_provider() at g_destroy_provider+0xae/frame 0xfffffe00004289f0
g_wither_washer() at g_wither_washer+0x87/frame 0xfffffe0000428a30
g_run_events() at g_run_events+0x3ca/frame 0xfffffe0000428a70
fork_exit() at fork_exit+0x84/frame 0xfffffe0000428ab0
fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000428ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---
KDB: enter: panic
[ thread pid 13 tid 100044 ]
Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
db> 


As noted, this is a build machine, and it was to be powered off for
the rest of the day anyway, so I don't need to get it up & running
immediately: I can poke at the ddb prompt, given some clues.

When running head, the system does not use ZFS (only UFS2+SU -- not
SUJ -- & tmpfs).

Same system had completed a source-based update for stable/11 from
r328392 to r328429 earlier today without incident (using a different
slice of the boot drive).

Peace,
david
-- 
David H. Wolfskill				david_at_catwhisker.org
"unfortunately, no trust!” -- well, of course!  You reap what you sow.

See http://www.catwhisker.org/~david/publickey.gpg for my public key.