Re: panic in softdep_slowdown()

On Wed, Jan 28, 2015 at 09:22:30PM +0300, Gleb Smirnoff wrote:
> On Wed, Jan 28, 2015 at 12:48:42PM +0200, Konstantin Belousov wrote:
> K> > Stopped at      softdep_slowdown+0x1d3: idivl   %ecx,%eax
> K> > db> bt
> K> > Tracing pid 49 tid 100045 td 0xfffff800026ee000
> K> > softdep_slowdown() at softdep_slowdown+0x1d3/frame 0xfffffe001eb5f2b0
> K> > ffs_truncate() at ffs_truncate+0x1be/frame 0xfffffe001eb5f640
> K> > ufs_setattr() at ufs_setattr+0x4e5/frame 0xfffffe001eb5f6a0
> K> > VOP_SETATTR_APV() at VOP_SETATTR_APV+0x22a/frame 0xfffffe001eb5f710
> K> > VOP_SETATTR() at VOP_SETATTR+0x45/frame 0xfffffe001eb5f760
> K> > vn_truncate() at vn_truncate+0x196/frame 0xfffffe001eb5f870
> K> > fo_truncate() at fo_truncate+0x41/frame 0xfffffe001eb5f8b0
> K> > kern_ftruncate() at kern_ftruncate+0x16d/frame 0xfffffe001eb5f920
> K> > sys_ftruncate() at sys_ftruncate+0x27/frame 0xfffffe001eb5f940
> K> > syscallenter() at syscallenter+0x46e/frame 0xfffffe001eb5f9b0
> K> > amd64_syscall() at amd64_syscall+0x1f/frame 0xfffffe001eb5fab0
> K> > Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe001eb5fab0
> K> > --- syscall (480, FreeBSD ELF64, sys_ftruncate), rip = 0x800b511fa, rsp = 0x7fffffffe998, rbp = 0x7fffffffeb90 ---
> K> > db> call doadump
> K> > Dumping 60 out of 495 MB:..27%..54%..80%
> K> > Dump complete
> K> > = 0
> K> > db>
> K> > 
> K> > I've got the core file.
> K> 
> K> At least the source line for the panic is needed.
> K> Also, print out the value of stat_flush_threads.
> 
> (kgdb) fr 11
> #11 0xffffffff80895d63 in softdep_slowdown (vp=0xfffff800028011d8)
>     at /usr/src/ifnet/sys/ufs/ffs/ffs_softdep.c:13055
> 13055           if (dep_current[D_DIRREM] < max_softdeps_hard / 2 &&
> (kgdb) p dep_current
> $1 = {1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 
>   0, 0, 0, 0}
> (kgdb) p max_softdeps_hard
> $2 = 153357
> (kgdb) p *ump
> $4 = {um_mountp = 0xfffff80002707330, um_dev = 0xfffff800026cbc00, 
>   um_cp = 0xfffff80002717480, um_bo = 0xfffff8000271edb8, 
>   um_devvp = 0xfffff8000271ece8, um_fstype = 2, um_fs = 0xfffff8000273b000, 
>   um_extattr = {uepm_lock = {lock_object = {lo_name = 0x0, lo_flags = 0, 
>         lo_data = 0, lo_witness = 0x0}, sx_lock = 0}, uepm_list = {
>       lh_first = 0x0}, uepm_ucred = 0x0, uepm_flags = 0}, um_nindir = 4096, 
>   um_bptrtodb = 3, um_seqinc = 8, um_lock = {lock_object = {
>       lo_name = 0xffffffff80a53d30 "FFS", lo_flags = 16973824, lo_data = 0, 
>       lo_witness = 0xfffffe00008e3400}, mtx_lock = 4}, um_fsckpid = 0, 
>   um_softdep = 0xfffff800027a0200, um_quotas = {0x0, 0x0}, um_cred = {0x0, 
>     0x0}, um_btime = {0, 0}, um_itime = {0, 0}, um_qflags = "\000", 
>   um_savedmaxfilesize = 0, um_candelete = 0, um_writesuspended = 0, 
>   um_balloc = 0xffffffff8086eb90 <ffs_balloc_ufs2>, 
>   um_blkatoff = 0xffffffff808a8170 <ffs_blkatoff>, 
>   um_truncate = 0xffffffff808717b0 <ffs_truncate>, 
>   um_update = 0xffffffff80871090 <ffs_update>, 
>   um_valloc = 0xffffffff808660c0 <ffs_valloc>, 
>   um_vfree = 0xffffffff808677b0 <ffs_vfree>, 
>   um_ifree = 0xffffffff808af420 <ffs_ifree>, 
>   um_rdonly = 0xffffffff808741c0 <ffs_rdonly>, 
>   um_snapgone = 0xffffffff80879b70 <ffs_snapgone>}
> (kgdb) p stat_flush_threads
> $5 = 1
> 
> I can't see where integer divide fault can happen with stat_flush_threads=1 :(

Look at the exact asm instruction which faulted, also look at the registers
content.

It might be hypervisor bug, after all.