Re: zfs/vfs lockups, via poudriere

On Sun, 2014-11-23 at 00:09 +0200, Andriy Gapon wrote:
> On 22/11/2014 21:20, Sean Bruno wrote:
> > bdrewery reported a vfs/zfs condition where operations will stall out
> > and block (rm, mv, file) during a poudriere build.  I've hit this now
> > and it seems to be alleviated by setting vfs.lookup_shared=0
> > 
> > I seem to be able to trivially reproduce this on my builders and want to
> > know if anyone is looking to diagnose this further.
> > 
> > original message:
> > https://lists.freebsd.org/pipermail/freebsd-fs/2014-September/020035.html
> > 
> > On my builders I see:
> > 
> > procstat -kka | grep zfs
> > 
> >     0 100666 kernel           zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe 
> >     3 100151 zfskern          arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe 
> >     3 100152 zfskern          l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a fork_trampoline+0xe 
> >     3 100657 zfskern          trim zroot       mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_trampoline+0xe 
> >     3 100675 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe 
> >     3 100676 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_trampoline+0xe 
> > 31071 100995 rm               -                mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb 
> > 31075 100693 mv               -                mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x4
> 
> The last line looks incomplete.
> 
> 
hrm ... cut-n-paste fail I guess.

procstat -kka | grep zfs

    0 100666 kernel           zfs_vn_rele_task mi_switch+0xe1 sleepq_wait+0x3a _sleep+0x2ad taskqueue_thread_loop+0xf5 fork_exit+0x9a fork_trampoline+0xe 
    3 100151 zfskern          arc_reclaim_thre mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad arc_reclaim_thread+0x288 fork_exit+0x9a fork_trampoline+0xe 
    3 100152 zfskern          l2arc_feed_threa mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad l2arc_feed_thread+0x16f fork_exit+0x9a fork_trampoline+0xe 
    3 100657 zfskern          trim zroot       mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad trim_thread+0x9e fork_exit+0x9a fork_trampoline+0xe 
    3 100675 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_wait+0x3a _cv_wait+0x190 txg_quiesce_thread+0x39b fork_exit+0x9a fork_trampoline+0xe 
    3 100676 zfskern          txg_thread_enter mi_switch+0xe1 sleepq_timedwait+0x3a _cv_timedwait_sbt+0x1ad txg_sync_thread+0x1dc fork_exit+0x9a fork_trampoline+0xe 
31071 100995 rm               -                mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0x9ab vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 zfs_lookup+0x45d zfs_freebsd_lookup+0x6d VOP_CACHEDLOOKUP_APV+0xa1 vfs_cache_lookup+0xd6 VOP_LOOKUP_APV+0xa1 lookup+0x5a1 namei+0x534 kern_rmdirat+0x8d amd64_syscall+0x3fb Xfast_syscall+0xfb 
31075 100693 mv               -                mi_switch+0xe1 sleepq_wait+0x3a sleeplk+0x18d __lockmgr_args+0xd5d vop_stdlock+0x3c VOP_LOCK1_APV+0xab _vn_lock+0x43 vputx+0x28a zfs_rename_unlock+0x3e zfs_freebsd_rename+0xe39 VOP_RENAME_APV+0xab kern_renameat+0x4a6 amd64_syscall+0x3fb Xfast_syscall+0xfb