On 8 Feb, Don Lewis wrote: > On 8 Feb, John Baldwin wrote: >> WITNESS has especially poor scalability issues and can get much worse as more >> lock classes and relationships are added to its tree due to how it tries to >> rebalance the tree every time it adds a new relationship (and each rebalance >> is O(n) I think). If you turn off witness via sysctl debug.witness.watch=0,. >> does the same weirdness persist? > > I'm in the middle of a portupgrade run to catch up with the latest perl > upgrade, so it'll be a while before I can try that experiment. Earlier > today I broke into DDB a bunch of times to get some stack traces to try > to figure out where the kernel was spending most of its time. Most of > the stack traces had this in common: > > critical_exit(c094d12c,eb27bbac,c0643c27,c0906a20,0) at critical_exit+0xb2 > _mtx_unlock_spin_flags(c0906a20,0,c083c0bd,325,c28515c0) at _mtx_unlock_spin_flags+0x8d > witness_checkorder(c2dad7bc,9,c0852ab7,9f) at witness_checkorder+0x29f > _mtx_lock_flags(c2dad7bc,0,c0852ab7,9f,22) at _mtx_lock_flags+0x5b > > where witness_checkorder+0x29f is here: > > /* > * If we know that the the lock we are acquiring comes after > * the lock we most recently acquired in the lock order tree, > * then there is no need for any further checks. > */ > if (isitmydescendant(w1, w)) { > mtx_unlock_spin(&w_mtx); > > and by far the majority of the callers were calling VM_OBJECT_LOCK(). > > I suspect that the tree has gotten rebalanced so that the "vm object" > lock suddenly takes a much longer time to find. I'll try to get > snapshots of the tree before and after the problem occurs. I forgot to try turning off debug.witness.watch, but I did a "show witness" when the system was slow and then after a reboot which made the system was fast again. It looks like the discovery of the FFS Lock -> standard object relationship, which causes witness to delete the direct link Giant -> standard object is what is throwing out the boat anchor. Without this direct parent/child relationship, isitmydescendant() has a much longer search to do. I just looked closely at the rebalancing code. Gak! Here are the outputs of show witness in the fast and slow cases: Fast: Sleep locks: 0 g_xdown -- last acquired _at_ /usr/src/sys/geom/geom_io.c:423 3 Giant -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:247 4 UMA lock -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1491 12 Malloc Stats -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:228 16 UMA pcpu -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1825 17 KMAP ENTRY -- last acquired _at_ /usr/src/sys/vm/uma_core.c:418 18 UMA zone -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1842 4 standard object -- last acquired _at_ /usr/src/sys/vm/vm_object.c:449 5 vm object_list -- last acquired _at_ /usr/src/sys/vm/vm_object.c:643 5 swap_pager swhash -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:1961 14 vm page queue mutex -- last acquired _at_ /usr/src/sys/vm/vm_fault.c:902 15 vnode interlock -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1898 16 Syncer mtx -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1617 16 vnode_free_list -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:590 16 cdev -- last acquired _at_ /usr/src/sys/kern/kern_conf.c:70 15 pmap -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2688 16 UMA pcpu -- (already displayed) 16 uma object -- last acquired _at_ /usr/src/sys/vm/uma_core.c:988 16 SYSMAPS -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2422 4 kernel linker -- last acquired _at_ /usr/src/sys/kern/kern_linker.c:461 4 eventhandler -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:213 5 eventhandler list -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:199 4 malloc -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:584 4 filedesc structure -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:814 6 accept -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:460 7 so_snd -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2112 8 tcp_hc_entry -- last acquired _at_ /usr/src/sys/netinet/tcp_hostcache.c:286 8 so_rcv -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2113 9 radix node head -- last acquired _at_ /usr/src/sys/net/route.c:148 10 ifnet -- last acquired _at_ /usr/src/sys/net/if.c:1056 10 rtentry -- last acquired _at_ /usr/src/sys/netinet/ip_output.c:822 11 network driver -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1756 12 if send queue -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1268 12 Malloc Stats -- (already displayed) 12 knlist lock for lockless objects -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453 12 system map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2317 13 kmem object -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:397 14 vm page queue mutex -- (already displayed) 13 kernel object -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3705 14 vm page queue mutex -- (already displayed) 11 rts_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 11 ifaddr -- last acquired _at_ /usr/src/sys/net/route.c:788 9 sellck -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:713 9 process lock -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:850 10 ktrace -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:354 10 sigacts -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:369 10 struct pargs.ref -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:1110 10 session -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:450 11 tty -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453 15 vnode interlock -- (already displayed) 11 uidinfo hash -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:1034 12 sleep mtxpool -- last acquired _at_ /usr/src/sys/kern/kern_prot.c:1749 12 uidinfo struct -- last acquired _at_ order list:0 13 allprison -- last acquired _at_ /usr/src/sys/kern/kern_jail.c:460 5 pipe mutex -- last acquired _at_ /usr/src/sys/kern/sys_pipe.c:1340 9 sellck -- (already displayed) 6 sigio lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:739 7 process group -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:257 9 process lock -- (already displayed) 4 rman -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:448 12 Malloc Stats -- (already displayed) 12 system map -- (already displayed) 4 kobj -- last acquired _at_ /usr/src/sys/kern/subr_kobj.c:298 4 devd -- last acquired _at_ /usr/src/sys/kern/subr_bus.c:497 9 sellck -- (already displayed) 4 bounce pages lock -- last acquired _at_ /usr/src/sys/i386/i386/busdma_machdep.c:1054 4 TID lock -- last acquired _at_ /usr/src/sys/kern/kern_thread.c:204 4 vm86 lock -- last acquired _at_ /usr/src/sys/i386/i386/vm86.c:581 4 acpica subsystem lock -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360 4 ithread -- last acquired _at_ /usr/src/sys/kern/kern_intr.c:275 4 ttylist -- last acquired _at_ /usr/src/sys/kern/tty.c:2799 11 tty -- (already displayed) 4 GEOM orphanage -- last acquired _at_ /usr/src/sys/geom/geom_event.c:188 4 ACPI semaphore -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300 4 taskqueue list -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:85 4 domain list -- last acquired _at_ /usr/src/sys/kern/uipc_domain.c:224 4 rman head -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:111 4 bdone lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3031 4 sf_buf -- last acquired _at_ /usr/src/sys/i386/i386/vm_machdep.c:680 4 taskqueue -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:194 4 bpf global lock -- last acquired _at_ /usr/src/sys/net/bpf.c:1446 5 bpf interface lock -- last acquired _at_ order list:0 6 bpf cdev lock -- last acquired _at_ order list:0 4 pseudofs -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86 4 nfsd_mtx -- last acquired _at_ /usr/src/sys/nfsserver/nfs_srvsock.c:811 8 so_rcv -- (already displayed) 4 if_cloners lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:199 4 Name Cache -- last acquired _at_ /usr/src/sys/kern/vfs_cache.c:347 15 vnode interlock -- (already displayed) 16 UMA pcpu -- (already displayed) 4 if_clone lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:321 4 tcp -- last acquired _at_ /usr/src/sys/netinet/tcp_subr.c:1359 5 tcpinp -- last acquired _at_ /usr/src/sys/netinet/tcp_input.c:747 6 so_glabel -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:273 6 random reseed -- last acquired _at_ /usr/src/sys/dev/random/yarrow.c:193 6 ip_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 6 arc4_mtx -- last acquired _at_ /usr/src/sys/libkern/arc4random.c:137 6 accept -- (already displayed) 4 pbuf mutex -- last acquired _at_ /usr/src/sys/vm/vm_pager.c:410 4 lo_mtx -- last acquired _at_ /usr/src/sys/net/if_loop.c:154 4 pfil_head_mtx -- last acquired _at_ /usr/src/sys/net/pfil.c:174 5 pfil_head_list lock -- last acquired _at_ /usr/src/sys/net/pfil.c:180 4 pseudofs_vncache -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243 15 vnode interlock -- (already displayed) 4 needsbuffer lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:313 4 Softdep Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:4420 5 buffer daemon lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:421 12 Malloc Stats -- (already displayed) 15 vnode interlock -- (already displayed) 16 UMA pcpu -- (already displayed) 4 runningbufspace lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:332 4 devstat -- last acquired _at_ /usr/src/sys/kern/subr_devstat.c:190 4 ACPI task queue -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118 4 mntid -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:415 5 mountlist -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:3078 6 struct mount mtx -- last acquired _at_ /usr/src/sys/kern/vfs_vnops.c:957 15 vnode interlock -- (already displayed) 4 buf queue lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:1544 15 vnode interlock -- (already displayed) 4 rtsock route_cb lock -- last acquired _at_ /usr/src/sys/net/rtsock.c:234 4 FFS Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:2316 4 fdesc -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:1425 4 g_disk_done -- last acquired _at_ /usr/src/sys/geom/geom_disk.c:198 5 bio queue -- last acquired _at_ /usr/src/sys/geom/geom_io.c:66 16 UMA pcpu -- (already displayed) 4 dirhash list -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348 5 dirhash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:456 4 rawcb -- last acquired _at_ /usr/src/sys/net/raw_cb.c:105 8 so_rcv -- (already displayed) 4 udp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:246 5 udp6inp -- last acquired _at_ /usr/src/sys/netinet6/udp6_usrreq.c:670 6 if_afdata -- last acquired _at_ /usr/src/sys/netinet6/scope6.c:272 6 ip6_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 6 accept -- (already displayed) 5 udpinp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:386 6 arc4_mtx -- (already displayed) 6 NFS reqq lock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:1049 7 Synch NFS reply posting -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:584 9 process lock -- (already displayed) 6 accept -- (already displayed) 4 ufs ihash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_ihash.c:156 15 vnode interlock -- (already displayed) 4 semid -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1002 5 sem -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1258 4 unp -- last acquired _at_ /usr/src/sys/kern/uipc_usrreq.c:329 6 accept -- (already displayed) 0 g_xup -- last acquired _at_ /usr/src/sys/geom/geom_io.c:484 3 Giant -- (already displayed) 0 arp_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 0 igmp_mtx -- last acquired _at_ /usr/src/sys/netinet/igmp.c:431 0 ipqlock -- last acquired _at_ /usr/src/sys/netinet/ip_input.c:1110 16 UMA pcpu -- (already displayed) 0 ip6qlock -- last acquired _at_ /usr/src/sys/netinet6/frag6.c:682 0 nfslock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_lock.c:123 0 GEOM topology -- last acquired _at_ /usr/src/sys/geom/geom_event.c:220 1 fdc lock -- last acquired _at_ /usr/src/sys/dev/fdc/fdc.c:777 2 callout_wait_lock -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:289 1 swapdev -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:2102 3 Giant -- (already displayed) 0 p_peers -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:250 0 ACPI PCI power methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207 0 ACPI root bus -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969 4 rman -- (already displayed) 4 ACPI semaphore -- (already displayed) 0 ACPI PCI bus methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214 1 ACPI PCI link -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972 2 kernel environment -- last acquired _at_ /usr/src/sys/kern/kern_environment.c:285 4 ACPI semaphore -- (already displayed) 12 Malloc Stats -- (already displayed) 16 UMA pcpu -- (already displayed) 0 ACPI CPU -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817 0 module subsystem sx lock -- last acquired _at_ /usr/src/sys/kern/kern_module.c:115 0 protect sysfilt_ops -- last acquired _at_ /usr/src/sys/kern/kern_event.c:669 0 sysctl lock -- last acquired _at_ /usr/src/sys/kern/kern_sysctl.c:1335 1 filelist lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2128 4 filedesc structure -- (already displayed) 1 addrsel_lock -- last acquired _at_ /usr/src/sys/netinet6/in6_src.c:1137 1 rip -- last acquired _at_ /usr/src/sys/netinet/raw_ip.c:866 1 allproc -- last acquired _at_ /usr/src/sys/kern/sched_4bsd.c:435 2 user map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2998 3 Giant -- (already displayed) 0 ddp_list_mtx -- last acquired _at_ order list:0 1 ddp_mtx -- last acquired _at_ order list:0 0 slip_mtx -- last acquired _at_ order list:0 1 slip sc_mtx -- last acquired _at_ order list:0 0 proctree -- last acquired _at_ /usr/src/sys/kern/tty.c:2076 1 allproc -- (already displayed) Slow: Sleep locks: 0 g_xdown -- last acquired _at_ /usr/src/sys/geom/geom_io.c:423 3 Giant -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:247 4 UMA lock -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1491 12 Malloc Stats -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:228 12 system map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2317 13 kmem object -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:397 14 vm page queue mutex -- last acquired _at_ /usr/src/sys/vm/vm_pageout.c:1433 15 vnode interlock -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1489 16 vnode_free_list -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:590 16 Syncer mtx -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1617 16 cdev -- last acquired _at_ /usr/src/sys/kern/kern_conf.c:70 15 pmap -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:1626 16 UMA pcpu -- last acquired _at_ /usr/src/sys/vm/uma_core.c:2251 17 KMAP ENTRY -- last acquired _at_ /usr/src/sys/vm/uma_core.c:418 18 UMA zone -- last acquired _at_ /usr/src/sys/vm/uma_core.c:2269 16 uma object -- last acquired _at_ /usr/src/sys/vm/uma_core.c:988 16 SYSMAPS -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2422 13 kernel object -- last acquired _at_ /usr/src/sys/vm/vm_object.c:449 14 vm page queue mutex -- (already displayed) 4 vm86 lock -- last acquired _at_ /usr/src/sys/i386/i386/vm86.c:581 4 eventhandler -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:213 5 eventhandler list -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:199 4 malloc -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:584 4 kernel linker -- last acquired _at_ /usr/src/sys/kern/kern_linker.c:1061 4 filedesc structure -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:814 6 accept -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:460 7 so_snd -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2112 8 tcp_hc_entry -- last acquired _at_ /usr/src/sys/netinet/tcp_hostcache.c:654 12 system map -- (already displayed) 8 so_rcv -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2113 9 sellck -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:713 9 radix node head -- last acquired _at_ /usr/src/sys/net/route.c:148 10 ifnet -- last acquired _at_ /usr/src/sys/net/if.c:1056 11 random reseed -- last acquired _at_ /usr/src/sys/dev/random/yarrow.c:193 11 arc4_mtx -- last acquired _at_ /usr/src/sys/libkern/arc4random.c:137 10 rtentry -- last acquired _at_ /usr/src/sys/netinet/ip_output.c:822 11 rts_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 11 network driver -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1756 12 Malloc Stats -- (already displayed) 12 knlist lock for lockless objects -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453 12 if send queue -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1268 12 system map -- (already displayed) 11 ifaddr -- last acquired _at_ /usr/src/sys/net/route.c:788 9 process lock -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:368 10 struct pargs.ref -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:1110 10 ktrace -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:354 10 sigacts -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:369 10 session -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:450 15 vnode interlock -- (already displayed) 11 tty -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453 11 uidinfo hash -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:1034 12 sleep mtxpool -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2099 12 uidinfo struct -- last acquired _at_ order list:0 13 allprison -- last acquired _at_ /usr/src/sys/kern/kern_jail.c:460 5 pipe mutex -- last acquired _at_ /usr/src/sys/kern/sys_pipe.c:1340 14 vm page queue mutex -- (already displayed) 9 sellck -- (already displayed) 6 sigio lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:739 7 process group -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:257 9 process lock -- (already displayed) 4 devd -- last acquired _at_ /usr/src/sys/kern/subr_bus.c:497 9 sellck -- (already displayed) 4 bounce pages lock -- last acquired _at_ /usr/src/sys/i386/i386/busdma_machdep.c:1054 4 kobj -- last acquired _at_ /usr/src/sys/kern/subr_kobj.c:298 4 TID lock -- last acquired _at_ /usr/src/sys/kern/kern_thread.c:204 4 rman -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:448 12 Malloc Stats -- (already displayed) 12 system map -- (already displayed) 4 bpf global lock -- last acquired _at_ /usr/src/sys/net/bpf.c:1446 5 bpf interface lock -- last acquired _at_ order list:0 6 bpf cdev lock -- last acquired _at_ order list:0 4 ACPI semaphore -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300 4 acpica subsystem lock -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360 4 GEOM orphanage -- last acquired _at_ /usr/src/sys/geom/geom_event.c:188 4 ithread -- last acquired _at_ /usr/src/sys/kern/kern_intr.c:275 4 ttylist -- last acquired _at_ /usr/src/sys/kern/tty.c:2799 11 tty -- (already displayed) 4 taskqueue list -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:85 4 rman head -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:111 4 taskqueue -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:194 4 domain list -- last acquired _at_ /usr/src/sys/kern/uipc_domain.c:224 4 bdone lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3031 4 sf_buf -- last acquired _at_ /usr/src/sys/i386/i386/vm_machdep.c:680 4 tcp -- last acquired _at_ /usr/src/sys/netinet/tcp_subr.c:1359 5 tcpinp -- last acquired _at_ /usr/src/sys/netinet/tcp_input.c:747 6 ip6_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 6 if_afdata -- last acquired _at_ /usr/src/sys/netinet6/scope6.c:272 6 so_glabel -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:273 6 ip_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 6 accept -- (already displayed) 4 if_cloners lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:199 4 pseudofs -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86 4 Name Cache -- last acquired _at_ /usr/src/sys/kern/vfs_cache.c:347 12 system map -- (already displayed) 4 if_clone lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:321 4 nfsd_mtx -- last acquired _at_ /usr/src/sys/nfsserver/nfs_srvsock.c:811 8 so_rcv -- (already displayed) 4 udp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:246 5 udp6inp -- last acquired _at_ /usr/src/sys/netinet6/udp6_usrreq.c:670 6 ip6_inq -- (already displayed) 6 if_afdata -- (already displayed) 6 accept -- (already displayed) 5 udpinp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:386 6 ip_inq -- (already displayed) 6 NFS reqq lock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:1049 7 Synch NFS reply posting -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:584 6 accept -- (already displayed) 4 runningbufspace lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:332 4 pfil_head_mtx -- last acquired _at_ /usr/src/sys/net/pfil.c:174 5 pfil_head_list lock -- last acquired _at_ /usr/src/sys/net/pfil.c:180 4 pbuf mutex -- last acquired _at_ /usr/src/sys/vm/vm_pager.c:410 4 lo_mtx -- last acquired _at_ /usr/src/sys/net/if_loop.c:154 4 pseudofs_vncache -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243 15 vnode interlock -- (already displayed) 4 rtsock route_cb lock -- last acquired _at_ /usr/src/sys/net/rtsock.c:234 4 ACPI task queue -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118 4 devstat -- last acquired _at_ /usr/src/sys/kern/subr_devstat.c:83 4 fdesc -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:1425 4 mntid -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:415 5 mountlist -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:3078 6 struct mount mtx -- last acquired _at_ /usr/src/sys/kern/vfs_vnops.c:984 15 vnode interlock -- (already displayed) 4 rawcb -- last acquired _at_ /usr/src/sys/net/raw_cb.c:105 8 so_rcv -- (already displayed) 4 unp -- last acquired _at_ /usr/src/sys/kern/uipc_usrreq.c:329 6 accept -- (already displayed) 4 FFS Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1044 5 needsbuffer lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:313 5 buf queue lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:1544 6 struct mount mtx -- (already displayed) 6 Softdep Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:402 7 buffer daemon lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:2151 12 Malloc Stats -- (already displayed) 12 system map -- (already displayed) 5 standard object -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3523 6 vm object_list -- last acquired _at_ /usr/src/sys/vm/vm_object.c:643 6 swap_pager swhash -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:1961 7 swapdev -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:2245 16 uma object -- (already displayed) 16 UMA pcpu -- (already displayed) 14 vm page queue mutex -- (already displayed) 4 g_disk_done -- last acquired _at_ /usr/src/sys/geom/geom_disk.c:198 5 bio queue -- last acquired _at_ /usr/src/sys/geom/geom_io.c:66 12 system map -- (already displayed) 4 rip -- last acquired _at_ /usr/src/sys/netinet/raw_ip.c:195 6 accept -- (already displayed) 4 semid -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1286 5 sem -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1258 4 dirhash list -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348 5 dirhash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349 4 ufs ihash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_ihash.c:156 15 vnode interlock -- (already displayed) 4 linux osname -- last acquired _at_ /usr/src/sys/modules/linux/../../compat/linux/linux_mib.c:231 4 ipqlock -- last acquired _at_ /usr/src/sys/netinet/ip_input.c:1110 16 UMA pcpu -- (already displayed) 0 g_xup -- last acquired _at_ /usr/src/sys/geom/geom_io.c:484 3 Giant -- (already displayed) 0 arp_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232 0 igmp_mtx -- last acquired _at_ /usr/src/sys/netinet/igmp.c:431 0 ip6qlock -- last acquired _at_ /usr/src/sys/netinet6/frag6.c:682 0 nfslock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_lock.c:123 0 GEOM topology -- last acquired _at_ /usr/src/sys/geom/geom_event.c:220 1 fdc lock -- last acquired _at_ /usr/src/sys/dev/fdc/fdc.c:777 2 callout_wait_lock -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:289 3 Giant -- (already displayed) 0 p_peers -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:250 0 ACPI PCI power methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207 0 ACPI root bus -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969 4 rman -- (already displayed) 4 ACPI semaphore -- (already displayed) 0 ACPI PCI bus methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214 1 ACPI PCI link -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972 4 ACPI semaphore -- (already displayed) 12 Malloc Stats -- (already displayed) 2 kernel environment -- last acquired _at_ /usr/src/sys/kern/kern_environment.c:285 16 UMA pcpu -- (already displayed) 0 ACPI CPU -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817 0 module subsystem sx lock -- last acquired _at_ /usr/src/sys/kern/kern_module.c:115 0 protect sysfilt_ops -- last acquired _at_ /usr/src/sys/kern/kern_event.c:669 0 sysctl lock -- last acquired _at_ /usr/src/sys/kern/kern_sysctl.c:1335 1 filelist lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2128 4 filedesc structure -- (already displayed) 1 addrsel_lock -- last acquired _at_ /usr/src/sys/netinet6/in6_src.c:1137 1 allproc -- last acquired _at_ /usr/src/sys/kern/sched_4bsd.c:435 2 user map -- last acquired _at_ /usr/src/sys/vm/vm_glue.c:176 3 Giant -- (already displayed) 0 ddp_list_mtx -- last acquired _at_ order list:0 1 ddp_mtx -- last acquired _at_ order list:0 0 slip_mtx -- last acquired _at_ order list:0 1 slip sc_mtx -- last acquired _at_ order list:0 0 proctree -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:591 1 allproc -- (already displayed)Received on Thu Feb 10 2005 - 10:04:21 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:28 UTC