Re: odd -CURRENT performance issue

From: Don Lewis <truckman_at_FreeBSD.org>
Date: Thu, 10 Feb 2005 03:04:12 -0800 (PST)
On  8 Feb, Don Lewis wrote:
> On  8 Feb, John Baldwin wrote:

>> WITNESS has especially poor scalability issues and can get much worse as more 
>> lock classes and relationships are added to its tree due to how it tries to 
>> rebalance the tree every time it adds a new relationship (and each rebalance 
>> is O(n) I think).  If you turn off witness via sysctl debug.witness.watch=0,. 
>> does the same weirdness persist?
> 
> I'm in the middle of a portupgrade run to catch up with the latest perl
> upgrade, so it'll be a while before I can try that experiment.  Earlier
> today I broke into DDB a bunch of times to get some stack traces to try
> to figure out where the kernel was spending most of its time.  Most of
> the stack traces had this in common:
> 
> critical_exit(c094d12c,eb27bbac,c0643c27,c0906a20,0) at critical_exit+0xb2
> _mtx_unlock_spin_flags(c0906a20,0,c083c0bd,325,c28515c0) at _mtx_unlock_spin_flags+0x8d
> witness_checkorder(c2dad7bc,9,c0852ab7,9f) at witness_checkorder+0x29f
> _mtx_lock_flags(c2dad7bc,0,c0852ab7,9f,22) at _mtx_lock_flags+0x5b
> 
> where witness_checkorder+0x29f is here:
> 
>         /*
>          * If we know that the the lock we are acquiring comes after
>          * the lock we most recently acquired in the lock order tree,
>          * then there is no need for any further checks.
>          */
>         if (isitmydescendant(w1, w)) {
>                 mtx_unlock_spin(&w_mtx);
> 
> and by far the majority of the callers were calling VM_OBJECT_LOCK().
> 
> I suspect that the tree has gotten rebalanced so that the "vm object"
> lock suddenly takes a much longer time to find.  I'll try to get
> snapshots of the tree before and after the problem occurs.

I forgot to try turning off debug.witness.watch, but I did a "show
witness" when the system was slow and then after a reboot which made the
system was fast again.  It looks like the discovery of the
	FFS Lock -> standard object
relationship, which causes witness to delete the direct link
	Giant -> standard object
is what is throwing out the boat anchor.  Without this direct
parent/child relationship, isitmydescendant() has a much longer search
to do.

I just looked closely at the rebalancing code.  Gak!

Here are the outputs of show witness in the fast and slow cases:

Fast:

Sleep locks:
0 g_xdown -- last acquired _at_ /usr/src/sys/geom/geom_io.c:423
3  Giant -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:247
4   UMA lock -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1491
12   Malloc Stats -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:228
16   UMA pcpu -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1825
17    KMAP ENTRY -- last acquired _at_ /usr/src/sys/vm/uma_core.c:418
18     UMA zone -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1842
4   standard object -- last acquired _at_ /usr/src/sys/vm/vm_object.c:449
5    vm object_list -- last acquired _at_ /usr/src/sys/vm/vm_object.c:643
5    swap_pager swhash -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:1961
14   vm page queue mutex -- last acquired _at_ /usr/src/sys/vm/vm_fault.c:902
15    vnode interlock -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1898
16     Syncer mtx -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1617
16     vnode_free_list -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:590
16     cdev -- last acquired _at_ /usr/src/sys/kern/kern_conf.c:70
15    pmap -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2688
16     UMA pcpu -- (already displayed)
16     uma object -- last acquired _at_ /usr/src/sys/vm/uma_core.c:988
16     SYSMAPS -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2422
4   kernel linker -- last acquired _at_ /usr/src/sys/kern/kern_linker.c:461
4   eventhandler -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:213
5    eventhandler list -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:199
4   malloc -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:584
4   filedesc structure -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:814
6    accept -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:460
7     so_snd -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2112
8      tcp_hc_entry -- last acquired _at_ /usr/src/sys/netinet/tcp_hostcache.c:286
8      so_rcv -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2113
9       radix node head -- last acquired _at_ /usr/src/sys/net/route.c:148
10       ifnet -- last acquired _at_ /usr/src/sys/net/if.c:1056
10       rtentry -- last acquired _at_ /usr/src/sys/netinet/ip_output.c:822
11        network driver -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1756
12         if send queue -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1268
12         Malloc Stats -- (already displayed)
12         knlist lock for lockless objects -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453
12         system map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2317
13          kmem object -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:397
14           vm page queue mutex -- (already displayed)
13          kernel object -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3705
14           vm page queue mutex -- (already displayed)
11        rts_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
11        ifaddr -- last acquired _at_ /usr/src/sys/net/route.c:788
9       sellck -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:713
9       process lock -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:850
10       ktrace -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:354
10       sigacts -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:369
10       struct pargs.ref -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:1110
10       session -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:450
11        tty -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453
15        vnode interlock -- (already displayed)
11        uidinfo hash -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:1034
12         sleep mtxpool -- last acquired _at_ /usr/src/sys/kern/kern_prot.c:1749
12         uidinfo struct -- last acquired _at_ order list:0
13          allprison -- last acquired _at_ /usr/src/sys/kern/kern_jail.c:460
5    pipe mutex -- last acquired _at_ /usr/src/sys/kern/sys_pipe.c:1340
9     sellck -- (already displayed)
6     sigio lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:739
7      process group -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:257
9       process lock -- (already displayed)
4   rman -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:448
12   Malloc Stats -- (already displayed)
12   system map -- (already displayed)
4   kobj -- last acquired _at_ /usr/src/sys/kern/subr_kobj.c:298
4   devd -- last acquired _at_ /usr/src/sys/kern/subr_bus.c:497
9    sellck -- (already displayed)
4   bounce pages lock -- last acquired _at_ /usr/src/sys/i386/i386/busdma_machdep.c:1054
4   TID lock -- last acquired _at_ /usr/src/sys/kern/kern_thread.c:204
4   vm86 lock -- last acquired _at_ /usr/src/sys/i386/i386/vm86.c:581
4   acpica subsystem lock -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360
4   ithread -- last acquired _at_ /usr/src/sys/kern/kern_intr.c:275
4   ttylist -- last acquired _at_ /usr/src/sys/kern/tty.c:2799
11   tty -- (already displayed)
4   GEOM orphanage -- last acquired _at_ /usr/src/sys/geom/geom_event.c:188
4   ACPI semaphore -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300
4   taskqueue list -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:85
4   domain list -- last acquired _at_ /usr/src/sys/kern/uipc_domain.c:224
4   rman head -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:111
4   bdone lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3031
4   sf_buf -- last acquired _at_ /usr/src/sys/i386/i386/vm_machdep.c:680
4   taskqueue -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:194
4   bpf global lock -- last acquired _at_ /usr/src/sys/net/bpf.c:1446
5    bpf interface lock -- last acquired _at_ order list:0
6     bpf cdev lock -- last acquired _at_ order list:0
4   pseudofs -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86
4   nfsd_mtx -- last acquired _at_ /usr/src/sys/nfsserver/nfs_srvsock.c:811
8    so_rcv -- (already displayed)
4   if_cloners lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:199
4   Name Cache -- last acquired _at_ /usr/src/sys/kern/vfs_cache.c:347
15   vnode interlock -- (already displayed)
16   UMA pcpu -- (already displayed)
4   if_clone lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:321
4   tcp -- last acquired _at_ /usr/src/sys/netinet/tcp_subr.c:1359
5    tcpinp -- last acquired _at_ /usr/src/sys/netinet/tcp_input.c:747
6     so_glabel -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:273
6     random reseed -- last acquired _at_ /usr/src/sys/dev/random/yarrow.c:193
6     ip_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
6     arc4_mtx -- last acquired _at_ /usr/src/sys/libkern/arc4random.c:137
6     accept -- (already displayed)
4   pbuf mutex -- last acquired _at_ /usr/src/sys/vm/vm_pager.c:410
4   lo_mtx -- last acquired _at_ /usr/src/sys/net/if_loop.c:154
4   pfil_head_mtx -- last acquired _at_ /usr/src/sys/net/pfil.c:174
5    pfil_head_list lock -- last acquired _at_ /usr/src/sys/net/pfil.c:180
4   pseudofs_vncache -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243
15   vnode interlock -- (already displayed)
4   needsbuffer lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:313
4   Softdep Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:4420
5    buffer daemon lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:421
12   Malloc Stats -- (already displayed)
15   vnode interlock -- (already displayed)
16   UMA pcpu -- (already displayed)
4   runningbufspace lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:332
4   devstat -- last acquired _at_ /usr/src/sys/kern/subr_devstat.c:190
4   ACPI task queue -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118
4   mntid -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:415
5    mountlist -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:3078
6     struct mount mtx -- last acquired _at_ /usr/src/sys/kern/vfs_vnops.c:957
15     vnode interlock -- (already displayed)
4   buf queue lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:1544
15   vnode interlock -- (already displayed)
4   rtsock route_cb lock -- last acquired _at_ /usr/src/sys/net/rtsock.c:234
4   FFS Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:2316
4   fdesc -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:1425
4   g_disk_done -- last acquired _at_ /usr/src/sys/geom/geom_disk.c:198
5    bio queue -- last acquired _at_ /usr/src/sys/geom/geom_io.c:66
16   UMA pcpu -- (already displayed)
4   dirhash list -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348
5    dirhash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:456
4   rawcb -- last acquired _at_ /usr/src/sys/net/raw_cb.c:105
8    so_rcv -- (already displayed)
4   udp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:246
5    udp6inp -- last acquired _at_ /usr/src/sys/netinet6/udp6_usrreq.c:670
6     if_afdata -- last acquired _at_ /usr/src/sys/netinet6/scope6.c:272
6     ip6_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
6     accept -- (already displayed)
5    udpinp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:386
6     arc4_mtx -- (already displayed)
6     NFS reqq lock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:1049
7      Synch NFS reply posting -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:584
9      process lock -- (already displayed)
6     accept -- (already displayed)
4   ufs ihash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_ihash.c:156
15   vnode interlock -- (already displayed)
4   semid -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1002
5    sem -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1258
4   unp -- last acquired _at_ /usr/src/sys/kern/uipc_usrreq.c:329
6    accept -- (already displayed)
0 g_xup -- last acquired _at_ /usr/src/sys/geom/geom_io.c:484
3  Giant -- (already displayed)
0 arp_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
0 igmp_mtx -- last acquired _at_ /usr/src/sys/netinet/igmp.c:431
0 ipqlock -- last acquired _at_ /usr/src/sys/netinet/ip_input.c:1110
16 UMA pcpu -- (already displayed)
0 ip6qlock -- last acquired _at_ /usr/src/sys/netinet6/frag6.c:682
0 nfslock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_lock.c:123
0 GEOM topology -- last acquired _at_ /usr/src/sys/geom/geom_event.c:220
1  fdc lock -- last acquired _at_ /usr/src/sys/dev/fdc/fdc.c:777
2   callout_wait_lock -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:289
1  swapdev -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:2102
3  Giant -- (already displayed)
0 p_peers -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:250
0 ACPI PCI power methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207
0 ACPI root bus -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969
4  rman -- (already displayed)
4  ACPI semaphore -- (already displayed)
0 ACPI PCI bus methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214
1  ACPI PCI link -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972
2   kernel environment -- last acquired _at_ /usr/src/sys/kern/kern_environment.c:285
4   ACPI semaphore -- (already displayed)
12  Malloc Stats -- (already displayed)
16  UMA pcpu -- (already displayed)
0 ACPI CPU -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817
0 module subsystem sx lock -- last acquired _at_ /usr/src/sys/kern/kern_module.c:115
0 protect sysfilt_ops -- last acquired _at_ /usr/src/sys/kern/kern_event.c:669
0 sysctl lock -- last acquired _at_ /usr/src/sys/kern/kern_sysctl.c:1335
1  filelist lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2128
4   filedesc structure -- (already displayed)
1  addrsel_lock -- last acquired _at_ /usr/src/sys/netinet6/in6_src.c:1137
1  rip -- last acquired _at_ /usr/src/sys/netinet/raw_ip.c:866
1  allproc -- last acquired _at_ /usr/src/sys/kern/sched_4bsd.c:435
2   user map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2998
3    Giant -- (already displayed)
0 ddp_list_mtx -- last acquired _at_ order list:0
1  ddp_mtx -- last acquired _at_ order list:0
0 slip_mtx -- last acquired _at_ order list:0
1  slip sc_mtx -- last acquired _at_ order list:0
0 proctree -- last acquired _at_ /usr/src/sys/kern/tty.c:2076
1  allproc -- (already displayed)


Slow:

Sleep locks:
0 g_xdown -- last acquired _at_ /usr/src/sys/geom/geom_io.c:423
3  Giant -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:247
4   UMA lock -- last acquired _at_ /usr/src/sys/vm/uma_core.c:1491
12   Malloc Stats -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:228
12   system map -- last acquired _at_ /usr/src/sys/vm/vm_map.c:2317
13    kmem object -- last acquired _at_ /usr/src/sys/vm/vm_kern.c:397
14     vm page queue mutex -- last acquired _at_ /usr/src/sys/vm/vm_pageout.c:1433
15      vnode interlock -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1489
16       vnode_free_list -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:590
16       Syncer mtx -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:1617
16       cdev -- last acquired _at_ /usr/src/sys/kern/kern_conf.c:70
15      pmap -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:1626
16       UMA pcpu -- last acquired _at_ /usr/src/sys/vm/uma_core.c:2251
17        KMAP ENTRY -- last acquired _at_ /usr/src/sys/vm/uma_core.c:418
18         UMA zone -- last acquired _at_ /usr/src/sys/vm/uma_core.c:2269
16       uma object -- last acquired _at_ /usr/src/sys/vm/uma_core.c:988
16       SYSMAPS -- last acquired _at_ /usr/src/sys/i386/i386/pmap.c:2422
13    kernel object -- last acquired _at_ /usr/src/sys/vm/vm_object.c:449
14     vm page queue mutex -- (already displayed)
4   vm86 lock -- last acquired _at_ /usr/src/sys/i386/i386/vm86.c:581
4   eventhandler -- last acquired _at_ /usr/src/sys/kern/subr_eventhandler.c:213
5    eventhandler list -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:199
4   malloc -- last acquired _at_ /usr/src/sys/kern/kern_malloc.c:584
4   kernel linker -- last acquired _at_ /usr/src/sys/kern/kern_linker.c:1061
4   filedesc structure -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:814
6    accept -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:460
7     so_snd -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2112
8      tcp_hc_entry -- last acquired _at_ /usr/src/sys/netinet/tcp_hostcache.c:654
12      system map -- (already displayed)
8      so_rcv -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:2113
9       sellck -- last acquired _at_ /usr/src/sys/kern/sys_generic.c:713
9       radix node head -- last acquired _at_ /usr/src/sys/net/route.c:148
10       ifnet -- last acquired _at_ /usr/src/sys/net/if.c:1056
11        random reseed -- last acquired _at_ /usr/src/sys/dev/random/yarrow.c:193
11        arc4_mtx -- last acquired _at_ /usr/src/sys/libkern/arc4random.c:137
10       rtentry -- last acquired _at_ /usr/src/sys/netinet/ip_output.c:822
11        rts_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
11        network driver -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1756
12         Malloc Stats -- (already displayed)
12         knlist lock for lockless objects -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453
12         if send queue -- last acquired _at_ /usr/src/sys/dev/fxp/if_fxp.c:1268
12         system map -- (already displayed)
11        ifaddr -- last acquired _at_ /usr/src/sys/net/route.c:788
9       process lock -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:368
10       struct pargs.ref -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:1110
10       ktrace -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:354
10       sigacts -- last acquired _at_ /usr/src/sys/kern/subr_sleepqueue.c:369
10       session -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:450
15        vnode interlock -- (already displayed)
11        tty -- last acquired _at_ /usr/src/sys/kern/kern_event.c:1453
11        uidinfo hash -- last acquired _at_ /usr/src/sys/kern/kern_resource.c:1034
12         sleep mtxpool -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2099
12         uidinfo struct -- last acquired _at_ order list:0
13          allprison -- last acquired _at_ /usr/src/sys/kern/kern_jail.c:460
5    pipe mutex -- last acquired _at_ /usr/src/sys/kern/sys_pipe.c:1340
14    vm page queue mutex -- (already displayed)
9     sellck -- (already displayed)
6     sigio lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:739
7      process group -- last acquired _at_ /usr/src/sys/kern/kern_proc.c:257
9       process lock -- (already displayed)
4   devd -- last acquired _at_ /usr/src/sys/kern/subr_bus.c:497
9    sellck -- (already displayed)
4   bounce pages lock -- last acquired _at_ /usr/src/sys/i386/i386/busdma_machdep.c:1054
4   kobj -- last acquired _at_ /usr/src/sys/kern/subr_kobj.c:298
4   TID lock -- last acquired _at_ /usr/src/sys/kern/kern_thread.c:204
4   rman -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:448
12   Malloc Stats -- (already displayed)
12   system map -- (already displayed)
4   bpf global lock -- last acquired _at_ /usr/src/sys/net/bpf.c:1446
5    bpf interface lock -- last acquired _at_ order list:0
6     bpf cdev lock -- last acquired _at_ order list:0
4   ACPI semaphore -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:300
4   acpica subsystem lock -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSynch.c:360
4   GEOM orphanage -- last acquired _at_ /usr/src/sys/geom/geom_event.c:188
4   ithread -- last acquired _at_ /usr/src/sys/kern/kern_intr.c:275
4   ttylist -- last acquired _at_ /usr/src/sys/kern/tty.c:2799
11   tty -- (already displayed)
4   taskqueue list -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:85
4   rman head -- last acquired _at_ /usr/src/sys/kern/subr_rman.c:111
4   taskqueue -- last acquired _at_ /usr/src/sys/kern/subr_taskqueue.c:194
4   domain list -- last acquired _at_ /usr/src/sys/kern/uipc_domain.c:224
4   bdone lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3031
4   sf_buf -- last acquired _at_ /usr/src/sys/i386/i386/vm_machdep.c:680
4   tcp -- last acquired _at_ /usr/src/sys/netinet/tcp_subr.c:1359
5    tcpinp -- last acquired _at_ /usr/src/sys/netinet/tcp_input.c:747
6     ip6_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
6     if_afdata -- last acquired _at_ /usr/src/sys/netinet6/scope6.c:272
6     so_glabel -- last acquired _at_ /usr/src/sys/kern/uipc_socket.c:273
6     ip_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
6     accept -- (already displayed)
4   if_cloners lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:199
4   pseudofs -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_fileno.c:86
4   Name Cache -- last acquired _at_ /usr/src/sys/kern/vfs_cache.c:347
12   system map -- (already displayed)
4   if_clone lock -- last acquired _at_ /usr/src/sys/net/if_clone.c:321
4   nfsd_mtx -- last acquired _at_ /usr/src/sys/nfsserver/nfs_srvsock.c:811
8    so_rcv -- (already displayed)
4   udp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:246
5    udp6inp -- last acquired _at_ /usr/src/sys/netinet6/udp6_usrreq.c:670
6     ip6_inq -- (already displayed)
6     if_afdata -- (already displayed)
6     accept -- (already displayed)
5    udpinp -- last acquired _at_ /usr/src/sys/netinet/udp_usrreq.c:386
6     ip_inq -- (already displayed)
6     NFS reqq lock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:1049
7      Synch NFS reply posting -- last acquired _at_ /usr/src/sys/nfsclient/nfs_socket.c:584
6     accept -- (already displayed)
4   runningbufspace lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:332
4   pfil_head_mtx -- last acquired _at_ /usr/src/sys/net/pfil.c:174
5    pfil_head_list lock -- last acquired _at_ /usr/src/sys/net/pfil.c:180
4   pbuf mutex -- last acquired _at_ /usr/src/sys/vm/vm_pager.c:410
4   lo_mtx -- last acquired _at_ /usr/src/sys/net/if_loop.c:154
4   pseudofs_vncache -- last acquired _at_ /usr/src/sys/fs/pseudofs/pseudofs_vncache.c:243
15   vnode interlock -- (already displayed)
4   rtsock route_cb lock -- last acquired _at_ /usr/src/sys/net/rtsock.c:234
4   ACPI task queue -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/Osd/OsdSchedule.c:118
4   devstat -- last acquired _at_ /usr/src/sys/kern/subr_devstat.c:83
4   fdesc -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:1425
4   mntid -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:415
5    mountlist -- last acquired _at_ /usr/src/sys/kern/vfs_subr.c:3078
6     struct mount mtx -- last acquired _at_ /usr/src/sys/kern/vfs_vnops.c:984
15     vnode interlock -- (already displayed)
4   rawcb -- last acquired _at_ /usr/src/sys/net/raw_cb.c:105
8    so_rcv -- (already displayed)
4   unp -- last acquired _at_ /usr/src/sys/kern/uipc_usrreq.c:329
6    accept -- (already displayed)
4   FFS Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_vfsops.c:1044
5    needsbuffer lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:313
5    buf queue lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:1544
6     struct mount mtx -- (already displayed)
6     Softdep Lock -- last acquired _at_ /usr/src/sys/ufs/ffs/ffs_softdep.c:402
7      buffer daemon lock -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:2151
12     Malloc Stats -- (already displayed)
12     system map -- (already displayed)
5    standard object -- last acquired _at_ /usr/src/sys/kern/vfs_bio.c:3523
6     vm object_list -- last acquired _at_ /usr/src/sys/vm/vm_object.c:643
6     swap_pager swhash -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:1961
7      swapdev -- last acquired _at_ /usr/src/sys/vm/swap_pager.c:2245
16     uma object -- (already displayed)
16     UMA pcpu -- (already displayed)
14    vm page queue mutex -- (already displayed)
4   g_disk_done -- last acquired _at_ /usr/src/sys/geom/geom_disk.c:198
5    bio queue -- last acquired _at_ /usr/src/sys/geom/geom_io.c:66
12   system map -- (already displayed)
4   rip -- last acquired _at_ /usr/src/sys/netinet/raw_ip.c:195
6    accept -- (already displayed)
4   semid -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1286
5    sem -- last acquired _at_ /usr/src/sys/kern/sysv_sem.c:1258
4   dirhash list -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:348
5    dirhash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_dirhash.c:349
4   ufs ihash -- last acquired _at_ /usr/src/sys/ufs/ufs/ufs_ihash.c:156
15   vnode interlock -- (already displayed)
4   linux osname -- last acquired _at_ /usr/src/sys/modules/linux/../../compat/linux/linux_mib.c:231
4   ipqlock -- last acquired _at_ /usr/src/sys/netinet/ip_input.c:1110
16   UMA pcpu -- (already displayed)
0 g_xup -- last acquired _at_ /usr/src/sys/geom/geom_io.c:484
3  Giant -- (already displayed)
0 arp_inq -- last acquired _at_ /usr/src/sys/net/netisr.c:232
0 igmp_mtx -- last acquired _at_ /usr/src/sys/netinet/igmp.c:431
0 ip6qlock -- last acquired _at_ /usr/src/sys/netinet6/frag6.c:682
0 nfslock -- last acquired _at_ /usr/src/sys/nfsclient/nfs_lock.c:123
0 GEOM topology -- last acquired _at_ /usr/src/sys/geom/geom_event.c:220
1  fdc lock -- last acquired _at_ /usr/src/sys/dev/fdc/fdc.c:777
2   callout_wait_lock -- last acquired _at_ /usr/src/sys/kern/kern_timeout.c:289
3  Giant -- (already displayed)
0 p_peers -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:250
0 ACPI PCI power methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci.c:207
0 ACPI root bus -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi.c:969
4  rman -- (already displayed)
4  ACPI semaphore -- (already displayed)
0 ACPI PCI bus methods -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pcib.c:214
1  ACPI PCI link -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_pci_link.c:972
4   ACPI semaphore -- (already displayed)
12  Malloc Stats -- (already displayed)
2   kernel environment -- last acquired _at_ /usr/src/sys/kern/kern_environment.c:285
16  UMA pcpu -- (already displayed)
0 ACPI CPU -- last acquired _at_ /usr/src/sys/modules/acpi/acpi/../../../dev/acpica/acpi_cpu.c:817
0 module subsystem sx lock -- last acquired _at_ /usr/src/sys/kern/kern_module.c:115
0 protect sysfilt_ops -- last acquired _at_ /usr/src/sys/kern/kern_event.c:669
0 sysctl lock -- last acquired _at_ /usr/src/sys/kern/kern_sysctl.c:1335
1  filelist lock -- last acquired _at_ /usr/src/sys/kern/kern_descrip.c:2128
4   filedesc structure -- (already displayed)
1  addrsel_lock -- last acquired _at_ /usr/src/sys/netinet6/in6_src.c:1137
1  allproc -- last acquired _at_ /usr/src/sys/kern/sched_4bsd.c:435
2   user map -- last acquired _at_ /usr/src/sys/vm/vm_glue.c:176
3    Giant -- (already displayed)
0 ddp_list_mtx -- last acquired _at_ order list:0
1  ddp_mtx -- last acquired _at_ order list:0
0 slip_mtx -- last acquired _at_ order list:0
1  slip sc_mtx -- last acquired _at_ order list:0
0 proctree -- last acquired _at_ /usr/src/sys/kern/kern_exit.c:591
1  allproc -- (already displayed)
Received on Thu Feb 10 2005 - 10:04:21 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:28 UTC