Hello Am 21.11.2008 um 01:01 schrieb Thomas: > Hello > > I encounter a deadlock while running a few rsync processes mirroring > remote data to my local zfs pool. After a few hours my system is > starting more and more vsftpd sessions without closing any inactive > ftp sessions. I can't kill any rsync and vsftpd processes with "kill > -9". Even shutdown -r now does not work. > > I got a few hunderts vsftpd processes like this > > 61346 root 1 57 0 7880K 1692K zfs 1 0:00 0.00% > vsftpd > 61481 root 1 68 0 7880K 1696K zfs 1 0:00 0.00% > vsftpd > 61354 root 1 65 0 7880K 1692K zfs 1 0:00 0.00% > vsftpd > 61480 root 1 68 0 7880K 1696K zfs 0 0:00 0.00% > vsftpd > 61600 root 1 69 0 7880K 1704K zfs 1 0:00 0.00% > vsftpd > 61599 root 1 68 0 7880K 1704K zfs 1 0:00 0.00% > vsftpd > > Right now i'm building a debug kernel. Whats the best way to get > usefull information from this deadlock? The system itself is not > crashing and response well to ssh and i also have a serial console. > > The system is running 8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008 > (64bit) with zpool version 13. I use zfs only as data pool. The base > system is running on ufs2: > > Filesystem Size Used Avail Capacity Mounted on > /dev/da0s1a 496M 220M 236M 48% / > devfs 1.0K 1.0K 0B 100% /dev > /dev/da0s1g 169G 15G 141G 9% /disk1 > /dev/da0s1f 3.9G 17M 3.5G 0% /tmp > /dev/da0s1e 29G 3.7G 23G 14% /usr > /dev/da0s1d 19G 7.1G 11G 40% /var > pool 853G 0B 853G 0% /usr/local/data > pool/cvsup 858G 5.7G 853G 1% /usr/local/data/cvsup > pool/ftp 3.3T 2.5T 853G 75% /usr/local/data/ftp > pool/portsnap 853G 633M 853G 0% /usr/local/data/ > portsnap > pool/www 853G 90M 853G 0% /usr/local/data/www > > loader.conf: > vm.kmem_size="1G" > kern.maxfiles="65536" > kern.maxproc="20480" > net.inet.tcp.tcbhashsize="4096" > net.inet.tcp.hostcache.hashsize="1024" > vfs.zfs.arc_min="64M" > vfs.zfs.arc_max="768M" > vfs.zfs.prefetch_disable="1" > > I also tried to set vfs.zfs.zil_disable=1 but the problem still exist. > > I know there are few deadlock reports listed in the freebsd wiki. > Maybe we can trigger the root cause of the problem and fix it :) > > Regards > Thomas I recompiled my kernel with debug option. Maybe i have some useful data. I did most commands listed at http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-deadlocks.html . alltrace created a ridiculous long list. I added all output to a text file. http://www.bsdunix.ch/deadlock_2008-11-21.txt Just the ps output and a trace output as example: db> ps pid ppid pgrp uid state wmesg wchan cmd 21909 2551 21909 0 R+ CPU 0 sysctl 21865 21848 21831 1001 S select 0xffffff002a8164c0 rsync 21863 21844 21827 1001 S zfs 0xffffff00038a7cc8 rsync 21861 21847 21838 1001 S zfs 0xffffff00038a7cc8 rsync 21858 21845 21835 1001 S zfs 0xffffff00038a7cc8 rsync 21852 21841 21841 1001 S zfs 0xffffff00038a7cc8 perl 21850 21837 21837 1001 S zfs 0xffffff00038a7cc8 perl5.8.8 21848 21831 21831 1001 S wait 0xffffff0106a82430 sh 21847 21838 21838 1001 S wait 0xffffff001ec22860 sh 21846 21839 21839 0 S db->db_c 0xffffff0119b826b0 newsyslog 21845 21835 21835 1001 S wait 0xffffff003094f860 sh 21844 21827 21827 1001 S wait 0xffffff0138091000 sh 21841 21830 21841 1001 Ss wait 0xffffff0030936430 sh 21839 21832 21839 0 Ss wait 0xffffff01428a8860 sh 21838 21825 21838 1001 Ss wait 0xffffff0030935000 sh 21837 21829 21837 1001 Ss wait 0xffffff003094b430 sh 21835 21828 21835 1001 Ss wait 0xffffff01428a8430 sh 21832 1016 1016 0 S piperd 0xffffff003097fb20 cron 21831 21826 21831 1001 Ss wait 0xffffff01428a3430 sh 21830 1016 1016 0 S piperd 0xffffff01383f4858 cron 21829 1016 1016 0 S piperd 0xffffff0023dd42c8 cron 21828 1016 1016 0 S piperd 0xffffff003098b590 cron 21827 21824 21827 1001 Ss wait 0xffffff0030936000 sh 21826 1016 1016 0 S piperd 0xffffff01383f4000 cron 21825 1016 1016 0 S piperd 0xffffff000383e590 cron 21824 1016 1016 0 S piperd 0xffffff0023dd5858 cron 19454 19448 19448 0 S db->db_c 0xffffff0119b826b0 newsyslog 19448 19445 19448 0 Ss wait 0xffffff0106a80000 sh 19445 1016 1016 0 S piperd 0xffffff000383e858 cron 14894 14893 14894 1007 Ss db->db_c 0xffffff0119b826b0 cat 14893 14891 14891 1007 S select 0xffffff0018caa1c0 sshd 14891 997 14891 0 Ss sbwait 0xffffff0106a048b4 sshd 13493 13479 13479 1001 S zfs 0xffffff00038a7cc8 perl5.8.8 13490 13483 13483 0 S db->db_c 0xffffff0119b826b0 newsyslog 13483 13477 13483 0 Ss wait 0xffffff0138090860 sh 13479 13476 13479 1001 Ss wait 0xffffff0138296860 sh 13477 1016 1016 0 S piperd 0xffffff01383f62c8 cron 13476 1016 1016 0 S piperd 0xffffff0105ed82c8 cron 9316 9311 9311 0 S db->db_c 0xffffff0119b826b0 newsyslog 9311 9308 9311 0 Ss wait 0xffffff0138294430 sh 9308 1016 1016 0 S piperd 0xffffff000383cb20 cron 7843 943 943 0 S zfs 0xffffff00038a7cc8 rsync 7767 7764 7764 1001 S zfs 0xffffff00038a7cc8 perl 7764 7761 7764 1001 Ss wait 0xffffff001ec21860 sh 7761 1016 1016 0 S piperd 0xffffff0105ed8590 cron 7693 943 943 0 S zfs 0xffffff00038a7cc8 rsync 7490 943 943 0 S zfs 0xffffff00038a7cc8 rsync 7263 943 943 0 S zfs 0xffffff00038a7cc8 rsync 6758 943 943 0 S zfs 0xffffff00038a7cc8 rsync 6630 943 943 0 S zfs 0xffffff00038a7cc8 rsync 6582 943 943 0 S zfs 0xffffff00038a7cc8 rsync 6529 6524 6524 0 S db->db_c 0xffffff0119b826b0 newsyslog 6527 6520 6520 1001 S zfs 0xffffff00038a7cc8 perl5.8.8 6524 6518 6524 0 Ss wait 0xffffff001ec23860 sh 6520 6517 6520 1001 Ss wait 0xffffff001ec05430 sh 6518 1016 1016 0 S piperd 0xffffff0023dd5000 cron 6517 1016 1016 0 S piperd 0xffffff000383c590 cron 6478 943 943 0 S zfs 0xffffff00038a7cc8 rsync 6053 943 943 0 S zfs 0xffffff00038a7cc8 rsync 5788 5786 5786 0 S zfs 0xffffff00038a7cc8 vsftpd 5786 823 5786 0 Ss sbwait 0xffffff0138272b34 vsftpd 5785 5783 5783 0 S zfs 0xffffff00038a7cc8 vsftpd 5783 823 5783 0 Ss sbwait 0xffffff01382f1634 vsftpd 5766 5764 5764 0 S zfs 0xffffff00038a7cc8 vsftpd 5764 823 5764 0 Ss sbwait 0xffffff01381b63b4 vsftpd 5753 5751 5751 0 S zfs 0xffffff00038a7cc8 vsftpd 5751 823 5751 0 Ss sbwait 0xffffff01382703b4 vsftpd 5744 5742 5742 0 S zfs 0xffffff00038a7cc8 vsftpd 5742 823 5742 0 Ss sbwait 0xffffff01381b6134 vsftpd 5733 5731 5731 0 S zfs 0xffffff00038a7cc8 vsftpd 5731 823 5731 0 Ss sbwait 0xffffff0138142db4 vsftpd 5730 5728 5728 0 S zfs 0xffffff00038a7cc8 vsftpd 5728 823 5728 0 Ss sbwait 0xffffff01381b5634 vsftpd 5727 5724 5720 1001 S zfs 0xffffff00038a7cc8 rsync 5724 5720 5720 1001 S wait 0xffffff013808d000 sh 5720 5717 5720 1001 Ss wait 0xffffff0037e6b000 sh 5717 1016 1016 0 S piperd 0xffffff00038d2000 cron 5714 943 943 0 S zfs 0xffffff00038a7cc8 rsync 5713 5711 5711 0 S zfs 0xffffff00038a7cc8 vsftpd 5711 823 5711 0 Ss sbwait 0xffffff01381b7634 vsftpd 5709 5707 5707 0 S zfs 0xffffff00038a7cc8 vsftpd 5707 823 5707 0 Ss sbwait 0xffffff01380d2b34 vsftpd 5706 5704 5704 0 S zfs 0xffffff00038a7cc8 vsftpd 5704 823 5704 0 Ss sbwait 0xffffff01381413b4 vsftpd 5696 5694 5694 0 S zfs 0xffffff00038a7cc8 vsftpd 5694 823 5694 0 Ss sbwait 0xffffff0138142634 vsftpd 5669 943 943 0 S zfs 0xffffff00038a7cc8 rsync 5627 5625 5598 1007 S zio->io_ 0xffffff00372c3df8 sh 5626 5600 5598 1007 S piperd 0xffffff00038d2858 grep 5625 5600 5598 1007 S wait 0xffffff013808e430 sh 5610 5597 5610 1007 Ss piperd 0xffffff000383e2c8 sendmail 5600 5599 5598 1007 S wait 0xffffff0037fec860 sh 5599 5598 5598 1007 S wait 0xffffff0037ca2430 lockf 5598 5597 5598 1007 Ss wait 0xffffff0037e67000 sh 5597 1016 1016 0 S piperd 0xffffff0105ed7b20 cron 5596 5594 5594 0 S zfs 0xffffff00038a7cc8 vsftpd 5594 823 5594 0 Ss sbwait 0xffffff00376a93b4 vsftpd 5584 5582 5582 0 S zfs 0xffffff00038a7cc8 vsftpd 5582 823 5582 0 Ss sbwait 0xffffff0037f66634 vsftpd 5570 5568 5568 0 S zfs 0xffffff00038a7cc8 vsftpd 5568 823 5568 0 Ss sbwait 0xffffff0037f6eb34 vsftpd 5472 5470 5470 0 S zfs 0xffffff00038a7cc8 vsftpd 5470 823 5470 0 Ss sbwait 0xffffff0037ff6634 vsftpd 5459 5457 5457 0 S zfs 0xffffff00038a7cc8 vsftpd 5457 823 5457 0 Ss sbwait 0xffffff0037ff9db4 vsftpd 5442 5440 5440 0 S zfs 0xffffff00038a7cc8 vsftpd 5440 823 5440 0 Ss sbwait 0xffffff0037ec3b34 vsftpd 5399 5397 5397 0 S zfs 0xffffff00038a7cc8 vsftpd 5397 823 5397 0 Ss sbwait 0xffffff0037ec48b4 vsftpd 5375 5358 5358 0 S zfs 0xffffff00038a7cc8 vsftpd 5374 5372 5372 0 S zfs 0xffffff00038a7cc8 vsftpd 5372 823 5372 0 Ss sbwait 0xffffff0037eb73b4 vsftpd 5367 5362 5362 0 S zfs 0xffffff00038a7cc8 vsftpd 5366 5364 5364 1004 S zio->io_ 0xffffff0003993b28 vsftpd 5364 823 5364 0 Ss sbwait 0xffffff0037eb6634 vsftpd 5362 823 5362 0 Ss sbwait 0xffffff0037eb6db4 vsftpd 5358 823 5358 0 Ss sbwait 0xffffff0037eb8134 vsftpd 5356 5353 5353 1004 S zio->io_ 0xffffff00372c82b8 vsftpd 5353 823 5353 0 Ss sbwait 0xffffff0037e0e634 vsftpd 5349 5347 5347 1004 S db->db_c 0xffffff0121effb10 vsftpd 5347 823 5347 0 Ss sbwait 0xffffff0037dc8db4 vsftpd 5344 5342 5342 1004 S db->db_c 0xffffff0121effb10 vsftpd 5342 823 5342 0 Ss sbwait 0xffffff0037e0edb4 vsftpd 5328 5326 5326 1004 S zfs 0xffffff0007de0308 vsftpd 5326 823 5326 0 Ss sbwait 0xffffff0037e10134 vsftpd 5323 5321 5321 1004 S db->db_c 0xffffff0121effb10 vsftpd 5321 823 5321 0 Ss sbwait 0xffffff0037d2db34 vsftpd 5318 5301 5301 1004 S db->db_c 0xffffff0121effb10 vsftpd 5301 823 5301 0 Ss sbwait 0xffffff0037dc7db4 vsftpd 5297 5295 5295 1004 S db->db_c 0xffffff0121effb10 vsftpd 5295 823 5295 0 Ss sbwait 0xffffff0037cb68b4 vsftpd 5288 5286 5286 1004 S db->db_c 0xffffff0121effb10 vsftpd 5286 823 5286 0 Ss sbwait 0xffffff0037d2cb34 vsftpd 5285 5283 5283 1004 S db->db_c 0xffffff0121effb10 vsftpd 5283 823 5283 0 Ss sbwait 0xffffff0037d2c8b4 vsftpd 5258 5256 5256 1004 S db->db_c 0xffffff0121effb10 vsftpd 5256 823 5256 0 Ss sbwait 0xffffff0037d2c3b4 vsftpd 5251 5249 5249 1004 S zio->io_ 0xffffff0037368b28 vsftpd 5249 823 5249 0 Ss sbwait 0xffffff0037ce78b4 vsftpd 5246 5244 5244 1004 S db->db_c 0xffffff0121effb10 vsftpd 5244 823 5244 0 Ss sbwait 0xffffff0037cb43b4 vsftpd 5226 5224 5224 1004 S zfs 0xffffff0007de0308 vsftpd 5224 823 5224 0 Ss sbwait 0xffffff0037ce7634 vsftpd 5223 5221 5221 1004 S db->db_c 0xffffff0121effb10 vsftpd 5221 823 5221 0 Ss sbwait 0xffffff0037afadb4 vsftpd 5189 5187 5187 1004 S db->db_c 0xffffff0121effb10 vsftpd 5187 823 5187 0 Ss sbwait 0xffffff0037c98db4 vsftpd 5186 5184 5184 1004 S zfs 0xffffff0007de0308 vsftpd 5184 823 5184 0 Ss sbwait 0xffffff0037cd8134 vsftpd 5134 5132 5132 1004 S db->db_c 0xffffff0121effb10 vsftpd 5132 823 5132 0 Ss sbwait 0xffffff0037bc0b34 vsftpd 5131 5129 5129 1004 S zfs 0xffffff0007de0308 vsftpd 5129 823 5129 0 Ss sbwait 0xffffff0037bc73b4 vsftpd 5128 5126 5126 1004 S zio->io_ 0xffffff01103f7858 vsftpd 5126 823 5126 0 Ss sbwait 0xffffff0037bc0634 vsftpd 5092 5090 5090 1004 S db->db_c 0xffffff0121effb10 vsftpd 5090 823 5090 0 Ss sbwait 0xffffff0037bd4634 vsftpd 5073 5071 5071 1004 S zio->io_ 0xffffff0003a7a2b8 vsftpd 5071 823 5071 0 Ss sbwait 0xffffff0037af8b34 vsftpd 5066 5064 5064 1004 S db->db_c 0xffffff0121effb10 vsftpd 5064 823 5064 0 Ss sbwait 0xffffff0037aa2db4 vsftpd 5044 5042 5042 1004 S db->db_c 0xffffff0121effb10 vsftpd 5042 823 5042 0 Ss sbwait 0xffffff0037aa2134 vsftpd 5041 943 943 65534 S db->db_c 0xffffff012a1b1e90 rsync 5040 5038 5038 1004 S db->db_c 0xffffff0121effb10 vsftpd 5038 823 5038 0 Ss sbwait 0xffffff0037aa2634 vsftpd 5007 5005 5005 1004 S db->db_c 0xffffff0121effb10 vsftpd 5005 823 5005 0 Ss sbwait 0xffffff00379c28b4 vsftpd 5003 5001 5001 1004 S db->db_c 0xffffff0121effb10 vsftpd 5001 823 5001 0 Ss sbwait 0xffffff0037a20634 vsftpd 5000 4998 4998 1004 S db->db_c 0xffffff0121effb10 vsftpd 4998 823 4998 0 Ss sbwait 0xffffff0037a20db4 vsftpd 4983 4981 4981 1004 S db->db_c 0xffffff0121effb10 vsftpd 4981 823 4981 0 Ss sbwait 0xffffff003787d8b4 vsftpd 4979 4977 4977 1004 S db->db_c 0xffffff0121effb10 vsftpd 4977 823 4977 0 Ss sbwait 0xffffff00379f0db4 vsftpd 4973 4970 4973 1003 S zio->io_ 0xffffff0133e8d858 cvsup 4970 4967 4958 0 S wait 0xffffff00379eb860 su 4967 4966 4958 0 S wait 0xffffff0007c11430 sh 4966 4960 4958 0 S wait 0xffffff0018402000 lockf 4960 4958 4958 0 S wait 0xffffff003786a000 sh 4958 4956 4958 0 Ss wait 0xffffff003787b000 sh 4956 1016 1016 0 S piperd 0xffffff000383eb20 cron 4952 4950 4950 1004 S db->db_c 0xffffff0130177e90 vsftpd 4950 823 4950 0 Ss sbwait 0xffffff003787c134 vsftpd 4933 943 943 65534 S db->db_c 0xffffff012a1b1e90 rsync 4894 4892 4892 1004 S db->db_c 0xffffff0121effb10 vsftpd 4892 823 4892 0 Ss sbwait 0xffffff0037957db4 vsftpd 4882 4880 4880 1004 S zfsvfs-> 0xffffff00079b1800 vsftpd 4880 823 4880 0 Ss sbwait 0xffffff0123edb3b4 vsftpd 4861 4860 4799 0 S zio->io_ 0xffffff0133bb72b8 df 4860 4816 4799 0 S wait 0xffffff002393a000 sh 4818 4817 4799 0 S piperd 0xffffff0105ed62c8 mail 4817 4802 4799 0 S wait 0xffffff0037868430 sh 4816 4802 4799 0 S wait 0xffffff0037868000 sh 4802 4799 4799 0 S wait 0xffffff00631cb860 sh 4799 4796 4799 0 Ss wait 0xffffff003787a000 sh 4796 1016 1016 0 S piperd 0xffffff0105ed6858 cron 4770 4765 4765 0 S zio->io_ 0xffffff0003ae7858 newsyslog 4765 4762 4765 0 Ss wait 0xffffff00631b4860 sh 4762 1016 1016 0 S piperd 0xffffff000383c2c8 cron 4756 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd 4755 970 970 1002 S zio->io_ 0xffffff0133bad588 cvsupd 4754 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd 4753 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd 4740 970 970 1002 S zio->io_ 0xffffff0133dcfb28 cvsupd 4739 970 970 1002 S db->db_c 0xffffff0103cef250 cvsupd 4710 4708 4708 1004 S db->db_c 0xffffff0130177e90 vsftpd 4708 823 4708 0 Ss sbwait 0xffffff00376f8634 vsftpd 4676 4674 4674 1004 S db->db_c 0xffffff0121effb10 vsftpd 4674 823 4674 0 Ss sbwait 0xffffff00376f93b4 vsftpd 4673 4671 4671 1004 S db->db_c 0xffffff0121effb10 vsftpd 4671 823 4671 0 Ss sbwait 0xffffff0037653db4 vsftpd 4670 4668 4668 1004 S db->db_c 0xffffff0121effb10 vsftpd 4668 823 4668 0 Ss sbwait 0xffffff00376a7b34 vsftpd 4667 4665 4665 1004 S db->db_c 0xffffff0121effb10 vsftpd 4665 823 4665 0 Ss sbwait 0xffffff00376a9134 vsftpd 4664 4662 4662 1004 S db->db_c 0xffffff0121effb10 vsftpd 4662 823 4662 0 Ss sbwait 0xffffff0037651b34 vsftpd 4661 4659 4659 1004 S db->db_c 0xffffff0121effb10 vsftpd 4659 823 4659 0 Ss sbwait 0xffffff00376523b4 vsftpd 4647 4645 4645 1004 S db->db_c 0xffffff0121effb10 vsftpd 4645 823 4645 0 Ss sbwait 0xffffff00375e3634 vsftpd 4644 4642 4642 1004 S db->db_c 0xffffff0121effb10 vsftpd 4642 823 4642 0 Ss sbwait 0xffffff00375f4134 vsftpd 4640 4638 4638 1004 S db->db_c 0xffffff0121effb10 vsftpd 4638 823 4638 0 Ss sbwait 0xffffff00375f4db4 vsftpd 4637 4635 4635 1004 S zio->io_ 0xffffff0003a8ddf8 vsftpd 4635 823 4635 0 Ss sbwait 0xffffff0088393634 vsftpd 4588 4586 4586 1004 S db->db_c 0xffffff0121effb10 vsftpd 4586 823 4586 0 Ss sbwait 0xffffff0037560634 vsftpd 4567 4565 4565 1004 S db->db_c 0xffffff0121effb10 vsftpd 4565 823 4565 0 Ss sbwait 0xffffff003753cb34 vsftpd 4553 943 943 65534 S zio->io_ 0xffffff0133bf8b28 rsync 4528 4526 4526 1004 S db->db_c 0xffffff0121effb10 vsftpd 4526 823 4526 0 Ss sbwait 0xffffff0007c318b4 vsftpd 4514 4512 4512 1004 S db->db_c 0xffffff0121effb10 vsftpd 4512 823 4512 0 Ss sbwait 0xffffff00883958b4 vsftpd 4509 4505 4505 1004 S db->db_c 0xffffff0121effb10 vsftpd 4505 823 4505 0 Ss sbwait 0xffffff00883938b4 vsftpd 4499 4497 4497 1004 S db->db_c 0xffffff0121effb10 vsftpd 4497 823 4497 0 Ss sbwait 0xffffff00181f7134 vsftpd 4486 4484 4484 1004 S db->db_c 0xffffff0121effb10 vsftpd 4484 823 4484 0 Ss sbwait 0xffffff0007e78134 vsftpd 4472 970 970 1002 S zio->io_ 0xffffff0133c12b28 cvsupd 4421 4419 4419 1004 S db->db_c 0xffffff0121effb10 vsftpd 4419 823 4419 0 Ss sbwait 0xffffff0123edd634 vsftpd 4412 4410 4410 1004 S db->db_c 0xffffff0121effb10 vsftpd 4410 823 4410 0 Ss sbwait 0xffffff0123739134 vsftpd 4395 4393 4393 1004 S db->db_c 0xffffff0121effb10 vsftpd 4393 823 4393 0 Ss sbwait 0xffffff0018c49b34 vsftpd 4363 4358 4358 1004 S db->db_c 0xffffff0121effb10 vsftpd 4362 4360 4360 1004 S zio->io_ 0xffffff0133b282b8 vsftpd 4360 823 4360 0 Ss sbwait 0xffffff0123739634 vsftpd 4358 823 4358 0 Ss sbwait 0xffffff014e6b3634 vsftpd 4354 4341 4341 1004 S zio->io_ 0xffffff01104612b8 vsftpd 4341 823 4341 0 Ss sbwait 0xffffff0123738134 vsftpd 4337 4335 4335 1004 S zio->io_ 0xffffff0133ae0858 vsftpd 4335 823 4335 0 Ss sbwait 0xffffff00181f63b4 vsftpd 4322 943 943 65534 S zio->io_ 0xffffff003700b588 rsync 4304 970 970 1002 SL zio->io_ 0xffffff011040d2b8 cvsupd 4298 4296 4296 1004 S zio->io_ 0xffffff01103b6df8 vsftpd 4296 823 4296 0 Ss sbwait 0xffffff00181f6134 vsftpd 2551 2549 2551 0 Ss+ pause 0xffffff0007c134d0 csh 2549 997 2549 0 Ss select 0xffffff00079a0dc0 sshd 2470 2469 2466 1001 S+ zfs 0xffffff0139a87a58 rsync 2469 2467 2466 1001 S+ zio->io_ 0xffffff00372c7df8 rsync 2467 2466 2466 1001 S+ wait 0xffffff0018403430 sh 2466 2465 2466 1001 S+ wait 0xffffff0018403000 sh 2465 1206 2465 0 S+ wait 0xffffff00038b2430 su 1206 1190 1206 0 Ss+ pause 0xffffff0018402900 csh 1190 997 1190 0 Ss select 0xffffff0023914240 sshd 1089 1 1089 0 Ss+ tty inpu 0xffffff00036310b8 getty 1088 1 1088 0 Ss+ tty inpu 0xffffff000372d8b8 getty 1087 1 1087 0 Ss+ tty inpu 0xffffff00037318b8 getty 1086 1 1086 0 Ss+ tty inpu 0xffffff000373bcb8 getty 1085 1 1085 0 Ss+ tty inpu 0xffffff000373b8b8 getty 1084 1 1084 0 Ss+ tty inpu 0xffffff000373c8b8 getty 1083 1 1083 0 Ss+ tty inpu 0xffffff000373c4b8 getty 1082 1 1082 0 Ss+ tty inpu 0xffffff000373a8b8 getty 1081 1 1081 0 Ss+ tty inpu 0xffffff000373a4b8 getty 1047 1 1047 0 Ss select 0xffffff0007ac0e40 inetd 1016 1 1016 0 Ss nanslp 0xffffffff808423e8 cron 1010 1 1010 25 Ss pause 0xffffff000383d4d0 sendmail 1006 1 1006 0 Ss select 0xffffff000741d240 sendmail 997 1 997 0 Ss select 0xffffff0007ac0540 sshd 970 1 970 1002 Ss select 0xffffff00038abbc0 cvsupd 949 1 948 80 S zio->io_ 0xffffff0133e0fb28 lighttpd 943 1 943 0 Ss select 0xffffff0003951640 rsync 899 1 899 0 Ss select 0xffffff0007a3d4c0 ntpd 841 1 840 0 S select 0xffffff000741c340 snmpd 823 1 823 0 Ss accept 0xffffff0007c2aa5e vsftpd 703 1 703 0 Ss select 0xffffff0007ac1140 syslogd 534 1 534 0 Ss select 0xffffff00079890c0 devd 171 0 0 0 SL tq->tq_d 0xffffff0007be5600 [zil_clean] 170 0 0 0 SL tq->tq_d 0xffffff00039232a0 [zil_clean] 169 0 0 0 SL tq->tq_d 0xffffff0003923180 [zil_clean] 168 0 0 0 SL tq->tq_d 0xffffff0003923060 [zil_clean] 167 0 0 0 SL tq->tq_d 0xffffff000390e4e0 [zil_clean] 165 0 0 0 SL tx->tx_q 0xffffff0003b6e668 [txg_thread_enter] 164 0 0 0 SL tx->tx_c 0xffffff00039124b0 [txg_thread_enter] 163 0 0 0 SL vgeom:io 0xffffff0003bc5110 [vdev:worker da7] 162 0 0 0 SL vgeom:io 0xffffff000394f490 [vdev:worker da6] 161 0 0 0 SL vgeom:io 0xffffff000394f510 [vdev:worker da5] 160 0 0 0 SL vgeom:io 0xffffff000394f690 [vdev:worker da4] 159 0 0 0 SL vgeom:io 0xffffff000394f710 [vdev:worker da3] 158 0 0 0 SL vgeom:io 0xffffff000394f810 [vdev:worker da2] 157 0 0 0 SL vgeom:io 0xffffff000394f910 [vdev:worker da1] 156 0 0 0 SL tq->tq_d 0xffffff000390e600 [spa_zio] 155 0 0 0 SL tq->tq_d 0xffffff000390e720 [spa_zio] 154 0 0 0 SL tq->tq_d 0xffffff000390e840 [spa_zio] 153 0 0 0 SL tq->tq_d 0xffffff000390e960 [spa_zio] 152 0 0 0 SL tq->tq_d 0xffffff000390f4e0 [spa_zio] 151 0 0 0 SL tq->tq_d 0xffffff000390f180 [spa_zio] 150 0 0 0 SL zfsvfs-> 0xffffff00079b11e0 [spa_zio] 149 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 148 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 147 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 146 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 145 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 144 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 143 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 142 0 0 0 SL tq->tq_d 0xffffff000390eba0 [spa_zio] 141 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 140 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 139 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 138 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 137 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 136 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 135 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 134 0 0 0 SL tq->tq_d 0xffffff000390ecc0 [spa_zio] 133 0 0 0 SL tq->tq_d 0xffffff000390ede0 [spa_zio] 132 0 0 0 SL tq->tq_d 0xffffff000390f2a0 [spa_zio] 131 0 0 0 SL tq->tq_d 0xffffff000390f3c0 [spa_zio] 97 0 0 0 SL l2arc_fe 0xffffffff80d134e0 [l2arc_feed_thread] 96 0 0 0 SL arc_recl 0xffffffff80d0af00 [arc_reclaim_thread] 93 0 0 0 SL tq->tq_d 0xffffff000390f060 [system_taskq] 92 0 0 0 SL tq->tq_d 0xffffff000390f060 [system_taskq] 24 0 0 0 SL sdflush 0xffffffff80a130b8 [softdepflush] 23 0 0 0 SL zio->io_ 0xffffff0133c0e588 [syncer] 22 0 0 0 SL vlruwt 0xffffff000365d000 [vnlru] 21 0 0 0 SL psleep 0xffffffff80a02fdc [bufdaemon] 9 0 0 0 SL pgzero 0xffffffff80a14a2c [pagezero] 8 0 0 0 SL psleep 0xffffffff80a13dc8 [vmdaemon] 7 0 0 0 SL psleep 0xffffffff80a13d8c [pagedaemon] 6 0 0 0 SL waiting_ 0xffffffff80a062a8 [sctp_iterator] 20 0 0 0 SL usbevt 0xffffff00015a9420 [usb4] 19 0 0 0 SL usbevt 0xfffffffe4046c420 [usb3] 18 0 0 0 SL usbevt 0xfffffffe4046a420 [usb2] 17 0 0 0 SL usbevt 0xfffffffe40468420 [usb1] 16 0 0 0 SL usbtsk 0xffffffff8083e0e8 [usbtask- dr] 15 0 0 0 SL usbtsk 0xffffffff8083e0c0 [usbtask- hc] 14 0 0 0 SL usbevt 0xfffffffe40466420 [usb0] 5 0 0 0 SL ccb_scan 0xffffffff80822e60 [xpt_thrd] 13 0 0 0 SL - 0xffffffff808420c4 [yarrow] 4 0 0 0 SL - 0xffffffff8083e848 [g_down] 3 0 0 0 SL - 0xffffffff8083e840 [g_up] 2 0 0 0 SL - 0xffffffff8083e830 [g_event] 12 0 0 0 RL (threaded) intr 100040 I [swi0: uart uart] 100039 I [irq1: atkbd0] 100038 I [irq20: atapci1] 100037 I [irq14: ata0] 100032 I [irq22: uhci1 uhci3] 100028 I [irq23: uhci0 uhci+] 100025 I [irq18: arcmsr0] 100024 I [irq9: acpi0] 100022 I [swi5: +] 100021 I [swi2: cambio] 100015 I [swi6: task queue] 100014 I [swi6: Giant taskq] 100008 I [swi1: net] 100007 I [swi4: clock] 100006 Run CPU 1 [swi4: clock] 100005 I [swi3: vm] 11 0 0 0 RL (threaded) idle 100004 CanRun [idle: cpu0] 100003 CanRun [idle: cpu1] 1 0 1 0 SLs wait 0xffffff0001442860 [init] 10 0 0 0 SL audit_wo 0xffffffff80a124f0 [audit] 0 0 0 0 SLs (threaded) kernel 100027 D - 0xffffff000159f280 [em1 taskq] 100026 D - 0xffffff00015a0000 [em0 taskq] 100023 D - 0xffffff000152b180 [thread taskq] 100019 D - 0xffffff000152b900 [acpi_task_2] 100018 D - 0xffffff000152b900 [acpi_task_1] 100017 D - 0xffffff000152b900 [acpi_task_0] 100016 D - 0xffffff000152b980 [kqueue taskq] 100012 D - 0xffffff0001455080 [firmware taskq] 100000 D sched 0xffffffff8083e940 [swapper] db> trace 4509 Tracing pid 4509 tid 100186 td 0xffffff0063ba4720 sched_switch() at sched_switch+0x180 mi_switch() at mi_switch+0x21d sleepq_switch() at sleepq_switch+0x123 sleepq_wait() at sleepq_wait+0x4d _cv_wait() at _cv_wait+0x17a dbuf_read() at dbuf_read+0x33b dmu_buf_hold() at dmu_buf_hold+0xcc zap_lockdir() at zap_lockdir+0x53 zap_lookup_norm() at zap_lookup_norm+0x45 zap_lookup() at zap_lookup+0x2e zfs_dirent_lock() at zfs_dirent_lock+0x501 zfs_dirlook() at zfs_dirlook+0x69 zfs_lookup() at zfs_lookup+0x1f4 zfs_freebsd_lookup() at zfs_freebsd_lookup+0x81 VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0xaf vfs_cache_lookup() at vfs_cache_lookup+0xf0 VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0xb7 lookup() at lookup+0x516 namei() at namei+0x53d vn_open_cred() at vn_open_cred+0x1a7 kern_openat() at kern_openat+0x169 syscall() at syscall+0x1e7 Xfast_syscall() at Xfast_syscall+0xab --- syscall (5, FreeBSD ELF64, open), rip = 0x800a4dd0c, rsp = 0x7fffffffe988, rbp = 0xfa0 --- I'm not a kernel debug expert. Kust let me know if you need additional information. Regards ThomasReceived on Fri Nov 21 2008 - 07:56:07 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:37 UTC