Re: deadlock with zfs?

Hello

> Am 21.11.2008 um 09:39 schrieb Nikolay Denev:
> 
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>>
>> On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote:
>>
>>> Hello
>>>
>>> I encounter a deadlock while running a few rsync processes mirroring
>>> remote data to my local zfs pool. After a few hours my system is
>>> starting more and more vsftpd sessions without closing any inactive
>>> ftp sessions. I can't kill any rsync and vsftpd processes with "kill
>>> -9". Even shutdown -r now does not work.
>>>
>>> I got a few hunderts vsftpd processes like this
>>>
>>> 61346 root          1  57    0  7880K  1692K zfs     1   0:00  0.00%
>>> vsftpd
>>> 61481 root          1  68    0  7880K  1696K zfs     1   0:00  0.00%
>>> vsftpd
>>> 61354 root          1  65    0  7880K  1692K zfs     1   0:00  0.00%
>>> vsftpd
>>> 61480 root          1  68    0  7880K  1696K zfs     0   0:00  0.00%
>>> vsftpd
>>> 61600 root          1  69    0  7880K  1704K zfs     1   0:00  0.00%
>>> vsftpd
>>> 61599 root          1  68    0  7880K  1704K zfs     1   0:00  0.00%
>>> vsftpd
>>>
>>> Right now i'm building a debug kernel. Whats the best way to get
>>> usefull information from this deadlock? The system itself is not
>>> crashing and response well to ssh and i also have a serial console.
>>>
>>> The system is running  8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008
>>> (64bit) with zpool version 13. I use zfs only as data pool. The base
>>> system is running on ufs2:
>>>
>>> Filesystem       Size    Used   Avail Capacity  Mounted on
>>> /dev/da0s1a      496M    220M    236M    48%    /
>>> devfs            1.0K    1.0K      0B   100%    /dev
>>> /dev/da0s1g      169G     15G    141G     9%    /disk1
>>> /dev/da0s1f      3.9G     17M    3.5G     0%    /tmp
>>> /dev/da0s1e       29G    3.7G     23G    14%    /usr
>>> /dev/da0s1d       19G    7.1G     11G    40%    /var
>>> pool             853G      0B    853G     0%    /usr/local/data
>>> pool/cvsup       858G    5.7G    853G     1%    /usr/local/data/cvsup
>>> pool/ftp         3.3T    2.5T    853G    75%    /usr/local/data/ftp
>>> pool/portsnap    853G    633M    853G     0%    /usr/local/data/portsnap
>>> pool/www         853G     90M    853G     0%    /usr/local/data/www
>>>
>>> loader.conf:
>>> vm.kmem_size="1G"
>>> kern.maxfiles="65536"
>>> kern.maxproc="20480"
>>> net.inet.tcp.tcbhashsize="4096"
>>> net.inet.tcp.hostcache.hashsize="1024"
>>> vfs.zfs.arc_min="64M"
>>> vfs.zfs.arc_max="768M"
>>> vfs.zfs.prefetch_disable="1"
>>>
>>> I also tried to set vfs.zfs.zil_disable=1 but the problem still exist.
>>>
>>> I know there are few deadlock reports listed in the freebsd wiki.
>>> Maybe we can trigger the root cause of the problem and fix it :)
>>>
>>> Regards
>>> Thomas
>>>
>>
>> Hi Thomas,
>>
>> from my recent experience it seems that tuning vm.kmem_size is no
>> longer required,
>> maybe you can try without it.
> 
> Sure, i rebooted the system without vm.kmem_size. It takes a a few hours
> until i got deadlocks.

Removing vm.kmem_size didn't fix my deadlock issue. It just took a bit
longer until the deadlock was triggered.

Maybe someone can check my deadlock debug output at
http://www.bsdunix.ch/deadlock_2008-11-21.txt


Regards,
Thomas