Re: deadlock with zfs?

Thomas Vogt



Am 21.11.2008 um 09:39 schrieb Nikolay Denev:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
> On 21 Nov, 2008, at 02:01 , Thomas Vogt wrote:
>
>> Hello
>>
>> I encounter a deadlock while running a few rsync processes  
>> mirroring remote data to my local zfs pool. After a few hours my  
>> system is starting more and more vsftpd sessions without closing  
>> any inactive ftp sessions. I can't kill any rsync and vsftpd  
>> processes with "kill -9". Even shutdown -r now does not work.
>>
>> I got a few hunderts vsftpd processes like this
>>
>> 61346 root          1  57    0  7880K  1692K zfs     1   0:00   
>> 0.00% vsftpd
>> 61481 root          1  68    0  7880K  1696K zfs     1   0:00   
>> 0.00% vsftpd
>> 61354 root          1  65    0  7880K  1692K zfs     1   0:00   
>> 0.00% vsftpd
>> 61480 root          1  68    0  7880K  1696K zfs     0   0:00   
>> 0.00% vsftpd
>> 61600 root          1  69    0  7880K  1704K zfs     1   0:00   
>> 0.00% vsftpd
>> 61599 root          1  68    0  7880K  1704K zfs     1   0:00   
>> 0.00% vsftpd
>>
>> Right now i'm building a debug kernel. Whats the best way to get  
>> usefull information from this deadlock? The system itself is not  
>> crashing and response well to ssh and i also have a serial console.
>>
>> The system is running  8.0-CURRENT Thu Nov 20 00:15:46 UTC 2008  
>> (64bit) with zpool version 13. I use zfs only as data pool. The  
>> base system is running on ufs2:
>>
>> Filesystem       Size    Used   Avail Capacity  Mounted on
>> /dev/da0s1a      496M    220M    236M    48%    /
>> devfs            1.0K    1.0K      0B   100%    /dev
>> /dev/da0s1g      169G     15G    141G     9%    /disk1
>> /dev/da0s1f      3.9G     17M    3.5G     0%    /tmp
>> /dev/da0s1e       29G    3.7G     23G    14%    /usr
>> /dev/da0s1d       19G    7.1G     11G    40%    /var
>> pool             853G      0B    853G     0%    /usr/local/data
>> pool/cvsup       858G    5.7G    853G     1%    /usr/local/data/cvsup
>> pool/ftp         3.3T    2.5T    853G    75%    /usr/local/data/ftp
>> pool/portsnap    853G    633M    853G     0%    /usr/local/data/ 
>> portsnap
>> pool/www         853G     90M    853G     0%    /usr/local/data/www
>>
>> loader.conf:
>> vm.kmem_size="1G"
>> kern.maxfiles="65536"
>> kern.maxproc="20480"
>> net.inet.tcp.tcbhashsize="4096"
>> net.inet.tcp.hostcache.hashsize="1024"
>> vfs.zfs.arc_min="64M"
>> vfs.zfs.arc_max="768M"
>> vfs.zfs.prefetch_disable="1"
>>
>> I also tried to set vfs.zfs.zil_disable=1 but the problem still  
>> exist.
>>
>> I know there are few deadlock reports listed in the freebsd wiki.  
>> Maybe we can trigger the root cause of the problem and fix it :)
>>
>> Regards
>> Thomas
>>
>
> Hi Thomas,
>
> from my recent experience it seems that tuning vm.kmem_size is no  
> longer required,
> maybe you can try without it.

Sure, i rebooted the system without vm.kmem_size. It takes a a few  
hours until i got deadlocks.

Regards
Thomas