Re: zfs hang

On Fri, Oct 10, 2014 at 02:35:14AM +0100, Steven Hartland wrote:
> 
> ----- Original Message ----- 
> From: "Steve Wills" <swills_at_freebsd.org>
> To: "Andriy Gapon" <avg_at_freebsd.org>
> Cc: <current_at_freebsd.org>; <fs_at_freebsd.org>
> Sent: Friday, October 10, 2014 2:27 AM
> Subject: Re: zfs hang
> 
> 
> > On Wed, Oct 08, 2014 at 08:55:26AM +0300, Andriy Gapon wrote:
> >> On 08/10/2014 03:40, Steve Wills wrote:
> >> > Hi,
> >> > 
> >> > Not sure which thread this belongs to, but I have a zfs hang on one of my boxes
> >> > running r272152. Running procstat -kka looks like:
> >> > 
> >> > http://pastebin.com/szZZP8Tf
> >> > 
> >> > My zpool commands seem to be hung in spa_errlog_lock while others are hung in
> >> > zfs_lookup. Suggestions?
> >> 
> >> There are several threads in zio_wait.  If this is their permanent state then
> >> there is some problem with I/O somewhere below ZFS.
> > 
> > Thanks for the feedback. It seems one of my disks is dying, I rebooted and it
> > came up OK, but today I got:
> > 
> >  panic: I/O to pool 'rpool' appears to be hung on vdev guid ..... at '/dev/ada0p3'
> > 
> > I have screenshots and backtrace if anyone is interested. Dying drives
> > shouldn't cause panic, right?
> 
> Its the deadman timer kicking in so yes, thats expected.
> 
> The following sysctls control this behaviour if you want to try and recover:
> vfs.zfs.deadman_synctime_ms: 1000000
> vfs.zfs.deadman_checktime_ms: 5000
> vfs.zfs.deadman_enabled: 1

Ah, ok. This pool has two disks, mirrored. I think one of them is dying, the
BIOS gives a SMART error on startup, but it still uses the disk fine. From what
I read of the zfs deadman design, it's for when the controller is acting up. So
I'm confused. Maybe this means both disks are dying?

Steve