Re: zpool scrub causes geli vdev to detach

From: Pawel Jakub Dawidek <pjd_at_FreeBSD.org>
Date: Thu, 11 Oct 2007 19:38:46 +0200
On Thu, Oct 11, 2007 at 07:05:17PM +0200, Fabian Keil wrote:
> On:
> 
> FreeBSD TP51.local 7.0-CURRENT FreeBSD 7.0-CURRENT
> #0: Thu Oct 11 09:21:19 CEST 2007
> fk_at_TP51.local:/usr/obj/usr/src/sys/THINKPAD  i386
> 
> this zpool:
> 
> fk_at_TP51 ~ $sudo zpool status
>   pool: tank
>  state: ONLINE
>  scrub: scrub completed with 0 errors on Thu Oct 11 13:56:28 2007
> config:
> 
>         NAME          STATE     READ WRITE CKSUM
>         tank          ONLINE       0     0     0
>           ad0s3f.eli  ONLINE       0     0     0
>           ad0s2.eli   ONLINE       0     0     0
> 
> errors: No known data errors
> 
> worked without issues when it only consisted of ad0s3f.eli,
> but then I added ad0s2.eli and now scrubbing the pool leads to:
> 
> Unread portion of the kernel message buffer:
> GEOM_ELI: Detached ad0s2.eli on last close.
> GEOM_LABEL: Label for provider ad0s2 is msdosfs/?A.?,{(#0.
> panic: Function g_eli_orphan_spoil_assert() called for ad0s3f.eli.
> KDB: enter: panic
> panic: from debugger
> Uptime: 5m27s
> Physical memory: 1014 MB
> Dumping 120 MB: 105 89 73 57 41 25 9
> 
> #0  doadump () at pcpu.h:195
> 195     pcpu.h: No such file or directory.
>         in pcpu.h
> (kgdb) bt   
> #0  doadump () at pcpu.h:195
> #1  0xc05db8f3 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
> #2  0xc05dbb1c in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:563
> #3  0xc04a40c7 in db_panic (addr=Could not find the frame base for "db_panic".
> ) at /usr/src/sys/ddb/db_command.c:433
> #4  0xc04a4825 in db_command_loop () at /usr/src/sys/ddb/db_command.c:401
> #5  0xc04a6255 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_main.c:222
> #6  0xc06014a4 in kdb_trap (type=3, code=0, tf=0xf4bcac1c) at /usr/src/sys/kern/subr_kdb.c:502
> #7  0xc0838d2b in trap (frame=0xf4bcac1c) at /usr/src/sys/i386/i386/trap.c:621
> #8  0xc082120b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
> #9  0xc0601602 in kdb_enter (msg=0xc0889693 "panic") at cpufunc.h:60
> #10 0xc05dbb05 in panic (fmt=0xc3e9a0e8 "Function %s() called for %s.") at /usr/src/sys/kern/kern_shutdown.c:547
> #11 0xc3e92d65 in ?? ()
> #12 0xc3e9a0e8 in ?? ()
> #13 0xc3e99d84 in ?? ()
> #14 0xc3e0d890 in ?? ()
> #15 0xf4bcacac in ?? ()
> #16 0xc058eff5 in g_spoil_event (arg=0xc3c92940, flag=-945983104) at /usr/src/sys/geom/geom_subr.c:903
> (kgdb) f 16
> #16 0xc058eff5 in g_spoil_event (arg=0xc3c92940, flag=-945983104) at /usr/src/sys/geom/geom_subr.c:903
> 903                     cp->geom->spoiled(cp);
> (kgdb) l
> 898                     if (!cp->spoiled)
> 899                             continue;
> 900                     cp->spoiled = 0;
> 901                     if (cp->geom->spoiled == NULL)
> 902                             continue;
> 903                     cp->geom->spoiled(cp);
> 904                     g_topology_assert();
> 905             }
> 906     }
> 907     
> 
> As a workaround I'm currently using:
> 
> --- sys/geom/eli/g_eli.c.orig   2007-10-11 13:35:53.744592827 +0200
> +++ sys/geom/eli/g_eli.c        2007-10-11 13:38:41.687509241 +0200
> _at__at_ -451,10 +451,14 _at__at_
>         gp = sc->sc_geom;
>         pp = LIST_FIRST(&gp->provider);
>         strlcpy(ppname, pp->name, sizeof(ppname));
> +
> +       G_ELI_DEBUG(0, "Ignored request to detach %s on last close.", ppname);
> +        /*
>         error = g_eli_destroy(sc, 1);
>         KASSERT(error == 0, ("Cannot detach %s on last close (error=%d).",
>             ppname, error));
>         G_ELI_DEBUG(0, "Detached %s on last close.", ppname);
> +        */
>  }
>  
>  int
> 
> and with that scrubbing finishes without errors (see status above),
> but triggers the G_ELI_DEBUG message once or twice:
> 
> Oct 11 13:46:30 TP51 sudo:       fk : TTY=ttyv0 ; PWD=/home/fk ; USER=root ; COMMAND=/etc/rc.d/zfs start
> Oct 11 13:46:31 TP51 kernel: WARNING: ZFS is considered to be an experimental feature in FreeBSD.
> Oct 11 13:46:31 TP51 kernel: ZFS filesystem version 6
> Oct 11 13:46:31 TP51 kernel: ZFS storage pool version 6
> Oct 11 13:46:32 TP51 kernel: GEOM_ELI: Ignored request to detach ad0s3f.eli on last close.
> Oct 11 13:46:32 TP51 last message repeated 3 times
> Oct 11 13:46:32 TP51 kernel: GEOM_ELI: Ignored request to detach ad0s2.eli on last close.
> Oct 11 13:46:44 TP51 sudo:       fk : TTY=ttyv0 ; PWD=/home/fk ; USER=root ; COMMAND=/sbin/zpool status
> Oct 11 13:47:12 TP51 sudo:       fk : TTY=ttyv0 ; PWD=/home/fk ; USER=root ; COMMAND=/sbin/zpool scrub tank
> Oct 11 13:47:13 TP51 kernel: GEOM_ELI: Ignored request to detach ad0s3f.eli on last close.
> Oct 11 13:47:13 TP51 kernel: GEOM_ELI: Ignored request to detach ad0s3f.eli on last close.
> 
> Not sure if it helps, but the zdevs are:
> 
> Geom name: ad0s3f.eli
> EncryptionAlgorithm: AES-CBC
> KeyLength: 128
> Crypto: software
> UsedKey: 0
> Flags: RW-DETACH, W-OPEN
> Providers:
> 1. Name: ad0s3f.eli
>    Mediasize: 8881561600 (8.3G)
>    Sectorsize: 4096
>    Mode: r1w1e1
> Consumers:
> 1. Name: ad0s3f
>    Mediasize: 8881565184 (8.3G)
>    Sectorsize: 512
>    Mode: r1w1e1
> 
> Geom name: ad0s2.eli
> EncryptionAlgorithm: AES-CBC
> KeyLength: 128
> Crypto: software
> UsedKey: 0
> Flags: RW-DETACH, W-OPEN
> Providers:
> 1. Name: ad0s2.eli
>    Mediasize: 9932263424 (9.3G)
>    Sectorsize: 4096
>    Mode: r1w1e1
> Consumers:
> 1. Name: ad0s2
>    Mediasize: 9932267520 (9.3G)
>    Sectorsize: 512
>    Mode: r1w1e1
> 
> 
> Is there an obvious solution, or should I file a PR about this?

Please file PR, but let me explain.

GELI's detach-on-last-close mechanism is a general purpose mechanism, it
may not work correctly with ZFS, because ZFS sometimes closes and reopen
providers, which will make GELI to detach. In other words you shouldn't
configure detach-on-last-close for ZFS components. It shouldn't panic
still.

-- 
Pawel Jakub Dawidek                       http://www.wheel.pl
pjd_at_FreeBSD.org                           http://www.FreeBSD.org
FreeBSD committer                         Am I Evil? Yes, I Am!

Received on Thu Oct 11 2007 - 15:39:12 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:19 UTC