ggated, gmirror + zfs panic

From: Mike Wolman <mike_at_nux.co.uk>
Date: Thu, 29 Mar 2007 17:47:01 +0100 (BST)
Hi,

I am experimenting with ggate, gmirror and zfs to create a redundant file 
system using 2 machines.

I currently do not have 10 spare drives so i have been using md devices to test 
the setup with.

If I run through the below commands i am able to crash the machine every time. 
I have a vmcore available but think this should be easy to reproduce.

On Secondary Machine

# dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k

# mdconfig -a -t vnode -f /home/mike-play/file0 -u 0
# mdconfig -a -t vnode -f /home/mike-play/file1 -u 1
# mdconfig -a -t vnode -f /home/mike-play/file2 -u 2
# mdconfig -a -t vnode -f /home/mike-play/file3 -u 3
# mdconfig -a -t vnode -f /home/mike-play/file4 -u 4

edit /etc/gg.exports

192.168.5.0/24 RW /dev/md0
192.168.5.0/24 RW /dev/md1
192.168.5.0/24 RW /dev/md2
192.168.5.0/24 RW /dev/md3
192.168.5.0/24 RW /dev/md4

Start ggated
# ggated -v


On Primary Machine

# dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k

# mdconfig -a -t vnode -f /home/mike-play/file0 -u 0
# mdconfig -a -t vnode -f /home/mike-play/file1 -u 1
# mdconfig -a -t vnode -f /home/mike-play/file2 -u 2
# mdconfig -a -t vnode -f /home/mike-play/file3 -u 3
# mdconfig -a -t vnode -f /home/mike-play/file4 -u 4

connect up the ggated exports:

# ggatec create -u 0 192.168.5.212 /dev/md0
# ggatec create -u 1 192.168.5.212 /dev/md1
# ggatec create -u 2 192.168.5.212 /dev/md2
# ggatec create -u 3 192.168.5.212 /dev/md3
# ggatec create -u 4 192.168.5.212 /dev/md4


Setup the individual mirrors for each device:

# gmirror label -v -b prefer zmirror0 /dev/ggate0
# gmirror insert -p90 zmirror0 /dev/md0
# gmirror label -v -b prefer zmirror1 /dev/ggate1
# gmirror insert -p90 zmirror1 /dev/md1
# gmirror label -v -b prefer zmirror2 /dev/ggate2
# gmirror insert -p90 zmirror2 /dev/md2
# gmirror label -v -b prefer zmirror3 /dev/ggate3
# gmirror insert -p90 zmirror3 /dev/md3
# gmirror label -v -b prefer zmirror4 /dev/ggate4
# gmirror insert -p90 zmirror4 /dev/md4


Create zfs pool from these mirrors:

# zpool create tank raidz2 \
/dev/mirror/zmirror0 \
/dev/mirror/zmirror1 \
/dev/mirror/zmirror2 \
/dev/mirror/zmirror3 \
/dev/mirror/zmirror4

At this point the primary machine panics and enters the kernel debugger.

If i exclude gmirror from the setup and only use the secondary machines md 
devices the zpool create works as expected:

On Secondary machine:

# dd if=/dev/zero of=/home/mike-play/file0 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file1 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file2 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file3 bs=1k count=1000k
# dd if=/dev/zero of=/home/mike-play/file4 bs=1k count=1000k
# mdconfig -a -t vnode -f /home/mike-play/file0 -u 0
# mdconfig -a -t vnode -f /home/mike-play/file1 -u 1
# mdconfig -a -t vnode -f /home/mike-play/file2 -u 2
# mdconfig -a -t vnode -f /home/mike-play/file3 -u 3
# mdconfig -a -t vnode -f /home/mike-play/file4 -u 4

edit /etc/gg.exports

192.168.5.0/24 RW /dev/md0
192.168.5.0/24 RW /dev/md1
192.168.5.0/24 RW /dev/md2
192.168.5.0/24 RW /dev/md3
192.168.5.0/24 RW /dev/md4

Start ggated
# ggated -v


On Primary machine:

# ggatec create -u 0 192.168.5.212 /dev/md0
# ggatec create -u 1 192.168.5.212 /dev/md1
# ggatec create -u 2 192.168.5.212 /dev/md2
# ggatec create -u 3 192.168.5.212 /dev/md3
# ggatec create -u 4 192.168.5.212 /dev/md4

# zpool create tank raidz2 \
/dev/ggate0 \
/dev/ggate1 \
/dev/ggate2 \
/dev/ggate3 \
/dev/ggate4

# zpool list
NAME                    SIZE    USED   AVAIL    CAP  HEALTH     ALTROOT
tank                   4.84G    206K   4.84G     0%  ONLINE     -



f7# dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT-200703 #0: Wed Mar 28 12:34:58 BST 2007
     mike_at_f7.eros.office:/usr/obj/usr/src/sys/GENERIC
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Athlon(tm) XP 3200+ (2194.51-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x6a0  Stepping = 0

Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
   AMD Features=0xc0400800<SYSCALL,MMX+,3DNow!+,3DNow!>
real memory  = 1056899072 (1007 MB)
avail memory = 1024897024 (977 MB)
ACPI APIC Table: <VKT400 AWRDACPI>
ioapic0 <Version 0.3> irqs 0-23 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <VKT400 AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
acpi_button1: <Sleep Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
agp0: <VIA 8235/8237 (Apollo KM400/KM400A) host to PCI bridge> on hostb0
pcib1: <PCI-PCI bridge> at device 1.0 on pci0
pci1: <PCI bus> on pcib1
vgapci0: <VGA-compatible display> mem 
0xd8000000-0xdbffffff,0xdc000000-0xdcffffff irq 16 at device 0.0 on pci1
re0: <RealTek 8169S Single-chip Gigabit Ethernet> port 0x9000-0x90ff mem 
0xde010000-0xde0100ff irq 16 at device 8.0 on pci0
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 
1000baseT-FDX, auto
re0: Ethernet address: 00:0e:2e:92:7d:0a
re0: [FILTER]
fwohci0: <VIA Fire II (VT6306)> port 0x9400-0x947f mem 0xde011000-0xde0117ff 
irq 17 at device 9.0 on pci0
fwohci0: [ITHREAD]
fwohci0: OHCI version 1.0 (ROM=1)
fwohci0: No. of Isochronous channels is 4.
fwohci0: EUI64 00:30:67:00:00:04:c4:15
fwohci0: Phy 1394a available S400, 3 ports.
fwohci0: Link S400, max_rec 2048 bytes.
firewire0: <IEEE1394(FireWire) bus> on fwohci0
fwe0: <Ethernet over FireWire> on firewire0
if_fwe0: Fake Ethernet address: 02:30:67:04:c4:15
fwe0: Ethernet address: 02:30:67:04:c4:15
fwe0: if_start running deferred for Giant
sbp0: <SBP-2/SCSI over FireWire> on firewire0
fwohci0: Initiate bus reset
fwohci0: node_id=0xc800ffc0, gen=1, CYCLEMASTER mode
firewire0: 1 nodes, maxhop <= 0, cable IRM = 0 (me)
firewire0: bus manager 0 (me)
atapci0: <VIA 6420 SATA150 controller> port 
0x9800-0x9807,0x9c00-0x9c03,0xa000-0xa007,0xa400-0xa403,0xa800-0xa80f,0xac00-0xacff 
irq 20 at device 15.0 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
atapci1: <VIA 8237 UDMA133 controller> port 
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xb000-0xb00f at device 15.1 on pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
uhci0: <VIA 83C572 USB controller> port 0xb400-0xb41f irq 21 at device 16.0 on 
pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <VIA 83C572 USB controller> on uhci0
usb0: USB revision 1.0
uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <VIA 83C572 USB controller> port 0xb800-0xb81f irq 21 at device 16.1 on 
pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <VIA 83C572 USB controller> on uhci1
usb1: USB revision 1.0
uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <VIA 83C572 USB controller> port 0xbc00-0xbc1f irq 21 at device 16.2 on 
pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <VIA 83C572 USB controller> on uhci2
usb2: USB revision 1.0
uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <VIA 83C572 USB controller> port 0xc000-0xc01f irq 21 at device 16.3 on 
pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <VIA 83C572 USB controller> on uhci3
usb3: USB revision 1.0
uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <VIA VT6202 USB 2.0 controller> mem 0xde012000-0xde0120ff irq 21 at 
device 16.4 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <VIA VT6202 USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
isab0: <PCI-ISA bridge> at device 17.0 on pci0
isa0: <ISA bus> on isab0
pci0: <multimedia, audio> at device 17.5 (no driver attached)
vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xc800-0xc8ff mem 
0xde013000-0xde0130ff irq 23 at device 18.0 on pci0
miibus1: <MII bus> on vr0
ukphy0: <Generic IEEE 802.3u media interface> PHY 1 on miibus1
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
vr0: using obsoleted if_watchdog interface
vr0: Ethernet address: 00:e0:4c:d5:10:a5
vr0: [ITHREAD]
acpi_tz0: <Thermal Zone> on acpi0
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: [FILTER]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xc8000-0xcffff pnpid ORM0000 on isa0
ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
ppc0: [GIANT-LOCKED]
ppc0: [ITHREAD]
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounter "TSC" frequency 2194511372 Hz quality 800
Timecounters tick every 1.000 msec
ad0: 39205MB <Maxtor 6E040L0 NAR61HA0> at ata0-master UDMA133
acd0: CDROM <Lite-On LTN486S 48x Max/YDS6> at ata1-slave UDMA33
Trying to mount root from ufs:/dev/ad0s1a
WARNING: / was not properly dismounted
WARNING: /tmp was not properly dismounted
WARNING: /usr was not properly dismounted
WARNING: /var was not properly dismounted
ZFS filesystem version 3
ZFS storage pool version 3
Received on Thu Mar 29 2007 - 15:11:30 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:07 UTC