Re: zfs kernel messages

From: Olli Hauer <ohauer_at_gmx.de>
Date: Tue, 23 Oct 2007 21:27:56 +0200
Pawel Jakub Dawidek wrote:
> On Mon, Oct 22, 2007 at 12:15:50AM +0200, Olli Hauer wrote:
>> while playing a little with zfs i get the following
>> kernel messages at ttyv0 (no panic/freeze happend)
>>
>> (captured vi ssh from other system)
>> # vidcontrol -P < /dev/ttyv0
>>
>> uma_zalloc_arg: zone "256" with the following non-sleepable locks held:
>> exclusive sleep mutex struct mount mtx r = 0 (0xc64e7d3c) locked _at_ 
>> /usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:206
>> KDB: stack backtrace:
>> db_trace_self_wrapper(c0a9c175,e72bb6e4,c078428d,c0a9c538,e72bb6f8,...) at 
>> db_trace_self_wrapper+0x26
>> kdb_backtrace(c0a9c538,e72bb6f8,4,1,0,...) at kdb_backtrace+0x29
>> witness_warn(5,0,c0aba40d,c0ac3e77,1,...) at witness_warn+0x1cd
>> uma_zalloc_arg(c146d1e0,0,102,2,c64e7d0c,...) at uma_zalloc_arg+0x34
>> malloc(94,c0b4c4e0,102,c64e7d0c,e72bb790,...) at malloc+0xd2
>> crget(c6985300,c0b4c4e0,c64e7d0c,e72bb7d4,c431b9dc,...) at crget+0x23
>> crdup(c3f07600,0,c439444c,ce,c2,...) at crdup+0xc
>> domount(c450a420,c6ee4330,c43995db,cc152900,e72bb810,...) at domount+0x20c
>> zfsctl_snapdir_lookup(e72bbaa0,e72bbaa0,c450a420,2,c456e660,...) at 
>> zfsctl_snapdir_lookup+0x362
>> VOP_LOOKUP_APV(c439d5c0,e72bbaa0,c450a420,c0aa409d,19b,...) at 
>> VOP_LOOKUP_APV+0xa5
>> lookup(e72bbb48,c0aa409d,c6,bf,c730102c,...) at lookup+0x58e
>> namei(e72bbb48,e72bbb94,60,0,c450a420,...) at namei+0x34b
>> kern_lstat(c450a420,28220318,0,e72bbc18,2d4738cb,...) at kern_lstat+0x4f
>> lstat(c450a420,e72bbcfc,8,c0a9ed42,c0b469d0,...) at lstat+0x2f
>> syscall(e72bbd38) at syscall+0x2b3
>> Xint0x80_syscall() at Xint0x80_syscall+0x20
>> --- syscall (190, FreeBSD ELF32, lstat), eip = 0x2815548b, esp = 
>> 0xbfbfe93c, ebp = 0xbfbfe9c8 ---
> 
> That's easy to fix. Try this patch:
> 
> 	http://people.freebsd.org/~pjd/patches/opensolaris_vfs.c.patch
> 
> Thanks for the report.
> 

Sorry for the late responce.
Patch applied, rebuild/installed kernel, reboot same test output is below.

Let me know If you need more information or if you have new instructions.


***** after this test I wait 5 min then try to reboot the system *****

Syncing disks, vnodes remaining...2 2 2 2 0 0 done
All buffers synced.

***** system hang, wait 10 minutes than push the power button once *****

Shutting down demon process:.
Shutting down local daemons:.
Writing entropy file:.

Oct 23.... init: some process would not die; ps axl advised
Waiting (max60 seconds) for system process `vnlu` to stop... timed out
Waiting (max60 seconds) for system process `bufdaemon` to stop... timed out
Waiting (max60 seconds) for system process `syncer` to stop... timed out
...


***** system power off *****
***** start system again (see dmesg) *****

...




***** output of the ls test *****

# vidcontrol -P < /dev/ttyv0

lock order reversal:
  1st 0xc4cea568 struct mount mtx (struct mount mtx) _at_ /usr/src/sys/modules/zfs/../../compat/opensolaris/kern/opensolaris_vfs.c:209
  2nd 0xc3ee9010 sleep mtxpool (sleep mtxpool) _at_ /usr/src/sys/kern/kern_resource.c:1266
KDB: stack backtrace:
db_trace_self_wrapper(c0a9c175,e7318734,c078510e,c0a9e63c,c3ee9010,...) at db_trace_self_wrapper+0x26
kdb_backtrace(c0a9e63c,c3ee9010,c0a982df,c0a982df,c0a98e54,...) at kdb_backtrace+0x29
witness_checkorder(c3ee9010,9,c0a98e54,4f2,38,...) at witness_checkorder+0x6de
_mtx_lock_flags(c3ee9010,0,c0a98e54,4f2,c4c32d00,...) at _mtx_lock_flags+0xbc
uifree(c3f08c20,c4967220,c4cea538,e73187d4,c431b9df,...) at uifree+0x2d
crfree(c4c32d00,0,c439446c,d1,c3,...) at crfree+0x54
domount(c48f8840,c4967220,c43995fb,c469e260,e7318810,...) at domount+0x20f
zfsctl_snapdir_lookup(e7318aa0,e7318aa0,c48f8840,2,c4967330,...) at zfsctl_snapdir_lookup+0x362
VOP_LOOKUP_APV(c439d5e0,e7318aa0,c48f8840,c0aa409d,19b,...) at VOP_LOOKUP_APV+0xa5
lookup(e7318b48,c0aa409d,c6,bf,c4cd772c,...) at lookup+0x58e
namei(e7318b48,c48f8840,c0bb51d4,c48f8840,e7318b4c,...) at namei+0x34b
kern_lstat(c48f8840,282111b8,0,e7318c18,c0aa5520,...) at kern_lstat+0x4f
lstat(c48f8840,e7318cfc,8,c0a9f13d,c0b469d0,...) at lstat+0x2f
syscall(e7318d38) at syscall+0x2b3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (190, FreeBSD ELF32, lstat), eip = 0x2819c48b, esp = 0xbfbfe57c, ebp = 0xbfbfe608 ---


***** output of zpool test *****
# zpool status
   pool: tank
  state: ONLINE
  scrub: none requested
config:

         NAME        STATE     READ WRITE CKSUM
         tank        ONLINE       0     0     0
           mirror    ONLINE       0     0     0
             twed0   ONLINE       0     0     0
             twed1   ONLINE       0     0     0

errors: No known data errors

# zpool scrub tank
# zpool status
   pool: tank
  state: ONLINE
  scrub: scrub completed with 0 errors on Tue Oct 23 21:15:42 2007
config:

         NAME        STATE     READ WRITE CKSUM
         tank        ONLINE       0     0     0
           mirror    ONLINE       0     0     0
             twed0   ONLINE       0     0     0
             twed1   ONLINE       0     0     0

errors: No known data errors


# uname -a
FreeBSD amd.uni.vrs 7.0-CURRENT-200710 FreeBSD 7.0-CURRENT-200710 #0: Tue Oct 23 20:12:59 CEST 2007     root_at_amd.uni.vrs:/usr/obj/usr/src/sys/GENERIC  i386

# dmesg
Copyright (c) 1992-2007 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-CURRENT-200710 #0: Tue Oct 23 20:12:59 CEST 2007
     root_at_amd.uni.vrs:/usr/obj/usr/src/sys/GENERIC
WARNING: WITNESS option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Sempron(tm)   2300+ (1580.46-MHz 686-class CPU)
   Origin = "AuthenticAMD"  Id = 0x681  Stepping = 1
   Features=0x383fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
   AMD Features=0xc0480800<SYSCALL,MP,MMX+,3DNow!+,3DNow!>
real memory  = 1073676288 (1023 MB)
avail memory = 1037058048 (989 MB)
mptable_probe: MP Config Table has bad signature: \^O8\^O
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <Nvidia AWRDACPI> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, 3fef0000 (3) failed
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
cpu0: <ACPI CPU> on acpi0
acpi_button0: <Power Button> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
Correcting nForce2 C1 CPU disconnect hangs
agp0: <NVIDIA nForce2 AGP Controller> on hostb0
pci0: <memory, RAM> at device 0.1 (no driver attached)
pci0: <memory, RAM> at device 0.2 (no driver attached)
pci0: <memory, RAM> at device 0.3 (no driver attached)
pci0: <memory, RAM> at device 0.4 (no driver attached)
pci0: <memory, RAM> at device 0.5 (no driver attached)
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
pci0: <serial bus, SMBus> at device 1.1 (no driver attached)
ohci0: <OHCI (generic) USB controller> mem 0xe0001000-0xe0001fff irq 3 at device 2.0 on pci0
ohci0: [GIANT-LOCKED]
ohci0: [ITHREAD]
usb0: OHCI version 1.0, legacy support
usb0: <OHCI (generic) USB controller> on ohci0
usb0: USB revision 1.0
uhub0: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 3 ports with 3 removable, self powered
ohci1: <OHCI (generic) USB controller> mem 0xe0002000-0xe0002fff irq 5 at device 2.1 on pci0
ohci1: [GIANT-LOCKED]
ohci1: [ITHREAD]
usb1: OHCI version 1.0, legacy support
usb1: <OHCI (generic) USB controller> on ohci1
usb1: USB revision 1.0
uhub1: <nVidia OHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 3 ports with 3 removable, self powered
ehci0: <NVIDIA nForce2 USB 2.0 controller> mem 0xe0003000-0xe00030ff irq 11 at device 2.2 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb2: EHCI version 1.0
usb2: companion controllers, 4 ports each: usb0 usb1
usb2: <NVIDIA nForce2 USB 2.0 controller> on ehci0
usb2: USB revision 2.0
uhub2: <nVidia EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb2
uhub2: 6 ports with 6 removable, self powered
pci0: <multimedia, audio> at device 6.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> at device 8.0 on pci0
pci1: <ACPI PCI bus> on pcib1
twe0: <3ware Storage Controller. Driver version 1.50.01.002> port 0xd000-0xd00f mem 0xdf800000-0xdf80000f,0xdf000000-0xdf7fffff irq 11 at device 6.0 on pci1
twe0: [GIANT-LOCKED]
twe0: [ITHREAD]
twe0: 4 ports, Firmware FE7X 1.05.00.068, BIOS BE7X 1.08.00.048
xl0: <3Com 3c905C-TX Fast Etherlink XL> port 0xd400-0xd47f mem 0xdf801000-0xdf80107f irq 5 at device 7.0 on pci1
miibus0: <MII bus> on xl0
ukphy0: <Generic IEEE 802.3u media interface> PHY 24 on miibus0
ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
xl0: Ethernet address: 00:04:75:bf:81:11
xl0: [ITHREAD]
atapci0: <nVidia nForce2 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xf000-0xf00f at device 9.0 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
pcib2: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci2: <ACPI PCI bus> on pcib2
vgapci0: <VGA-compatible display> mem 0xdc000000-0xdcffffff,0xd0000000-0xd7ffffff irq 5 at device 0.0 on pci2
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
sio0: type 16550A
sio0: [FILTER]
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: [ITHREAD] 

psm0: model Generic PS/2 mouse, device ID 0 

pmtimer0 on isa0 

orm0: <ISA Option ROMs> at iomem 0xc0000-0xca7ff,0xcc000-0xccfff,0xcd000-0xcd7ff pnpid ORM0000 on isa0 

ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0 

ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode 

ppc0: FIFO with 16/16/16 bytes threshold 

ppbus0: <Parallel port bus> on ppc0 

ppi0: <Parallel I/O> on ppbus0 

plip0: <PLIP network interface> on ppbus0 

lpt0: <Printer> on ppbus0 

lpt0: Interrupt-driven port 

ppc0: [GIANT-LOCKED] 

ppc0: [ITHREAD] 

sc0: <System console> at flags 0x100 on isa0 

sc0: VGA <16 virtual consoles, flags=0x300> 

sio1: configured irq 3 not in bitmap of probed irqs 0 

sio1: port may not be enabled 

vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 

Timecounter "TSC" frequency 1580462894 Hz quality 800 

Timecounters tick every 1.000 msec 

ad0: 305245MB <Hitachi HDT725032VLAT80 V54OA4NA> at ata0-master UDMA133 

ad1: 78167MB <Maxtor 6Y080L0 YAR41BW0> at ata0-slave UDMA133 

ad2: 238475MB <HDT722525DLAT80 V44OA70A> at ata1-master UDMA133 

acd0: DVDR <HL-DT-ST DVDRAM GSA-4163B/A102> at ata1-slave UDMA33 

twed0: <Unit 2, JBOD, Normal> on twe0 

twed0: 305245MB (625142448 sectors) 

twed1: <Unit 3, JBOD, Normal> on twe0 

twed1: 305245MB (625142448 sectors) 

WARNING: WITNESS option enabled, expect reduced performance. 

Trying to mount root from ufs:/dev/ad0s1a 

WARNING: / was not properly dismounted 

WARNING: /tmp was not properly dismounted 

WARNING: /usr was not properly dismounted 

WARNING: /var was not properly dismounted
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
ZFS filesystem version 6
ZFS storage pool version 6
Received on Tue Oct 23 2007 - 17:28:25 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:20 UTC