Re: Kernel panic on all versions of FreeBSD

2009/8/14 Alexey Tarasov <lexasoft_at_gmail.com>:
> Hello.
>
> I have more than 30 Supermicro servers with same config and on all of them I
> get kernel panic.
>
> Here is the screenshots of kgdb output:
> http://lexasoft.ru/metamphetamine/Panic/Picture%2073.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2074.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2075.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2076.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2077.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2078.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2079.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2080.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2080.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2082.png
> http://lexasoft.ru/metamphetamine/Panic/Picture%2083.png
>
> Here is dmesg output:
>
> Copyright (c) 1992-2009 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 7.2-STABLE #0: Wed Aug 12 01:06:04 MSD 2009
>   root_at_st2.srv.getthebit.com:/usr/obj/usr/src/sys/STORAGE
> WARNING: WITNESS option enabled, expect reduced performance.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Pentium(R) 4 CPU 3.00GHz (2992.52-MHz K8-class CPU)
>  Origin = "GenuineIntel"  Id = 0xf65  Stepping = 5
>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>  Features2=0xe59d<SSE3,DTES64,MON,DS_CPL,EST,TM2,CNXT-ID,CX16,xTPR,PDCM>
>  AMD Features=0x20100800<SYSCALL,NX,LM>
>  AMD Features2=0x1<LAHF>
>  TSC: P-state invariant
>  Logical CPUs per core: 2
> usable memory = 2132484096 (2033 MB)
> avail memory  = 2056134656 (1960 MB)
> ACPI APIC Table: <PTLTD    APIC  >
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> cpu0 (BSP): APIC ID:  0
> cpu1 (AP/HT): APIC ID:  1
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> ioapic1 <Version 2.0> irqs 24-47 on motherboard
> kbd1 at kbdmux0
> acpi0: <PTLTD   RSDT> on motherboard
> acpi0: [ITHREAD]
> acpi0: Power Button (fixed)
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pcib2: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
> pci9: <ACPI PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci9
> pci10: <ACPI PCI bus> on pcib3
> pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
> pci13: <ACPI PCI bus> on pcib4
> em0: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x4000-0x401f mem
> 0xe8000000-0xe801ffff irq 16 at device 0.0 on pci13
> em0: Using MSI interrupt
> em0: [FILTER]
> em0: Ethernet address: 00:30:48:91:93:40
> pcib5: <ACPI PCI-PCI bridge> irq 16 at device 28.5 on pci0
> pci14: <ACPI PCI bus> on pcib5
> em1: <Intel(R) PRO/1000 Network Connection 6.9.6> port 0x5000-0x501f mem
> 0xe8200000-0xe821ffff irq 17 at device 0.0 on pci14
> em1: Using MSI interrupt
> em1: [FILTER]
> em1: Ethernet address: 00:30:48:91:93:41
> uhci0: <UHCI (generic) USB controller> port 0x3000-0x301f irq 23 at device
> 29.0 on pci0
> uhci0: [GIANT-LOCKED]
> uhci0: [ITHREAD]
> usb0: <UHCI (generic) USB controller> on uhci0
> usb0: USB revision 1.0
> uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> uhub0: 2 ports with 2 removable, self powered
> uhci1: <UHCI (generic) USB controller> port 0x3020-0x303f irq 19 at device
> 29.1 on pci0
> uhci1: [GIANT-LOCKED]
> uhci1: [ITHREAD]
> usb1: <UHCI (generic) USB controller> on uhci1
> usb1: USB revision 1.0
> uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
> uhub1: 2 ports with 2 removable, self powered
> uhci2: <UHCI (generic) USB controller> port 0x3040-0x305f irq 18 at device
> 29.2 on pci0
> uhci2: [GIANT-LOCKED]
> uhci2: [ITHREAD]
> usb2: <UHCI (generic) USB controller> on uhci2
> usb2: USB revision 1.0
> uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
> uhub2: 2 ports with 2 removable, self powered
> uhci3: <UHCI (generic) USB controller> port 0x3060-0x307f irq 16 at device
> 29.3 on pci0
> uhci3: [GIANT-LOCKED]
> uhci3: [ITHREAD]
> usb3: <UHCI (generic) USB controller> on uhci3
> usb3: USB revision 1.0
> uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
> uhub3: 2 ports with 2 removable, self powered
> ehci0: <Intel 82801GB/R (ICH7) USB 2.0 controller> mem 0xe8600000-0xe86003ff
> irq 23 at device 29.7 on pci0
> ehci0: [GIANT-LOCKED]
> ehci0: [ITHREAD]
> usb4: EHCI version 1.0
> usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
> usb4: <Intel 82801GB/R (ICH7) USB 2.0 controller> on ehci0
> usb4: USB revision 2.0
> uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
> uhub4: 8 ports with 8 removable, self powered
> pcib6: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci15: <ACPI PCI bus> on pcib6
> vgapci0: <VGA-compatible display> port 0x6000-0x60ff mem
> 0xe0000000-0xe7ffffff,0xe8300000-0xe830ffff irq 16 at device 0.0 on pci15
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel ICH7 UDMA100 controller> port
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x30a0-0x30af at device 31.1 on pci0
> ata0: <ATA channel 0> on atapci0
> ata0: [ITHREAD]
> atapci1: <Intel ICH7 SATA300 controller> port
> 0x30e8-0x30ef,0x30dc-0x30df,0x30e0-0x30e7,0x30d8-0x30db,0x30b0-0x30bf mem
> 0xe8600400-0xe86007ff irq 19 at device 31.2 on pci0
> atapci1: [ITHREAD]
> ata2: <ATA channel 0> on atapci1
> ata2: [ITHREAD]
> ata3: <ATA channel 1> on atapci1
> ata3: [ITHREAD]
> pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> atkbd0: [ITHREAD]
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: configured irq 4 not in bitmap of probed irqs 0
> sio0: port may not be enabled
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
> acpi0
> sio0: type 16550A
> sio0: [FILTER]
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> sio1: configured irq 3 not in bitmap of probed irqs 0
> sio1: port may not be enabled
> sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
> sio1: type 16550A
> sio1: [FILTER]
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: does not respond
> device_attach: fdc0 attach returned 6
> ppc0: <Parallel port> port 0x378-0x37f,0x778-0x77f irq 7 drq 3 on acpi0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/9 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> ppbus0: [ITHREAD]
> plip0: <PLIP network interface> on ppbus0
> plip0: WARNING: using obsoleted IFF_NEEDSGIANT flag
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> ppc0: [GIANT-LOCKED]
> ppc0: [ITHREAD]
> cpu0: <ACPI CPU> on acpi0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr f2700000f27
> device_attach: est0 attach returned 6
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> cpu1: <ACPI CPU> on acpi0
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> est: CPU supports Enhanced Speedstep, but is not recognized.
> est: cpu_vendor GenuineIntel, msr f2700000f27
> device_attach: est1 attach returned 6
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
> fdc0: does not respond
> device_attach: fdc0 attach returned 6
> orm0: <ISA Option ROMs> at iomem
> 0xc0000-0xcafff,0xcb000-0xcbfff,0xcc000-0xccfff on isa0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> Timecounters tick every 1.000 msec
> acd0: CDROM <CD-224E-N/1.AA> at ata0-slave UDMA33
> ad4: 381554MB <Seagate ST3400632NS 5.01> at ata2-master SATA150
> ad5: 381554MB <Seagate ST3400632NS 5.01> at ata2-slave SATA150
> ad6: 381554MB <Seagate ST3400632NS 5.01> at ata3-master SATA150
> ad7: 476940MB <Seagate ST3500630AS 3.AAK> at ata3-slave SATA150
> SMP: AP CPU #1 Launched!
> WARNING: WITNESS option enabled, expect reduced performance.
> Trying to mount root from ufs:/dev/ad4s1a
> WARNING: / was not properly dismounted
> WARNING: /data1 was not properly dismounted
> WARNING: /data2 was not properly dismounted
> WARNING: /data3 was not properly dismounted
> WARNING: /data4 was not properly dismounted
> WARNING: /tmp was not properly dismounted
> WARNING: /usr was not properly dismounted
> /usr: mount pending error: blocks 176 files 26
> WARNING: /var was not properly dismounted
> /var: mount pending error: blocks 44 files 11
> em0: link state changed to UP
>
> I have vmcore in /var/crash folder. What debug information is needed to
> debug this problem?

As you got a dump, could you please list with kgdb the precise point
within witness_lock() where the offending spinlock unlocking happens?
Can you show the state of the offending spinlock?

Attilio


-- 
Peace can only be achieved by understanding - A. Einstein