RE: 7.0-BETA4 - witness_warn on boot

From: Michael Jung <mikej_at_paymentallianceintl.com>
Date: Mon, 7 Jan 2008 10:01:30 -0500
 

-----Original Message-----
From: Scott Long [mailto:scottl_at_samsco.org] 
Sent: Sunday, December 16, 2007 2:48 AM
To: Michael Jung
Cc: freebsd-current_at_freebsd.org
Subject: Re: 7.0-BETA4 - witness_warn on boot

This simple patch should fix it.  This code path is pretty rare, though,
I'm frankly shocked that you're running into it.

--- cam_xpt.c   20 Oct 2007 23:23:12 -0000      1.191
+++ cam_xpt.c   16 Dec 2007 07:46:22 -0000
_at__at_ -3862,6 +3862,7 _at__at_
                                                    &work_ccb->ccb_h,
                                                    xpt_links.stqe);

+                               mtx_unlock(&xsoftc.xpt_lock);
                                 continue;
                         } else {
                                 /*

Scott


Michael Jung wrote:
> I was running 7.0-BETA4 and had problems with SEG_FAULT 12 in g_up so 
> I rebuilt a stock GENERIC kernel with these additional options:
> 
>  
> 
> options     KDB
> 
> option      DDB
> 
> options     INVARIANTS
> 
> options     INVARIANT_SUPPORT
> 
> options     WITNESS
> 
> options     DEBUG_LOCKS
> 
> options     DEBUG_VFS_LOCKS
> 
> options     DIAGNOSTIC
> 
>  
> 
> Here is the build date:
> 
>  
> 
> FreeBSD charon.confluentasp.com 7.0-BETA4 FreeBSD 7.0-BETA4 #6: Tue 
> Dec
> 11 09:05:50 EST 2007
> mikej_at_charon.confluentasp.com:/usr/obj/usr/src/sys/CHARON  i386
> 
> (I cvsup'd source on this day just prior to kernel build - 
> tag=RELENG_7)
> 
>  
> 
> Now when my SCSI array is plugged into the AHA-2944 controller the 
> kernel panics on boot.  If the array is not plugged
> 
> into the controller the system boots and runs fine.
> 
>  
> 
> I've read the handbook and tried to follow what posts I read as to 
> what would be of use in trouble shooting this.
> 
> Please don't shoot the messenger. I'll be glad to supply and 
> additional information - this box is not in production
> 
> so I'm willing to try anything.
> 
>  
> 
> Just to note, the controller and array had been in production under 
> 6.2-RELEASE and prior versions and has been running
> 
> for years without any problems.
> 
>  
> 
> Thanks.
> 
>  
> 
> --mikej
> 
>  
> 
> KDB: debugger backends: ddb
> 
> KDB: current backend: ddb
> 
> Copyright (c) 1992-2007 The FreeBSD Project.
> 
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 
> 1994
> 
>             The Regents of the University of California. All rights 
> reserved.
> 
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> 
> FreeBSD 7.0-BETA4 #6: Tue Dec 11 09:05:50 EST 2007
> 
>     mikej_at_charon.confluentasp.com:/usr/obj/usr/src/sys/CHARON
> 
> WARNING: WITNESS option enabled, expect reduced performance.
> 
> WARNING: DIAGNOSTIC option enabled, expect reduced performance.
> 
> Timecounter "i8254" frequency 1193182 Hz quality 0
> 
> CPU: Mobile AMD Sempron(tm) Processor 3000+ (1799.81-MHz 686-class 
> CPU)
> 
>   Origin = "AuthenticAMD"  Id = 0x20fc2  Stepping = 2
> 
>  
> Features=0x78bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PG
> E, MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2>
> 
>   Features2=0x1<SSE3>
> 
>   AMD Features=0xc2500800<SYSCALL,NX,MMX+,FFXSR,3DNow!+,3DNow!>
> 
>   AMD Features2=0x1<LAHF>
> 
> real memory  = 1005453312 (958 MB)
> 
> avail memory = 969691136 (924 MB)
> 
> ACPI APIC Table: <VIAK8M AWRDACPI>
> 
> WITNESS: spin lock intrcnt not in order list
> 
> ioapic0 <Version 0.3> irqs 0-23 on motherboard
> 
> kbd1 at kbdmux0
> 
> ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
> RF5413)
> 
> acpi0: <VIAK8M AWRDACPI> on motherboard
> 
> acpi0: [ITHREAD]
> 
> acpi0: Power Button (fixed)
> 
> acpi0: reservation of 0, a0000 (3) failed
> 
> acpi0: reservation of 100000, 3bde0000 (3) failed
> 
> Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
> 
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
> 
> cpu0: <ACPI CPU> on acpi0
> 
> powernow0: <PowerNow! K8> on cpu0
> 
> acpi_button0: <Power Button> on acpi0
> 
> acpi_button1: <Sleep Button> on acpi0
> 
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> 
> pci0: <ACPI PCI bus> on pcib0
> 
> agp0: <VIA 8380 host to PCI bridge> on hostb0
> 
> pcib1: <PCI-PCI bridge> at device 1.0 on pci0
> 
> pci1: <PCI bus> on pcib1
> 
> vgapci0: <VGA-compatible display> mem
> 0xf0000000-0xf3ffffff,0xf4000000-0xf4ffffff irq 16 at device 0.0 on 
> pci1
> 
> pcib2: <PCI-PCI bridge> at device 8.0 on pci0
> 
> pci2: <PCI bus> on pcib2
> 
> ahc0: <Adaptec 4944 Ultra SCSI adapter> port 0xc000-0xc0ff mem 
> 0xf6013000-0xf6013fff irq 16 at device 4.0 on pci2
> 
> ahc0: [ITHREAD]
> 
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> 
> ahc1: <Adaptec 4944 Ultra SCSI adapter> port 0xc400-0xc4ff mem 
> 0xf6010000-0xf6010fff irq 17 at device 5.0 on pci2
> 
> ahc1: [ITHREAD]
> 
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> 
> ahc2: <Adaptec 4944 Ultra SCSI adapter> port 0xc800-0xc8ff mem 
> 0xf6011000-0xf6011fff irq 18 at device 6.0 on pci2
> 
> ahc2: [ITHREAD]
> 
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> 
> ahc3: <Adaptec 4944 Ultra SCSI adapter> port 0xcc00-0xccff mem 
> 0xf6012000-0xf6012fff irq 19 at device 7.0 on pci2
> 
> ahc3: [ITHREAD]
> 
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> 
> atapci0: <VIA 6420 SATA150 controller> port 
> 0xe100-0xe107,0xe700-0xe703,0xe800-0xe807,0xe900-0xe903,0xe000-0xe00f,
> 0x d000-0xd0ff irq 20 at device 15.0 on pci0
> 
> atapci0: [ITHREAD]
> 
> ata2: <ATA channel 0> on atapci0
> 
> ata2: [ITHREAD]
> 
> ata3: <ATA channel 1> on atapci0
> 
> ata3: [ITHREAD]
> 
> atapci1: <VIA 8237 UDMA133 controller> port 
> 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xe200-0xe20f at device 15.1 on 
> pci0
> 
> ata0: <ATA channel 0> on atapci1
> 
> ata0: [ITHREAD]
> 
> ata1: <ATA channel 1> on atapci1
> 
> ata1: [ITHREAD]
> 
> uhci0: <VIA 83C572 USB controller> port 0xe300-0xe31f irq 21 at device

> 16.0 on pci0
> 
> uhci0: [GIANT-LOCKED]
> 
> uhci0: [ITHREAD]
> 
> usb0: <VIA 83C572 USB controller> on uhci0
> 
> usb0: USB revision 1.0
> 
> uhub0: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
> 
> uhub0: 2 ports with 2 removable, self powered
> 
> uhci1: <VIA 83C572 USB controller> port 0xe400-0xe41f irq 21 at device
> 16.1 on pci0
> 
> uhci1: [GIANT-LOCKED]
> 
> uhci1: [ITHREAD]
> 
> usb1: <VIA 83C572 USB controller> on uhci1
> 
> usb1: USB revision 1.0
> 
> uhub1: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
> 
> uhub1: 2 ports with 2 removable, self powered
> 
> uhci2: <VIA 83C572 USB controller> port 0xe500-0xe51f irq 21 at device
> 16.2 on pci0
> 
> uhci2: [GIANT-LOCKED]
> 
> uhci2: [ITHREAD]
> 
> usb2: <VIA 83C572 USB controller> on uhci2
> 
> usb2: USB revision 1.0
> 
> uhub2: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
> 
> uhub2: 2 ports with 2 removable, self powered
> 
> uhci3: <VIA 83C572 USB controller> port 0xe600-0xe61f irq 21 at device
> 16.3 on pci0
> 
> uhci3: [GIANT-LOCKED]
> 
> uhci3: [ITHREAD]
> 
> usb3: <VIA 83C572 USB controller> on uhci3
> 
> usb3: USB revision 1.0
> 
> uhub3: <VIA UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
> 
> uhub3: 2 ports with 2 removable, self powered
> 
> ehci0: <VIA VT6202 USB 2.0 controller> mem 0xf6100000-0xf61000ff irq 
> 21 at device 16.4 on pci0
> 
> ehci0: [GIANT-LOCKED]
> 
> ehci0: [ITHREAD]
> 
> usb4: EHCI version 1.0
> 
> usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
> 
> usb4: <VIA VT6202 USB 2.0 controller> on ehci0
> 
> usb4: USB revision 2.0
> 
> uhub4: <VIA EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
> 
> uhub4: 8 ports with 8 removable, self powered
> 
> isab0: <PCI-ISA bridge> at device 17.0 on pci0
> 
> isa0: <ISA bus> on isab0
> 
> pci0: <multimedia, audio> at device 17.5 (no driver attached)
> 
> vr0: <VIA VT6102 Rhine II 10/100BaseTX> port 0xdc00-0xdcff mem 
> 0xf6101000-0xf61010ff irq 23 at device 18.0 on pci0
> 
> vr0: Quirks: 0x0
> 
> miibus0: <MII bus> on vr0
> 
> rlphy0: <RTL8201L 10/100 media interface> PHY 1 on miibus0
> 
> rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> 
> vr0: using obsoleted if_watchdog interface
> 
> vr0: Ethernet address: 00:e0:4d:0a:1b:e6
> 
> vr0: [ITHREAD]
> 
> acpi_tz0: <Thermal Zone> on acpi0
> 
> fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on 
> acpi0
> 
> fdc0: [FILTER]
> 
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> 
> sio0: configured irq 4 not in bitmap of probed irqs 0
> 
> sio0: port may not be enabled
> 
> sio0: configured irq 4 not in bitmap of probed irqs 0
> 
> sio0: port may not be enabled
> 
> sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 
> on acpi0
> 
> sio0: type 16550A, console
> 
> sio0: [FILTER]
> 
> atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
> 
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> 
> kbd0 at atkbd0
> 
> atkbd0: [GIANT-LOCKED]
> 
> atkbd0: [ITHREAD]
> 
> psm0: <PS/2 Mouse> irq 12 on atkbdc0
> 
> psm0: [GIANT-LOCKED]
> 
> psm0: [ITHREAD]
> 
> psm0: model IntelliMouse, device ID 3
> 
> pmtimer0 on isa0
> 
> orm0: <ISA Option ROM> at iomem 0xcc000-0xcc7ff pnpid ORM0000 on isa0
> 
> ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
> 
> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
> 
> ppbus0: <Parallel port bus> on ppc0
> 
> lpt0: <Printer> on ppbus0
> 
> lpt0: Interrupt-driven port
> 
> ppi0: <Parallel I/O> on ppbus0
> 
> plip0: <PLIP network interface> on ppbus0
> 
> ppc0: [GIANT-LOCKED]
> 
> ppc0: [ITHREAD]
> 
> sc0: <System console> at flags 0x100 on isa0
> 
> sc0: VGA <16 virtual consoles, flags=0x300>
> 
> sio1: configured irq 3 not in bitmap of probed irqs 0
> 
> sio1: port may not be enabled
> 
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on 
> isa0
> 
> Timecounter "TSC" frequency 1799808459 Hz quality 800
> 
> Timecounters tick every 1.000 msec
> 
> Waiting 5 seconds for SCSI devices to settle
> 
> ad4: 381554MB <WDC WD4000AAKS-00TMA0 12.01C01> at ata2-master SATA150
> 
> suspending ithread with the following locks held:
> 
> exclusive sleep mutex XPT lock r = 0 (0xc0b883a4) locked _at_
> /usr/src/sys/cam/cam_xpt.c:3852
> 
> panic: witness_warn
> 
> cpuid = 0
> 
> KDB: enter: panic
> 
> [thread pid 18 tid 100016 ]
> 
> Stopped at      kdb_enter+0x32: leave
> 
> db> ps
> 
>   pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
> 
>    53     0     0     0  SL      waiting_ 0xc0c084cc [sctp_iterator]
> 
>    52     0     0     0  WL                          [irq7: ppc0]
> 
>    51     0     0     0  WL                          [irq12: psm0]
> 
>    50     0     0     0  WL                          [irq1: atkbd0]
> 
>    49     0     0     0  WL                          [swi0: sio]
> 
>    48     0     0     0  SL      -        0xc401f23c [fdc0]
> 
>    47     0     0     0  SL      cooling  0xc3fedad4 [acpi_cooling0]
> 
>    46     0     0     0  SL      tzpoll   0xc0d85d40 [acpi_thermal]
> 
>    45     0     0     0  WL                          [irq23: vr0]
> 
>    44     0     0     0  SL      usbevt   0xc3fe4210 [usb4]
> 
>    43     0     0     0  SL      usbevt   0xc3ffb210 [usb3]
> 
>    42     0     0     0  SL      usbevt   0xc3ff5210 [usb2]
> 
>    41     0     0     0  SL      usbevt   0xc3fee210 [usb1]
> 
>    40     0     0     0  SL      usbtsk   0xc0bb7174 [usbtask-dr]
> 
>    39     0     0     0  SL      usbtsk   0xc0bb7160 [usbtask-hc]
> 
>    38     0     0     0  SL      usbevt   0xc3fc8210 [usb0]
> 
>    37     0     0     0  WL                          [irq21: uhci0
> uhci*]
> 
>    36     0     0     0  WL                          [irq15: ata1]
> 
>    35     0     0     0  WL                          [irq14: ata0]
> 
> --More--
> 
>         
> 
>    34     0     0     0  WL                          [irq20: atapci0]
> 
>    33     0     0     0  SL      idle     0xc3ef1e00 [aic_recovery3]
> 
>    32     0     0     0  RL                          [irq19: ahc3]
> 
>    31     0     0     0  SL      idle     0xc3ef1e00 [aic_recovery3]
> 
>    30     0     0     0  SL      idle     0xc3ef1400 [aic_recovery2]
> 
>    29     0     0     0  RL                          [irq18: ahc2]
> 
>    28     0     0     0  SL      idle     0xc3ef1400 [aic_recovery2]
> 
>    27     0     0     0  SL      idle     0xc3ef0a00 [aic_recovery1]
> 
>    26     0     0     0  RL                          [irq17: ahc1]
> 
>    25     0     0     0  SL      idle     0xc3ef0a00 [aic_recovery1]
> 
>    24     0     0     0  SL      idle     0xc3ef0c00 [aic_recovery0]
> 
>    23     0     0     0  WL                          [irq16: ahc0]
> 
>    22     0     0     0  SL      idle     0xc3ef0c00 [aic_recovery0]
> 
>    21     0     0     0  WL                          [irq9: acpi0]
> 
>    20     0     0     0  SL      -        0xc3ed9580 [thread taskq]
> 
>    19     0     0     0  WL                          [swi5: +]
> 
>    18     0     0     0  RL      CPU 0               [swi2: cambio]
> 
>     9     0     0     0  SL      ccb_scan 0xc0b88374 [xpt_thrd]
> 
>     8     0     0     0  SL      -        0xc3ed9900 [acpi_task_2]
> 
>     7     0     0     0  SL      -        0xc3ed9900 [acpi_task_1]
> 
> --More--
> 
>         
> 
>     6     0     0     0  SL      -        0xc3ed9900 [acpi_task_0]
> 
>     5     0     0     0  SL      -        0xc3ed9980 [kqueue taskq]
> 
>    17     0     0     0  WL                          [swi6: task
queue]
> 
>    16     0     0     0  WL                          [swi6: Giant
taskq]
> 
>    15     0     0     0  SL      -        0xc0bb94d4 [yarrow]
> 
>     4     0     0     0  SL      -        0xc0bb754c [g_down]
> 
>     3     0     0     0  SL      -        0xc0bb7548 [g_up]
> 
>     2     0     0     0  SL      -        0xc0bb7540 [g_event]
> 
>    14     0     0     0  WL                          [swi3: vm]
> 
>    13     0     0     0  RL                          [swi4: clock sio]
> 
>    12     0     0     0  WL                          [swi1: net]
> 
>    11     0     0     0  RL                          [idle: cpu0]
> 
>     1     0     0     0  ?L                          [swapper]
> 
>    10     0     0     0  SL      audit_wo 0xc0c11ad4 [audit]
> 
>     0     0     0     0  SLs     conifhk  0xc0b599ec [swapper]
> 
> db> show reg
> 
> cs                0x20
> 
> ds                0x28
> 
> es                0x28
> 
> fs                 0x8
> 
> ss                0x28
> 
> eax               0x12
> 
> ecx         0xc1434bde
> 
> edx                  0
> 
> ebx              0x100
> 
> esp         0xe175cc5c
> 
> ebp         0xe175cc64
> 
> esi                0x1
> 
> edi         0xc3dea210
> 
> eip         0xc0777ca2  kdb_enter+0x32
> 
> efl            0x80282
> 
> kdb_enter+0x32: leave
> 
> db> show locks
> 
> exclusive sleep mutex XPT lock r = 0 (0xc0b883a4) locked _at_
> /usr/src/sys/cam/cam_xpt.c:3852
> 
> db> show lockedvnods
> 
> Locked vnodes
> 
> db> bt
> 
> Tracing pid 18 tid 100016 td 0xc3dea210
> 
> kdb_enter(c0aa2af2,0,c0aa7718,e175cc9c,0,...) at kdb_enter+0x32
> 
> panic(c0aa7718,e175ccd8,0,0,0,...) at panic+0x124
> 
> witness_warn(2,0,c0a9f7d0,471,c3ed97e4,...) at witness_warn+0x19c
> 
> ithread_loop(c3ed80d0,e175cd38,c0a9f445,2ea,c3ee42ac,...) at 
> ithread_loop+0x2ca
> 
> fork_exit(c0735670,c3ed80d0,e175cd38) at fork_exit+0xb8
> 
> fork_trampoline() at fork_trampoline+0x8
> 
> --- trap 0, eip = 0, esp = 0xe175cd70, ebp = 0 ---
> 
> db>
> 
> 
> CONFIDENTIALITY NOTE: This message is intended only for the use of the

> individual or entity to whom it is addressed and may contain 
> information that is privileged, confidential, and exempt from 
> disclosure under applicable law. If the reader of this message is not 
> the intended recipient, you are hereby notified that any 
> dissemination, distribution or copying of this communication is 
> strictly prohibited. If you have received this transmission in error, 
> please notify us by telephone at (502) 212-4001 or notify us at PAI , 
> Dept. 99, 11857 Commonwealth Drive, Louisville, KY  40299.  Thank you.
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list 
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
"freebsd-current-unsubscribe_at_freebsd.org"

Scott,

Any chance on getting this committed?  It's been working great but
requires me to 
patch after every cvsup.

Thanks.

--mikej

CONFIDENTIALITY NOTE: This message is intended only for the use
of the individual or entity to whom it is addressed and may contain 
information that is privileged, confidential, and exempt from 
disclosure under applicable law. If the reader of this message is 
not the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication 
is strictly prohibited. If you have received this transmission 
in error, please notify us by telephone at (502) 212-4001 or 
notify us at PAI , Dept. 99, 11857 Commonwealth Drive, 
Louisville, KY  40299.  Thank you.
Received on Mon Jan 07 2008 - 14:01:49 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:25 UTC