Re: 5.2-CURRENT constantly crashing

From: Robert Watson <rwatson_at_FreeBSD.org>
Date: Thu, 22 Jul 2004 08:37:50 -0400 (EDT)
On Thu, 22 Jul 2004, Jan Lentfer wrote:

> i am running 5.2-CURRENT on an old 4x PPRO (200) Amdahl EnVista Server. 
> The system is constantly crashing under heavy load for about 2-3 month
> now.  Since all drives are connected to a Mylex Raid Controller, I can't
> get any kernel dumps :(. 
> 
> The system fails with "panic: page fault. fatal trap 12. fault code =
> supervisor read, page not present". 
> 
> Since this is old hardware I am not quite sure if this is a hardware or
> software problem, but the system used to run stable with 5.1-RELEASE. 
> 
> I don't have a serial console either, so i can't copy and paste the
> panic warning. Any way this can be sent over ssh so I can post the
> complete kernel panic??

Not easily.  The most useful thing up front would be to know what the
faulting address is, and to have you turn the instruction pointer in the
panic into an address in the kernel.  If you can get into DDB and get a
stack trace, a list of the names of the symbols (and ideally the offsets)
would be good.  The best answer would be if you could set up a serial
console temporarily, as it will both save you trouble and increase
accuracy.

Thanks!

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research


> 
> This is the output from systat -vmstat from the last crash (don't know if it 
> helps):
> 
>     2 users    Load 11.96 12.03 10.58                  Jul 22 10:23
> 
> Mem:KB    REAL            VIRTUAL                     VN PAGER  SWAP PAGER
>         Tot   Share      Tot    Share    Free         in  out     in  out
> Act  226024   12128   585524    18944   27160 count
> All  482280   43796  3256020   172068         pages    1
>                                                      1508 zfod   Interrupts
> Proc:r  p  d  s  w    Csw  Trp  Sys  Int  Sof  Flt     96 cow     242 total
>      3    14113 12   2044 1884 1740  350   40 1671  94240 wire        1: atkb
>                                                    257092 act         5: ahc1
> 15.8%Sys   1.1%Intr 51.0%User 23.6%Nice  8.4%Idl   130564 inact       6: fdc0
> |    |    |    |    |    |    |    |    |    |       1992 cache   129 8: rtc
> ========>>>>>>>>>>>>>>>>>>>>>>>>>>------------      25832 free        9: ahc0
>                                                           daefr     7 10: xl0
> Namei         Name-cache    Dir-cache                1257 prcfr     6 11: mlx
>     Calls     hits    %     hits    %                     react       13: npx
>     11579    11522  100                                   pdwak   100 0: clk
>                                                           pdpgs
> Disks mlxd0 mlxd1   cd0   fd0 pass0                       intrn
> KB/t   5.21  0.00  0.00  0.00  0.00                 61888 buf
> tps       5     0     0     0     0                   105 dirtybuf
> MB/s   0.03  0.00  0.00  0.00  0.00                 35957 desiredvnodes
> % busy    4     0     0     0     0                  8989 numvnodes
>                                                      1193 freevnodes
> 
> 
> 
> 
> amdahl# uname -a
> FreeBSD amdahl.bio.tu-darmstadt.de 5.2-CURRENT FreeBSD 5.2-CURRENT #2: Thu Jul  
> 8 12:37:12 CEST 2004     
> lentferj_at_amdahl.bio.tu-darmstadt.de:/usr/obj/usr/src/sys/PPRO_SMP  i386
> 
> 
> 
> 
> amdahl# cat /usr/src/sys/i386/conf/PPRO_SMP
> 
> machine         i386
> #cpu            I486_CPU
> #cpu            I586_CPU
> cpu             I686_CPU
> ident           PPRO_SMP
> 
> # Turn on Filesystem Quotas
> options         QUOTA
> 
> # Enable IP Filtering
> options IPFIREWALL
> options IPDIVERT
> options IPFIREWALL_DEFAULT_TO_ACCEPT
> options DUMMYNET
> 
> #To statically compile in device wiring instead of /boot/device.hints
> #hints          "GENERIC.hints"         #Default places to look for devices.
> 
> #makeoptions    DEBUG=-g                #Build kernel with gdb(1) debug 
> symbols
> 
> options         SCHED_4BSD              #4BSD scheduler
> options         INET                    #InterNETworking
> options         INET6                   #IPv6 communications protocols
> options         FFS                     #Berkeley Fast Filesystem
> options         SOFTUPDATES             #Enable FFS soft updates support
> options         UFS_ACL                 #Support for access control lists
> options         UFS_DIRHASH             #Improve performance on big 
> directories
> options         MD_ROOT                 #MD is a potential root device
> options         NFSCLIENT               #Network Filesystem Client
> options         NFSSERVER               #Network Filesystem Server
> options         NFS_ROOT                #NFS usable as root device, requires 
> NFSCLIENT
> options         MSDOSFS                 #MSDOS Filesystem
> options         CD9660                  #ISO 9660 Filesystem
> options         PROCFS                  #Process filesystem (requires 
> PSEUDOFS)
> options         PSEUDOFS                #Pseudo-filesystem framework
> options         COMPAT_43               #Compatible with BSD 4.3 [KEEP THIS!]
> options         COMPAT_FREEBSD4         #Compatible with FreeBSD4
> options         SCSI_DELAY=15000        #Delay (in ms) before probing SCSI
> options         KTRACE                  #ktrace(1) support
> options         SYSVSHM                 #SYSV-style shared memory
> options         SYSVMSG                 #SYSV-style message queues
> options         SYSVSEM                 #SYSV-style semaphores
> options         _KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time 
> extensions
> options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
> options         AHC_REG_PRETTY_PRINT    # Print register bitfields in debug
>                                         # output.  Adds ~128k to driver.
> options         AHD_REG_PRETTY_PRINT    # Print register bitfields in debug
>                                         # output.  Adds ~215k to driver.
> 
> # Debugging for use in -current
> #options        DDB                     #Enable the kernel debugger
> #options        INVARIANTS              #Enable calls of extra sanity checking
> options         INVARIANT_SUPPORT       #Extra sanity checks of internal 
> structures, required by INVARIANTS
> #options        WITNESS                 #Enable checks to detect deadlocks and 
> cycles
> #options        WITNESS_SKIPSPIN        #Don't run witness on spinlocks for 
> speed
> 
> # To make an SMP kernel, the next two are needed
> options         SMP                     # Symmetric MultiProcessor Kernel
> #options        APIC_IO                 # Symmetric (APIC) I/O
> device          apic
> 
> device          isa
> device          eisa
> device          pci
> 
> # Floppy drives
> device          fdc
> 
> # ATA and ATAPI devices
> device          ata
> device          atadisk                 # ATA disk drives
> device          atapicd                 # ATAPI CDROM drives
> device          atapifd                 # ATAPI floppy drives
> device          atapist                 # ATAPI tape drives
> options         ATA_STATIC_ID           #Static device numbering
> 
> # SCSI Controllers
> #device         ahb             # EISA AHA1742 family
> device          ahc             # AHA2940 and onboard AIC7xxx devices
> #device         ahd             # AHA39320/29320 and onboard AIC79xx devices
> #device         amd             # AMD 53C974 (Tekram DC-390(T))
> #device         isp             # Qlogic family
> #device         mpt             # LSI-Logic MPT-Fusion
> #device         ncr             # NCR/Symbios Logic
> #device         sym             # NCR/Symbios Logic (newer chipsets + those of 
> `ncr')
> #device         trm             # Tekram DC395U/UW/F DC315U adapters
> 
> #device         adv             # Advansys SCSI adapters
> #device         adw             # Advansys wide SCSI adapters
> #device         aha             # Adaptec 154x SCSI adapters
> #device         aic             # Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
> #device         bt              # Buslogic/Mylex MultiMaster SCSI adapters
> 
> #device         ncv             # NCR 53C500
> #device         nsp             # Workbit Ninja SCSI-3
> #device         stg             # TMC 18C30/18C50
> 
> # RAID controllers interfaced to the SCSI subsystem
> #device         asr             # DPT SmartRAID V, VI and Adaptec SCSI RAID
> #device         ciss            # Compaq Smart RAID 5*
> #device         dpt             # DPT Smartcache III, IV - See NOTES for 
> options!
> #device         iir             # Intel Integrated RAID
> #device         mly             # Mylex AcceleRAID/eXtremeRAID
> 
> # SCSI peripherals
> device          scbus           # SCSI bus (required)
> device          ch              # SCSI media changers
> device          da              # Direct Access (disks)
> device          sa              # Sequential Access (tape etc)
> device          cd              # CD
> device          pass            # Passthrough device (direct SCSI access)
> device          ses             # SCSI Environmental Services (and SAF-TE)
> 
> # RAID controllers
> #device         aac             # Adaptec FSA RAID
> #device         aacp            # SCSI passthrough for aac (requires CAM)
> #device         amr             # AMI MegaRAID
> #device         ida             # Compaq Smart RAID
> #device         ips             # IBM (Adaptec) ServeRAID
> device          mlx             # Mylex DAC960 family
> #device         pst             # Promise Supertrak SX6000
> #device         twe             # 3ware ATA RAID
> 
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device          atkbdc          # AT keyboard controller
> device          atkbd           # AT keyboard
> device          psm             # PS/2 mouse
> 
> device          vga             # VGA video card driver
> 
> device          splash          # Splash screen and screen saver support
> 
> # syscons is the default console driver, resembling an SCO console
> device          sc
> 
> # Enable this for the pcvt (VT220 compatible) console driver
> #device         vt
> #options        XSERVER                 # support for X server on a vt console
> #options        FAT_CURSOR              # start with block cursor
> 
> #device         agp             # support several AGP chipsets
> 
> # Floating point support - do not disable.
> device          npx
> 
> # Power management support (see NOTES for more options)
> #device         apm
> # Add suspend/resume support for the i8254.
> device          pmtimer
> 
> # PCCARD (PCMCIA) support
> # Pcmcia and cardbus bridge support
> #device         cbb                     # cardbus (yenta) bridge
> #device         pcic                    # ExCA ISA and PCI bridges
> #device         pccard                  # PC Card (16-bit) bus
> #device         cardbus                 # CardBus (32-bit) bus
> 
> # Serial (COM) ports
> device          sio             # 8250, 16[45]50 based serial ports
> 
> # Parallel port
> device          ppc
> device          ppbus           # Parallel port bus (required)
> device          lpt             # Printer
> device          plip            # TCP/IP over parallel
> device          ppi             # Parallel port interface device
> #device         vpo             # Requires scbus and da
> 
> 
> # PCI Ethernet NICs.
> #device         de              # DEC/Intel DC21x4x (``Tulip'')
> #device         em              # Intel PRO/1000 adapter Gigabit Ethernet Card
> #device         txp             # 3Com 3cR990 (``Typhoon'')
> #device         vx              # 3Com 3c590, 3c595 (``Vortex'')
> 
> # PCI Ethernet NICs that use the common MII bus controller code.
> # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
> device          miibus          # MII bus support
> #device         dc              # DEC/Intel 21143 and various workalikes
> #device         fxp             # Intel EtherExpress PRO/100B (82557, 82558)
> #device         pcn             # AMD Am79C97x PCI 10/100 (precedence over 
> 'lnc')
> #device         rl              # RealTek 8129/8139
> #device         sf              # Adaptec AIC-6915 (``Starfire'')
> #device         sis             # Silicon Integrated Systems SiS 900/SiS 7016
> #device         sk              # SysKonnect SK-984x and SK-982x gigabit 
> ethernet
> #device         ste             # Sundance ST201 (D-Link DFE-550TX)
> #device         ti              # Alteon Networks Tigon I/II gigabit ethernet
> #device         tl              # Texas Instruments ThunderLAN
> #device         tx              # SMC EtherPower II (83c170 ``EPIC'')
> #device         vr              # VIA Rhine, Rhine II
> #device         wb              # Winbond W89C840F
> device          xl              # 3Com 3c90x (``Boomerang'', ``Cyclone'')
> #device         bge             # Broadcom BCM570xx Gigabit Ethernet
> 
> # ISA Ethernet NICs.  pccard nics included.
> #device         cs              # Crystal Semiconductor CS89x0 NIC
> # 'device ed' requires 'device miibus'
> #device         ed              # NE[12]000, SMC Ultra, 3c503, DS8390 cards
> #device         ex              # Intel EtherExpress Pro/10 and Pro/10+
> #device         ep              # Etherlink III based cards
> #device         fe              # Fujitsu MB8696x based cards
> #device         ie              # EtherExpress 8/16, 3C507, StarLAN 10 etc.
> #device         lnc             # NE2100, NE32-VL Lance Ethernet cards
> #device         sn              # SMC's 9000 series of ethernet chips
> #device         xe              # Xircom pccard ethernet
> 
> # ISA devices that use the old ISA shims
> #device         le
> 
> # Wireless NIC cards
> #device         wlan            # 802.11 support
> #device         an              # Aironet 4500/4800 802.11 wireless NICs.
> #device         awi             # BayStack 660 and others
> #device         wi              # WaveLAN/Intersil/Symbol 802.11 wireless 
> NICs.
> #device         wl              # Older non 802.11 Wavelan wireless NIC.
> 
> # Pseudo devices - the number indicates how many units to allocate.
> device          random          # Entropy device
> device          loop            # Network loopback
> device          ether           # Ethernet support
> device          sl              # Kernel SLIP
> device          ppp             # Kernel PPP
> device          tun             # Packet tunnel.
> device          pty             # Pseudo-ttys (telnet etc)
> device          md              # Memory "disks"
> device          gif             # IPv6 and IPv4 tunneling
> device          faith           # IPv6-to-IPv4 relaying (translation)
> 
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> device          bpf             # Berkeley packet filter
> 
> # USB support
> #device         uhci            # UHCI PCI->USB interface
> #device         ohci            # OHCI PCI->USB interface
> #device         usb             # USB Bus (required)
> #device         udbp            # USB Double Bulk Pipe devices
> #device         ugen            # Generic
> #device         uhid            # "Human Interface Devices"
> #device         ukbd            # Keyboard
> #device         ulpt            # Printer
> #device         umass           # Disks/Mass storage - Requires scbus and da
> #device         ums             # Mouse
> #device         urio            # Diamond Rio 500 MP3 player
> #device         uscanner        # Scanners
> # USB Ethernet, requires mii
> #device         aue             # ADMtek USB ethernet
> #device         axe             # ASIX Electronics USB ethernet
> #device         cue             # CATC USB ethernet
> #device         kue             # Kawasaki LSI USB ethernet
> 
> # FireWire support
> #device         firewire        # FireWire bus code
> #device         sbp             # SCSI over FireWire (Requires scbus and da)
> #device         fwe             # Ethernet over FireWire (non-standard!)
> 
> 
> 
> 
> amdahl# dmesg
> Copyright (c) 1992-2004 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD 5.2-CURRENT #2: Thu Jul  8 12:37:12 CEST 2004
>     lentferj_at_amdahl.bio.tu-darmstadt.de:/usr/obj/usr/src/sys/PPRO_SMP
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Pentium Pro (198.95-MHz 686-class CPU)
>   Origin = "GenuineIntel"  Id = 0x619  Stepping = 9
>   Features=0xfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV>
> real memory  = 536870912 (512 MB)
> avail memory = 519884800 (495 MB)
> MPTable: <INTEL    ALDER       >
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP): APIC ID:  1
>  cpu2 (AP): APIC ID:  2
>  cpu3 (AP): APIC ID:  4
> ioapic0: Assuming intbase of 0
> ioapic0 <Version 1.1> irqs 0-15 on motherboard
> npx0: [FAST]
> npx0: <math processor> on motherboard
> npx0: INT 16 interface
> pcib0: <Intel 82454KX/GX (Orion) host to PCI bridge> at pcibus 0 on 
> motherboard
> pci0: <PCI bus> on pcib0
> xl0: <3Com 3c905B-TX Fast Etherlink XL> port 0xfc80-0xfcff mem 
> 0xfeaffc00-0xfeaffc7f irq 10 at device 13.0 on pci0
> miibus0: <MII bus> on xl0
> xlphy0: <3Com internal media interface> on miibus0
> xlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> xl0: Ethernet address: 00:01:02:a8:b4:cf
> xl0: [GIANT-LOCKED]
> eisab0: <PCI-EISA bridge> at device 14.0 on pci0
> eisa0: <EISA bus> on eisab0
> mainboard0: <INT31c0 (System Board)> on eisa0 slot 0
> isa0: <ISA bus> on eisab0
> pci0: <unknown> at device 15.0 (no driver attached)
> pci0: <memory, RAM> at device 20.0 (no driver attached)
> pcib1: <Intel 82454KX/GX (Orion) host to PCI bridge> at pcibus 1 on 
> motherboard
> pci1: <PCI bus> on pcib1
> mlx0: <Mylex version 3 RAID interface> port 0xec80-0xecff mem 
> 0xfe8ffc00-0xfe8ffc7f irq 11 at device 10.0 on pci1
> mlx0: [GIANT-LOCKED]
> mlx0: DAC960P/PD, 2 channels, firmware 3.51-0-04, 32MB RAM
> mlxd0: <Mylex System Drive> on mlx0
> mlxd0: 8188MB (16769024 sectors) RAID 0 (online)
> mlxd1: <Mylex System Drive> on mlx0
> mlxd1: 20455MB (41891840 sectors) RAID 0 (online)
> ahc0: <Adaptec aic7880 Ultra SCSI adapter> port 0xe800-0xe8ff mem 
> 0xfe8fe000-0xfe8fefff irq 9 at device 11.0 on pci1
> ahc0: Using left over BIOS settings
> ahc0: [GIANT-LOCKED]
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> ahc1: <Adaptec aic7880 Ultra SCSI adapter> port 0xe400-0xe4ff mem 
> 0xfe8fd000-0xfe8fdfff irq 5 at device 12.0 on pci1
> ahc1: Using left over BIOS settings
> ahc1: [GIANT-LOCKED]
> aic7880: Ultra Wide Channel A, SCSI Id=7, 16/253 SCBs
> cpu0 on motherboard
> cpu1 on motherboard
> cpu2 on motherboard
> cpu3 on motherboard
> orm0: <ISA Option ROMs> at iomem 
> 0xec000-0xeffff,0xea000-0xebfff,0xcf000-0xcf7ff,0xce800-0xcefff,0xc0000-0xc7fff 
> on isa0
> pmtimer0 on isa0
> ata0 at port 0x3f6,0x1f0-0x1f7 irq 14 on isa0
> ata1 at port 0x376,0x170-0x177 irq 15 on isa0
> atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
> atkbd0: <AT Keyboard> irq 1 on atkbdc0
> kbd0 at atkbd0
> atkbd0: [GIANT-LOCKED]
> fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> at port 
> 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0
> fdc0: FIFO enabled, 8 bytes threshold
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
> ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode
> ppc0: FIFO with 16/16/8 bytes threshold
> ppbus0: <Parallel port bus> on ppc0
> plip0: <PLIP network interface> on ppbus0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
> sio0: type 16550A
> sio1 at port 0x2f8-0x2ff irq 3 on isa0
> sio1: type 16550A
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> unknown: <PNP0303> can't assign resources (port)
> psmcpnp0: irq resource info is missing; assuming irq 12
> unknown: <PNP0700> can't assign resources (port)
> unknown: <PNP0400> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> Timecounters tick every 10.000 msec
> ipfw2 initialized, divert enabled, rule-based forwarding enabled, default to 
> accept, logging disabled
> Waiting 15 seconds for SCSI devices to settle
> SMP: AP CPU #1 Launched!
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #3 Launched!
> cd0 at ahc1 bus 0 target 0 lun 0
> cd0: <TEAC CD-ROM CD-516S 1.0D> Removable CD-ROM SCSI-2 device
> cd0: 10.000MB/s transfers (10.000MHz, offset 8)
> cd0: Attempt to query device size failed: NOT READY, Medium not present
> Mounting root from ufs:/dev/mlxd0s1a
> WARNING: / was not properly dismounted
> WARNING: /tmp was not properly dismounted
> /tmp: mount pending error: blocks 40 files 10
> WARNING: /usr was not properly dismounted
> /usr: superblock summary recomputed
> WARNING: /var was not properly dismounted
> /var: mount pending error: blocks 12 files 0
> /var: superblock summary recomputed
> WARNING: /blast was not properly dismounted
> Accounting enabled
> 
> 
> 
> Thanks,
> 
> 
> Jan 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> 
Received on Thu Jul 22 2004 - 10:38:34 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:02 UTC