sysctl spinning (was: Re: ps Causes Hard Hang)

From: Robert Watson <rwatson_at_freebsd.org>
Date: Wed, 3 Mar 2004 13:38:27 -0500 (EST)
On Wed, 3 Mar 2004, Cy Schubert wrote:

> I'm running 5 -CURRENT systems. My firewall system, using IPF, hard
> hangs every time ps is entered -- totally unresponsive, requiring either
> a power cycle or reset switch to bring it back to life. 
> 
> Before I start digging into this seriously I'd like to possibly get info
> from anyone who may have experienced this before. 

Alan Cox and I have both experienced this -- it's actually only a hard
hang if you're trying to use the syscons break to debugger, serial break
to debugger can get into DDB fine.  It looks like the sysctl code is
spinning in kernel, possibly due to looping waiting for a response other
than EAGAIN.  I'm wonder if it was the recent limits on locked memory
changes in sysctl, although at first we thought it might be the sleepq
changes (seems less likely now).  Because sysctl holds Giant, the other
CPUs are locked out of Giant-protected bits of the kernel (many of them),
including Syscons.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research


> 
> Dmesg:
> 
> Copyright (c) 1992-2004 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>         The Regents of the University of California. All rights reserved.
> FreeBSD 5.2-CURRENT #2: Wed Mar  3 06:30:44 PST 2004
>     root_at_cwsys:/export/obj/opt/src/cvs-current/src/sys/KOMQUATS
> Preloaded elf kernel "/boot/kernel/kernel" at 0xc075b000.
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Pentium/P54C (150.07-MHz 586-class CPU)
>   Origin = "GenuineIntel"  Id = 0x52c  Stepping = 12
>   Features=0x1bf<FPU,VME,DE,PSE,TSC,MSR,MCE,CX8>
> real memory  = 50331648 (48 MB)
> avail memory = 43896832 (41 MB)
> Intel Pentium detected, installing workaround for F00F bug
> npx0: [FAST]
> npx0: <math processor> on motherboard
> npx0: INT 16 interface
> pcibios: BIOS version 2.10
> Found $PIR table, 5 entries at 0xc00f0a80
> pcib0: <Host to PCI bridge> at pcibus 0 on motherboard
> pci0: <PCI bus> on pcib0
> $PIR: 0:9 INTA routed to irq 5
> $PIR: 0:10 INTA routed to irq 9
> $PIR: 0:11 INTA routed to irq 12
> $PIR: 0:12 INTA routed to irq 10
> isab0: <PCI-ISA bridge> at device 1.0 on pci0
> isa0: <ISA bus> on isab0
> atapci0: <Intel PIIX4 UDMA33 controller> port 0xe000-0xe00f at device 1.1 
> on pci0
> ata0: at 0x1f0 irq 14 on atapci0
> ata0: [MPSAFE]
> ata1: at 0x170 irq 15 on atapci0
> ata1: [MPSAFE]
> uhci0: <Intel 82371AB/EB (PIIX4) USB controller> port 0xd800-0xd81f at 
> device 1.2 on pci0
> $PIR: 0:1 INTD routed to irq 5
> usb0: <Intel 82371AB/EB (PIIX4) USB controller> on uhci0
> usb0: USB revision 1.0
> uhub0: Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1
> uhub0: 2 ports with 2 removable, self powered
> pci0: <bridge, PCI-unknown> at device 1.3 (no driver attached)
> ed0: <NE2000 PCI Ethernet (RealTek 8029)> port 0xd400-0xd41f irq 5 at 
> device 9.0 on pci0
> ed0: address 00:60:67:66:4d:86, type NE2000 (16 bit) 
> dc0: <ADMtek AN985 10/100BaseTX> port 0xd000-0xd0ff mem 
> 0xe7000000-0xe70003ff irq 9 at device 10.0 on pci0
> dc0: Ethernet address: 00:03:6d:15:e4:d1
> miibus0: <MII bus> on dc0
> ukphy0: <Generic IEEE 802.3u media interface> on miibus0
> ukphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
> ed1: <NE2000 PCI Ethernet (ProLAN)> port 0xb800-0xb81f irq 12 at device 
> 11.0 on pci0
> ed1: address 00:20:78:12:5c:0c, type NE2000 (16 bit) 
> pci0: <display, VGA> at device 12.0 (no driver attached)
> orm0: <Option ROMs> at iomem 0xdc000-0xdffff,0xc0000-0xc7fff on isa0
> pmtimer0 on isa0
> aha0 at port 0x330-0x333 irq 11 drq 5 on isa0
> aha0: AHA-1542CP FW Rev. D.0 (ID=46) SCSI Host Adapter, SCSI ID 7, 16 CCBs
> atkbdc0: <Keyboard controller (i8042)> at port 0x64,0x60 on isa0
> atkbd0: <AT Keyboard> flags 0x1 irq 1 on atkbdc0
> kbd0 at atkbd0
> fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> at port 
> 0x3f7,0x3f0-0x3f5 irq 6 drq 2 on isa0
> fdc0: FIFO enabled, 8 bytes threshold
> fd0: <1440-KB 3.5" drive> on fdc0 drive 0
> ppc0: <Parallel port> at port 0x378-0x37f irq 7 on isa0
> ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
> ppbus0: <Parallel port bus> on ppc0
> lpt0: <Printer> on ppbus0
> lpt0: Interrupt-driven port
> ppi0: <Parallel I/O> on ppbus0
> sc0: <System console> at flags 0x100 on isa0
> sc0: VGA <16 virtual consoles, flags=0x300>
> sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
> sio0: type 16550A
> sio1 at port 0x2f8-0x2ff irq 3 on isa0
> sio1: type 16550A
> vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
> unknown: <PNP0400> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> unknown: <PNP0501> can't assign resources (port)
> unknown: <PNP0700> can't assign resources (port)
> unknown: <PNP0303> can't assign resources (port)
> Timecounter "TSC" frequency 150066555 Hz quality 800
> Timecounters tick every 10.000 msec
> ad0: 2441MB <WDC AC22500L> [4960/16/63] at ata0-master WDMA2
> Waiting 15 seconds for SCSI devices to settle
> da0 at aha0 bus 0 target 0 lun 0
> da0: <SEAGATE SX910800N 8514> Fixed Direct Access SCSI-2 device 
> da0: 10.000MB/s transfers (10.000MHz, offset 8)
> da0: 8669MB (17755614 512 byte sectors: 64H 32S/T 8669C)
> da1 at aha0 bus 0 target 1 lun 0
> da1: <QUANTUM MAVERICK 540S 0901> Fixed Direct Access SCSI-2 device 
> da1: 10.000MB/s transfers (10.000MHz, offset 8)
> da1: 516MB (1057758 512 byte sectors: 64H 32S/T 516C)
> Mounting root from ufs:/dev/ad0s1a
> WARNING: / was not properly dismounted
> IP Filter: v3.4.31 initialized.  Default = pass all, Logging = enabled
> ed0: promiscuous mode enabled
> ed0: promiscuous mode disabled
> 
> cwfw$ kldstat
> Id Refs Address    Size     Name
>  1    5 0xc0400000 359a24   kernel
>  2    1 0xc163e000 3000     fdescfs.ko
>  3    1 0xc1642000 a000     unionfs.ko
>  4    1 0xc165d000 16000    ipl.ko
>  5    1 0xc1763000 4000     if_tun.ko
> cwfw$ 
> 
> #
> # GENERIC -- Generic kernel configuration file for FreeBSD/i386
> #
> # For more information on this file, please read the handbook section on
> # Kernel Configuration Files:
> #
> #    http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-
> config.html
> #
> # The handbook is also available locally in /usr/share/doc/handbook
> # if you've installed the doc distribution, otherwise always see the
> # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the
> # latest information.
> #
> # An exhaustive list of options and more detailed explanations of the
> # device lines is also present in the ../../conf/NOTES and NOTES files. 
> # If you are in doubt as to the purpose or necessity of a line, check first 
> # in NOTES.
> #
> # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.384.2.2 2003/05/31 15:18:41 
> scottl Exp $
> 
> machine		i386
> # cpu		I486_CPU
> cpu		I586_CPU
> cpu		I686_CPU
> ident		KOMQUATS
> 
> #To statically compile in device wiring instead of /boot/device.hints
> #hints		"GENERIC.hints"		#Default places to look for devices.
> 
> #makeoptions	DEBUG=-g		#Build kernel with gdb(1) debug symbols
> 
> options 	SCHED_4BSD		#4BSD scheduler
> # options		SCHED_ULE		#ULE scheduler
> options 	INET			#InterNETworking
> # options 	INET6			#IPv6 communications protocols
> options 	FFS			#Berkeley Fast Filesystem
> options 	SOFTUPDATES		#Enable FFS soft updates support
> options 	UFS_ACL			#Support for access control lists
> options 	UFS_DIRHASH		#Improve performance on big directories
> options 	MD_ROOT			#MD is a potential root device
> options 	NFSCLIENT		#Network Filesystem Client
> options 	NFSSERVER		#Network Filesystem Server
> options 	NFS_ROOT		#NFS usable as root device, requires NFSCLIENT
> options 	MSDOSFS			#MSDOS Filesystem
> options 	CD9660			#ISO 9660 Filesystem
> options 	PROCFS			#Process filesystem (requires PSEUDOFS)
> options 	PSEUDOFS		#Pseudo-filesystem framework
> options 	COMPAT_43		#Compatible with BSD 4.3 [KEEP THIS!]
> options 	COMPAT_FREEBSD4		#Compatible with FreeBSD4
> options 	SCSI_DELAY=15000	#Delay (in ms) before probing SCSI
> options 	KTRACE			#ktrace(1) support
> options 	SYSVSHM			#SYSV-style shared memory
> options 	SYSVMSG			#SYSV-style message queues
> options 	SYSVSEM			#SYSV-style semaphores
> # options	SHMALL=1025
> options		SHMALL=2049
> options 	_KPOSIX_PRIORITY_SCHEDULING #Posix P1003_1B real-time extensions
> options 	KBD_INSTALL_CDEV	# install a CDEV entry in /dev
> options 	AHC_REG_PRETTY_PRINT	# Print register bitfields in debug
> 					# output.  Adds ~128k to driver.
> options 	AHD_REG_PRETTY_PRINT	# Print register bitfields in debug
> 					# output.  Adds ~215k to driver.
> options		PFIL_HOOKS
> 
> # Debugging for use in -current
> ##options 	DDB			#Enable the kernel debugger
> ##options 	INVARIANTS		#Enable calls of extra sanity checking
> #options 	INVARIANTS		#Enable calls of extra sanity checking
> #options 	INVARIANT_SUPPORT	#Extra sanity checks of internal structures, 
> required by INVARIANTS
> ##options 	WITNESS			#Enable checks to detect deadlocks and cycles
> #options 	WITNESS			#Enable checks to detect deadlocks and cycles
> ##options 	WITNESS_SKIPSPIN	#Don't run witness on spinlocks for speed
> #options 	WITNESS_SKIPSPIN	#Don't run witness on spinlocks for speed
> 
> # To make an SMP kernel, the next two are needed
> #options 	SMP			# Symmetric MultiProcessor Kernel
> #options 	APIC_IO			# Symmetric (APIC) I/O
> 
> #
> # Add software watchdog routines.  This will add some sysctl OIDs that
> # can be used in combination with an external daemon to create a
> # software-based watchdog solution.
> #
> options		SW_WATCHDOG
> 
> device		isa
> device		eisa
> device		pci
> 
> # Floppy drives
> device		fdc
> 
> # ATA and ATAPI devices
> device		ata
> device		atadisk			# ATA disk drives
> device		atapicd			# ATAPI CDROM drives
> device		atapifd			# ATAPI floppy drives
> device		atapist			# ATAPI tape drives
> # device		atapicam		# emulate ATAPI devices as SCSI ditto via CAM
> 					# needs CAM to be present (scbus & pass)
> options 	ATA_STATIC_ID		#Static device numbering
> 
> # SCSI Controllers
> # device		ahb		# EISA AHA1742 family
> device		ahc		# AHA2940 and onboard AIC7xxx devices
> # device		ahd		# AHA39320/29320 and onboard AIC79xx devices
> # device		amd		# AMD 53C974 (Tekram DC-390(T))
> # device		isp		# Qlogic family
> # device		mpt		# LSI-Logic MPT-Fusion
> #device		ncr		# NCR/Symbios Logic
> # device		sym		# NCR/Symbios Logic (newer chipsets + those of `ncr')
> # device		trm		# Tekram DC395U/UW/F DC315U adapters
> 
> # device		adv		# Advansys SCSI adapters
> # device		adw		# Advansys wide SCSI adapters
> device		aha		# Adaptec 154x SCSI adapters
> # device		aic		# Adaptec 15[012]x SCSI adapters, AIC-6[23]60.
> # device		bt		# Buslogic/Mylex MultiMaster SCSI adapters
> 
> # device		ncv		# NCR 53C500
> # device		nsp		# Workbit Ninja SCSI-3
> # device		stg		# TMC 18C30/18C50
> 
> # RAID controllers interfaced to the SCSI subsystem
> # device		asr		# DPT SmartRAID V, VI and Adaptec SCSI RAID
> # device		ciss		# Compaq Smart RAID 5*
> # device		dpt		# DPT Smartcache III, IV - See NOTES for options!
> # device		iir		# Intel Integrated RAID
> # device		mly		# Mylex AcceleRAID/eXtremeRAID
> 
> # SCSI peripherals
> device		scbus		# SCSI bus (required)
> device		ch		# SCSI media changers
> device		da		# Direct Access (disks)
> device		sa		# Sequential Access (tape etc)
> device		cd		# CD
> device		pass		# Passthrough device (direct SCSI access)
> device		ses		# SCSI Environmental Services (and SAF-TE)
> 
> # RAID controllers
> # device		aac		# Adaptec FSA RAID
> # device		aacp		# SCSI passthrough for aac (requires CAM)
> # device		amr		# AMI MegaRAID
> # device		ida		# Compaq Smart RAID
> # device		ips		# IBM (Adaptec) ServeRAID
> # device		mlx		# Mylex DAC960 family
> # device		pst		# Promise Supertrak SX6000
> # device		twe		# 3ware ATA RAID
> 
> # atkbdc0 controls both the keyboard and the PS/2 mouse
> device		atkbdc		# AT keyboard controller
> device		atkbd		# AT keyboard
> device		psm		# PS/2 mouse
> 
> device		vga		# VGA video card driver
> 
> device		splash		# Splash screen and screen saver support
> 
> # syscons is the default console driver, resembling an SCO console
> device		sc
> 
> # Enable this for the pcvt (VT220 compatible) console driver
> #device		vt
> #options 	XSERVER			# support for X server on a vt console
> #options 	FAT_CURSOR		# start with block cursor
> 
> device		agp		# support several AGP chipsets
> 
> # Floating point support - do not disable.
> device		npx
> 
> # Power management support (see NOTES for more options)
> #device		apm
> # Add suspend/resume support for the i8254.
> device		pmtimer
> 
> # PCCARD (PCMCIA) support
> # Pcmcia and cardbus bridge support
> # device		cbb			# cardbus (yenta) bridge
> #device		pcic			# ExCA ISA and PCI bridges
> # device		pccard			# PC Card (16-bit) bus
> # device		cardbus			# CardBus (32-bit) bus
> 
> # Serial (COM) ports
> device		sio		# 8250, 16[45]50 based serial ports
> 
> # Parallel port
> device		ppc
> device		ppbus		# Parallel port bus (required)
> device		lpt		# Printer
> # device		plip		# TCP/IP over parallel
> device		ppi		# Parallel port interface device
> #device		vpo		# Requires scbus and da
> 
> 
> # PCI Ethernet NICs.
> # device		de		# DEC/Intel DC21x4x (``Tulip'')
> # device		em		# Intel PRO/1000 adapter Gigabit Ethernet Card
> # device		txp		# 3Com 3cR990 (``Typhoon'')
> # device		vx		# 3Com 3c590, 3c595 (``Vortex'')
> 
> # PCI Ethernet NICs that use the common MII bus controller code.
> # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs!
> device		miibus		# MII bus support
> device		dc		# DEC/Intel 21143 and various workalikes
> # device		fxp		# Intel EtherExpress PRO/100B (82557, 82558)
> # device		pcn		# AMD Am79C97x PCI 10/100 (precedence over 'lnc')
> # device		rl		# RealTek 8129/8139
> # device		sf		# Adaptec AIC-6915 (``Starfire'')
> # device		sis		# Silicon Integrated Systems SiS 900/SiS 7016
> # device		sk		# SysKonnect SK-984x and SK-982x gigabit ethernet
> # device		ste		# Sundance ST201 (D-Link DFE-550TX)
> # device		ti		# Alteon Networks Tigon I/II gigabit ethernet
> # device		tl		# Texas Instruments ThunderLAN
> # device		tx		# SMC EtherPower II (83c170 ``EPIC'')
> # device		vr		# VIA Rhine, Rhine II
> # device		wb		# Winbond W89C840F
> # device		xl		# 3Com 3c90x (``Boomerang'', ``Cyclone'')
> # device		bge		# Broadcom BCM570xx Gigabit Ethernet
> 
> # ISA Ethernet NICs.  pccard nics included.
> device		cs		# Crystal Semiconductor CS89x0 NIC
> # 'device ed' requires 'device miibus'
> device		ed		# NE[12]000, SMC Ultra, 3c503, DS8390 cards
> # device		ex		# Intel EtherExpress Pro/10 and Pro/10+
> # device		ep		# Etherlink III based cards
> # device		fe		# Fujitsu MB8696x based cards
> # device		ie		# EtherExpress 8/16, 3C507, StarLAN 10 etc.
> # device		lnc		# NE2100, NE32-VL Lance Ethernet cards
> # device		sn		# SMC's 9000 series of ethernet chips
> # device		xe		# Xircom pccard ethernet
> 
> # ISA devices that use the old ISA shims
> #device		le
> 
> # Wireless NIC cards
> # device		wlan		# 802.11 support
> # device		an		# Aironet 4500/4800 802.11 wireless NICs. 
> # device		awi		# BayStack 660 and others
> # device		wi		# WaveLAN/Intersil/Symbol 802.11 wireless NICs.
> # #device		wl		# Older non 802.11 Wavelan wireless NIC.
> 
> # Pseudo devices - the number indicates how many units to allocate.
> device		random		# Entropy device
> device		loop		# Network loopback
> device		ether		# Ethernet support
> # device		sl		# Kernel SLIP
> # device		ppp		# Kernel PPP
> # device		tun		# Packet tunnel.
> device		pty		# Pseudo-ttys (telnet etc)
> device		md		# Memory "disks"
> # device		gif		# IPv6 and IPv4 tunneling
> # device		faith		# IPv6-to-IPv4 relaying (translation)
> 
> # The `bpf' device enables the Berkeley Packet Filter.
> # Be aware of the administrative consequences of enabling this!
> device		bpf		# Berkeley packet filter
> 
> # USB support
> device		uhci		# UHCI PCI->USB interface
> device		ohci		# OHCI PCI->USB interface
> device		usb		# USB Bus (required)
> #device		udbp		# USB Double Bulk Pipe devices
> device		ugen		# Generic
> device		uhid		# "Human Interface Devices"
> device		ukbd		# Keyboard
> device		ulpt		# Printer
> device		umass		# Disks/Mass storage - Requires scbus and da
> device		ums		# Mouse
> device		urio		# Diamond Rio 500 MP3 player
> device		uscanner	# Scanners
> # USB Ethernet, requires mii
> device		aue		# ADMtek USB ethernet
> device		axe		# ASIX Electronics USB ethernet
> device		cue		# CATC USB ethernet
> device		kue		# Kawasaki LSI USB ethernet
> 
> # FireWire support
> # device		firewire	# FireWire bus code
> # device		sbp		# SCSI over FireWire (Requires scbus and da)
> # device		fwe		# Ethernet over FireWire (non-standard!)
> #
> # SMB bus
> #
> # System Management Bus support is provided by the 'smbus' device.
> # Access to the SMBus device is via the 'smb' device (/dev/smb*),
> # which is a child of the 'smbus' device.
> #
> # Supported devices:
> # smb		standard io through /dev/smb*
> #
> # Supported SMB interfaces:
> # iicsmb	I2C to SMB bridge with any iicbus interface
> # bktr		brooktree848 I2C hardware interface
> # intpm		Intel PIIX4 Power Management Unit
> # alpm		Acer Aladdin-IV/V/Pro2 Power Management Unit
> # ichsmb	Intel ICH SMBus controller chips (82801AA, 82801AB, 82801BA)
> #
> device		smbus		# Bus support, required for smb below.
> 
> # device		intpm
> # device		alpm
> device		ichsmb
> 
> device		smb
> 
> #
> # I2C Bus
> #
> # Philips i2c bus support is provided by the `iicbus' device.
> #
> # Supported devices:
> # ic	i2c network interface
> # iic	i2c standard io
> # iicsmb i2c to smb bridge. Allow i2c i/o with smb commands.
> #
> # Supported interfaces:
> # pcf	Philips PCF8584 ISA-bus controller
> # bktr	brooktree848 I2C software interface
> #
> # Other:
> # iicbb	generic I2C bit-banging code (needed by lpbb, bktr)
> #
> device		iicbus		# Bus support, required for ic/iic/iicsmb below.
> device		iicbb
> 
> device		ic
> device		iic
> device		iicsmb		# smb over i2c bridge
> 
> # device		pcf0	at isa? port 0x320 irq 5
> 
> Would enabling DDB in the event of a hard hang help determine the problem?
> 
> Cheers,
> --
> Cy Schubert <Cy.Schubert_at_komquats.com>        http://www.komquats.com/
> BC Government                     .                       FreeBSD UNIX
> Cy.Schubert_at_osg.gov.bc.ca         .                     cy_at_FreeBSD.org
> http://www.gov.bc.ca/             .            http://www.FreeBSD.org/
> 
> 
> 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> 
Received on Wed Mar 03 2004 - 09:39:40 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:45 UTC