Re: system 20% busy at all times?

On Feb 19, 2013, at 10:36 AM, "Eggert, Lars" <lars_at_netapp.com> wrote:

> Hi,
> 
> I have a system running -CURRENT that in top(1) is showing ~20% CPU usage for the system at all times. Any ideas what could be causing this, or how I would go about diagnosing this further? Nothing in the logs.
> 
> Thanks,
> Lars
> 
> PS: dmesg attached, in case it helps:
> 
> Copyright (c) 1992-2013 The FreeBSD Project.
> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
> 	The Regents of the University of California. All rights reserved.
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 10.0-CURRENT #11 r+2fc9b3d: Tue Feb 12 19:32:15 CET 2013
>    elars_at_stanley.muccbc.hq.netapp.com:/home/elars/obj/usr/home/elars/src/sys/FAS3270 amd64
> FreeBSD clang version 3.2 (tags/RELEASE_32/final 170710) 20121221
> CPU: Intel(R) Xeon(R) CPU           E5240  _at_ 3.00GHz (3000.17-MHz K8-class CPU)
>  Origin = "GenuineIntel"  Id = 0x1067a  Family = 0x6  Model = 0x17  Stepping = 10
>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>  Features2=0xc0ce3bd<SSE3,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,DCA,SSE4.1,XSAVE,OSXSAVE>
>  AMD Features=0x20100800<SYSCALL,NX,LM>
>  AMD Features2=0x1<LAHF>
>  TSC: P-state invariant, performance statistics
> real memory  = 18253611008 (17408 MB)
> avail memory = 16526143488 (15760 MB)
> Event timer "LAPIC" quality 400
> ACPI APIC Table: <PTLTD  CARNEGIE>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 2 package(s) x 2 core(s)
> cpu0 (BSP): APIC ID:  0
> cpu1 (AP): APIC ID:  1
> cpu2 (AP): APIC ID:  6
> cpu3 (AP): APIC ID:  7
> ioapic0 <Version 2.0> irqs 0-23 on motherboard
> kbd0 at kbdmux0
> ctl: CAM Target Layer loaded
> smbios0: <System Management BIOS> at iomem 0xf6c00-0xf6c1e on motherboard
> smbios0: Version: 2.5
> cryptosoft0: <software crypto> on motherboard
> acpi0: <PTLTD CARNEGIE> on motherboard
> acpi0: Power Button (fixed)
> cpu0: <ACPI CPU> on acpi0
> ACPI Error: [\134_SB_.PCI0.LPC0.BCMD] Namespace lookup failure, AE_NOT_FOUND (20130117/psargs-393)
> ACPI Error: Method parse/execution failed [\134_PR_.CPU0._OSC] (Node 0xfffffe0007630c00), AE_NOT_FOUND (20130117/psparse-560)
> ACPI Error: [\134_SB_.PCI0.LPC0.BCMD] Namespace lookup failure, AE_NOT_FOUND (20130117/psargs-393)
> ACPI Error: Method parse/execution failed [\134_PR_.CPU0._OSC] (Node 0xfffffe0007630c00), AE_NOT_FOUND (20130117/psparse-560)
> ACPI Error: Method parse/execution failed [\134_PR_.CPU0._PDC] (Node 0xfffffe0007630c40), AE_NOT_FOUND (20130117/psparse-560)
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
> Event timer "RTC" frequency 32768 Hz quality 0
> attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
> Timecounter "i8254" frequency 1193182 Hz quality 0
> Event timer "i8254" frequency 1193182 Hz quality 100
> Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
> acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
> pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
> pci0: <ACPI PCI bus> on pcib0
> pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
> pci1: <ACPI PCI bus> on pcib1
> pci1: <network, ethernet> at device 0.0 (no driver attached)
> pcib2: <PCI-PCI bridge> at device 3.0 on pci0
> pci2: <PCI bus> on pcib2
> pcib3: <ACPI PCI-PCI bridge> at device 4.0 on pci0
> pci3: <ACPI PCI bus> on pcib3
> pcib4: <ACPI PCI-PCI bridge> mem 0xdeb00000-0xdeb1ffff irq 16 at device 0.0 on pci3
> pci4: <ACPI PCI bus> on pcib4
> pcib4: no PRT entry for 4.4.INTA
> pcib4: no PRT entry for 4.5.INTA
> pcib4: no PRT entry for 4.8.INTA
> pcib5: <PCI-PCI bridge> irq 5 at device 4.0 on pci4
> pci5: <PCI bus> on pcib5
> pcib6: <PCI-PCI bridge> irq 10 at device 5.0 on pci4
> pci6: <PCI bus> on pcib6
> pcib4: no PRT entry for 4.5.INTA
> pcib4: no PRT entry for 4.5.INTB
> ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.0> mem 0xdec00000-0xdec7ffff,0xded00000-0xded03fff irq 10 at device 0.0 on pci6
> ix0: Using MSIX interrupts with 5 vectors
> ix0: Ethernet address: 90:e2:ba:2b:3b:6c
> ix0: PCI Express Bus: Speed 5.0Gb/s Width x8
> ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 2.5.0> mem 0xdec80000-0xdecfffff,0xded04000-0xded07fff irq 11 at device 0.1 on pci6
> ix1: Using MSIX interrupts with 5 vectors
> ix1: Ethernet address: 90:e2:ba:2b:3b:6d
> ix1: PCI Express Bus: Speed 5.0Gb/s Width x8
> pcib7: <ACPI PCI-PCI bridge> irq 5 at device 8.0 on pci4
> pci7: <ACPI PCI bus> on pcib7
> pcib8: <PCI-PCI bridge> at device 5.0 on pci0
> pci8: <PCI bus> on pcib8
> pcib9: <PCI-PCI bridge> at device 6.0 on pci0
> pci9: <PCI bus> on pcib9
> pcib10: <PCI-PCI bridge> mem 0xdee00000-0xdee1ffff irq 16 at device 0.0 on pci9
> pci10: <PCI bus> on pcib10
> pcib11: <PCI-PCI bridge> irq 16 at device 0.0 on pci10
> pci11: <PCI bus> on pcib11
> pcib12: <PCI-PCI bridge> mem 0xdef00000-0xdef1ffff irq 16 at device 0.0 on pci11
> pci12: <PCI bus> on pcib12
> pcib13: <PCI-PCI bridge> irq 17 at device 1.0 on pci12
> pci13: <PCI bus> on pcib13
> pcib14: <PCI-PCI bridge> irq 16 at device 4.0 on pci12
> pci14: <PCI bus> on pcib14
> pcib15: <PCI-PCI bridge> irq 17 at device 5.0 on pci12
> pci15: <PCI bus> on pcib15
> pcib16: <PCI-PCI bridge> irq 16 at device 8.0 on pci12
> pci16: <PCI bus> on pcib16
> pcib17: <PCI-PCI bridge> at device 0.0 on pci16
> pci17: <PCI bus> on pcib17
> pcib18: <PCI-PCI bridge> at device 0.0 on pci17
> pci18: <PCI bus> on pcib18
> em0: <Intel(R) PRO/1000 Network Connection 7.3.2> mem 0xdf020000-0xdf03ffff,0xdf000000-0xdf01ffff irq 16 at device 0.0 on pci18
> em0: Using an MSI interrupt
> em0: Ethernet address: 00:1b:21:a8:a5:58
> em1: <Intel(R) PRO/1000 Network Connection 7.3.2> mem 0xdf060000-0xdf07ffff,0xdf040000-0xdf05ffff irq 17 at device 0.1 on pci18
> em1: Using an MSI interrupt
> em1: Ethernet address: 00:1b:21:a8:a5:59
> pcib19: <PCI-PCI bridge> at device 1.0 on pci17
> pci19: <PCI bus> on pcib19
> em2: <Intel(R) PRO/1000 Network Connection 7.3.2> mem 0xdf120000-0xdf13ffff,0xdf100000-0xdf11ffff irq 17 at device 0.0 on pci19
> em2: Using an MSI interrupt
> em2: Ethernet address: 00:1b:21:a8:a5:5a
> em3: <Intel(R) PRO/1000 Network Connection 7.3.2> mem 0xdf160000-0xdf17ffff,0xdf140000-0xdf15ffff irq 18 at device 0.1 on pci19
> em3: Using an MSI interrupt
> em3: Ethernet address: 00:1b:21:a8:a5:5b
> pcib20: <PCI-PCI bridge> irq 17 at device 9.0 on pci12
> pci20: <PCI bus> on pcib20
> pcib21: <PCI-PCI bridge> mem 0xdf200000-0xdf21ffff irq 17 at device 0.0 on pci20
> pci21: <PCI bus> on pcib21
> pcib22: <PCI-PCI bridge> irq 17 at device 4.0 on pci21
> pci22: <PCI bus> on pcib22
> pcib23: <PCI-PCI bridge> irq 18 at device 5.0 on pci21
> pci23: <PCI bus> on pcib23
> pci23: <mass storage, SAS> at device 0.0 (no driver attached)
> pcib24: <PCI-PCI bridge> irq 17 at device 8.0 on pci21
> pci24: <PCI bus> on pcib24
> pci24: <mass storage, SAS> at device 0.0 (no driver attached)
> pcib25: <PCI-PCI bridge> irq 16 at device 4.0 on pci10
> pci25: <PCI bus> on pcib25
> pcib26: <PCI-PCI bridge> irq 17 at device 5.0 on pci10
> pci26: <PCI bus> on pcib26
> pci26: <serial bus, Fibre Channel> at device 0.0 (no driver attached)
> pci26: <serial bus, Fibre Channel> at device 0.1 (no driver attached)
> pcib27: <PCI-PCI bridge> irq 18 at device 6.0 on pci10
> pci27: <PCI bus> on pcib27
> pci27: <mass storage, SAS> at device 0.0 (no driver attached)
> pcib28: <PCI-PCI bridge> at device 7.0 on pci0
> pci28: <PCI bus> on pcib28
> pci0: <base peripheral> at device 8.0 (no driver attached)
> uhci0: <Intel 82801I (ICH9) USB controller> port 0x1820-0x183f irq 16 at device 26.0 on pci0
> usbus0 on uhci0
> ehci0: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xdfa01000-0xdfa013ff irq 18 at device 26.7 on pci0
> usbus1: EHCI version 1.0
> usbus1 on ehci0
> pcib29: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
> pci29: <ACPI PCI bus> on pcib29
> em4: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x3000-0x301f mem 0xdf720000-0xdf73ffff,0xdf700000-0xdf71ffff irq 16 at device 0.0 on pci29
> em4: Using an MSI interrupt
> em4: Ethernet address: 00:a0:98:30:91:24
> em5: <Intel(R) PRO/1000 Network Connection 7.3.2> port 0x3020-0x303f mem 0xdf760000-0xdf77ffff,0xdf740000-0xdf75ffff irq 17 at device 0.1 on pci29
> em5: Using an MSI interrupt
> em5: Ethernet address: 00:a0:98:30:91:25
> pcib30: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
> pci30: <ACPI PCI bus> on pcib30
> bge0: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004201> mem 0xdf800000-0xdf80ffff irq 16 at device 0.0 on pci30
> bge0: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
> miibus0: <MII bus> on bge0
> brgphy0: <BCM5750 1000BASE-T media interface> PHY 1 on miibus0
> brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> bge0: Ethernet address: 00:a0:98:30:91:28
> pcib31: <ACPI PCI-PCI bridge> irq 17 at device 28.5 on pci0
> pci31: <ACPI PCI bus> on pcib31
> bge1: <Broadcom NetXtreme Gigabit Ethernet Controller, ASIC rev. 0x004201> mem 0xdf900000-0xdf90ffff irq 17 at device 0.0 on pci31
> bge1: CHIP ID 0x00004201; ASIC REV 0x04; CHIP REV 0x42; PCI-E
> miibus1: <MII bus> on bge1
> brgphy1: <BCM5750 1000BASE-T media interface> PHY 1 on miibus1
> brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
> bge1: Ethernet address: 00:a0:98:30:91:29
> uhci1: <Intel 82801I (ICH9) USB controller> port 0x1840-0x185f irq 16 at device 29.0 on pci0
> usbus2 on uhci1
> ehci1: <Intel 82801I (ICH9) USB 2.0 controller> mem 0xdfa01400-0xdfa017ff irq 16 at device 29.7 on pci0
> usbus3: EHCI version 1.0
> usbus3 on ehci1
> pcib32: <ACPI PCI-PCI bridge> at device 30.0 on pci0
> pci32: <ACPI PCI bus> on pcib32
> isab0: <PCI-ISA bridge> at device 31.0 on pci0
> isa0: <ISA bus> on isab0
> ichsmb0: <Intel 82801I (ICH9) SMBus controller> port 0x1860-0x187f mem 0xdfa01800-0xdfa018ff irq 17 at device 31.3 on pci0
> smbus0: <System Management Bus> on ichsmb0
> smb0: <SMBus generic I/O> on smbus0
> pci0: <dasp> at device 31.6 (no driver attached)
> acpi_button0: <Power Button> on acpi0
> uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
> uart0: console (9600,n,8,1)
> uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
> ichwd0 on isa0
> ichwd0: ICH WDT present but disabled in BIOS or hardware
> device_attach: ichwd0 attach returned 6
> ipmi0: <IPMI System Interface> on isa0
> ipmi0: KCS mode found at io 0xca2 alignment 0x1 on isa
> ichwd0 at port 0x1030-0x1037,0x1060-0x107f on isa0
> ichwd0: ICH WDT present but disabled in BIOS or hardware
> device_attach: ichwd0 attach returned 6
> orm0: <ISA Option ROMs> at iomem 0xc0800-0xc17ff,0xc1800-0xc27ff,0xdc000-0xdffff on isa0
> coretemp0: <CPU On-Die Thermal Sensors> on cpu0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> coretemp1: <CPU On-Die Thermal Sensors> on cpu1
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> coretemp2: <CPU On-Die Thermal Sensors> on cpu2
> est2: <Enhanced SpeedStep Frequency Control> on cpu2
> p4tcc2: <CPU Frequency Thermal Control> on cpu2
> coretemp3: <CPU On-Die Thermal Sensors> on cpu3
> est3: <Enhanced SpeedStep Frequency Control> on cpu3
> p4tcc3: <CPU Frequency Thermal Control> on cpu3
> Timecounters tick every 1.000 msec
> usbus0: 12Mbps Full Speed USB v1.0
> usbus1: 480Mbps High Speed USB v2.0
> ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
> ugen0.1: <Intel> at usbus0
> uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
> ugen1.1: <Intel> at usbus1
> uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
> usbus2: 12Mbps Full Speed USB v1.0
> usbus3: 480Mbps High Speed USB v2.0
> ugen2.1: <Intel> at usbus2
> uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
> ugen3.1: <Intel> at usbus3
> uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
> ipmi0: IPMI device rev. 1, firmware rev. 1.00, version 2.0
> ipmi0: Number of channels 0
> ipmi0: Attached watchdog
> bootpc_init: wired to interface 'em4'
> Sending DHCP Discover packet from interface em4 (00:a0:98:30:91:24)
> uhub0: 2 ports with 2 removable, self powered
> uhub2: 2 ports with 2 removable, self powered
> uhub1: 2 ports with 2 removable, self powered
> uhub3: 2 ports with 2 removable, self powered
> ugen3.2: <Micron Technology> at usbus3
> umass0: <Micron Technology Real SSD eUSB 2GB, class 0/0, rev 2.00/11.10, addr 2> on usbus3
> umass0:  SCSI over Bulk-Only; quirks = 0x4000
> umass0:1:0:-1: Attached to scbus1
> da0 at umass-sim0 bus 0 scbus1 target 0 lun 0
> da0: <MICRON eUSB DISK 1110> Fixed Direct Access SCSI-0 device 
> da0: 40.000MB/s transfers
> da0: 1936MB (3964928 512 byte sectors: 255H 63S/T 246C)
> Received DHCP Offer packet on em4 from 0.0.0.0 (accepted) (no root path)
> Sending DHCP Request packet from interface em4 (00:a0:98:30:91:24)
> Received DHCP Ack packet on em4 from 0.0.0.0 (accepted) (got root path)
> em4 at 10.11.12.5 server 0.0.0.0
> subnet mask 255.255.255.0 router 10.11.12.13 rootfs 10.11.12.13:/usr/home/elars/dst 
> Adjusted interface em4
> SMP: AP CPU #3 Launched!
> SMP: AP CPU #2 Launched!
> SMP: AP CPU #1 Launched!
> Timecounter "TSC-low" frequency 1500084729 Hz quality 1000
> hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/2/40/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/40/0x67<INT,USR,SYS,REA,WRI>
> Trying to mount root from nfs: []...
> NFS ROOT: 10.11.12.13:/usr/home/elars/dst


What about reviewing top(1) ?

or possibly ps(1) aufx



At least you should be able to see what takes up CPU:
- system
- user processes
- interrupts



Kindly attach at least your ps output :)