Tuning KVA_PAGES / kmem?

From: Stefan Bethke <stb_at_lassitu.de>
Date: Tue, 20 Apr 2004 13:14:52 +0200
Hi,

I'm struggling a bit with a box I'm setting up right now: 2x Xeon 2.8  
GHz with 6 GB RAM. Main disk is a RAID 5 on 3ware controller. I'm  
running -current from around Easter.

Occasionally, the machine panics with:
panic: kmem_malloc(4096): kmem_map too small: 275251200 total allocated
at line 337 in file /usr/src/sys/vm/vm_kern.c

I had stability problems when I was using PAE, so I disabled that for  
the moment; even when running GENERIC, I get this panic.

I dug through the archives, and I found some hints that with >>2 GB  
RAM, the auto-tuning might produce undesirable results, so I might need  
to tune KVA_PAGES to reserve more address space for the kernel, or  
fiddle with VM_KMEM_SIZE_MAX, VM_KMEM_SIZE_SCALE, or some vm sysctls.

Can someone point me in the right direction?


Thanks,
Stefan

=========== sysctl vm =============
vm.vmtotal:
System wide totals computed every five seconds: (values in kilobytes)
===============================================
Processes:              (RUNQ: 3 Disk Wait: 1 Page Wait: 0 Sleep: 34)
Virtual Memory:         (Total: 2098618K, Active 128332K)
Real Memory:            (Total: 2174136K Active 77996K)
Shared Virtual Memory:  (Total: 11576K Active: 6316K)
Shared Real Memory:     (Total: 10532K Active: 5948K)
Free Memory Pages:      1811516K

vm.loadavg: { 4.05 4.00 4.08 }
vm.v_free_min: 6361
vm.v_free_target: 26823
vm.v_free_reserved: 1379
vm.v_inactive_target: 40234
vm.v_cache_min: 26823
vm.v_cache_max: 53646
vm.v_pageout_free_min: 34
vm.pageout_algorithm: 0
vm.swap_enabled: 1
vm.kmem_size: 209715200
vm.swap_async_max: 4
vm.dmmax: 32
vm.nswapdev: 1
vm.swap_idle_threshold1: 2
vm.swap_idle_threshold2: 10
vm.v_free_severe: 3870
vm.stats.sys.v_swtch: 41683765
vm.stats.sys.v_trap: 98922116
vm.stats.sys.v_syscall: 628250901
vm.stats.sys.v_intr: 22170437
vm.stats.sys.v_soft: 16178982
vm.stats.vm.v_vm_faults: 87981696
vm.stats.vm.v_cow_faults: 1666527
vm.stats.vm.v_cow_optim: 4490
vm.stats.vm.v_zfod: 85148244
vm.stats.vm.v_ozfod: 84790971
vm.stats.vm.v_swapin: 0
vm.stats.vm.v_swapout: 0
vm.stats.vm.v_swappgsin: 0
vm.stats.vm.v_swappgsout: 0
vm.stats.vm.v_vnodein: 121341
vm.stats.vm.v_vnodeout: 0
vm.stats.vm.v_vnodepgsin: 269302
vm.stats.vm.v_vnodepgsout: 0
vm.stats.vm.v_intrans: 33
vm.stats.vm.v_reactivated: 115830
vm.stats.vm.v_pdwakeups: 0
vm.stats.vm.v_pdpages: 0
vm.stats.vm.v_dfree: 0
vm.stats.vm.v_pfree: 3307919
vm.stats.vm.v_tfree: 89201119
vm.stats.vm.v_page_size: 4096
vm.stats.vm.v_page_count: 996725
vm.stats.vm.v_free_reserved: 1379
vm.stats.vm.v_free_target: 26823
vm.stats.vm.v_free_min: 6361
vm.stats.vm.v_free_count: 452621
vm.stats.vm.v_wire_count: 61828
vm.stats.vm.v_active_count: 18866
vm.stats.vm.v_inactive_target: 40234
vm.stats.vm.v_inactive_count: 463270
vm.stats.vm.v_cache_count: 0
vm.stats.vm.v_cache_min: 26823
vm.stats.vm.v_cache_max: 53646
vm.stats.vm.v_pageout_free_min: 34
vm.stats.vm.v_interrupt_free_min: 2
vm.stats.vm.v_forks: 72171
vm.stats.vm.v_vforks: 6433
vm.stats.vm.v_rforks: 2
vm.stats.vm.v_kthreads: 154
vm.stats.vm.v_forkpages: 5914990
vm.stats.vm.v_vforkpages: 800591
vm.stats.vm.v_rforkpages: 118
vm.stats.vm.v_kthreadpages: 0
vm.stats.misc.zero_page_count: 349073
vm.stats.misc.cnt_prezero: 85182803
vm.max_proc_mmap: 34952
vm.msync_flush_flags: 3
vm.idlezero_enable: 1
vm.idlezero_maxrun: 16
vm.max_launder: 32
vm.pageout_stats_max: 26823
vm.pageout_full_stats_interval: 20
vm.pageout_stats_interval: 5
vm.pageout_stats_free_max: 5
vm.swap_idle_enabled: 0
vm.defer_swapspace_pageouts: 0
vm.disable_swapspace_pageouts: 0
vm.pageout_lock_miss: 0
vm.zone:
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS

FFS2 dinode:     256,        0, 179487,  13863,  1091123
FFS1 dinode:     128,        0,      0,      0,        0
FFS inode:       140,        0, 179487,  13909,  1091123
SWAPMETA:        276,   121576,      0,      0,        0
unpcb:           140,    25620,     14,     98,    22136
ripcb:           180,    25608,      0,     44,        1
tcpreass:         20,     1737,      0,    386,       23
hostcache:        88,    15390,      3,    177,       12
syncache:        104,    15390,      0,     76,        5
tcptw:            56,     5183,      1,    354,      423
tcpcb:           372,    25600,     11,     59,      547
inpcb:           180,    25608,     12,    120,      547
udpcb:           180,    25608,      2,    108,     1420
socket:          240,    25600,     30,    114,    24106
KNOTE:            64,        0,      0,    310,     1380
PIPE:            376,        0,      3,     97,    40955
DIRHASH:        1024,        0,   1162,    374,     1662
NFSNODE:         452,        0,      0,      0,        0
NFSMOUNT:        424,        0,      0,      0,        0
L VFS Cache:     291,        0,   2223,   2821,    22980
S VFS Cache:      68,        0, 188772,  13303,  1246054
NAMEI:          1024,        0,      3,     45,  8161192
VNODEPOLL:        60,        0,      0,      0,        0
VNODE:           260,        0, 192750,    180,   192750
ata_request:     180,        0,      0,     66,       27
g_bio:           136,        0,      0,   5829,  2658969
VMSPACE:         236,        0,     41,    163,    78606
UPCALL:           44,        0,      0,      0,        0
KSE:              84,        0,    360,     91,      360
KSEGRP:          128,        0,    360,    105,      360
TID:             140,        0,      0,      0,        0
THREAD:          336,        0,    360,     36,      360
PROC:            440,        0,    194,    166,    78759
Files:            68,        0,    111,    420,  1715901
4096:           4096,        0,    235,    160,    78847
2048:           2048,        0,    128,   2888,     5858
1024:           1024,        0,     35,    109,     7646
512:             512,        0,    345,    159,    39721
256:             256,        0,    359,  16621,   345303
128:             128,        0,   2104,  16279,   749702
64:               64,        0,   3388,   8082,   365044
32:               32,        0,   1466,  16861,   665787
16:               16,        0,   1000,    673,    84504
DP fakepg:        72,        0,      0,      0,        0
PV ENTRY:         28,  2248680,  31168,  15452, 119085363
MAP ENTRY:        60,        0,   1312,   1196,  3376100
KMAP ENTRY:       60,    57354,    391,   1787,   281213
MAP:             160,        0,      9,     41,        2
VM OBJECT:       132,        0, 178885,  15485,  1678591
128 Bucket:      524,        0,   1274,      0,        0
64 Bucket:       268,        0,    100,      5,        0
32 Bucket:       140,        0,    108,      4,        0
16 Bucket:        76,        0,     54,     50,        0
UMA Hash:        128,        0,      2,     29,        0
UMA Slabs:        34,        0,   2429,    101,        0
UMA Zones:       432,        0,     55,      8,        0

vm.kvm_size: 1069543424
vm.kvm_free: 398454784
=========== sysctl vm =============

=========== netstat -m ============
322/832/51200 mbufs in use (current/peak/max):
           322 mbufs allocated to data
320/608/25600 mbuf clusters in use (current/peak/max)
0/1/6656 sfbufs in use (current/peak/max)
64000 KBytes allocated to network (1.1% in use, 2.2% wired)
0 requests for memory denied
0 requests for memory delayed
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
=========== netstat -m ============


========= kernel config ===========
###
### Kernel configuarion file for majestix.tallence.de
###

machine         i386
cpu                     I686_CPU
ident           MAJESTIX

###
### General options: protocols, file systems
###

options         SCHED_ULE               # ULE scheduler
options         INET                    # InterNETworking
options         FFS                     # Berkeley Fast Filesystem
options         SOFTUPDATES             # Enable FFS soft updates  
support
options         UFS_ACL                 # Support for access control  
lists
options         UFS_DIRHASH             # Improve performance on big  
directories
options         MD_ROOT                 # MD is a potential root device
options         NFSCLIENT               # Network Filesystem Client
options         NFSSERVER               # Network Filesystem Server
options         NFS_ROOT                # NFS usable as /, requires  
NFSCLIENT
options         MSDOSFS                 # MSDOS Filesystem
options         CD9660                  # ISO 9660 Filesystem
options         PROCFS                  # Process filesystem (requires  
PSEUDOFS)
options         PSEUDOFS                # Pseudo-filesystem framework
options         COMPAT_43               # Compatible with BSD 4.3 [KEEP  
THIS!]
options         COMPAT_FREEBSD4         # Compatible with FreeBSD4
options         SCSI_DELAY=15000        # Delay (in ms) before probing  
SCSI
options         KTRACE                  # ktrace(1) support
options         SYSVSHM                 # SYSV-style shared memory
options         SYSVMSG                 # SYSV-style message queues
options         SYSVSEM                 # SYSV-style semaphores
options         _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time  
extensions
options         KBD_INSTALL_CDEV        # install a CDEV entry in /dev
options         AHC_REG_PRETTY_PRINT    # Print register bitfields in  
debug
                                         # output.  Adds ~128k to driver.
options         AHD_REG_PRETTY_PRINT    # Print register bitfields in  
debug
                                         # output.  Adds ~215k to driver.
options         PFIL_HOOKS              # pfil(9) framework


###
### Machine-specific:
###

options         SMP             # Symmetric MultiProcessor Kernel
device          apic            # I/O APIC

# PAE is incompatible with modules
#options                PAE                     # Physical Address  
Extensions Kernel
device          acpi
makeoptions     NO_MODULES=yes


###
### Busses and drivers
###

device          isa
device          eisa
device          pci

# SCSI peripherals
device          scbus           # SCSI bus (required for SCSI)
device          ch              # SCSI media changers
device          da              # Direct Access (disks)
device          sa              # Sequential Access (tape etc)
device          cd              # CD
device          pass            # Passthrough device (direct SCSI  
access)
device          ses             # SCSI Environmental Services (and  
SAF-TE)

device          miibus          # MII bus support


device          fdc

device          ata
device          atadisk         # ATA disk drives
device          ataraid         # ATA RAID drives
device          atapicd         # ATAPI CDROM drives
device          atapifd         # ATAPI floppy drives
device          atapist         # ATAPI tape drives
options         ATA_STATIC_ID   # Static device numbering

device          ahc             # AHA2940 and onboard AIC7xxx devices
device          ahd             # AHA39320/29320 and onboard AIC79xx  
devices

device          twe             # 3ware ATA RAID

device          atkbdc          # AT keyboard controller
device          atkbd           # AT keyboard
device          psm             # PS/2 mouse
device          vga             # VGA video card driver
device          sc

device          npx

device          sio             # 8250, 16[45]50 based serial ports

device          em              # Intel PRO/1000 adapter Gigabit  
Ethernet Card
device          fxp             # Intel EtherExpress PRO/100B (82557,  
82558)
device          random          # Entropy device
device          loop            # Network loopback
device          ether           # Ethernet support
device          tun             # Packet tunnel.
device          pty             # Pseudo-ttys (telnet etc)
device          md              # Memory "disks"
device          bpf             # Berkeley packet filter

pseudo-device   pf      1
========= kernel config ===========

============= dmesg ===============
Copyright (c) 1992-2004 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
         The Regents of the University of California. All rights  
reserved.
FreeBSD 5.2-CURRENT #1: Thu Apr 15 12:14:11 GMT 2004
     root_at_majestix.tallence.de:/usr/obj/usr/src/sys/MAJESTIX
Preloaded elf kernel "/boot/kernel/kernel" at 0xc07df000.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(TM) CPU 2.80GHz (2799.21-MHz 686-class CPU)
   Origin = "GenuineIntel"  Id = 0xf25  Stepping = 5
    
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE 
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
   Hyperthreading: 2 logical CPUs
real memory  = 4160684032 (3967 MB)
avail memory = 4078157824 (3889 MB)
ACPI APIC Table: <A M I  OEMAPIC >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
  cpu0 (BSP): APIC ID:  0
  cpu1 (AP): APIC ID:  1
  cpu2 (AP): APIC ID:  6
  cpu3 (AP): APIC ID:  7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
ioapic4 <Version 2.0> irqs 96-119 on motherboard
Pentium Pro MTRR support enabled
random: <entropy source, Software, Yarrow>
acpi0: <A M I OEMRSDT> on motherboard
acpi0: [GIANT-LOCKED]
pcibios: BIOS version 2.10
Found $PIR table, 13 entries at 0xc00f47e0
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
acpi_cpu0: <CPU> on acpi0
acpi_cpu1: <CPU> on acpi0
acpi_cpu2: <CPU> on acpi0
acpi_cpu3: <CPU> on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pcib1: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P5 -  
AE_NOT_FOUND
pci5: <ACPI PCI bus> on pcib1
pci5: <base peripheral, interrupt controller> at device 28.0 (no driver  
attached)
pcib2: <ACPI PCI-PCI bridge> at device 29.0 on pci5
pci7: <ACPI PCI bus> on pcib2
em0: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port  
0xd880-0xd8bf mem 0xfe9a0000-0xfe9bffff irq 48 at device 1.0 on pci7
em0: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9a0000
em0: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xd880
em0: [GIANT-LOCKED]
em0: Ethernet address: 00:e0:81:28:97:ea
em0:  Speed:N/A  Duplex:N/A
em1: <Intel(R) PRO/1000 Network Connection, Version - 1.7.25> port  
0xdc00-0xdc3f mem 0xfe9e0000-0xfe9fffff irq 49 at device 1.1 on pci7
em1: Bus reserved 0x20000 bytes for rid 0x10 type 3 at 0xfe9e0000
em1: Bus reserved 0x40 bytes for rid 0x20 type 4 at 0xdc00
em1: [GIANT-LOCKED]
em1: Ethernet address: 00:e0:81:28:97:eb
em1:  Speed:N/A  Duplex:N/A
pci5: <base peripheral, interrupt controller> at device 30.0 (no driver  
attached)
pcib3: <ACPI PCI-PCI bridge> at device 31.0 on pci5
pci6: <ACPI PCI bus> on pcib3
twe0: <3ware 7000 series Storage Controller. Driver version  
1.50.01.000> port 0xcc00-0xcc0f mem 0xfe000000-0xfe7fffff irq 24 at  
device 3.0 on pci6
twe0: Bus reserved 0x10 bytes for rid 0x10 type 4 at 0xcc00
twe0: [GIANT-LOCKED]
twe0: 8 ports, Firmware FE7S 1.05.00.063, BIOS BE7X 1.08.00.048
pcib4: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pcib4: could not get PCI interrupt routing table for \\_SB_.PCI0.P0P2 -  
AE_NOT_FOUND
pci2: <ACPI PCI bus> on pcib4
pci2: <base peripheral, interrupt controller> at device 28.0 (no driver  
attached)
pcib5: <ACPI PCI-PCI bridge> at device 29.0 on pci2
pci4: <ACPI PCI bus> on pcib5
pci2: <base peripheral, interrupt controller> at device 30.0 (no driver  
attached)
pcib6: <ACPI PCI-PCI bridge> at device 31.0 on pci2
pci3: <ACPI PCI bus> on pcib6
pci0: <serial bus, USB> at device 29.0 (no driver attached)
pci0: <serial bus, USB> at device 29.1 (no driver attached)
pci0: <serial bus, USB> at device 29.2 (no driver attached)
pcib7: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci1: <ACPI PCI bus> on pcib7
fxp0: <Intel 82551 Pro/100 Ethernet> port 0x8080-0x80bf mem  
0xfd280000-0xfd29ffff,0xfd2fd000-0xfd2fdfff irq 19 at device 1.0 on  
pci1
fxp0: Bus reserved 0x1000 bytes for rid 0x10 type 3 at 0xfd2fd000
miibus0: <MII bus> on fxp0
inphy0: <i82555 10/100 media interface> on miibus0
inphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
fxp0: Ethernet address: 00:e0:81:28:97:d8
fxp0: [GIANT-LOCKED]
pci1: <display, VGA> at device 2.0 (no driver attached)
atapci0: <Promise PDC20318 SATA150 controller> port  
0x8400-0x847f,0x8480-0x848f,0x8c00-0x8c3f mem  
0xfd2a0000-0xfd2bffff,0xfd2fe000-0xfd2fefff irq 18 at device 3.0 on  
pci1
atapci0: rid 0x20 says memory, driver wants 4 failed.
atapci0: Bus reserved 0x1000 bytes for rid 0x1c type 3 at 0xfd2fe000
ata2: at 0xfd2fe000 on atapci0
ata3: at 0xfd2fe000 on atapci0
ata4: at 0xfd2fe000 on atapci0
ata5: at 0xfd2fe000 on atapci0
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci1: <Intel ICH3 UDMA100 controller> port 0xffa0-0xffaf at device  
31.1 on pci0
atapci1: Bus reserved 0x10 bytes for rid 0x20 type 4 at 0xffa0
ata0: at 0x1f0 irq 14 on atapci1
ata1: at 0x170 irq 15 on atapci1
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_button1: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x64,0x60 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0 port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1 port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
fdc0: <Enhanced floppy controller (i82077, NE72065 or clone)> port  
0x3f7,0x3f0-0x3f5 irq 6 drq 2 on acpi0
fdc0: FIFO enabled, 8 bytes threshold
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
cpu0 on motherboard
cpu1 on motherboard
cpu2 on motherboard
cpu3 on motherboard
npx0: [FAST]
npx0: <math processor> on motherboard
npx0: INT 16 interface
orm0: <Option ROMs> at iomem 0xcb800-0xcc7ff,0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on  
isa0
Timecounters tick every 10.000 msec
acpi_cpu: throttling enabled, 8 steps (100% to 12.5%), currently 100.0%
acd0: DVDROM <DV-25E> at ata0-master PIO4
ad10: 239372MB <Maxtor 7Y250M0> [486344/16/63] at ata5-master UDMA133
twed0: <Unit 0, RAID5, Normal> on twe0
twed0: 718114MB (1470698112 sectors)
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
Mounting root from ufs:/dev/twed0s1a
============= dmesg ===============


-- 
Stefan Bethke <stb_at_lassitu.de>   Fon +49 170 346 0140
Received on Tue Apr 20 2004 - 02:14:56 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:51 UTC