Re: device driver memory leak in 5.1-20030726?

From: Mark Blackman <mark_at_exonetric.com>
Date: Sat, 26 Jul 2003 21:48:25 +0100
A backtrace: (where and where full) for those who can decipher them
uma_core.c seems to have been the trigger.

(kgdb) where
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
#1  0xc032cc4c in boot (howto=260) at  
/usr/src/sys/kern/kern_shutdown.c:372
#2  0xc032cfd7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
#3  0xc0163e22 in db_panic () at /usr/src/sys/ddb/db_command.c:449
#4  0xc0163da2 in db_command (last_cmdp=0xc05c6b40, cmd_table=0x0,
     aux_cmd_tablep=0xc054de7c, aux_cmd_tablep_end=0xc054de94)
     at /usr/src/sys/ddb/db_command.c:346
#5  0xc0163ec5 in db_command_loop () at  
/usr/src/sys/ddb/db_command.c:471
#6  0xc0166dc5 in db_trap (type=3, code=0) at  
/usr/src/sys/ddb/db_trap.c:73
#7  0xc04b454c in kdb_trap (type=3, code=0, regs=0xcc464aa4)
     at /usr/src/sys/i386/i386/db_interface.c:172
#8  0xc04c5e1d in trap (frame=
       {tf_fs = -1047855080, tf_es = -867827696, tf_ds = 16, tf_edi = 1,  
tf_esi = -1068224493, tf_ebp = -867808528, tf_isp = -867808560, tf_ebx  
= 0, tf_edx = 0, tf_ecx = -1067232032, tf_eax = 18, tf_trapno = 3,  
tf_err = 0, tf_eip = -1068808188, tf_cs = 8, tf_eflags = 646, tf_esp =  
-1068208597, tf_ss = -1068312245})
     at /usr/src/sys/i386/i386/trap.c:580
#9  0xc04b5f38 in calltrap () at {standard input}:102
#10 0xc032cf65 in panic (
     fmt=0xc0543013 "kmem_malloc(%ld): kmem_map too small: %ld total  
allocated")
     at /usr/src/sys/kern/kern_shutdown.c:534
#11 0xc047dee0 in kmem_malloc (map=0xc082f0b0, size=4096, flags=2)
     at /usr/src/sys/vm/vm_kern.c:339
#12 0xc048ee87 in page_alloc (zone=0xc083aee0, bytes=0, pflag=0x0,  
wait=0)
---Type <return> to continue, or q <return> to quit---
     at /usr/src/sys/vm/uma_core.c:806
#13 0xc048ebbf in slab_zalloc (zone=0xc083aee0, wait=2)
     at /usr/src/sys/vm/uma_core.c:711
#14 0xc048fd58 in uma_zone_slab (zone=0xc083aee0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1503
#15 0xc048ff96 in uma_zalloc_bucket (zone=0xc083aee0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1606
#16 0xc048fbf9 in uma_zalloc_arg (zone=0xc083aee0, udata=0x0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1434
#17 0xc0321543 in malloc (size=3229855456, type=0xc0583a80, flags=258)
     at /usr/src/sys/vm/uma.h:229
#18 0xc03325f5 in sigacts_alloc () at /usr/src/sys/kern/kern_sig.c:2719
#19 0xc03173ce in fork1 (td=0xc18bce40, flags=20, pages=0,  
procp=0xcc464cd8)
     at /usr/src/sys/kern/kern_fork.c:414
#20 0xc0316c2b in fork (td=0xc18bce40, uap=0xcc464d10)
     at /usr/src/sys/kern/kern_fork.c:102
#21 0xc04c6753 in syscall (frame=
       {tf_fs = 134938671, tf_es = 134873135, tf_ds = -1078001617,  
tf_edi = 6, tf_esi = 135030952, tf_ebp = -1077937480, tf_isp =  
-867807884, tf_ebx = 135016448, tf_edx = 3, tf_ecx = -1077937680,  
tf_eax = 2, tf_trapno = 12, tf_err = 2, tf_eip = 673679423, tf_cs = 31,  
tf_eflags = 531, tf_esp = -1077937732, tf_ss = 47})
     at /usr/src/sys/i386/i386/trap.c:1008
#22 0xc04b5f8d in Xint0x80_syscall () at {standard input}:144
---Can't read userspace from dump, or kernel process---

(kgdb) where full
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:240
No locals.
#1  0xc032cc4c in boot (howto=260) at  
/usr/src/sys/kern/kern_shutdown.c:372
No locals.
#2  0xc032cfd7 in panic () at /usr/src/sys/kern/kern_shutdown.c:550
	td = (struct thread *) 0xc18bce40
	bootopt = 260
	newpanic = 0
	ap = 0xcc464924 "‹IFâ=\026¿\004HK¿"
	buf = "kmem_malloc(4096): kmem_map too small: 112951296 total  
allocated", '\0' <repeats 191 times>
#3  0xc0163e22 in db_panic () at /usr/src/sys/ddb/db_command.c:449
No locals.
#4  0xc0163da2 in db_command (last_cmdp=0xc05c6b40, cmd_table=0x0,
     aux_cmd_tablep=0xc054de7c, aux_cmd_tablep_end=0xc054de94)
     at /usr/src/sys/ddb/db_command.c:346
	cmd = (struct command *) 0xc04dedfc
	t = 0
	modif =  
"\0t\\¿hid¿lIFÃ\r\0\0\0‡Tc¿\r\0\0\0\001\0\0\0\214IFÃF£J¿‡:b¿\aK\0  
`Uc¿‡]a¿†t\\¿x\0\0\0†t\\¿hid¿∞IFÃa[\026¿\222ZP¿PZ\026¿\0\0\0\0\020\0\0\0 
hid¿†t\\¿∂S\026¿†t\\¿–l\\¿x\0\0\0\003\0\0"
	addr = -1068808188
	count = -1
	have_addr = 0
---Type <return> to continue, or q <return> to quit---
	result = 0
#5  0xc0163ec5 in db_command_loop () at  
/usr/src/sys/ddb/db_command.c:471
No locals.
#6  0xc0166dc5 in db_trap (type=3, code=0) at  
/usr/src/sys/ddb/db_trap.c:73
	bkpt = 0
#7  0xc04b454c in kdb_trap (type=3, code=0, regs=0xcc464aa4)
     at /usr/src/sys/i386/i386/db_interface.c:172
	ef = 70
	ddb_mode = 1
#8  0xc04c5e1d in trap (frame=
       {tf_fs = -1047855080, tf_es = -867827696, tf_ds = 16, tf_edi = 1,  
tf_esi = -1068224493, tf_ebp = -867808528, tf_isp = -867808560, tf_ebx  
= 0, tf_edx = 0, tf_ecx = -1067232032, tf_eax = 18, tf_trapno = 3,  
tf_err = 0, tf_eip = -1068808188, tf_cs = 8, tf_eflags = 646, tf_esp =  
-1068208597, tf_ss = -1068312245})
     at /usr/src/sys/i386/i386/trap.c:580
	td = (struct thread *) 0xc18bce40
	p = (struct proc *) 0xc19c7d3c
	sticks = 3224514865
	i = 0
	ucode = 0
	type = 3
	code = 0
	eva = 0
#9  0xc04b5f38 in calltrap () at {standard input}:102
---Type <return> to continue, or q <return> to quit---
No locals.
#10 0xc032cf65 in panic (
     fmt=0xc0543013 "kmem_malloc(%ld): kmem_map too small: %ld total  
allocated")
     at /usr/src/sys/kern/kern_shutdown.c:534
	td = (struct thread *) 0xc18bce40
	bootopt = 256
	newpanic = 1
	ap = 0x0
	buf = "kmem_malloc(4096): kmem_map too small: 112951296 total  
allocated", '\0' <repeats 191 times>
#11 0xc047dee0 in kmem_malloc (map=0xc082f0b0, size=4096, flags=2)
     at /usr/src/sys/vm/vm_kern.c:339
	offset = 710
	i = 3229855456
	entry = 0xcc464b7c
	addr = 3233144832
	m = 0x2
	pflags = -1065111820
#12 0xc048ee87 in page_alloc (zone=0xc083aee0, bytes=0, pflag=0x0,  
wait=0)
     at /usr/src/sys/vm/uma_core.c:806
	p = (void *) 0x0
#13 0xc048ebbf in slab_zalloc (zone=0xc083aee0, wait=2)
     at /usr/src/sys/vm/uma_core.c:711
	slab = 0xc76f24c8
---Type <return> to continue, or q <return> to quit---
	mem = (u_int8_t *) 0xc083aef4 "¨7X¿\227uO¿\235IT¿"
	flags = 2 '\002'
	i = 2
#14 0xc048fd58 in uma_zone_slab (zone=0xc083aee0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1503
	slab = 0x0
#15 0xc048ff96 in uma_zalloc_bucket (zone=0xc083aee0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1606
	bucket = 0xc192d400
	slab = 0xc083aef4
#16 0xc048fbf9 in uma_zalloc_arg (zone=0xc083aee0, udata=0x0, flags=258)
     at /usr/src/sys/vm/uma_core.c:1434
	item = (void *) 0xc18bce40
	cache = 0xc083afa8
	bucket = 0x0
	cpu = 0
#17 0xc0321543 in malloc (size=3229855456, type=0xc0583a80, flags=258)
     at /usr/src/sys/vm/uma.h:229
	indx = 8
	va = 0xc05eff60 "LHX¿¶“R¿¶“R¿"
	zone = 0xc083aee0
	ksp = (struct malloc_type *) 0xc0583a80
#18 0xc03325f5 in sigacts_alloc () at /usr/src/sys/kern/kern_sig.c:2719
No locals.
---Type <return> to continue, or q <return> to quit---
#19 0xc03173ce in fork1 (td=0xc18bce40, flags=20, pages=0,  
procp=0xcc464cd8)
     at /usr/src/sys/kern/kern_fork.c:414
	p2 = (struct proc *) 0xc1920974
	pptr = (struct proc *) 0x0
	uid = 3247573364
	newproc = (struct proc *) 0xc1920974
	trypid = 669
	ok = 669
	curfail = 0
	pidchecked = 99999
	lastfail = {tv_sec = 0, tv_usec = 0}
	fd = (struct filedesc *) 0xc19c7da8
	fdtol = (struct filedesc_to_leader *) 0x165
	p1 = (struct proc *) 0xc19c7d3c
	td2 = (struct thread *) 0x246
	ke2 = (struct kse *) 0x29d
	kg2 = (struct ksegrp *) 0x23
	newsigacts = (struct sigacts *) 0x0
	error = 35
#20 0xc0316c2b in fork (td=0xc18bce40, uap=0xcc464d10)
     at /usr/src/sys/kern/kern_fork.c:102
	error = 0
	p2 = (struct proc *) 0xc18bce40
#21 0xc04c6753 in syscall (frame=
---Type <return> to continue, or q <return> to quit---
       {tf_fs = 134938671, tf_es = 134873135, tf_ds = -1078001617,  
tf_edi = 6, tf_esi = 135030952, tf_ebp = -1077937480, tf_isp =  
-867807884, tf_ebx = 135016448, tf_edx = 3, tf_ecx = -1077937680,  
tf_eax = 2, tf_trapno = 12, tf_err = 2, tf_eip = 673679423, tf_cs = 31,  
tf_eflags = 531, tf_esp = -1077937732, tf_ss = 47})
     at /usr/src/sys/i386/i386/trap.c:1008
	params = 0xbfbff9c0---Can't read userspace from dump, or kernel  
process---

(kgdb)
(kgdb) quit

On Saturday, July 26, 2003, at 09:06 PM, Mark Blackman wrote:

> Hi all,
>
> I'm seeing the same 'kmem_malloc(4096): kmem_map too small: XXXXX  
> total allocated'
> messages that a few other have reported.
>
> Now, I understand that setting kern.vm.kmem.size larger is supposed to
> help, but I'm using a 128M Celeron-650 i386 system with no unusual
> devices (expect perhaps a Speedtouch ADSL modem) and I've progressively
> set the kern.vm.kmem.size to larger and larger values, starting at
> 64MB, then 96MB and finally 128MB.
>
> As I approached the physical memory size of the machine (128MB),
> the panic problem failed to reappear, but I got another problem  
> whereby the kernel
> appeared to take over all of memory (i.e. processes were gradually
> all getting swapped out, but no other process seemed to be taking
> the memory) within about 30 minutes of boot-up.
>
> I noticed in the final minutes of the case where kmem.size=128MB (i.e.  
> all
> of physical RAM), that kern.malloc was reporting 100M of 'devbuf'  
> memory
> allocations and that it was gradually increasing at about 25k per
> second. I can't believe this is normal behaviour, but I'm no
> expert. I believe the devbuf allocations are specifically for
> device drivers.
>
> From these symptoms, I'm speculating that one or more device drivers
> are producing kernel memory leaks and either triggering the
> 'kmem_map too small' messages or pushing all of the userland processes
> out of the way. Is this a reasonable interpretation?
>
> Does anyone else see symptoms that might lead to this conclusion?
>
> As a side note, I also briefly witnessed scrolling
> errors like 'ad0: out of memory in start'.
>
> I have no idea if this implies the 'ad' driver is an issue.
>
> Regards,
> Mark Blackman
> Exonetric Consulting
>
Received on Sat Jul 26 2003 - 11:48:31 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:16 UTC