Re: Today's -current panics

Robert Watson wrote:
> On Fri, 11 Jun 2004, othermark wrote:
>> I get a very similar stack track traversing through sossend(), under
>> heavy NFS load on a 1GB machine.  Note the panic message here, and the
>> peculiarity that previous incarnations of -current did not panic under
>> similar load.  It is highly reproduceable via a 'make installworld' via
>> NFS with /usr/src and /usr/obj mounted.  The NFS serving machine will
>> always panic using vanilla GENERIC:
> 
> Hrmm.  It's certainly similar, but it's not clear to me that it's
> necessarily related (although I wouldn't preclude that).  Could you tell
> me what date you cvsup'd this tree?

Jun 10 07:35 PST

> Also, since you have a dump (yay!), could you try running vmstat -m,
> vmstat -z, and netstat -mb against the dump?  It could be that a bug was
> introduced in nfsd that leaks mbufs or the like, or it could be a nit in
> mbuma.


[root_at_pippin GENERIC]$ vmstat -z -M /var/crash/vmcore.3
vmstat: not implemented

DOH! (same goes for -m)

[root_at_pippin GENERIC]$ netstat -mb -M /var/crash/vmcore.3
netstat: calloc: cannot allocate memory for mbtypes seen flag: Cannot
allocate memory

DOH!

I'm very willing to help further so I can continue to do nfs installs
of -current for my test harness.

The NFS bits panic looks like this (same panic message):

(kgdb) symbol-file kernel.debug
Reading symbols from kernel.debug...done.
(kgdb) exec-file kernel
(kgdb) core-file /var/crash/vmcore.4
panic: kmem_malloc(4096): kmem_map too small: 40894464 total allocated
panic messages:
---
panic: kmem_malloc(4096): kmem_map too small: 40894464 total allocated
cpuid = 0;
Debugger("panic")
panic: from debugger
cpuid = 0;
Uptime: 7m9s
Dumping 127 MB
 16 32 48 64 80 96 112
---
Reading symbols from /boot/kernel/acpi.ko...done.
Loaded symbols for /boot/kernel/acpi.ko
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:236
236             dumping++;
(kgdb) bt
#0  doadump () at /usr/src/sys/kern/kern_shutdown.c:236
#1  0xc064ffd9 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:370
#2  0xc06503dd in panic () at /usr/src/sys/kern/kern_shutdown.c:548
#3  0xc0469902 in db_panic () at /usr/src/sys/ddb/db_command.c:453
#4  0xc0469862 in db_command (last_cmdp=0xc09250a0, cmd_table=0x0,
    aux_cmd_tablep=0xc08a6f00, aux_cmd_tablep_end=0xc08a6f18)
    at /usr/src/sys/ddb/db_command.c:348
#5  0xc04699a5 in db_command_loop () at /usr/src/sys/ddb/db_command.c:475
#6  0xc046cb05 in db_trap (type=3, code=0) at /usr/src/sys/ddb/db_trap.c:73
#7  0xc080132c in kdb_trap (type=3, code=0, regs=0xc80f28cc)
    at /usr/src/sys/i386/i386/db_interface.c:159
#8  0xc0817948 in trap (frame=
      {tf_fs = -1066991592, tf_es = -1064763376, tf_ds = -1064828912, tf_edi
= -
1064721812, tf_esi = 1, tf_ebp = -938530536, tf_isp = -938530568, tf_ebx =
0, tf
_edx = 0, tf_ecx = -1056882688, tf_eax = 18, tf_trapno = 3, tf_err = 0,
tf_eip =
 -1065347531, tf_cs = 8, tf_eflags = 642, tf_esp = -1064702569, tf_ss =
-1064817
463}) at /usr/src/sys/i386/i386/trap.c:579
#9  0xc0801635 in Debugger (msg=0x0) at machine/cpufunc.h:56
#10 0xc0650376 in panic (
    fmt=0xc089a26c "kmem_malloc(%ld): kmem_map too small: %ld total
allocated")
    at /usr/src/sys/kern/kern_shutdown.c:532
#11 0xc07c43cb in kmem_malloc (map=0xc103a0a0, size=4096, flags=258)
    at /usr/src/sys/vm/vm_kern.c:324
#12 0xc07d59a7 in page_alloc (zone=0xc1021b00, bytes=0, pflag=0x0, wait=0)
    at /usr/src/sys/vm/uma_core.c:892
#13 0xc07d562d in slab_zalloc (zone=0xc1021b00, wait=258)
    at /usr/src/sys/vm/uma_core.c:789
#14 0xc07d6d4a in uma_zone_slab (zone=0xc1021b00, flags=2)
    at /usr/src/sys/vm/uma_core.c:1772
#15 0xc07d70a1 in uma_zalloc_internal (zone=0xc1021b00, udata=0x0, flags=2)
    at /usr/src/sys/vm/uma_core.c:1936
#16 0xc07d6c52 in uma_zalloc_arg (zone=0xc1021b00, udata=0xc80f2b14,
flags=2)
    at /usr/src/sys/vm/uma_core.c:1711
#17 0xc0741728 in nfsrv_read (nfsd=0xc1704000, slp=0x0, td=0xc137d2c0,
    mrq=0xc80f2c90) at /usr/src/sys/sys/mbuf.h:334
#18 0xc075203a in nfssvc_nfsd (td=0x0)
    at /usr/src/sys/nfsserver/nfs_syscalls.c:465
#19 0xc07518bd in nfssvc (td=0xc137d2c0, uap=0xc80f2d14)
    at /usr/src/sys/nfsserver/nfs_syscalls.c:181
---Type <return> to continue, or q <return> to quit---
#20 0xc08182f0 in syscall (frame=
      {tf_fs = -1078001617, tf_es = 47, tf_ds = -1078001617, tf_edi =
-107794054
0, tf_esi = 4, tf_ebp = -1077941432, tf_isp = -938529420, tf_ebx = 0, tf_edx
= 6
72430424, tf_ecx = 25, tf_eax = 155, tf_trapno = 12, tf_err = 2, tf_eip =
671893
695, tf_cs = 31, tf_eflags = 646, tf_esp = -1077941460, tf_ss = 47})
    at /usr/src/sys/i386/i386/trap.c:1004
#21 0x280c48bf in ?? ()
---Can't read userspace from dump, or kernel process---

-- 
othermark
atkin901 at nospam dot yahoo dot com
(!wired)?(coffee++):(wired);