Re: ps -axl during textdumps occasionally segfaults with a HUGE ps.core

On Fri, 14 Aug 2009 11:05:05 +0200
Thomas Backman <serenity_at_exscape.org> wrote:

> Looks like you're right!
> I tried the same deal:
> [root_at_chaos ~]# time ps -axl -M /var/crash/vmcore.45.NMAP_SCAN
> Segmentation fault: 11 (core dumped)
> 
> real    0m46.005s
> user    0m0.000s
> sys     0m7.753s
> 
> (All the time taken, according to the hard drive noise, was to save  
> the core dump, which existed long before it returned to the shell.
> 
> [root_at_chaos ~]# gdb /bin/ps ps.core
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "amd64-marcel-freebsd"...(no debugging  
> symbols found)...
> Core was generated by `ps'.
> Program terminated with signal 11, Segmentation fault.
> Reading symbols from /lib/libm.so.5...(no debugging symbols  
> found)...done.
> Loaded symbols for /lib/libm.so.5
> Reading symbols from /lib/libkvm.so.5...(no debugging symbols  
> found)...done.
> Loaded symbols for /lib/libkvm.so.5
> Reading symbols from /lib/libc.so.7...(no debugging symbols  
> found)...done.
> Loaded symbols for /lib/libc.so.7
> Reading symbols from /libexec/ld-elf.so.1...(no debugging symbols  
> found)...done.
> Loaded symbols for /libexec/ld-elf.so.1
> #0  0x0000000800960b9b in strlen () from /lib/libc.so.7
> (gdb) bt
> #0  0x0000000800960b9b in strlen () from /lib/libc.so.7
> #1  0x0000000800959812 in open () from /lib/libc.so.7
> #2  0x00000008008f0546 in vsnprintf () from /lib/libc.so.7
> #3  0x0000000800772d79 in _kvm_err () from /lib/libkvm.so.5
> #4  0x00000008007707f7 in kvm_getprocs () from /lib/libkvm.so.5
> #5  0x0000000000405322 in uname ()
> #6  0x0000000000401f0e in ?? ()
> #7  0x0000000800539000 in ?? ()
> #8  0x0000000000000000 in ?? ()
> #9  0x0000000000000000 in ?? ()
> ...
> #639 0x9066669066669066 in ?? ()
> #640 0x00007fffffffec38 in ?? ()
> #641 0x0000000000000004 in ?? ()
> #642 0x00007fffffffec60 in ?? ()
> #643 0x0000000000000012 in ?? ()
> Cannot access memory at address 0x800000000000
> 
> Crash in strlen() this time, rather than bcopy(), but uname() in
> still the root cause, I guess...?
> 

I managed to get a full backtrace and can at least see what's causing
the crash: it seems it's stepping past the nlist array and calls
vsnprintf with a bad argument. kvm_nlist returns -1 to report that the
symbol table couldn't be read, but the code assumes it has returned a
positive number to indicate that there's an invalid entry, so it starts
searching for that entry where n_type is 0.

tau# gdb ps
GNU gdb 6.1.1 [FreeBSD]
[...]
(gdb) break kvm_proc.c:631
No source file named kvm_proc.c.
Make breakpoint pending on future shared library load? (y or [n]) y
Breakpoint 1 (kvm_proc.c:631) pending.
(gdb) run -ax -M /var/crash/vmcore.3
Starting program: /bin/ps -ax -M /var/crash/vmcore.3
Breakpoint 2 at 0x80076f2f8: file /usr/src/lib/libkvm/kvm_proc.c, line
631. Pending breakpoint "kvm_proc.c:631" resolved

Program received signal SIGSEGV, Segmentation fault.
0x000000080096340b in strlen (str=Variable "str" is not available.
) at /usr/src/lib/libc/string/strlen.c:88
88		    if (*p == '\0')
(gdb) bt
#0  0x000000080096340b in strlen (str=Variable "str" is not available.
) at /usr/src/lib/libc/string/strlen.c:88
#1  0x000000080095c082 in __vfprintf (fp=0x7fffffffd9a0,
fmt0=0x800773915 "%s: no such symbol", ap=0x7fffffffdb10)
at /usr/src/lib/libc/stdio/vfprintf.c:825 #2  0x00000008008cc696 in
vsnprintf (str=Variable "str" is not available. )
at /usr/src/lib/libc/stdio/vsnprintf.c:70 #3  0x0000000800772e89 in
_kvm_err (kd=Variable "kd" is not available. )
at /usr/src/lib/libkvm/kvm.c:104 #4  0x0000000800770907 in kvm_getprocs
(kd=0x800b02300, op=8, arg=0, cnt=0x7fffffffdf1c)
at /usr/src/lib/libkvm/kvm_proc.c:561 #5  0x0000000000405322 in main
(argc=4, argv=0x7fffffffe9a8) at /usr/src/bin/ps/ps.c:511 (gdb) frame 4
#4  0x0000000800770907 in kvm_getprocs (kd=0x800b02300, op=8, arg=0,
cnt=0x7fffffffdf1c) at /usr/src/lib/libkvm/kvm_proc.c:561
561				_kvm_err(kd, kd->program, (gdb) list
556			nl[5].n_name = 0; 557	
558			if (kvm_nlist(kd, nl) != 0) {
559				for (p = nl; p->n_type != 0; ++p)
560					;
561				_kvm_err(kd, kd->program,
562					 "%s: no such symbol",
p->n_name); 563				return (0);
564			}
565			if (KREAD(kd, nl[0].n_value, &nprocs)) {
(gdb) print nl
$1 = {{n_name = 0x8007738ef "_nprocs", n_type = 240 'ð', n_other = -1
'ÿ', n_desc = -1, n_value = 34365215744}, { n_name = 0x8007738f7
"_allproc", n_type = 160 ' ', n_other = -100 '\234', n_desc = 80,
n_value = 0}, { n_name = 0x800773900 "_zombproc", n_type = 57 '9',
n_other = 2 '\002', n_desc = 81, n_value = 34367538496}, { n_name =
0x80077390a "_ticks", n_type = 74 'J', n_other = 0 '\0', n_desc = 0,
n_value = 34365215744}, { n_name = 0x800773911 "_hz", n_type = 168 '¨',
n_other = -23 'é', n_desc = -1, n_value = 140737488349576}, {n_name =
0x0, n_type = 1 '\001', n_other = 0 '\0', n_desc = 0, n_value =
34365024109}} 

-- 
Bruce Cran