Re: FreeBSD 8.0-BETA2/amd64 crashes on SMP under load

On Friday 31 July 2009 12:40:26 pm Marcel Moolenaar wrote:
> 
> On Jul 31, 2009, at 5:46 AM, John Baldwin wrote:
> 
> > On Thursday 30 July 2009 1:16:32 pm Marcel Moolenaar wrote:
> >>
> >> On Jul 30, 2009, at 10:09 AM, Anton Shterenlikht wrote:
> >>
> >>> On Thu, Jul 30, 2009 at 10:02:29AM -0700, Marcel Moolenaar wrote:
> >>>>
> >>>> On Jul 30, 2009, at 2:05 AM, Anton Shterenlikht wrote:
> >>>>> By the way, are these two FreeBSD docs up to date:
> >>>>>
> >>>>>
> > 
http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/advanced.html#KERNEL-PANIC-TROUBLESHOOTING
> >>>>>
> >>>>>
> > 
http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerneldebug.html
> >>>>>
> >>>>> In particular, it is still true that minidump is a default dump
> >>>>> type?
> >>>>
> >>>> ia64 doesn't yet have minidumps. In fact, changes to GDB that
> >>>> happened a year ago or so broke the ability to read ia64 core
> >>>> files. I plan on implementing minidumps after 8.0-RELEASE.
> >>>
> >>> you mean, even if I get a dump, I wouldn't be able to read it?
> >>
> >> Correct.
> >>
> >> The change that broke it is:
> >> http://svn.FreeBSD.org/viewvc/base?view=revision&revision=178670
> >
> > Wait, how did that break ia64 but not other architectures?
> 
> hob% sudo kgdb -n 1
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "ia64-marcel-freebsd"...
> GDB can't read core files on this machine.
> (kgdb) info target
> Symbols from "/usr/obj/nfs/freebsd/base/head/sys/HOB/kernel.debug".
> Local exec file:
> 	`/usr/obj/nfs/freebsd/base/head/sys/HOB/kernel.debug',
>          file type elf64-ia64-little.
> 	Entry point: 0xe000000004068000
> 	0xe000000004000190 - 0xe00000000400019d is .interp
> 	0xe0000000040001a0 - 0xe00000000400b100 is .hash
> 	0xe00000000400b100 - 0xe000000004034cc8 is .dynsym
> 	0xe000000004034cc8 - 0xe0000000040505f3 is .dynstr
> 	0xe0000000040505f8 - 0xe000000004061518 is .rela.data
> 	0xe000000004061518 - 0xe000000004062ad8 is .rela.got
> 	0xe000000004062ad8 - 0xe000000004062af0 is .rela.sdata
> 	0xe000000004068000 - 0xe00000000469fcc0 is .text
> 	0xe00000000469fcc0 - 0xe000000004715884 is .rodata
> 	0xe000000004715890 - 0xe00000000471f350 is .opd
> 	0xe00000000471f350 - 0xe00000000475c0c0 is .IA_64.unwind_info
> 	0xe00000000475c0c0 - 0xe00000000478bc40 is .IA_64.unwind
> 	0xe00000000478e000 - 0xe0000000047f1868 is .data
> 	0xe0000000047f1868 - 0xe0000000047f3a70 is set_sysctl_set
> 	0xe0000000047f3a80 - 0xe0000000047f4780 is set_pcpu
> 	0xe0000000047f4780 - 0xe0000000047f5c48 is set_sysinit_set
> 	0xe0000000047f5c48 - 0xe0000000047f6610 is set_sysuninit_set
> 	0xe0000000047f6610 - 0xe0000000047f70a0 is set_modmetadata_set
> 	0xe0000000047f70a0 - 0xe0000000047f70b8 is set_kdb_dbbe_set
> 	0xe0000000047f70b8 - 0xe0000000047f70c8 is set_gdb_dbgport_set
> ---Type <return> to continue, or q <return> to quit---
> 	0xe0000000047f70c8 - 0xe0000000047f70e0 is set_cons_set
> 	0xe0000000047f70e0 - 0xe0000000047f7200 is .dynamic
> 	0xe0000000047f7200 - 0xe000000004810bc8 is .got
> 	0xe000000004810bc8 - 0xe0000000048124dc is .sdata
> 	0xe0000000048124e0 - 0xe000000004814a00 is .sbss
> 	0xe000000004814a00 - 0xe0000000049d52f0 is .bss
> (kgdb)
> 
> 
> The stratum of the kgdb target was changed from highest (thread_stratum)
> to lowest (core_stratum). As such, rather than hide kernel core file
> details from GDB we became dependent upon them.
> 
> Simply changing the stratum from core_stratum to thread_stratum shows
> what's going on:
> 
> hob% sudo kgdb -n 1
> GNU gdb 6.1.1 [FreeBSD]
> Copyright 2004 Free Software Foundation, Inc.
> GDB is free software, covered by the GNU General Public License, and  
> you are
> welcome to change it and/or distribute copies of it under certain  
> conditions.
> Type "show copying" to see the conditions.
> There is absolutely no warranty for GDB.  Type "show warranty" for  
> details.
> This GDB was configured as "ia64-marcel-freebsd"...
> 
> warning: "/var/crash/vmcore.1": no core file handler recognizes  
> format, using default
> 
> 
> In other words: kgdb was designed to hide FreeBSD specifics from the  
> core GDB
> code, because core GDB doesn't know how to deal with all FreeBSD  
> details.
> Revision 178670 created a stronger dependency on core GDB and as such  
> broke
> architectures that core GDB doesn't support for FreeBSD.

So gdb doesn't work for core dumps on regular processes under ia64?  Still,
kgdb doesn't actually use the regular core target at all, it uses its own
target that uses libkvm to service the actual memory I/O that normally goes
to the core target.  The previous code was very hacky and didn't allow you
to use the 'file' and 'core' commands as a result.

-- 
John Baldwin