Re: panic: in_pcblookup_local (?)

On Monday, April 29, 2013 8:35:52 pm Glen Barber wrote:
> On Mon, Apr 29, 2013 at 12:24:06PM -0400, John Baldwin wrote:
> > On Sunday, April 28, 2013 12:02:56 am Glen Barber wrote:
> > > On Sat, Apr 27, 2013 at 10:17:32AM +0200, Ian FREISLICH wrote:
> > > > Hi
> > > > 
> > > > I've been getting the following panic on recent current r249717.
> > > > Sadly the crashdump is useless.
> > > > 
> > > 
> > > I just saw similar panic on 10-CURRENT r249588.
> > > 
> > > > Fatal trap 9: general protection fault while in kernel mode
> > > > cpuid = 15; apic id = 0f
> > > > instruction pointer     = 0x20:0xffffffff80546fbc
> > > > stack pointer           = 0x28:0xffffff846b677770
> > > > frame pointer           = 0x28:0xffffff846b6777b0
> > > > code segment            = base 0x0, limit 0xfffff, type 0x1b
> > > >                         = DPL 0, pres 1, long 1, def32 0, gran 1
> > > > processor eflags        = interrupt enabled, resume, IOPL = 0
> > > > current process         = 4361 (zabbix_agentd)
> > > 
> > > Hmm..  This is interests me.  In my case, cf-agent was the current
> > > process.
> > > 
> > > Backtrace of my panic follows.  Any pointers on how to debug this
> > > further would be appreciated.
> > > 
> > > Glen
> > > 
> > > Script started on Sat Apr 27 23:53:53 2013
> > > root_at_orion:/usr/obj/usr/src/sys/ORION # kgdb ./kernel.debug 
> > /var/crash/vmcore.4
> > > GNU gdb 6.1.1 [FreeBSD]
> > > Copyright 2004 Free Software Foundation, Inc.
> > > GDB is free software, covered by the GNU General Public License, and you are
> > > welcome to change it and/or distribute copies of it under certain 
> > conditions.
> > > Type "show copying" to see the conditions.
> > > There is absolutely no warranty for GDB.  Type "show warranty" for details.
> > > This GDB was configured as "amd64-marcel-freebsd"...
> > > 
> > > Unread portion of the kernel message buffer:
> > > 
> > > 
> > > Fatal trap 9: general protection fault while in kernel mode
> > > cpuid = 1; apic id = 01
> > > instruction pointer	= 0x20:0xffffffff80736cec
> > > stack pointer	        = 0x28:0xffffff81aad4e760
> > > frame pointer	        = 0x28:0xffffff81aad4e7a0
> > > code segment		= base 0x0, limit 0xfffff, type 0x1b
> > > 			= DPL 0, pres 1, long 1, def32 0, gran 1
> > > processor eflags	= interrupt enabled, resume, IOPL = 0
> > > current process		= 78664 (cf-agent)
> > > trap number		= 9
> > > panic: general protection fault
> > > cpuid = 1
> > > KDB: stack backtrace:
> > > #0 0xffffffff80642a56 at kdb_backtrace+0x66
> > > #1 0xffffffff80606eeb at panic+0x13b
> > > #2 0xffffffff808e3b10 at trap_fatal+0x290
> > > #3 0xffffffff808e4331 at trap+0x241
> > > #4 0xffffffff808cdbb3 at calltrap+0x8
> > > #5 0xffffffff807371d8 at in_pcb_lport+0x128
> > > #6 0xffffffff8073745a at in_pcbbind_setup+0x16a
> > > #7 0xffffffff80737d8e at in_pcbconnect_setup+0x71e
> > > #8 0xffffffff80737df9 at in_pcbconnect_mbuf+0x59
> > > #9 0xffffffff807bf29f at udp_connect+0x11f
> > > #10 0xffffffff80680615 at kern_connectat+0x275
> > > #11 0xffffffff80680731 at sys_connect+0x41
> > > #12 0xffffffff808e32cb at amd64_syscall+0x63b
> > > #13 0xffffffff808cde97 at Xfast_syscall+0xf7
> > > Uptime: 3d19h38m52s
> > > (ada0:ahcich0:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
> > > (ada0:ahcich0:0:0:0): CAM status: CCB request is in progress
> > > (ada0:ahcich0:0:0:0): Error 5, Retries exhausted
> > > (ada0:ahcich0:0:0:0): Synchronize cache failed
> > > (ada1:ahcich1:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
> > > (ada1:ahcich1:0:0:0): CAM status: CCB request is in progress
> > > (ada1:ahcich1:0:0:0): Error 5, Retries exhausted
> > > (ada1:ahcich1:0:0:0): Synchronize cache failed
> > > (ada2:ahcich4:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
> > > (ada2:ahcich4:0:0:0): CAM status: CCB request is in progress
> > > (ada2:ahcich4:0:0:0): Error 5, Retries exhausted
> > > (ada2:ahcich4:0:0:0): Synchronize cache failed
> > > (ada3:ahcich5:0:0:0): FLUSHCACHE48. ACB: ea 00 00 00 00 40 00 00 00 00 00 00
> > > (ada3:ahcich5:0:0:0): CAM status: CCB request is in progress
> > > (ada3:ahcich5:0:0:0): Error 5, Retries exhausted
> > > (ada3:ahcich5:0:0:0): Synchronize cache failed
> > > Dumping 1014 out of 6049 
> > MB:..2%..12%..21%..32%..42%..51%..62%..71%..81%..92%
> > > 
> > > Reading symbols from /boot/kernel/zfs.ko.symbols...done.
> > > Loaded symbols for /boot/kernel/zfs.ko.symbols
> > > Reading symbols from /boot/kernel/opensolaris.ko.symbols...done.
> > > Loaded symbols for /boot/kernel/opensolaris.ko.symbols
> > > #0  doadump (textdump=<value optimized out>) at pcpu.h:231
> > > 231		__asm("movq %%gs:%1,%0" : "=r" (td)
> > > (kgdb) frame 6
> > > #6  0xffffffff80736cec in in_pcblookup_local (pcbinfo=0xffffffff80dc9180, 
> > laddr=
> > >       {s_addr = 50374848}, lport=339, lookupflags=1, 
> > cred=0xfffffe016cdad100)
> > >     at /usr/src/sys/netinet/in_pcb.c:1438
> > > 1438			LIST_FOREACH(phd, porthash, phd_hash) {
> > > (kgdb) list *0xffffffff80736cec
> > > 0xffffffff80736cec is in in_pcblookup_local 
> > (/usr/src/sys/netinet/in_pcb.c:1439).
> > > 1434			 * port hash list.
> > > 1435			 */
> > > 1436			porthash = &pcbinfo->ipi_porthashbase[INP_PCBPORTHASH(lport,
> > > 1437			    pcbinfo->ipi_porthashmask)];
> > > 1438			LIST_FOREACH(phd, porthash, phd_hash) {
> > > 1439				if (phd->phd_port == lport)
> > > 1440					break;
> > > 1441			}
> > > 1442			if (phd != NULL) {
> > > 1443				/*
> > 
> > Can you see what 'phd' and 'porthash' are?  If kgdb can't see them you can
> > reconstruct what 'porthash' should be (you know the lport arg to this routine
> > and can get the relevant fields from 'pcbinfo').
> > 
> 
> I'm not sure if the output included makes much sense.  If I did not do
> something correctly, please let me know.
> 
> Glen
> 
> Script started on Mon Apr 29 20:27:39 2013
> root_at_orion:/usr/obj/usr/src/sys/ORION #	kgdb ./kernel.debug /var/crash/vmcore.4
> [...]
> #0  doadump (textdump=<value optimized out>) at pcpu.h:231
> 231		__asm("movq %%gs:%1,%0" : "=r" (td)
> (kgdb) frame 6
> #6  0xffffffff80736cec in in_pcblookup_local (pcbinfo=0xffffffff80dc9180, laddr=
>       {s_addr = 50374848}, lport=339, lookupflags=1, cred=0xfffffe016cdad100)
>     at /usr/src/sys/netinet/in_pcb.c:1438
> 1438			LIST_FOREACH(phd, porthash, phd_hash) {
> (kgdb) p *phd
> Cannot access memory at address 0x9e17b100fffffe00
> (kgdb) p *porthash
> No symbol "porthash" in current context.
> (kgdb) p &pcbinfo->ipi_porthashbase[INP_PCBPORTHASH(lport,pcbinfo->ipi_porthashmask)]
> No symbol "INP_PCBPORTHASH" in current context.
> (kgdb) p pcbinfo->ipi_porthashmask
> $1 = 127
> (kgdb) p *pcbinfo->ipi_porthashmask
> Cannot access memory at address 0x7f
> (kgdb) p *lport
> Cannot access memory at address 0x153
> (kgdb) p lport
> $2 = 339
> (kgdb) quit
> root_at_orion:/usr/obj/usr/src/sys/ORION #	^D

Try 'p phd' to start.  INP_PCBPORTHASH is a macro, so you will have to do it by hand:

'p pcbinfo->ipi_porthashbase[lport & pcbinfo->ipi_porthashmask]'

(That should be what 'porthash' is.)

-- 
John Baldwin