Re: panic in rt_check_fib()

From: Robert Watson <rwatson_at_FreeBSD.org>
Date: Sat, 13 Sep 2008 15:15:38 +0100 (BST)
On Sat, 13 Sep 2008, Robert Watson wrote:

> On Fri, 5 Sep 2008, Giorgos Keramidas wrote:
>
>> A kernel that I built last night to test Ed's "packet mode" for ptys 
>> included all the changes up to 182743 panics with:
>
> I had an identical panic on 7-STABLE last night:
>
> db> bt
> Tracing pid 782 tid 100091 td 0xc4496440
> kdb_enter_why(c0b25ea1,c0b25ea1,c0b24c19,e6772978,0,...) at 
> kdb_enter_why+0x3a
> panic(c0b24c19,c0b32d59,c0b32d7a,633,c436c9b0,...) at panic+0x12c
> _mtx_lock_sleep(c436ddf4,c4496440,0,c0b32d7a,633,...) at _mtx_lock_sleep+0x4a
> _mtx_lock_flags(c436ddf4,0,c0b32d7a,633,c436ca14,...) at _mtx_lock_flags+0xd1

This is actually from i386/machine/pcpu.h, line 194:

static __inline struct thread *
__curthread(void)
{
         struct thread *td;

         __asm __volatile("movl %%fs:0,%0" : "=r" (td));
         return (td);
}

> rt_check_fib(e6772a0c,e6772a28,c424ea90,0,e6772a1c,...) at rt_check_fib+0x2b4

(kgdb) l *rt_check_fib+0x2b4
0xc0827104 is in rt_check_fib (/usr/src/sys/net/route.c:1587).
warning: Source file is more recent than executable.

1582			if ((rt->rt_flags & RTF_UP) == 0) {
1583				RTFREE_LOCKED(rt);	/* unlock gwroute */
1584				rt = rt0;
1585				rt0->rt_gwroute = NULL;
1586			lookup:
1587				RT_UNLOCK(rt0);
1588	/* XXX MRT link level looked up in table 0 */
1589				rt = rtalloc1_fib(rt->rt_gateway, 1, 0UL, 0);
1590				if (rt == rt0) {
1591					RT_REMREF(rt0);

> in_rt_check(e6772a0c,e6772a28,c424ea90,0,0,...) at in_rt_check+0x26

(kgdb) l *in_rt_check+0x26
0xc08576f6 is in in_rt_check (/usr/src/sys/netinet/in_rmx.c:473).
468	int
469	in_rt_check(struct rtentry **lrt, struct rtentry **lrt0,
470		struct sockaddr *dst, u_int fibnum)
471	{
472		return (rt_check_fib(lrt, lrt0, dst, fibnum));
473	}
474
475	void
476	in_rtredirect(struct sockaddr *dst,
477		struct sockaddr *gateway,

> arpresolve(c4040000,c436c9b0,c4240800,c424ea90,e6772a42,...) at 
> arpresolve+0xb9

(kgdb) l *arpresolve+0xb9
0xc084ddc9 is in arpresolve (/usr/src/sys/netinet/if_ether.c:379).
warning: Source file is more recent than executable.

374				ETHER_MAP_IP_MULTICAST(&SIN(dst)->sin_addr, 
desten);
375				return (0);
376			}
377			fibnum = M_GETFIB(m);
378		}
379
380		if (rt0 != NULL) {
381			/* Look for a cached arp (ll) entry. */
382			error = in_rt_check(&rt, &rt0, dst, fibnum);
383			if (error) {

Looks like I'm using an older version of if_ether.c than I have checked out.

> ether_output(c4040000,c4240800,c424ea90,c436c9b0,c450b9d8,...) at 
> ether_output+0x7e

(kgdb) l *ether_output+0x7e
0xc081774e is in ether_output (/usr/src/sys/net/if_ethersubr.c:175).
warning: Source file is more recent than executable.

170
171		hlen = ETHER_HDR_LEN;
172		switch (dst->sa_family) {
173	#ifdef INET
174		case AF_INET:
175			error = arpresolve(ifp, rt0, m, dst, edst);
176			if (error)
177				return (error == EWOULDBLOCK ? 0 : error);
178			type = htons(ETHERTYPE_IP);
179			break;

Robert N M Watson
Computer Laboratory
University of Cambridge

> ip_output(c4240800,0,e6772ab0,0,0,...) at ip_output+0xa34
> udp_send(c44f74b0,0,c4240800,c4514ac0,0,...) at udp_send+0x58b
> sosend_dgram(c44f74b0,c4514ac0,e6772bd4,c4240800,0,...) at sosend_dgram+0x352
> sosend(c44f74b0,c4514ac0,e6772bd4,0,0,...) at sosend+0x54
> kern_sendit(c4496440,20,e6772c58,0,0,...) at kern_sendit+0x106
> sendit(0,1,e6772c54,28,c426a090,...) at sendit+0x162
> sendmsg(c4496440,e6772cfc,c,c4496630,c0bd53c0,...) at sendmsg+0x78
> syscall(e6772d38) at syscall+0x2b3
> Xint0x80_syscall() at Xint0x80_syscall+0x20
>
> Unfortunately, I was unable to successfully get a crashdump -- not entirely 
> sure why as it seemed to go to disk ok.
>
> Robert N M Watson
> Computer Laboratory
> University of Cambridge
>
>
>> 
>> ========================================================================
>> 
>> root_at_kobe:/var/crash# kgdb /boot/kernel/kernel vmcore.5
>> GNU gdb 6.1.1 [FreeBSD]
>> Copyright 2004 Free Software Foundation, Inc.
>> GDB is free software, covered by the GNU General Public License, and you 
>> are
>> welcome to change it and/or distribute copies of it under certain 
>> conditions.
>> Type "show copying" to see the conditions.
>> There is absolutely no warranty for GDB.  Type "show warranty" for details.
>> This GDB was configured as "i386-marcel-freebsd"...
>> 
>> Unread portion of the kernel message buffer:
>> panic: _mtx_lock_sleep: recursed on non-recursive mutex rtentry _at_ 
>> /home/build/src/sys/net/route.c:1742
>> 
>> cpuid = 0
>> Uptime: 5m26s
>> Physical memory: 2026 MB
>> Dumping 80 MB: 65 49 33 17 1
>> 
>> Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from 
>> /boot/kernel/snd_hda.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/snd_hda.ko
>> Reading symbols from /boot/kernel/sound.ko...Reading symbols from 
>> /boot/kernel/sound.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/sound.ko
>> Reading symbols from /boot/kernel/if_iwn.ko...Reading symbols from 
>> /boot/kernel/if_iwn.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/if_iwn.ko
>> Reading symbols from /boot/kernel/acpi.ko...Reading symbols from 
>> /boot/kernel/acpi.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/acpi.ko
>> Reading symbols from /boot/kernel/snake_saver.ko...Reading symbols from 
>> /boot/kernel/snake_saver.ko.symbols...done.
>> done.
>> Loaded symbols for /boot/kernel/snake_saver.ko
>> #0  doadump () at pcpu.h:221
>> 221     pcpu.h: No such file or directory.
>>        in pcpu.h
>> (kgdb) list
>> 216     in pcpu.h
>> (kgdb) bt
>> #0  doadump () at pcpu.h:221
>> #1  0xc05e13ac in boot (howto=260) at 
>> /home/build/src/sys/kern/kern_shutdown.c:418
>> #2  0xc05e1678 in panic (fmt=Variable "fmt" is not available.
>> ) at /home/build/src/sys/kern/kern_shutdown.c:572
>> #3  0xc05d3fda in _mtx_lock_sleep (m=0xc573eba4, tid=3314466816, opts=0, 
>> file=0xc08f457a "/home/build/src/sys/net/route.c", line=1742) at 
>> /home/build/src/sys/kern/kern_mutex.c:310
>> #4  0xc05d422f in _mtx_lock_flags (m=0xc573eba4, opts=0, file=0xc08f457a 
>> "/home/build/src/sys/net/route.c", line=1742) at 
>> /home/build/src/sys/kern/kern_mutex.c:182
>> #5  0xc0694ad8 in rt_check_fib (lrt=0xe7c299ec, lrt0=0xe7c29a08, 
>> dst=0xc5550710, fibnum=0) at /home/build/src/sys/net/route.c:1742
>> #6  0xc06caf36 in in_rt_check (lrt=0xe7c299ec, lrt0=0xe7c29a08, 
>> dst=0xc5550710, fibnum=0) at /home/build/src/sys/netinet/in_rmx.c:472
>> #7  0xc06c0ecd in arpresolve (ifp=0xc51fd800, rt0=0xc573eca8, m=0xc59c2200, 
>> dst=0xc5550710, desten=0xe7c29a22 "") at 
>> /home/build/src/sys/netinet/if_ether.c:388
>> #8  0xc0689a9e in ether_output (ifp=0xc51fd800, m=0xc59c2200, 
>> dst=0xc5550710, rt0=0xc573eca8) at 
>> /home/build/src/sys/net/if_ethersubr.c:183
>> #9  0xc06d1bf1 in ip_output (m=0xc59c2200, opt=0x0, ro=0xe7c29aa8, 
>> flags=Variable "flags" is not available.
>> ) at /home/build/src/sys/netinet/ip_output.c:563
>> #10 0xc073ecfb in udp_send (so=0xc573b498, flags=0, m=0xc59c2200, 
>> addr=0xc597e2f0, control=0x0, td=0xc58ec000) at 
>> /home/build/src/sys/netinet/udp_usrreq.c:1060
>> #11 0xc064530f in sosend_dgram (so=0xc573b498, addr=0xc597e2f0, 
>> uio=0xe7c29bd4, top=0xc59c2200, control=0x0, flags=Variable "flags" is not 
>> available.
>> ) at /home/build/src/sys/kern/uipc_socket.c:1059
>> #12 0xc0643054 in sosend (so=0xc573b498, addr=0xc597e2f0, uio=0xe7c29bd4, 
>> top=0x0, control=0x0, flags=0, td=0xc58ec000) at 
>> /home/build/src/sys/kern/uipc_socket.c:1292
>> #13 0xc064bf15 in kern_sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, 
>> flags=0, control=0x0, segflg=UIO_USERSPACE) at 
>> /home/build/src/sys/kern/uipc_syscalls.c:782
>> #14 0xc064c121 in sendit (td=0xc58ec000, s=516, mp=0xe7c29c54, flags=0) at 
>> /home/build/src/sys/kern/uipc_syscalls.c:719
>> #15 0xc064c1d1 in sendmsg (td=0xc58ec000, uap=0xe7c29cf8) at 
>> /home/build/src/sys/kern/uipc_syscalls.c:915
>> #16 0xc0884d13 in syscall (frame=0xe7c29d38) at 
>> /home/build/src/sys/i386/i386/trap.c:1090
>> #17 0xc0869020 in Xint0x80_syscall () at 
>> /home/build/src/sys/i386/i386/exception.s:261
>> #18 0x00000033 in ?? ()
>> Previous frame inner to this frame (corrupt stack?)
>> (kgdb)
>> 
>> ========================================================================
>> 
>> From the limited testing I could do today it seems that the following
>> changes might be useful to track down why this is happening:
>> 
>> /head_at_182698 -> ok so far
>> /head_at_182743 -> panic
>> 
>> I don't see any rt_check_fib() changes in this commit range, so it may
>> be false that /head_at_182698 is ok.  It just doesn't panic immediately
>> when I try to bring up my re0 interface and set the default route.
>> 
>> - Giorgos
>> 
>> 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>
Received on Sat Sep 13 2008 - 12:15:39 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:35 UTC