Re: PCcard NIC insert: "Fatal trap 12: page fault while in kernel mode"

On Friday 30 July 2010 22:43:12 David Wolfskill wrote:
> I thought I had mentioned this a while back, as I've been seeing it
> since at least 13 July, but a quick check didn't convince me otherwise.
>
> So I reproduced the problem yesterday, as of r210598: Thu Jul 29
> 17:54:37 PDT 2010.
>
> Symptom is that I get a panic on insert (or kernel probe, if it's
> inserted already at boot time) of a PCcard NIC.


VIMAGE kernels do not properly set curvnet context when dynamically attaching 
devices (such as USB or pccard NICs), that's why the dereferencing V_if_index 
fails.  You can use the "show pcpu" and "show vnets" DDB commands to 
determine whether curvnet context is properly set or not.

bz's vimage tree from p4://depot/user/bz/vimage/... aims at fixing those and 
many other initialization issues with VIMAGE kernels, though I don't know how 
stable it is currently...

Cheers,

Marko


> I've attached the core.txt file associated with the dump; here's the
> backtrace from it:
>
> #0  doadump () at pcpu.h:231
> 231     pcpu.h: No such file or directory.
>         in pcpu.h
> (kgdb) #0  doadump () at pcpu.h:231
> #1  0xc0890bfe in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416
> #2  0xc0890ed2 in panic (fmt=Variable "fmt" is not available.
> ) at /usr/src/sys/kern/kern_shutdown.c:590
> #3  0xc04d8647 in db_panic (addr=Could not find the frame base for
> "db_panic". ) at /usr/src/sys/ddb/db_command.c:478
> #4  0xc04d8c71 in db_command (last_cmdp=0xc0e08edc, cmd_table=0x0,
> dopager=1) at /usr/src/sys/ddb/db_command.c:445
> #5  0xc04d8dca in db_command_loop () at /usr/src/sys/ddb/db_command.c:498
> #6  0xc04daced in db_trap (type=12, code=0) at
> /usr/src/sys/ddb/db_main.c:229 #7  0xc08c37b6 in kdb_trap (type=12, code=0,
> tf=0xc52a2a9c)
>     at /usr/src/sys/kern/subr_kdb.c:535
> #8  0xc0beb7af in trap_fatal (frame=0xc52a2a9c, eva=24)
>     at /usr/src/sys/i386/i386/trap.c:936
> #9  0xc0bebcdc in trap (frame=0xc52a2a9c) at
> /usr/src/sys/i386/i386/trap.c:326 #10 0xc0bd303c in calltrap () at
> /usr/src/sys/i386/i386/exception.s:166 #11 0xc0944a35 in
> ifindex_alloc_locked (idxp=0xc52a2b16)
>     at /usr/src/sys/net/if.c:262
> #12 0xc0945232 in if_alloc (type=6 '\006') at /usr/src/sys/net/if.c:403
> #13 0xc11ad9ce in an_attach (sc=0xc8e69000, flags=0)
>     at /usr/src/sys/modules/an/../../dev/an/if_an.c:683
> #14 0xc11b205a in an_pccard_attach (dev=0xc7e63380)
>     at /usr/src/sys/modules/an/../../dev/an/if_an_pccard.c:146
> #15 0xc08be12f in device_attach (dev=0xc7e63380) at device_if.h:178
> #16 0xc06e556b in pccard_probe_and_attach_child (dev=0x22,
> child=0xc7e63380, pf=0xc8e5b280) at /usr/src/sys/dev/pccard/pccard.c:297
> #17 0xc06e5a1b in pccard_attach_card (dev=0xc79b5d80)
>     at /usr/src/sys/dev/pccard/pccard.c:249
> #18 0xc064225a in exca_insert (exca=0xc783d804) at card_if.h:83
> #19 0xc06ea26f in cbb_event_thread (arg=0xc783d800)
>     at /usr/src/sys/dev/pccbb/pccbb.c:557
> #20 0xc08664a8 in fork_exit (callout=0xc06e9fd0 <cbb_event_thread>,
>     arg=0xc783d800, frame=0xc52a2d28) at /usr/src/sys/kern/kern_fork.c:843
> #21 0xc0bd30b4 in fork_trampoline () at
> /usr/src/sys/i386/i386/exception.s:273
>
>
> I've been looking around, mostly in frame #11:
>
> (kgdb) frame 11
> #11 0xc0944a35 in ifindex_alloc_locked (idxp=0xc52a2b16) at
> /usr/src/sys/net/if.c:262 262             for (idx = 1; idx <= V_if_index;
> idx++) {
> (kgdb) list -
> 252     ifindex_alloc_locked(u_short *idxp)
> 253     {
> 254             u_short idx;
> 255
> 256             IFNET_WLOCK_ASSERT();
> 257
> 258             /*
> 259              * Try to find an empty slot below V_if_index.  If we fail,
> take the 260              * next slot.
> 261              */
> (kgdb) list
> 262             for (idx = 1; idx <= V_if_index; idx++) {
> 263                     if (V_ifindex_table[idx].ife_ifnet == NULL)
> 264                             break;
> 265             }
> 266
> 267             /* Catch if_index overflow. */
> 268             if (idx < 1)
> 269                     return (ENOSPC);
> 270             if (idx > V_if_index)
> 271                     V_if_index = idx;
> (kgdb)
>
> but I confess I have little expertise for doing much there myself.
>
> I had been advised to sprinkle printf()s in there, which I had tried;
> they appeared to demonstrate that the IFNET_WLOCK_ASSERT() on line 256
> succeeded, but the printf() that I stuck in at the beginning of the for
> loop on present line 263 didn't generate any output that I saw.
>
> (I have since taken them out, and the dump in question reflects the
> reverted state of sys/net/if.c.)
>
> The panic appears to be quite reproducible, and I'm more than willing
> to experiment.  (I find that the least iinconvenient way to reproduce
> the panic is to boot normally -- without a PCcard -- then "shutdown
> now", re-mount all the mounted file systems as read-only, then
> insert the PCcard.  Trying to catch it during the kernel probes loses,
> as apparently the swap space hasn't yet been added to the system -- at
> least, trying to "panic" merely whines that it has no place to record
> the dump.  In contrast, creating the panic immediately after shutdown
> doesn't suffer that problem, as swap had been added earlier.)
>
> Thanks!
>
> Peace,
> david