Re: kernel panic by enabling net.inet.ip.random_id

From: Shawn Webb <shawn.webb_at_hardenedbsd.org>
Date: Wed, 6 Jan 2016 09:46:08 -0500
(kgdb) list *(0xffffffff80b5de9e)
0xffffffff80b5de9e is in ip_fillid (/usr/src/sys/netinet/ip_id.c:237).
warning: Source file is more recent than executable.

232             new_id = 0;
233             do {
234                     if (new_id != 0)
235                             V_random_id_collisions++;
236                     arc4rand(&new_id, sizeof(new_id), 0);
237             } while (bit_test(V_id_bits, new_id) || new_id == 0);
238             bit_clear(V_id_bits, V_id_array[V_array_ptr]);
239             bit_set(V_id_bits, new_id);
240             V_id_array[V_array_ptr] = new_id;
241             V_array_ptr++;

This is the change I made to ip_id.c that caused the underlying kernel
panic:
https://github.com/HardenedBSD/hardenedBSD/commit/52d5a93b92097e7a79be8d2e0eb9c1a58b8337d1

Ideally, we should be able to just toggle that variable and all would be
well. But it seems with the VIMAGE work, something is preventing that.

Thanks,

Shawn

On Tue, Jan 05, 2016 at 06:22:34PM -0800, Adrian Chadd wrote:
> try list *(0x[address]) .
> 
> That line is mtx_unlock(), which makes no sense (as mtx_lock succeeded fine.)
> 
> 
> -a
> 
> 
> On 5 January 2016 at 18:13, Shawn Webb <shawn.webb_at_hardenedbsd.org> wrote:
> > Thanks for the quick reply! Here's some more debugging output:
> >
> > === Begin Log ===
> > (kgdb) bt
> > #0  doadump (textdump=0) at pcpu.h:221
> > #1  0xffffffff8037c78b in db_dump (dummy=<value optimized out>, dummy2=false, dummy3=0, dummy4=0x0) at /usr/src/sys/ddb/db_command.c:533
> > #2  0xffffffff8037c57e in db_command (cmd_table=0x0) at /usr/src/sys/ddb/db_command.c:440
> > #3  0xffffffff8037c314 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493
> > #4  0xffffffff8037edab in db_trap (type=<value optimized out>, code=0) at /usr/src/sys/ddb/db_main.c:251
> > #5  0xffffffff80a5c563 in kdb_trap (type=12, code=0, tf=<value optimized out>) at /usr/src/sys/kern/subr_kdb.c:654
> > #6  0xffffffff80e6b7e1 in trap_fatal (frame=0xfffffe02c33894d0, eva=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:829
> > #7  0xffffffff80e6ba2d in trap_pfault (frame=0xfffffe02c33894d0, usermode=<value optimized out>) at /usr/src/sys/amd64/amd64/trap.c:684
> > #8  0xffffffff80e6b15f in trap (frame=0xfffffe02c33894d0) at /usr/src/sys/amd64/amd64/trap.c:435
> > #9  0xffffffff80e4af97 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234
> > #10 0xffffffff80b5de9e in ip_fillid (ip=0xfffff8000ef8cb88) at /usr/src/sys/netinet/ip_id.c:237
> > #11 0xffffffff80b6c41b in ip_output (m=<value optimized out>, opt=<value optimized out>, ro=<value optimized out>, flags=0, imo=0x0, inp=0xfffff8000e66e960) at /usr/src/sys/netinet/ip_output.c:268
> > #12 0xffffffff80bf0612 in udp_send (so=<value optimized out>, flags=<value optimized out>, m=<value optimized out>, addr=0x0, control=<value optimized out>, td=0xfffff8000ef8cb88) at /usr/src/sys/netinet/udp_usrreq.c:1517
> > #13 0xffffffff80aa3872 in sosend_dgram (so=0xfffff8000e6422e8, addr=0x0, uio=<value optimized out>, top=0xfffff8000ef8cb00, control=0x0, flags=<value optimized out>, td=0xffffffff81bef2ec) at /usr/src/sys/kern/uipc_socket.c:1164
> > #13 0xffffffff80aa3872 in sosend_dgram (so=0xfffff8000e6422e8, addr=0x0, uio=<value optimized out>, top=0xfffff8000ef8cb00, control=0x0, flags=<value optimized out>, td=0xffffffff81bef2ec) at /usr/src/sys/kern/uipc_socket.c:1164
> > #14 0xffffffff80aaa03b in kern_sendit (td=0xfffff8000e4cd9c0, s=6, mp=<value optimized out>, flags=0, control=0x0, segflg=UIO_USERSPACE) at /usr/src/sys/kern/uipc_syscalls.c:906
> > #15 0xffffffff80aaa336 in sendit (td=0xfffff8000e4cd9c0, s=<value optimized out>, mp=0xfffffe02c3389970, flags=3980) at /usr/src/sys/kern/uipc_syscalls.c:833
> > #16 0xffffffff80aaa1fd in sys_sendto (td=0x0, uap=<value optimized out>) at /usr/src/sys/kern/uipc_syscalls.c:957
> > #17 0xffffffff80e6bfdb in amd64_syscall (td=0xfffff8000e4cd9c0, traced=0) at subr_syscall.c:135
> > #18 0xffffffff80e4b27b in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:394
> > #19 0x000003e339782e8a in ?? ()
> > (kgdb) x/i 0xffffffff80b5de9e
> > 0xffffffff80b5de9e <ip_fillid+142>:     movzbl (%rax,%rcx,1),%esi
> > (kgdb) info reg
> > rax            0x0      0
> > rbx            0x0      0
> > rcx            0x0      0
> > rdx            0x0      0
> > rsi            0x0      0
> > rdi            0x0      0
> > rbp            0xfffffe02c3388fe0       0xfffffe02c3388fe0
> > rsp            0xfffffe02c3388fc8       0xfffffe02c3388fc8
> > r8             0x0      0
> > r9             0x0      0
> > r10            0x0      0
> > r11            0x0      0
> > r12            0xffffffff817c0b80       -2122577024
> > r13            0xffffffff817c1470       -2122574736
> > r14            0x1      1
> > r15            0x4      4
> > rip            0xffffffff80a1fae3       0xffffffff80a1fae3 <doadump+51>
> > eflags         0x0      0
> > cs             0x0      0
> > ss             0x0      0
> > ds             0x0      0
> > es             0x0      0
> > fs             0x0      0
> > gs             0x0      0
> > === End Log ===
> >
> > Thanks,
> >
> > Shawn
> >
> > On Tue, Jan 05, 2016 at 06:06:41PM -0800, Adrian Chadd wrote:
> >> looks like a null pointer deference. What's kgdb show at that IP?
> >>
> >>
> >> -a
> >>
> >>
> >> On 5 January 2016 at 17:57, Shawn Webb <shawn.webb_at_hardenedbsd.org> wrote:
> >> > Hey All,
> >> >
> >> > Here's a kernel panic I'm experiencing by enabling net.inet.ip.random_id
> >> > at boot.
> >> >
> >> > I'm on latest HEAD on amd64 in bhyve. I'll soon-ish be testing on native
> >> > hardware with VIMAGE enabled.
> >> >
> >> > === Begin Log ===
> >> > Kernel page fault with the following non-sleepable locks held:
> >> > exclusive sleep mutex ip_id_mtx (ip_id_mtx) r = 0 (0xffffffff81c54830) locked _at_ /usr/src/sys/netinet/ip_id.c:227
> >> > stack backtrace:
> >> > #0 0xffffffff80a79620 at witness_debugger+0x70
> >> > #1 0xffffffff80a7a937 at witness_warn+0x3d7
> >> > #2 0xffffffff80e6b887 at trap_pfault+0x57
> >> > #3 0xffffffff80e6b15f at trap+0x4bf
> >> > #4 0xffffffff80e4af97 at calltrap+0x8
> >> > #5 0xffffffff80b6c41b at ip_output+0x16b
> >> > #6 0xffffffff80b68e82 at icmp_reflect+0x5b2
> >> > #7 0xffffffff80b6883f at icmp_error+0x46f
> >> > #8 0xffffffff80beeb12 at udp_input+0x982
> >> > #9 0xffffffff80b69d1d at ip_input+0x17d
> >> > #10 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> >> > #11 0xffffffff80afecce at ether_demux+0x15e
> >> > #12 0xffffffff80affa14 at ether_nh_input+0x344
> >> > #13 0xffffffff80b08ba1 at netisr_dispatch_src+0x81
> >> > #14 0xffffffff80afefcf at ether_input+0x4f
> >> > #15 0xffffffff8089a5c3 at vtnet_rxq_eof+0x823
> >> > #16 0xffffffff8089b2ce at vtnet_rx_vq_intr+0x4e
> >> > #17 0xffffffff809e9ba6 at intr_event_execute_handlers+0x96
> >> >
> >> >
> >> > Fatal trap 12: page fault while in kernel mode
> >> > cpuid = 6; apic id = 06
> >> > fault virtual address   = 0x5bd
> >> > fault code              = supervisor read data, page not present
> >> > instruction pointer     = 0x20:0xffffffff80b5de9e
> >> > stack pointer           = 0x28:0xfffffe02b8d483e0
> >> > frame pointer           = 0x28:0xfffffe02b8d48410
> >> > code segment            = base 0x0, limit 0xfffff, type 0x1b
> >> >                         = DPL 0, pres 1, long 1, def32 0, gran 1
> >> > processor eflags        = interrupt enabled, resume, IOPL = 0
> >> > current process         = 12 (irq265: virtio_pci0)
> >> > [ thread pid 12 tid 100040 ]
> >> > Stopped at      ip_fillid+0x8e: movzbl  (%rax,%rcx,1),%esi
> >> > === End Log ===
> >> >
> >> > Thanks,
> >> >
> >> > --
> >> > Shawn Webb
> >> > HardenedBSD
> >> >
> >> > GPG Key ID:          0x6A84658F52456EEE
> >> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE
> >
> > --
> > Shawn Webb
> > HardenedBSD
> >
> > GPG Key ID:          0x6A84658F52456EEE
> > GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

-- 
Shawn Webb
HardenedBSD

GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE

Received on Wed Jan 06 2016 - 13:46:12 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:02 UTC