Re: -CURRENT panics on intensive fs operations.

On Sun, 2007-02-25 at 07:52 +0200, Kostik Belousov wrote:
> On Sun, Feb 25, 2007 at 12:41:44AM -0500, Alexandre Sunny Kovalenko wrote:
> > On Sat, 2007-02-24 at 21:55 +0200, Kostik Belousov wrote:
> > > On Thu, Feb 22, 2007 at 10:07:45AM -0500, Alexandre Sunny Kovalenko wrote:
> > > > On Tue, 2007-02-20 at 09:48 -0500, Alexandre "Sunny" Kovalenko wrote:
> > > > > On Wed, 2007-02-14 at 20:14 -0500, Kris Kennaway wrote:
> > > > > > On Wed, Feb 14, 2007 at 07:48:51PM -0500, Alexandre Sunny Kovalenko wrote:
> > > > > > > On Tue, 2007-02-13 at 20:17 -0500, Kris Kennaway wrote:
> > > > > > > > On Tue, Feb 13, 2007 at 08:02:39PM -0500, Alexandre Sunny Kovalenko wrote:
> > > > > > > > > I can reliably panic -CURRENT (Feb 11, noon EST) with the something that
> > > > > > > > > excersises the file system. I have currently settled on (cd /usr/ports;
> > > > > > > > > make clean), but it all started out as doing some "emerges" to test the
> > > > > > > > > latest linuxolator. In the case of the "make clean" I have seen it
> > > > > > > > > crashing as early as /usr/ports/audio and as late
> > > > > > > > > as /usr/ports/textproc. 
> > > > > > > > > 
> > > > I am still not capable to get good backtrace from the kernel dump, but I
> > > > have managed to hook up remote console to this machine, so here are
> > > > results:
> > > > 
> > > > db> bt
> > > > Tracing pid 33 tid 100032 td 0xc4cee510
> > > > kdb_enter(c067c69d) at kdb_enter+0x2b
> > > > panic(c0667ba3,c306d5c0,c306d5c0,e38a2cfc,c0619fd9,...) at panic+0x11c
> > > > vm_pageq_remove_nowakeup(c306d5c0,c061a0b8,e38a2d04,c061a0ee,e38a2d24,...) at vm_pageq_remove_nowakeup+0x35
> > > > vm_page_zero_idle(e38a2d24,c04c7fe4,0,e38a2d38,c4ef8900,...) at
> > > > vm_page_zero_idle+0x49
> > > > vm_pagezero(0,e38a2d38) at vm_pagezero+0x36
> > > > fork_exit(c061a0b8,0,e38a2d38) at fork_exit+0xac
> > > > fork_trampoline() at fork_trampoline+0x8
> > > > --- trap 0, eip = 0, esp = 0xe38a2d70, ebp = 0 ---
> > > > db> ps
> > > >   pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
> > > > <snip>
> > > >    33     0     0     0  RL      CPU 0               [pagezero]
> > > > <snip>
> > > > 
> > > > ... and (hopefully) relevant bits from the source
> > > > (kgdb) list *vm_pageq_remove_nowakeup+0x35
> > > > 0xc06192f9 is in vm_pageq_remove_nowakeup
> > > > (/usr/src/sys/vm/vm_pageq.c:223).
> > > > 218             struct vpgqueues *pq;
> > > > 219
> > > > 220             if (queue != PQ_NONE) {
> > > > 221                     pq = &vm_page_queues[queue];
> > > > 222                     VM_PAGE_SETQUEUE2(m, PQ_NONE);
> > > > 223                     TAILQ_REMOVE(&pq->pl, m, pageq);
> > > There, please, show the output of "p/x *m" and "p/x *pq".
> > > 
> > Unfortunately, with the latest -CURRENT the end result is different:
> > 
> > Kernel page fault with the following non-sleepable locks held:
> > exclusive sleep mutex sellck r = 0 (0xc0741284) locked
> > _at_ /usr/src/sys/kern/sys_generic.c:776
> > KDB: stack backtrace:
> > db_trace_self_wrapper(c067f599) at db_trace_self_wrapper+0x25
> > kdb_backtrace(1,c4fb46c0,c,e5447ab4,e5447aa8,...) at kdb_backtrace+0x29
> > witness_warn(5,0,c069ed6f) at witness_warn+0x192
> > trap(e5447ab4) at trap+0x10f
> > calltrap() at calltrap+0x6
> > --- trap 0xc, eip = 0x80bfe48d, esp = 0xe5447af4, ebp = 0xe5447c54 ---
> > kernload(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at -0x7f401b73
> > select(c4e02d80,e5447d00) at select+0x44
> > syscall(e5447d38) at syscall+0x256
> > Xint0x80_syscall() at Xint0x80_syscall+0x20
> > --- syscall (93, FreeBSD ELF32, select), eip = 0x2815f273, esp =
> > 0xbfbfe82c, ebp = 0xbfbfee48 ---
> > 
> > 
> > Fatal trap 12: page fault while in kernel mode
> > cpuid = 0; apic id = 00
> > fault virtual address   = 0x80bfe48d
> > fault code              = supervisor read, page not present
> > instruction pointer     = 0x20:0x80bfe48d
> > stack pointer           = 0x28:0xe5447af4
> > frame pointer           = 0x28:0xe5447c54
> > code segment            = base 0x0, limit 0xfffff, type 0x1b
> >                         = DPL 0, pres 1, def32 1, gran 1
> > processor eflags        = interrupt enabled, resume, IOPL = 0
> > current process         = 520 (powerd)
> > [thread pid 520 tid 100052 ]
> > Stopped at      -0x7f401b73:    *** error reading from address 80bfe48d
> > ***
> > db> 
> > 
> > db> where
> > Tracing pid 520 tid 100052 td 0xc4e02d80
> > kern_select(c4e02d80,5,bfbfedb0,0,bfbfedb0,...) at kern_select+0x4e5
> > select(c4e02d80,e5447d00) at select+0x44
> > syscall(e5447d38) at syscall+0x256
> > Xint0x80_syscall() at Xint0x80_syscall+0x20
> > --- syscall (93, FreeBSD ELF32, select), eip = 0x2815f273, esp =
> > 0xbfbfe82c, ebp = 0xbfbfee48 ---
> > db> 
> > 
> > what should I look for here?
> 
> Did you used some memory tester ? This looks like (random) memory corruption.
memtest86 was running for about 7 hours without a hiccup. If there are
any better ones, I would appreciate a suggestion.

> 
> Reported fault address 0x80bfe48d belongs to user part of VA. Could you,
> please, show the source line that corresponds the m in
> your compiled kernel ?
(kgdb) l *kern_select+0x4e5
0xc050a5dd is in kern_select (/usr/src/sys/kern/sys_generic.c:812).
807             if (error == 0)
808                     goto retry;
809
810     done:
811             clear_selinfo_list(td);
812             mtx_lock_spin(&sched_lock);
813             td->td_flags &= ~TDF_SELECT;
814             mtx_unlock_spin(&sched_lock);
815             mtx_unlock(&sellock);
816
(kgdb) p/x sched_lock
$1 = {mtx_object = {lo_name = 0xc067bf73, lo_type = 0xc067bf73, lo_flags
= 0x8b0000, 
    lo_witness_data = {lod_list = {stqe_next = 0xc0704310}, lod_witness
= 0xc0704310}}, 
  mtx_lock = 0x4, mtx_recurse = 0x0}
(kgdb)  

-- 
Alexandre "Sunny" Kovalenko