Re: diskless/unionfs panics

On Sat, 23 Apr 2005, Danny Braniss wrote:

> > On Fri, 22 Apr 2005, Danny Braniss wrote:
> >
> > > hi,
> > > 	after much debugging, it seems that the main problem with unionfs is
> > > that if it's called early in the boot process it will panic the kernel:
> > >
> > > trap 12: page fault while in kernel mode
> > > cpuid = 0; apic id = 00
> > > fault virtual address   = 0x0
> > > fault code              = supervisor read, page not present
> > > instruction pointer     = 0x8:0xffffffff8038e3f5
> > > stack pointer           = 0x10:0xffffffffb1eac7b0
> > > frame pointer           = 0x10:0xffffffffb1eac7e0
> > > code segment            = base 0x0, limit 0xfffff, type 0x1b
> > >                         = DPL 0, pres 1, long 1, def32 0, gran 1
> > > processor eflags        = interrupt enabled, resume, IOPL = 0
> > > current process         = 213 (sh)
> > > [thread pid 213 tid 100066 ]
> > > Stopped at      _mtx_lock_flags+0x35:   cmpq    $0x80779d40,0(%rdi)
> >
> > unintialized mutex, probably, although it looks like it'd be the vm page
> > queue mutex which should be init'd by then.
> >
> > Is this -CURRENT?
> yes, cvs'ed a few days ago (but the problem is not new).
>
> >
> > > db> tr
> > > Tracing pid 213 tid 100066 td 0xffffff007b9b1000
> > > _mtx_lock_flags() at _mtx_lock_flags+0x35
> > > exec_map_first_page() at exec_map_first_page+0x60
> >
> > If you have a debug kernel for this around, load it into gdb and 'disass
> > exec_map_first_page' and look around offset 96 to see if its referencing a
> > mutex (mtx) near there.
>
> arghh, gdb, is there a quick guide for this? im almost there, but
> can't sync speed (the console is at 38400).

Oh, don't bother trying to attach directly to the kernel, just look at the
kernel.debug binary , if you've got one.  If not, put

makeoptions	DEBUG=-g

in your kernel config so you have one next time.

>
> >
> > > kern_execve() at kern_execve+0x2a0
> > > execve() at execve+0x5d
> > > syscall() at syscall+0x4ab
> > > Xfast_syscall() at Xfast_syscall+0xa8
> > > --- syscall (59, FreeBSD ELF64, execve), rip = 0x80090630c, rsp =
> > > 0x7fffffffcbf8, rbp = 0 ---
> > >
> > > doing the unionfs stuff sometime later - after the single user prompt - seems
> > > do be ok.
> > >
> > > another thing, there are some LOR caused by the unionfs & md:
> > > extarct of rc.d/initdiskless:
> > >
> > > if [ -e /conf/union ]; then
> > >     if ! sysctl vfs.uniondebug >/dev/null 2>&1; then
> > > 	echo Loading unionfs
> > > 	kldload unionfs
> > >     fi
> > >     echo Doing UNIONFS
> > >     mount_md 4096 /conf/etc
> > >     chmod 755 /conf/etc
> > >     mount_unionfs /conf/etc /etc
> > >     ls -R /etc > /dev/null
> > >     touch /etc/.sentinel
> > >     md_created_etc=created
> > > fi
> > > ...
> > >
> > > Loading unionfs
> > > lock order reversal
> > >  1st 0xffffff007c3b00f0 system map (system map) _at_ /r+d/6.0/src/sys/vm/vm_map.c:
> > > 1100
> > >  2nd 0xffffff00623d8620 vm object (standard object) _at_
> > > /r+d/6.0/src/sys/vm/vm_map.c:897
> > > KDB: stack backtrace:
> > > witness_checkorder() at witness_checkorder+0x5f1
> > > _mtx_lock_flags() at _mtx_lock_flags+0x4a
> > > vm_map_insert() at vm_map_insert+0x115
> > > vm_map_find() at vm_map_find+0x9d
> > > link_elf_load_file() at link_elf_load_file+0x811
> > > linker_load_module() at linker_load_module+0x50e
> > > kldload() at kldload+0xc3
> > > syscall() at syscall+0x4ab
> > > Xfast_syscall() at Xfast_syscall+0xa8
> > > --- syscall (304, FreeBSD ELF64, kldload), rip = 0x80067950c, rsp =
> > > 0x7fffffffedf8, rbp = 0x7fffffffee68 ---
> > >
> > > Doing UNIONFS
> > > lock order reversal
> > >  1st 0xffffffff80892040 vm page queue mutex (vm page queue mutex) _at_
> > > /r+d/6.0/src/sys/kern/vfs_bio.c:1485
> > >  2nd 0xffffff0061dc98b0 vnode interlock (vnode interlock) _at_
> > > /r+d/6.0/src/sys/kern/vfs_subr.c:1992
> > > KDB: stack backtrace:
> > > witness_checkorder() at witness_checkorder+0x5f1
> > > _mtx_lock_flags() at _mtx_lock_flags+0x4a
> > > vdrop() at vdrop+0x31
> > > vm_page_remove() at vm_page_remove+0x126
> > > vm_page_free_toq() at vm_page_free_toq+0x61
> > > vm_page_free() at vm_page_free+0x1e
> > > vfs_vmio_release() at vfs_vmio_release+0x18f
> > > brelse() at brelse+0x792
> > > ffs_mount() at ffs_mount+0x121b
> > > vfs_donmount() at vfs_donmount+0xaa3
> > > kernel_mount() at kernel_mount+0xaf
> > > ffs_cmount() at ffs_cmount+0x92
> > > mount() at mount+0x132
> > > syscall() at syscall+0x1fb
> > > Xfast_syscall() at Xfast_syscall+0xa8
> > > --- syscall (21, FreeBSD ELF64, mount), rip = 0x80067e68c, rsp =
> > > 0x7fffffffdf98, rbp = 0x7fffffffea58 ---
> > >
> > >
> > > _______________________________________________
> > > freebsd-current_at_freebsd.org mailing list
> > > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> > >
> >
> > --
> > Doug White                    |  FreeBSD: The Power to Serve
> > dwhite_at_gumbysoft.com          |  www.FreeBSD.org
>
>

-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite_at_gumbysoft.com          |  www.FreeBSD.org