Strange kernel trap 12 with vm_page_splay() on FreeBSD/i386 SMP 7.0-RC1

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Recently I have encountered a strange kernel trap 12, which always end
up at vm_page_splay() like this, the fault va and IP vary from time to time:

Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 03
fault virtual address	= 0xfff9a4b5
fault code		= supervisor read, page not present
instruction pointer	= 0x20:0xc073eec7
stack pointer	        = 0x28:0xe66fa94c
frame pointer	        = 0x28:0xe66fa9a4
code segment		= base 0x0, limit 0xfffff, type 0x1b
			= DPL 0, pres 1, def32 1, gran 1
processor eflags	= interrupt enabled, resume, IOPL = 0
current process		= 1123 (mysqld)
trap number		= 12
panic: page fault
cpuid = 3
Uptime: 10m6s
Physical memory: 1015 MB
Dumping 130 MB: 115 99 83 67 51 35 19 3

And the backtrace was:

#0  doadump () at pcpu.h:195
#1  0xc0566b37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0566df9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0777eec in trap_fatal (frame=0xe66fa90c, eva=4294550709) at
/usr/src/sys/i386/i386/trap.c:899
#4  0xc0778150 in trap_pfault (frame=0xe66fa90c, usermode=0,
eva=4294550709) at /usr/src/sys/i386/i386/trap.c:812
#5  0xc0778aa2 in trap (frame=0xe66fa90c) at
/usr/src/sys/i386/i386/trap.c:490
#6  0xc075f47b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc073eec7 in vm_page_splay (pindex=Unhandled dwarf expression
opcode 0x93
) at /usr/src/sys/vm/vm_page.c:576
#8  0xc073efad in vm_page_lookup (object=0xc4513a2c, pindex=Unhandled
dwarf expression opcode 0x93
) at /usr/src/sys/vm/vm_page.c:759
#9  0xc05ca4f6 in allocbuf (bp=0xd7d20f20, size=16384) at
/usr/src/sys/kern/vfs_bio.c:2884
#10 0xc05cddcd in getblk (vp=0xc44a5330, blkno=4379, size=16384,
slpflag=0, slptimeo=0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/vfs_bio.c:2662
#11 0xc05d08d7 in cluster_read (vp=0xc44a5330, filesize=104590932,
lblkno=4380, size=16384, cred=0x0, totread=5240, seqcount=0, bpp=0xe66fab80)
    at /usr/src/sys/kern/vfs_cluster.c:118
#12 0xc0717d20 in ffs_read (ap=0xe66fabc8) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:511
#13 0xc0782c72 in VOP_READ_APV (vop=0xc07f7140, a=0xe66fabc8) at
vnode_if.c:637
#14 0xc05ee5f4 in vn_read (fp=0xc3e09c60, uio=0xe66fac60,
active_cred=0xc4142500, flags=0, td=0xc41d9880) at vnode_if.h:344
#15 0xc059b5e6 in dofileread (td=0xc41d9880, fd=106, fp=0xc3e09c60,
auio=0xe66fac60, offset=-1, flags=0) at file.h:242
#16 0xc059b958 in kern_readv (td=0xc41d9880, fd=106, auio=0xe66fac60) at
/usr/src/sys/kern/sys_generic.c:192
#17 0xc059ba3f in read (td=0xc41d9880, uap=0xe66facfc) at
/usr/src/sys/kern/sys_generic.c:108
#18 0xc0778489 in syscall (frame=0xe66fad38) at
/usr/src/sys/i386/i386/trap.c:1035
#19 0xc075f4e0 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

Another one:

#0  doadump () at pcpu.h:195
#1  0xc0566b37 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2  0xc0566df9 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3  0xc0777eec in trap_fatal (frame=0xe68fd90c, eva=9437986) at
/usr/src/sys/i386/i386/trap.c:899
#4  0xc0778150 in trap_pfault (frame=0xe68fd90c, usermode=0,
eva=9437986) at /usr/src/sys/i386/i386/trap.c:812
#5  0xc0778aa2 in trap (frame=0xe68fd90c) at
/usr/src/sys/i386/i386/trap.c:490
#6  0xc075f47b in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7  0xc073ef09 in vm_page_splay (pindex=Unhandled dwarf expression
opcode 0x93
) at /usr/src/sys/vm/vm_page.c:590
#8  0xc073efad in vm_page_lookup (object=0xc4623934, pindex=Unhandled
dwarf expression opcode 0x93
) at /usr/src/sys/vm/vm_page.c:759
#9  0xc05ca4f6 in allocbuf (bp=0xd7d0b640, size=16384) at
/usr/src/sys/kern/vfs_bio.c:2884
#10 0xc05cddcd in getblk (vp=0xc4827aa0, blkno=63, size=16384,
slpflag=0, slptimeo=0, flags=Variable "flags" is not available.
) at /usr/src/sys/kern/vfs_bio.c:2662
#11 0xc05d08d7 in cluster_read (vp=0xc4827aa0, filesize=3393192,
lblkno=64, size=16384, cred=0x0, totread=15352, seqcount=0, bpp=0xe68fdb80)
    at /usr/src/sys/kern/vfs_cluster.c:118
#12 0xc0717d20 in ffs_read (ap=0xe68fdbc8) at
/usr/src/sys/ufs/ffs/ffs_vnops.c:511
#13 0xc0782c72 in VOP_READ_APV (vop=0xc07f7140, a=0xe68fdbc8) at
vnode_if.c:637
#14 0xc05ee5f4 in vn_read (fp=0xc45941b0, uio=0xe68fdc60,
active_cred=0xc3e35000, flags=0, td=0xc4f05660) at vnode_if.h:344
#15 0xc059b5e6 in dofileread (td=0xc4f05660, fd=176, fp=0xc45941b0,
auio=0xe68fdc60, offset=-1, flags=0) at file.h:242
#16 0xc059b958 in kern_readv (td=0xc4f05660, fd=176, auio=0xe68fdc60) at
/usr/src/sys/kern/sys_generic.c:192
#17 0xc059ba3f in read (td=0xc4f05660, uap=0xe68fdcfc) at
/usr/src/sys/kern/sys_generic.c:108
#18 0xc0778489 in syscall (frame=0xe68fdd38) at
/usr/src/sys/i386/i386/trap.c:1035
#19 0xc075f4e0 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#20 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)

Because this is observed on two servers, it smells like a software
issue, but I have not yet completely ruled hardware factor out.  Anyone
has some debugging hints for this?

Cheers,
- --
Xin LI <delphij_at_delphij.net>	http://www.delphij.net/
FreeBSD - The Power to Serve!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.4 (FreeBSD)

iD8DBQFHfZIvi+vbBBjt66ARAqqnAJ9QgJnKER7GmxmehDUeG4oUI+9JBgCfdjrv
0RGvBHUAUUp8si/Mc9Z6qck=
=DD59
-----END PGP SIGNATURE-----