Kris Kennaway wrote: > On Sat, Jun 16, 2007 at 12:51:41PM +0930, Benjamin Close wrote: > >> Kris Kennaway wrote: >> >>> On Fri, Jun 15, 2007 at 11:00:04PM +0930, Benjamin Close wrote: >>> >>> >>>> Hi All, >>>> Whilst running out of memory compiling Xorg (scanPCI is a killer) I >>>> discovered a quick way to deadlock the system: >>>> >>>> dd if=/dev/zero of=somefileonzfs bs=something count=something >>>> mdconfig -a -f something >>>> swapon /dev/md0 >>>> >>>> Then do something that needs swap.. instant deadlock. The system is >>>> still responsive but all disk access become hung. >>>> >>>> Known issue? If so is there a way we can warn users/prevent users from >>>> doing it? >>>> >>>> >>> Enable DEBUG_VFS_LOCKS and DEBUG_LOCKS, then break to DDB when the >>> deadlock occurs and do 'show lockedvnods'. >>> >>> >> Ok, enabled the above and this time got: >> >> swap_pager: indefinite wait buffer: bufobj: 0, blkno: 312865, size: 16384 >> >> just before the deadlock: >> >> Show locked vnods returns (hand transcribed) >> >> 0xffffff002c3ab5d0: tagz zfs, type VREG >> usecout 1, writecount 1, refcount 2 mountedhere 0 >> flags() >> v_object 0xffffff002f047c80 ref 0 pages 0 >> lock type zfs: EXCL (count 1) by thread 0xffffff0030573360 (pid 1188) >> > > What was needed was the backtrace that should have been also displayed > here. > > oops, made the kernel, forgot to install it, new trace below - looks like this one is not directly zil related. >> Pid 1188 is: >> >> 1188 0 0 0 SL zfs:(&zi 0xffffff000208fd58 [md0] >> > > Also a trace of the current backtrace of this. However based on the > single byte of information 'i' I predict that this is due to zil (ZFS > intent logging). I have turned this off on my machines because of > deadlock conditions during low memory, which is also going to be true > on your system. Try adding to /boot/loader.conf > > vfs.zfs.zil_disable=1 > > So this is a known issue then. >> called doadump and though it went through the motions, savecore didn't >> find anything saved. >> Not sure what you need debugging wise, let me know. >> >> System is Intel Core 2 duo, running in SMP amd64, updated Friday 15th June. >> The box has 1G physical ram, 1G dedicated swap partition, but needs an >> extra 300M to compile xf86ScanPCI.c (freaky!). >> > > Or just use -O0 on this file. > In the end I did use -O0 but figured the report would be more worthwhile, it's certainly a issue for when zfs becomes non experimental. Here's the latest trace: 0xffffff0023daa20: tag zfs, type VREG usecount 1, writecount 1, refcount 2, mounted here0 flags() vobject 0xffffff002df1cbb8 ref 0 pages 0 lock type zfs: EXCL (count 1) by thread 0xffffff002ed5b000 (pid 1033) #0 0xfffffff8045f62b at _lockmgr+0x4cb #1 at ... VOP_LOCK1_APV+0x79 #2 at ... _vn_lock+0x94 #3 at ... mdstart_vnode+0x15f #4 at ... md_kthread+0x111 #5 at ... fork_exit #6 at ... fork_trampoline 1033 0 0 0 SL vmwait 0xffffffff80a573b8 [md0] Trace 1033 sched_switch()+0x15e mi_switch+0x231 sleepq_switch+0xc7 sleepq_wait+0x44 _sleep+0x327 kmem_malloc+0x2a2 uma_large_malloc+0x4a malloc+0x12d arc_get_data_buf+0x36e arc_buf_alloc+0xe4 arc_read+0fa dbuf_read+0x43a dmu_tx_check_ioerr+09a dmu_tx_count_write+0x178 dmu_tx_hold_write+0x4a zfs_freebsd_write+0x399 VOP_WRITE_APV+0xe5 mdstart_vnod()+0x1a5 md_kthread+0x111 fork_exit fork_trampoline Looks like zfs is wanting to write but in order to do it needs memory so asks md for it. md needs to page something out in order to be able to provide it. Since md is zfs file backed - deadlock. Got a dump this time but the stack appears corrupt .... hmm, haven't got a valid dump for a while now, wonder if something else is up. mdstart_vnode+0x15f: 564 vn_lock(vp,KL_EXCLUSIVE | LK_RETY, td ); Cheers, BenjaminReceived on Sat Jun 16 2007 - 07:57:32 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:12 UTC