I upgraded a box to a recent 6.x this morning, and ran into the following apparently deadlock while backing up a partitionusing dump -L. It appears to have gotten stuck during mksnap_ffs. Some interpretation below, but the high level summary is that it looks to me like we have a lock order issue between the snapshot write lock for the file system and vnode locks, which occurs because vrele() calls VOP_INACTIVE with the vnode lock held, and ufs_inactive() then proceeds to get a write reference to the file system. It could be a fix is to grab the write reference to the file system in vrele() before the vnode lock, to aovid grabbing the write reference in ufs_inactive(). Robert N M Watson www:~> more backup.csh #!/bin/csh set DATE=`date +"%Y%m%d"` dump -L -f - -a -u -0 /home/pnn | gzip -9 > ${DATE}-backup.dump.gz www:~> df Filesystem 1K-blocks Used Avail Capacity Mounted on /dev/twed0s1a 507630 103502 363518 22% / devfs 1 1 0 100% /dev /dev/twed0s1f 89359640 370128 81840742 0% /home/pnn /dev/twed0s1e 16244334 3211132 11733656 21% /usr /dev/twed0s1d 8122126 1541978 5930378 21% /var www:~> ./backup.csh load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k load: 0.00 cmd: mksnap_ffs 2993 [ufs] 0.00u 3.88s 0% 496k On the serial console: telnet> send break KDB: enter: Line break on console [thread pid 11 tid 100004 ] Stopped at kdb_enter+0x2b: nop db> show lockedvnods Locked vnodes 0xc13c49b4: tag ufs, type VDIR usecount 153, writecount 0, refcount 1 mountedhere 0 flags (VV_ROOT) v_object 0 lock type ufs: EXCL (count 1) by thread 0xc1807a10 (pid 3057) with 152 pend ing ino 2, on dev twed0s1f (229, 9) 0xc15268a0: tag ufs, type VDIR usecount 2, writecount 0, refcount 1 mountedhere 0 flags () v_object 0 lock type ufs: EXCL (count 1) by thread 0xc137e000 (pid 3062) with 1 pendin g ino 3085322, on dev twed0s1f (229, 9) 0xc163bbdc: tag ufs, type VDIR usecount 2, writecount 0, refcount 1 mountedhere 0 flags () v_object 0 lock type ufs: EXCL (count 1) by thread 0xc162dcf0 (pid 3056) with 1 pendin g ino 3133686, on dev twed0s1f (229, 9) 0xc17e933c: tag ufs, type VREG usecount 1, writecount 0, refcount 0 mountedhere 0 flags () v_object 0xc17e7210 lock type ufs: EXCL (count 1) by thread 0xc1996450 (pid 3045) with 1 pendin g ino 3134802, on dev twed0s1f (229, 9) 0xc18419b4: tag ufs, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 flags () v_object 0xc1826000 lock type ufs: EXCL (count 1) by thread 0xc19b2450 (pid 3047) ino 3132762, on dev twed0s1f (229, 9) 0xc1ac933c: tag ufs, type VREG usecount 1, writecount 0, refcount 865 mountedhere 0 flags () v_object 0 lock type ufs: EXCL (count 1) by thread 0xc162d8a0 (pid 2993) ino 9891841, on dev twed0s1f (229, 9) 0xc19fd9b4: tag ufs, type VREG usecount 0, writecount 0, refcount 0 mountedhere 0 flags () v_object 0xc1a93948 lock type ufs: EXCL (count 1) by thread 0xc1807b80 (pid 2609) ino 3197600, on dev twed0s1f (229, 9) db> ps pid proc uid ppid pgrp flag stat wmesg wchan cmd 3296 c1cf93f8 1062 3292 3292 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3295 c1cf91fc 1062 3293 3293 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3293 c1cf9be8 1062 3290 3293 0004000 [SLPQ wait 0xc1cf9be8][SLP] sh 3292 c1d66000 1062 3289 3292 0004000 [SLPQ wait 0xc1d66000][SLP] sh 3290 c19b0000 0 390 390 0000000 [SLPQ piperd 0xc1655480][SLP] cron 3289 c1d661fc 0 390 390 0000000 [SLPQ piperd 0xc19a3d80][SLP] cron 3271 c1d66be8 1062 3268 3268 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3268 c1d69000 1062 3267 3268 0004000 [SLPQ wait 0xc1d69000][SLP] sh 3267 c1d669ec 0 390 390 0000000 [SLPQ piperd 0xc13aac00][SLP] cron 3247 c1cf95f4 1062 3245 3245 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3245 c1cf99ec 1062 3243 3245 0004000 [SLPQ wait 0xc1cf99ec][SLP] sh 3243 c1cf9de4 0 390 390 0000000 [SLPQ piperd 0xc13aa480][SLP] cron 3227 c1d66de4 1062 3222 3222 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3226 c1c8f1fc 1062 3223 3223 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3223 c18063f8 1062 3218 3223 0004000 [SLPQ wait 0xc18063f8][SLP] sh 3222 c1805000 1062 3217 3222 0004000 [SLPQ wait 0xc1805000][SLP] sh 3218 c19ae000 0 390 390 0000000 [SLPQ piperd 0xc13a9a80][SLP] cron 3217 c18067f0 0 390 390 0000000 [SLPQ piperd 0xc13a9300][SLP] cron 3197 c1c8f3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3196 c1c8f5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3195 c1c8f7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3194 c1c8f9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3193 c1c8fbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3192 c1c8fde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3191 c1cf6000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3190 c1cf61fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3189 c1cf63f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3188 c1cf65f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3187 c1cf67f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3186 c1cf69ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3185 c1cf6be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3184 c1cf6de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3183 c1cf9000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3182 c1c8e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3181 c1c8e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3180 c1c8e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3179 c1c1e3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3178 c1c1e5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3177 c1c1e7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3176 c1c1e9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3175 c1c1ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3174 c1c1ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3173 c1c88000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3172 c1c881fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3171 c1c883f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3170 c1c885f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3169 c1c887f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3168 c1c889ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3167 c1c88be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3166 c1c88de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3165 c1c8e000 1062 3163 3163 4000000 [SLPQ ufs 0xc13c4a78][SLP] sh 3163 c1c8e3f8 1062 3161 3163 0004000 [SLPQ wait 0xc1c8e3f8][SLP] sh 3161 c1c8e7f0 0 390 390 0000000 [SLPQ piperd 0xc19a3000][SLP] cron 3159 c1c8ebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3158 c1c8ede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3157 c1c8f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3156 c1bc37f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3155 c1bc39ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3154 c1bc3be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3153 c1bc3de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3152 c1c19000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3151 c1c191fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3150 c1c193f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3149 c1c195f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3148 c1c197f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3147 c1c199ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3146 c1c19be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3145 c1c19de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3144 c1c1a000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3143 c1c1a1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3142 c1c1a3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3141 c1c1a5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3140 c1c1a7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3139 c1c1a9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3138 c1c1abe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3137 c1c1ade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3136 c1c1e000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3135 c1c1e1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3134 c1bc1de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3133 c1bc1be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3132 c1b837f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3131 c1b839ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3130 c1b83be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3129 c1b83de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3128 c1bc1000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3127 c1bc11fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3126 c1bc13f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3125 c1bc15f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3124 c1bc17f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3123 c1bc19ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3120 c1bc3000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3119 c1bc31fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3118 c1bc33f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3117 c1bc35f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3116 c1abb7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3115 c1abb9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3114 c1abbbe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3113 c1abbde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3112 c1b82000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3111 c1b821fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3110 c1b823f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3109 c1b825f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3108 c1b827f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3107 c1b829ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3106 c1b82be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3105 c1b82de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3104 c1b83000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3103 c1b831fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3102 c1b833f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3101 c1b835f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3100 c19b07f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3099 c19b09ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3098 c19b0be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3097 c19b0de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3096 c1aba000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3095 c1aba1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3094 c1aba3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3093 c1aba5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3092 c1aba7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3091 c1aba9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3090 c1ababe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3089 c1abade4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3088 c1abb000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3087 c1abb1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3086 c1abb3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3085 c16291fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3084 c1abb5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3083 c1510be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3082 c162c3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3081 c18061fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3080 c19957f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3079 c18055f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3078 c19aede4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3077 c19959ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3076 c19aebe8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3075 c16297f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3074 c16293f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3073 c19ae5f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3072 c19ae1fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3071 c19951fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3070 c19ae7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3069 c12ab7f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3068 c1510de4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3067 c1805be8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3066 c1994000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3065 c19ae9ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3064 c137d3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3063 c150f000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3062 c1510000 80 2910 403 0004000 [SLPQ ufs 0xc163bca0][SLP] download.cgi 3060 c19953f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3059 c19947f0 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3058 c16299ec 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3057 c1806be8 80 403 403 0000100 [SLPQ ufs 0xc1526964][SLP] httpd 3056 c1994be8 80 403 403 0000100 [SLPQ ufs 0xc17e9400][SLP] httpd 3055 c1806000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3053 c137dde4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3051 c19b05f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3048 c19941fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 3047 c19b03f8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] httpd 3045 c1995be8 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] httpd 3036 c19945f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2993 c162c1fc 1000 2992 2987 0004102 [SLPQ ufs 0xc13c4a78][SLP] mksnap_ffs 2992 c162cbe8 1000 2990 2987 0004002 [SLPQ wait 0xc162cbe8][SLP] sh 2991 c1995de4 1000 2987 2987 0004002 [SLPQ piperd 0xc13a9780][SLP] gzip 2990 c150f5f4 1000 2987 2987 0004002 [SLPQ wait 0xc150f5f4][SLP] dump 2987 c1806de4 1000 1762 2987 0004002 [SLPQ pause 0xc1806e18][SLP] csh 2910 c1994de4 80 403 403 0000100 [SLPQ wait 0xc1994de4][SLP] httpd 2909 c18053f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2908 c19ae3f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2843 c1995000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2839 c19955f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2669 c1629000 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 2609 c18069ec 80 403 403 0000100 [SLPQ suspfs 0xc137086c][SLP] httpd 2598 c19943f8 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 1981 c18051fc 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 1899 c16295f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 1762 c162cde4 1000 1761 1762 0004002 [SLPQ pause 0xc162ce18][SLP] tcsh 1761 c162c000 1000 1759 1759 0000100 [SLPQ select 0xc078a744][SLP] sshd 1759 c1629be8 0 368 1759 0004100 [SLPQ sbwait 0xc1429480][SLP] sshd 1754 c18065f4 80 403 403 0000100 [SLPQ ufs 0xc13c4a78][SLP] httpd 526 c162c9ec 1062 516 526 0004100 [SLPQ piperd 0xc13a9900][SLP] sendmail 525 c162c7f0 1062 1 525 0004000 [SLPQ accept 0xc1429186][SLP] perl 516 c18057f0 0 390 390 0000000 [SLPQ piperd 0xc13a9c00][SLP] cron 447 c12ab5f4 0 1 447 0004002 [SLPQ ttyin 0xc1318410][SLP] getty 446 c15107f0 0 1 446 0004002 [SLPQ ttyin 0xc12f5810][SLP] getty 445 c15105f4 0 1 445 0004002 [SLPQ ttyin 0xc12f6810][SLP] getty 444 c15101fc 0 1 444 0004002 [SLPQ ttyin 0xc12f6410][SLP] getty 443 c150f9ec 0 1 443 0004002 [SLPQ ttyin 0xc12ef410][SLP] getty 442 c15103f8 0 1 442 0004002 [SLPQ ttyin 0xc12ef810][SLP] getty 441 c150f3f8 0 1 441 0004002 [SLPQ ttyin 0xc12efc10][SLP] getty 440 c150f1fc 0 1 440 0004002 [SLPQ ttyin 0xc12f5010][SLP] getty 439 c150fde4 0 1 439 0004002 [SLPQ ttyin 0xc12f5410][SLP] getty 427 c150fbe8 0 1 427 0000000 [SLPQ select 0xc078a744][SLP] inetd 403 c150f7f0 0 1 403 0000000 [SLPQ select 0xc078a744][SLP] httpd 390 c12ab9ec 0 1 390 0000000 [SLPQ nanslp 0xc0785dac][SLP] cron 378 c137d000 25 1 378 0000100 [SLPQ pause 0xc137d034][SLP] sendmail 374 c12abbe8 0 1 374 0000100 [SLPQ select 0xc078a744][SLP] sendmail 368 c137d1fc 0 1 368 0000100 [SLPQ select 0xc078a744][SLP] sshd 252 c12ab3f8 0 1 252 0000000 [SLPQ select 0xc078a744][SLP] syslogd 227 c12abde4 0 1 227 0000000 [SLPQ select 0xc078a744][SLP] devd 46 c137d5f4 0 0 0 0000204 [SLPQ - 0xcb30cd14][SLP] schedcpu 45 c137d7f0 0 0 0 0000204 [SLPQ - 0xc07923ec][SLP] nfsiod 3 44 c137d9ec 0 0 0 0000204 [SLPQ - 0xc07923e8][SLP] nfsiod 2 43 c137dbe8 0 0 0 0000204 [SLPQ - 0xc07923e4][SLP] nfsiod 1 42 c1243be8 0 0 0 0000204 [SLPQ - 0xc07923e0][SLP] nfsiod 0 41 c1243de4 0 0 0 0000204 [SLPQ vlruwt 0xc1243de4][SLP] vnlru 40 c12a8000 0 0 0 0000204 [SLPQ syncer 0xc0785b0c][SLP] syncer 39 c12a81fc 0 0 0 0000204 [SLPQ psleep 0xc078ac90][SLP] bufdaemon 38 c12a83f8 0 0 0 000020c [SLPQ pgzero 0xc07989e4][SLP] pagezero 9 c12a85f4 0 0 0 0000204 [SLPQ psleep 0xc0798534][SLP] vmdaemon 8 c12a87f0 0 0 0 0000204 [SLPQ psleep 0xc07984f0][SLP] pagedaemon 37 c12a89ec 0 0 0 0000204 [IWAIT] swi0: sio 7 c12a8be8 0 0 0 0000204 [SLPQ - 0xc120983c][SLP] fdc0 36 c12a8de4 0 0 0 0000204 [SLPQ usbtsk 0xc0780814][SLP] usbtask 35 c12ab000 0 0 0 0000204 [SLPQ usbevt 0xc12bb210][SLP] usb0 34 c12ab1fc 0 0 0 0000204 [IWAIT] swi5:+ 6 c12335f4 0 0 0 0000204 [SLPQ - 0xc1263dc0][SLP] thread taskq 33 c12337f0 0 0 0 0000204 [IWAIT] swi6:+ 32 c12339ec 0 0 0 0000204 [IWAIT] swi6: task queue 5 c1233be8 0 0 0 0000204 [SLPQ - 0xc128e040][SLP] kqueue taskq 31 c1233de4 0 0 0 0000204 [IWAIT] swi2: cambio 30 c1243000 0 0 0 0000204 [SLPQ - 0xc077e7a0][SLP] yarrow 4 c12431fc 0 0 0 0000204 [SLPQ - 0xc0783148][SLP] g_down 3 c12433f8 0 0 0 0000204 [SLPQ - 0xc0783144][SLP] g_up 2 c12435f4 0 0 0 0000204 [SLPQ - 0xc078313c][SLP] g_event 29 c12437f0 0 0 0 0000204 [IWAIT] swi3: vm 28 c12439ec 0 0 0 000020c [IWAIT] swi4: clock sio 27 c12191fc 0 0 0 0000204 [IWAIT] swi1: net 26 c12193f8 0 0 0 0000204 [IWAIT] irq15: ata1 25 c12195f4 0 0 0 0000204 [IWAIT] irq14: ata0 24 c12197f0 0 0 0 0000204 [IWAIT] irq13: 23 c12199ec 0 0 0 0000204 [IWAIT] irq12: 22 c1219be8 0 0 0 0000204 [IWAIT] irq11: 21 c1219de4 0 0 0 0000204 [IWAIT] irq10: 20 c1233000 0 0 0 0000204 [IWAIT] irq9: twe0 19 c12331fc 0 0 0 0000204 [IWAIT] irq8: rtc 18 c12333f8 0 0 0 0000204 [IWAIT] irq7: ppc0 17 c1211000 0 0 0 0000204 [IWAIT] irq6: fdc0 16 c12111fc 0 0 0 0000204 [IWAIT] irq5: bge0 15 c12113f8 0 0 0 0000204 [IWAIT] irq4: sio0 14 c12115f4 0 0 0 0000204 [IWAIT] irq3: uhci0 13 c12117f0 0 0 0 0000204 [IWAIT] irq1: atkbd0 12 c12119ec 0 0 0 0000204 [IWAIT] irq0: clk 11 c1211be8 0 0 0 000020c [CPU 0] idle 1 c1211de4 0 0 1 0004200 [SLPQ wait 0xc1211de4][SLP] init 10 c1219000 0 0 0 0000204 [SLPQ ktrace 0xc0783d38][SLP] ktrace 0 c07832a0 0 0 0 0000200 [SLPQ sched 0xc07832a0][SLP] swapper 517 c162c5f4 1062 516 517 0006000 zomb[INACTIVE] sh db> show pcpu cpuid = 0 curthread = 0xc12125c0: pid 11 "idle" curpcb = 0xc69b7da0 fpcurthread = none idlethread = 0xc12125c0: pid 11 "idle" APIC ID = 0 currentldt = 0x28 Joe 'sh' process in ufs: db> trace 3296 Tracing pid 3296 tid 100255 td 0xc1cfa730 sched_switch(c1cfa730,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c13c4a78,cbad385c,c0526901,c13c4a78,0) at sleepq_switch+0xe0 sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30 msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1 acquire(cbad38b4,1010040,600,c1cfa730,0) at acquire+0x9a lockmgr(c13c4a78,1010002,c13c49b4,c1cfa730,cbad38d4) at lockmgr+0x382 vop_stdlock(cbad3904) at vop_stdlock+0x1b VOP_LOCK_APV(c076f340,cbad3904,cbad38f4,c06c279e,cbad3904) at VOP_LOCK_APV+0x7e ffs_lock(cbad3904) at ffs_lock+0x3b VOP_LOCK_APV(c076ed60,cbad3904) at VOP_LOCK_APV+0x7e vn_lock(c13c49b4,10002,c1cfa730,2,c136a800) at vn_lock+0xc2 vget(c13c49b4,10002,c1cfa730,c1cfa730,c13c4bdc) at vget+0xc9 ufs_ihashget(c136a800,2,2,cbad3a00,c06e98f4) at ufs_ihashget+0x98 ffs_vget(c1370800,2,2,cbad3a00) at ffs_vget+0x37 ufs_root(c1370800,cbad3a3c,c1cfa730,1,1) at ufs_root+0x18 lookup(cbad3c34,c05197b6,1,c1cfa730,c06ea9fd) at lookup+0x593 namei(cbad3c34,c05197b6,c1cf9460,1,0) at namei+0x34e do_execve(c1cfa730,cbad3cc4,0,0,cbad3cc4) at do_execve+0x15d kern_execve(c1cfa730,cbad3cc4,0,c6295000,c6295000) at kern_execve+0x74 execve(c1cfa730,cbad3d14,3,0,282) at execve+0x32 syscall(2f,2f,2f,8067058,806708c) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (59, FreeBSD ELF32, execve), eip = 0x28121d5b, esp = 0xbfbfec1c, ebp = 0xbfbfec48 --- httpd holding a VV_ROOT vnode lock, blocked in ufs: db> trace 3057 Tracing pid 3057 tid 100107 td 0xc1807a10 sched_switch(c1807a10,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c1526964,cb707994,c0526901,c1526964,0) at sleepq_switch+0xe0 sleepq_wait(c1526964,0,0,c1526964,1010040) at sleepq_wait+0x30 msleep(c1526964,c0785244,50,c06f08fe,0) at msleep+0x2d1 acquire(cb7079ec,1010040,600,c1807a10,0) at acquire+0x9a lockmgr(c1526964,1010002,c15268a0,c1807a10,cb707a0c) at lockmgr+0x382 vop_stdlock(cb707a3c) at vop_stdlock+0x1b VOP_LOCK_APV(c076f340,cb707a3c,cb707a2c,c06c279e,cb707a3c) at VOP_LOCK_APV+0x7e ffs_lock(cb707a3c) at ffs_lock+0x3b VOP_LOCK_APV(c076ed60,cb707a3c) at VOP_LOCK_APV+0x7e vn_lock(c15268a0,10002,c1807a10,cb707b24,cb707c38) at vn_lock+0xc2 vget(c15268a0,2,c1807a10,1b8,c1807a10) at vget+0xc9 vfs_cache_lookup(cb707b24) at vfs_cache_lookup+0x1bd VOP_LOOKUP_APV(c076ed60,cb707b24) at VOP_LOOKUP_APV+0x7e lookup(cb707c10,cb707b70,1,c1807a10,1) at lookup+0x359 namei(cb707c10,cb707bc4,c0672ebb,c19cbad0,c0703dc4) at namei+0x34e kern_stat(c1807a10,81541f8,0,cb707c84) at kern_stat+0x35 stat(c1807a10,cb707d14,2,0,296) at stat+0x1b syscall(280b002f,2f,bfbf002f,81541f8,8170) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp = 0xbfbfe9dc, ebp = 0xbfbfea78 --- download.cgi blocked in ufs holding a directory vnode lock: db> trace 3062 Tracing pid 3062 tid 100050 td 0xc137e000 sched_switch(c137e000,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c163bca0,cb2e5844,c0526901,c163bca0,0) at sleepq_switch+0xe0 sleepq_wait(c163bca0,0,0,c163bca0,1010040) at sleepq_wait+0x30 msleep(c163bca0,c0784578,50,c06f08fe,0) at msleep+0x2d1 acquire(cb2e589c,1010040,600,c137e000,0) at acquire+0x9a lockmgr(c163bca0,1010002,c163bbdc,c137e000,cb2e58bc) at lockmgr+0x382 vop_stdlock(cb2e58ec) at vop_stdlock+0x1b VOP_LOCK_APV(c076f340,cb2e58ec,cb2e58dc,c06c279e,cb2e58ec) at VOP_LOCK_APV+0x7e ffs_lock(cb2e58ec) at ffs_lock+0x3b VOP_LOCK_APV(c076ed60,cb2e58ec) at VOP_LOCK_APV+0x7e vn_lock(c163bbdc,10002,c137e000,cb2e59d4,cb2e5c04) at vn_lock+0xc2 vget(c163bbdc,2,c137e000,1e0,c137e000) at vget+0xc9 vfs_cache_lookup(cb2e59d4) at vfs_cache_lookup+0x1bd VOP_LOOKUP_APV(c076ed60,cb2e59d4) at VOP_LOOKUP_APV+0x7e lookup(cb2e5bdc,cb2e5a76,1,c137e000,cb2e5a80) at lookup+0x359 namei(cb2e5bdc,0,d7ec0005,c137e000,c050f370) at namei+0x34e vn_open_cred(cb2e5bdc,cb2e5cdc,1a4,c19ace00,3) at vn_open_cred+0x27e vn_open(cb2e5bdc,cb2e5cdc,1a4,3,c07849f8) at vn_open+0x1e kern_open(c137e000,804a794,0,1,1b6) at kern_open+0xb6 open(c137e000,cb2e5d14,3,1,292) at open+0x1a syscall(2f,2f,2f,4,2814e280) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (5, FreeBSD ELF32, open), eip = 0x280ba23b, esp = 0xbfbfe10c, ebp = 0xbfbfe138 --- httpd holding a directory lock blocked in ufs: db> trace 3056 Tracing pid 3056 tid 100099 td 0xc162dcf0 sched_switch(c162dcf0,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c17e9400,cb6be994,c0526901,c17e9400,0) at sleepq_switch+0xe0 sleepq_wait(c17e9400,0,0,c17e9400,1010040) at sleepq_wait+0x30 msleep(c17e9400,c0784c38,50,c06f08fe,0) at msleep+0x2d1 acquire(cb6be9ec,1010040,600,c162dcf0,0) at acquire+0x9a lockmgr(c17e9400,1010002,c17e933c,c162dcf0,cb6bea0c) at lockmgr+0x382 vop_stdlock(cb6bea3c) at vop_stdlock+0x1b VOP_LOCK_APV(c076f340,cb6bea3c,cb6bea2c,c06c279e,cb6bea3c) at VOP_LOCK_APV+0x7e ffs_lock(cb6bea3c) at ffs_lock+0x3b VOP_LOCK_APV(c076ed60,cb6bea3c) at VOP_LOCK_APV+0x7e vn_lock(c17e933c,10002,c162dcf0,cb6beb24,cb6bec38) at vn_lock+0xc2 vget(c17e933c,2,c162dcf0,210,c162dcf0) at vget+0xc9 vfs_cache_lookup(cb6beb24) at vfs_cache_lookup+0x1bd VOP_LOOKUP_APV(c076ed60,cb6beb24) at VOP_LOOKUP_APV+0x7e lookup(cb6bec10,c17e9450,1,c162dcf0,75c) at lookup+0x359 namei(cb6bec10,c05197b6,c0798060,1,c06ea9fd) at namei+0x34e kern_stat(c162dcf0,8161ad8,0,cb6bec84) at kern_stat+0x35 stat(c162dcf0,cb6bed14,2,0,292) at stat+0x1b syscall(280b002f,2f,bfbf002f,8161ad8,8170) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (188, FreeBSD ELF32, stat), eip = 0x2839359b, esp = 0xbfbfe75c, ebp = 0xbfbfe7f8 --- httpd holding a file lock, blocked in suspfs from ufs_inactive: db> trace 3045 Tracing pid 3045 tid 100123 td 0xc1996450 sched_switch(c1996450,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c137086c,cb6ecb8c,c0526901,c137086c,0) at sleepq_switch+0xe0 sleepq_wait(c137086c,0,0,1,c153be38) at sleepq_wait+0x30 msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 vn_write_suspend_wait(c17e933c,c1370800,1,0,c06f2634) at vn_write_suspend_wait+0xe4 ufs_inactive(cb6ecc14) at ufs_inactive+0x186 VOP_INACTIVE_APV(c076ed60,cb6ecc14) at VOP_INACTIVE_APV+0x7e vrele(c17e933c,c17e7210,0,c07046ce,19d) at vrele+0x10e vm_object_vndeallocate(c17e7210) at vm_object_vndeallocate+0xbd vm_object_deallocate(c17e7210,c17e7210,0,c0703dc4,89e) at vm_object_deallocate+0x85 vm_map_entry_delete(c19cbbb8,c16258c4,284c8000,c19cbbb8,c16258c4) at vm_map_entry_delete+0x130 vm_map_delete(c19cbbb8,284c8000,284c9000,0,c1996450) at vm_map_delete+0x18f munmap(c1996450,cb6ecd14,2,3,292) at munmap+0xa3 syscall(280b002f,2f,bfbf002f,2831e058,815f2d8) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = 0xbfbfc6fc, ebp = 0xbfbfc718 --- Another httpd holding a file lock, blocked in suspfs from ufs_inactive: db> trace 3047 Tracing pid 3047 tid 100143 td 0xc19b2450 sched_switch(c19b2450,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c137086c,cb781b8c,c0526901,c137086c,0) at sleepq_switch+0xe0 sleepq_wait(c137086c,0,0,1,c141ec08) at sleepq_wait+0x30 msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 vn_write_suspend_wait(c18419b4,c1370800,1,0,c06f2634) at vn_write_suspend_wait+0xe4 ufs_inactive(cb781c14) at ufs_inactive+0x186 VOP_INACTIVE_APV(c076ed60,cb781c14) at VOP_INACTIVE_APV+0x7e vrele(c18419b4,c1826000,0,c07046ce,19d) at vrele+0x10e vm_object_vndeallocate(c1826000) at vm_object_vndeallocate+0xbd vm_object_deallocate(c1826000,c1826000,0,c0703dc4,89e) at vm_object_deallocate+0x85 vm_map_entry_delete(c19cbe10,c18bfd8c,284cd000,c19cbe10,c18bfd8c) at vm_map_entry_delete+0x130 vm_map_delete(c19cbe10,284cd000,284ce000,0,c19b2450) at vm_map_delete+0x18f munmap(c19b2450,cb781d14,2,3,292) at munmap+0xa3 syscall(280b002f,2f,bfbf002f,2831e058,815d228) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = 0xbfbfc6fc, ebp = 0xbfbfc718 --- mksnap_ffs, creating the snapshot, but blocked on a vnode lock: db> trace 2993 Tracing pid 2993 tid 100096 td 0xc162d8a0 sched_switch(c162d8a0,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c13c4a78,cb6b560c,c0526901,c13c4a78,0) at sleepq_switch+0xe0 sleepq_wait(c13c4a78,0,0,c13c4a78,1010040) at sleepq_wait+0x30 msleep(c13c4a78,c0784458,50,c06f08fe,0) at msleep+0x2d1 acquire(cb6b5664,1010040,600,c162d8a0,0) at acquire+0x9a lockmgr(c13c4a78,1010002,c13c49b4,c162d8a0,cb6b5684) at lockmgr+0x382 vop_stdlock(cb6b56b4) at vop_stdlock+0x1b VOP_LOCK_APV(c076f340,cb6b56b4,cb6b56a4,c06c279e,cb6b56b4) at VOP_LOCK_APV+0x7e ffs_lock(cb6b56b4) at ffs_lock+0x3b VOP_LOCK_APV(c076ed60,cb6b56b4) at VOP_LOCK_APV+0x7e vn_lock(c13c49b4,10002,c162d8a0) at vn_lock+0xc2 ffs_snapshot(c1370800,c14ccbc0,c14ccbc0,c1369b00,0) at ffs_snapshot+0x1242 ffs_mount(c1370800,c162d8a0,20000000,201000,c13c49b4) at ffs_mount+0x7a5 vfs_domount(c162d8a0,c13207b0,c141c030,1211000,c1320670) at vfs_domount+0x576 vfs_donmount(c162d8a0,1211000,cb6b5c14,c1422880,e) at vfs_donmount+0xce kernel_mount(c1320690,1211000,bfbfed33,0,0) at kernel_mount+0x6d ffs_cmount(c1320690,bfbfe578,1211000,c162d8a0,c076ea80) at ffs_cmount+0x5d mount(c162d8a0,cb6b5d14,4,2,206) at mount+0x156 syscall(2f,2f,2f,bfbfe7c8,bfbfed29) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (21, FreeBSD ELF32, mount), eip = 0x280b80db, esp = 0xbfbfe4ec, ebp = 0xbfbfebe0 --- Anothr httpd blocked in suspfs while holding a file vnode lock, trying to unmap: db> trace 2609 Tracing pid 2609 tid 100108 td 0xc1807b80 sched_switch(c1807b80,0,1) at sched_switch+0x17f mi_switch(1,0) at mi_switch+0x264 sleepq_switch(c137086c,cb70ab8c,c0526901,c137086c,0) at sleepq_switch+0xe0 sleepq_wait(c137086c,0,0,1,c14cae38) at sleepq_wait+0x30 msleep(c137086c,c1370844,29f,c06f2e66,0) at msleep+0x2d1 vn_write_suspend_wait(c19fd9b4,c1370800,1,0,c06f2634) at vn_write_suspend_wait+0xe4 ufs_inactive(cb70ac14) at ufs_inactive+0x186 VOP_INACTIVE_APV(c076ed60,cb70ac14) at VOP_INACTIVE_APV+0x7e vrele(c19fd9b4,c1a93948,0,c07046ce,19d) at vrele+0x10e vm_object_vndeallocate(c1a93948) at vm_object_vndeallocate+0xbd vm_object_deallocate(c1a93948,c1a93948,0,c0703dc4,89e) at vm_object_deallocate+0x85 vm_map_entry_delete(c18bd834,c18cb3b8,284c7000,c18bd834,c18cb3b8) at vm_map_entry_delete+0x130 vm_map_delete(c18bd834,284c7000,284c8000,0,c1807b80) at vm_map_delete+0x18f munmap(c1807b80,cb70ad14,2,37,296) at munmap+0xa3 syscall(2f,2f,bfbf002f,2831e058,815d0c8) at syscall+0x213 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (73, FreeBSD ELF32, munmap), eip = 0x28393cbb, esp = 0xbfbfc5cc, ebp = 0xbfbfc5e8 --- No WITNESS in this kernel. My first glance reading is that there's a deadlock between the snapshot lock and vnode locks, which ocurs because a vnode lock has been acquired as part of the mmap tear-down, which leads to vn_write_suspend() being called in ufs_inactive as the inode has a zero reference count so has to be GC'd, which involves writing to the file system. VOP_INACTIVE is called by vrele() with the vnode lock held. My understanding is that, generally, we acquire the file system snapshot lock before vnode locks as part of the VFS lock order. This could be a misinterpretation. I'll try to get a dump, but this is a twe box so I may not be able to. Robert N M WatsonReceived on Fri Feb 18 2005 - 13:51:26 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:28 UTC