head -r325997 panics: dump/restore: restore got ufs_dirbad ino . . .: mangled entry; cp -ax got: ffs_valloc: dup alloc (after some cylinder checksum failures)

From: Mark Millard <markmi_at_dsl-only.net>
Date: Fri, 24 Nov 2017 00:54:41 -0800
head -r323997 context, non-debug kernel (with
debug symbols).

This turned out to be for a USB SSD that was
dying (Retired Block Count had grown rather
large, other things looking okay).

I do not know if panics are an intended
result for the messed up data or not.


The panic information follows.

The USB ssd with /dev/gpt/FBSDUSSDroot on it had
just been freshly constructed via bsdconfig, UFS
with softupdate enabled but journaling disabled.

# mount -o noatime /dev/gpt/FBSDUSSDroot /mnt
# dump -C16 -b64 -0aL -f - / | (cd /mnt && restore -rf -)
  DUMP: Date of this level 0 dump: Thu Nov 23 16:26:00 2017
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping snapshot of /dev/gpt/FBSDFSSDroot (/) to standard output
  DUMP: mapping (Pass I) [regular files]
  DUMP: Cache 16 MB, blocksize = 65536
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 140989948 tape blocks.
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
warning: ./.snap: File exists
expected next file 9, got 8

Manually transcribed from the console screen
for the crash:

panic: ufs_dirbad ino 308183317 at offset 0: mangled entry.
cpuid = 21
time = 1511483271
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1c3449a2b0
vpanic() at panic+0x19c/frame 0xfffffe1c3449a330
panic() at panic+0x43/frame 0xfffffe1c3449a390
ufs_lokup_ino at ufs_lookup_ino+0xe7f/frame 0xfffffe1c3449a4a0
VOP_CACHEDLOOKUP_APV() at VOP_CACHEDLOOKUP_APV+0x77c/frame 0xffffe1c3449a4d0
vfs_cache_lookup() at vfs_cache_lookup+0xd6/frame 0xfffffe1c3449a530
VOP_LOOKUP_APV() at VOP_LOOKUP_APV+0x7c/frame 0xfffffe1c3449a560
lookup() at lookup+0x6c1/frame 0xfffffe1c3449a600
namei() at namei+0x48c/frame 0xfffffe1c3449a6d0
vn_open_cred() at vn_open_cred+0xd8/frame 0xfffffe1c3449a810
kern_openat at kern_openat+0x212/frame 0xfffffe1c3449a980
amd64_syscall() at amd64_syscall+0x9f7/frame 0xfffffe1c3499aab0
Xfast_syscall() at XFast_syscall+0xfb/frame 0xfffffe1c3499aab0
--- syscall (449, FreEBSD ELF64, sys_openat), rip = 0x80093be2a, rsp = 0x7fffffffd518, rpb=0x7fffffffd600 ---
KDB: enter: panic
[ thread pid 915 tid 100328 ]
Stopped at kdb_enter_0x3b: movq $0,kbd_why
db>


A retry via (newfs copied from a dumpfs -m output):

# newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -k 6408 -m 8 -o time -s 905969664 /dev/gpt/FBSDUSSDroot 
/dev/gpt/FBSDUSSDroot: 442368.0MB (905969664 sectors) block size 32768, fragment size 4096
	using 707 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
	with soft updates
super-block backups (for fsck_ffs -b #) at:
. . .
# mount -o noatime /dev/gpt/FBSDUSSDroot /mnt
FBSDFSSD# dump -C16 -b64 -0aL -f - / | (cd /mnt && restore -rf -)
  DUMP: Date of this level 0 dump: Thu Nov 23 17:02:18 2017
  DUMP: Date of last level 0 dump: the epoch
  DUMP: Dumping snapshot of /dev/gpt/FBSDFSSDroot (/) to standard output
  DUMP: mapping (Pass I) [regular files]
  DUMP: Cache 16 MB, blocksize = 65536
  DUMP: mapping (Pass II) [directories]
  DUMP: estimated 141064070 tape blocks.
  DUMP: dumping (Pass III) [directories]
  DUMP: dumping (Pass IV) [regular files]
warning: ./.snap: File exists
expected next file 5, got 4
  DUMP: 33.57% done, finished in 0:09 at Thu Nov 23 17:17:12 2017


did it again: bad dir ino 11316224 . (I'm not transcribing
this screen but it is very similar to the earlier one.)


Then, trying a cp -ax instead lead to a different
panic (crash info transcribed from screenshot of
console). . .

# newfs -O 2 -U -a 4 -b 32768 -d 32768 -e 4096 -f 4096 -g 16384 -h 64 -i 8192 -k 6408 -m 8 -o time -s 905969664 /dev/gpt/FBSDUSSDroot
/dev/gpt/FBSDUSSDroot: 442368.0MB (905969664 sectors) block size 32768, fragment size 4096
	using 707 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
	with soft updates
super-block backups (for fsck_ffs -b #) at:
. . .
# mount -o noatime /dev/gpt/FBSDUSSDroot /mnt
# cp -ax / /mnt

Eventually on the console got:

. . .
UFS /dev/gpt/FBSDUSSDroot (/mnt) cylinder checksum failed: cg 475, cgp:0x782bac7c != bp: 0x3c6d091f
UFS /dev/gpt/FBSDUSSDroot (/mnt) cylinder checksum failed: cg 477, cgp:0x782bac7c != bp: 0xcbfd1084
mode = 0116600, inum = 38603264, fs = /mnt
panic: ffs_valloc: dup alloc
cpuid = 27
time = 1511490715
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe1c344fe400
vpanic() at vpanic+0x19c/frame 0xfffffe1c344fe480
panic() at panic+0x43/frame 0xfffffe1c344fe4e0
ffs_valloc() at ffs_valloc+0x85c/frame 0xffffe1c344fe570
ufs_mkdir() at ufs_mkdir+0xcc/frame 0xfffffe1c344fe730
VOP_MKDIR_APV() at VOP_MKDIR_APV+0x82/frame 0xfffffe1c344fe760
kern_mkdirat() at kern_mkdirat+0x1e2/frame 0xfffffe1c344fe980
amd64_syscall() at amd64_syscall+0x9f7/frame 0xfffffe1c344feab0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xfffffe1c344feab0
--- syscall (136, FreBSD ELF64, sys_mkdir), rip = 0x80099a78a, rsp = 0x7fffffffd838, rbp = 0x7fffffffdb30 ---
KDB: enter: panic
[ thread pid 966 tid 100348 ]
Stopped at kdb_enter+0x3b: movq $0,kdb_why
db>




===
Mark Millard
markmi at dsl-only.net
Received on Fri Nov 24 2017 - 08:01:29 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:13 UTC