Re: GBDE + md() + ccd() = corruption?

I had a little more time to work on this - more background on the setup is
below.  Anyhow, I made up a new filesystem, starting making large text
files on a regular ufs slice, and copying them to a gbde encrypted fs
until I had a file that had the md5sum/diff inconsistencies - I copied
the inconsistent file from the gbde encrypted fs back to a regular ufs
slice (the md5's while files exist on a ufs slice are reproducable and
don't change per md5 "run" on the same file)- although both files
were the same size, diff and md5 both agreed they were different.  Due to
diff's memory issues, I used split on both the original and the
inconsistent copy of the text file, making a large number of 10mb (or so)
segments, ran md5 against both and diff against that output - one segment
was inconsistent, roughly somewhere between 502mb and 512mb into the text
file.  running diff against the two segments found an jumble of characters
on line 267717.  Although I can't say as I know what the jumble means, by
the general frequency/distribution of the characters I would say the
segment looks very much like that of cat'ing a file used as a
device through md() when gbde is attached, perhaps a bit of encrypted
data that wasn't decrypted.  But then again, I might be totally off here.

More background info:

# Setting up the device

R31# ls -la part*
-rw-r--r--  1 root  wheel  1175000000 Apr 16 15:48 part1
-rw-r--r--  1 root  wheel  1175000000 Apr 16 21:17 part2
-rw-r--r--  1 root  wheel  1175000000 Apr 16 15:46 part3
-rw-r--r--  1 root  wheel  1175000000 Apr 16 15:47 part4
R31# cat setup1
mdconfig -a -t vnode -f part1 -u 1
mdconfig -a -t vnode -f part2 -u 2
mdconfig -a -t vnode -f part3 -u 3
mdconfig -a -t vnode -f part4 -u 4
R31# ./setup1
R31# ccdconfig ccd0 0 none /dev/md1 /dev/md2 /dev/md3 /dev/md4
R31# gbde init /dev/ccd0
Enter new passphrase:
Reenter new passphrase:
Wrote key 0 at (insert large number here, don't remember off hand)
R31# newfs -m0% /dev/ccd0
(newfs output...)
R31# mount /dev/ccd0.bde /mnt

# Making some test files...

R31# cat /etc/fstab | grep usr
/dev/ad1s2f             /usr            ufs     rw              2       2
R31# pwd
/usr/dvd
R31# yes 'echo "abcdefg123456789" >> testline.txt'|sh
(wait a while, until file is ~100k, repeat for lack of patience with yes
'cat testline.txt >>file1.txt'|sh, wait a while, I made ~200mb, 400mb,
600mb, and a 800mb file, the 4th file at 800mb displayed the issue)
R31# cp file*.txt /mnt
R31# diff --brief /mnt/file4.txt /usr/dvd/file4.txt
Files /mnt/file4.txt and /usr/dvd/file4.txt differ
R31# cp /mnt/file4.txt /usr/dvd/file4_copy_from_gbde_fs.txt
R31# ls -la file4*
-rw-r--r--  1 root  wheel  864281088 Apr 16 18:51
file4_copy_from_gbde_fs.txt
-rw-r--r--  1 root  wheel  864281088 Apr 16 15:30 file4_original.txt
R31# md5 file4*
MD5 (file4_original.txt) = b8869c190d64566de6dbb565125fef6d
MD5 (file4_copy_from_gbde_fs.txt) = 053ec9fd8c63d6a20785646d9ee5c161
R31# diff --brief file4_c* file4_or*
Files file4_copy_from_gbde_fs.txt and file4_original.txt differ


#Splitting things up and locating the broken piece...

R31# mkdir split.file4_copy_from_gbde_fs
R31# mkdir split.file4_original
R31# cd split*s
R31# split -b10000k ../file4_copy_from_gbde_fs.txt
R31# md5 * > md5sum1.txt
R31# cd ../split*l
R31# split -b10000k ../file4_original.txt
R31# md5 * > md5sum2.txt
R31# diff md5sum2.txt ../split*fs/md5sum1.txt
1c1
< MD5 (md5sum2.txt) = d41d8cd98f00b204e9800998ecf8427e
---
> MD5 (md5sum1.txt) = d41d8cd98f00b204e9800998ecf8427e
48c48
< MD5 (xbu) = 1066b54bbbc235ae115284d7e55fedc9
---
> MD5 (xbu) = 9006ec9b31628ba748c8bc9c52a7279a
R31# diff xbu ../split*fs/xbu > xbu.diff

# Diff output from that last command is attached


Comments?  Suggestions?

- Matt