> On Mar 1, 2015, at 9:29 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > > On Sun, Mar 01, 2015 at 19:41:07 -0500, Dan Langille wrote: >> >>> On Mar 1, 2015, at 7:31 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: >>> >>> On Sun, Mar 01, 2015 at 19:15:05 -0500, Dan Langille wrote: >>>> >>>>> On Feb 17, 2015, at 1:36 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: >>>>> >>>>> On Sat, Feb 14, 2015 at 18:22:43 -0500, Dan Langille wrote: >>>>>> >>>>>>> On Feb 13, 2015, at 7:32 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: >>>>>>> >>>>>>> >>>>>>> I have a fairly large set of changes to the sa(4) driver and mt(1) driver >>>>>>> that I'm planning to commit in the near future. >>>>>>> >>>>>>> A description of the changes is here and below in this message. >>>>>>> >>>>>>> If you have tape hardware and the inclination, I'd appreciate testing and >>>>>>> feedback. >>>>>> >>>>>> I have a DLT 8000 and an SDLT 220. >>>>>> >>>>>> I don't have anything running current, but I have a spare machine which I could use for testing. >>>>>> >>>>>> Do you see any value is tests with that hardware? I'd be testing it via Bacula. >>>>>> >>>>>> disclosure: I'm the sysutils/bacula-* maintainer and a Bacula committer. >>>>>> >>>>> >>>>> Actually, yes. Bacula is a bit tricky to configure, so your trying it out >>>>> would be helpful if you have the time. >>>>> >>>>> In looking at the manuals for both the SDLT 220 and the DLT 8000, they both >>>>> claim to support long position information for the SCSI READ POSITION >>>>> command. >>>>> >>>>> You can see what I'm talking about by doing: >>>>> >>>>> mt eod >>>>> mt status >>>>> >>>>> On my DDS-4 tape drive, this shows: >>>>> >>>>> # mt -f /dev/nsa3 status >>>>> Drive: sa3: <SEAGATE DAT 06240-XXX 8071> Serial Number: HJ00YWY >>>>> --------------------------------- >>>>> Mode Density Blocksize bpi Compression >>>>> Current: 0x26:DDS-4 1024 bytes 97000 enabled (DCLZ) >>>>> --------------------------------- >>>>> Current Driver State: at rest. >>>>> --------------------------------- >>>>> Partition: 0 Calc File Number: -1 Calc Record Number: -1 >>>>> Residual: 0 Reported File Number: -1 Reported Record Number: -1 >>>>> Flags: None >>>>> >>>>> But on an LTO-5, which will give long position information, I get: >>>>> >>>>> [root_at_doc ~]# mt status >>>>> Drive: sa0: <IBM ULTRIUM-HH5 E4J1> >>>>> --------------------------------- >>>>> Mode Density Blocksize bpi Compression >>>>> Current: 0x58:LTO-5 variable 384607 enabled (0x1) >>>>> --------------------------------- >>>>> Current Driver State: at rest. >>>>> --------------------------------- >>>>> Partition: 0 Calc File Number: 2 Calc Record Number: -1 >>>>> Residual: 0 Reported File Number: 2 Reported Record Number: 32373 >>>>> Flags: None >>>>> >>>>> That, in combination with the changes I made to the position information >>>>> code in the driver, mean that even the old MTIOCGET ioctl should return an >>>>> accurate file number at end of data. e.g., on the LTO-5: >>>>> >>>>> [root_at_doc ~]# mt ostatus >>>>> Mode Density Blocksize bpi Compression >>>>> Current: 0x58:LTO-5 variable 384607 0x1 >>>>> ---------available modes--------- >>>>> 0: 0x58:LTO-5 variable 384607 0x1 >>>>> 1: 0x58:LTO-5 variable 384607 0x1 >>>>> 2: 0x58:LTO-5 variable 384607 0x1 >>>>> 3: 0x58:LTO-5 variable 384607 0x1 >>>>> --------------------------------- >>>>> Current Driver State: at rest. >>>>> --------------------------------- >>>>> File Number: 2 Record Number: -1 Residual Count -1 >>>>> >>>>> So the thing to try, in addition to just making sure that Bacula continues >>>>> to work properly, is to try setting this for the tape drive in >>>>> bacula-sd.conf: >>>>> >>>>> Hardware End of Medium = yes >>>>> >>>>> It looks like the Bacula tape program (btape) has a test mode, and it would >>>>> be good to run through the tests on one of the tape drives and see whether >>>>> they work, and whether the results are different before and after the >>>>> changes. I'm not sure how to enable the test mode. >>>> >>>> I have this in /usr/local/etc/bacula/bacula-sd.conf >>>> >>>> Device { >>>> Name = DLT >>>> Description = "QUANTUM DLT7000 1624" >>>> Media Type = DLT >>>> Archive Device = /dev/nsa1 >>>> >>>> Autochanger = YES >>>> Drive Index = 0 >>>> >>>> Offline On Unmount = no >>>> Hardware End of Medium = yes >>>> BSF at EOM = yes >>>> Backward Space Record = no >>>> Fast Forward Space File = no >>>> TWO EOF = yes >>>> } >>>> >>>> FYI, http://www.freebsddiary.org/digital-tl891.php (from 2006) has a btape test on this same model. >>>> >>>> Here's the test I ran tonight: >>>> >>>> [root_at_cuppy:/usr/home/dan] # btape -c /usr/local/etc/bacula/bacula-sd.conf /dev/nsa1 >>>> Tape block granularity is 1024 bytes. >>>> btape: butil.c:287-0 Using device: "/dev/nsa1" for writing. >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >>>> *test >>>> >>>> === Write, rewind, and re-read test === >>>> >>>> I'm going to write 10000 records and an EOF >>>> then write 10000 records and an EOF, then rewind, >>>> and re-read the data to verify that it is correct. >>>> >>>> This is an *essential* feature ... >>>> >>>> btape: btape.c:1152-0 Wrote 10000 blocks of 64412 bytes. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1168-0 Wrote 10000 blocks of 64412 bytes. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1210-0 Rewind OK. >>>> 10000 blocks re-read correctly. >>>> Got EOF on tape. >>>> 10000 blocks re-read correctly. >>>> === Test Succeeded. End Write, rewind, and re-read test === >>>> >>>> btape: btape.c:1277-0 Block position test >>>> btape: btape.c:1289-0 Rewind OK. >>>> Reposition to file:block 0:4 >>>> Block 5 re-read correctly. >>>> Reposition to file:block 0:200 >>>> Block 201 re-read correctly. >>>> Reposition to file:block 0:9999 >>>> Block 10000 re-read correctly. >>>> Reposition to file:block 1:0 >>>> Block 10001 re-read correctly. >>>> Reposition to file:block 1:600 >>>> Block 10601 re-read correctly. >>>> Reposition to file:block 1:9999 >>>> Block 20000 re-read correctly. >>>> === Test Succeeded. End Write, rewind, and re-read test === >>>> >>>> >>>> >>>> === Append files test === >>>> >>>> This test is essential to Bacula. >>>> >>>> I'm going to write one record in file 0, >>>> two records in file 1, >>>> and three records in file 2 >>>> >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1420-0 Now moving to end of medium. >>> >>> This is the critical piece. The test moves the tape to the end of the >>> medium. With hardware position information, you can tell what filemark >>> you're on. Without it, you can't. >>> >>>> btape: btape.c:622-0 tape_dev.c:345 ioctl MTIOCGET error on "DLT" (/dev/nsa1). ERR=No error: 0. >>>> We should be in file 3. I am at file 0. This is NOT correct!!!! >>>> >>>> Append test failed. Attempting again. >>>> Setting "Hardware End of Medium = no >>>> and "Fast Forward Space File = no >>>> and retrying append test. >>> >>> This is not surprsing, given that the drive doesn't support long read >>> position data. (It's a SCSI-2 device.) So that means that Bacula will >>> need to do it manually. >> >> Yes, I have nothing newer than SCSI-2. Even my SDLT is SCSI-2 but that >> tape library is hooked up to a different computer and was doing backups today. > > So, here is one thing that we can try to see whether these drives support > long position information, even though they only claim to be SCSI-2. If > they do, we can potentially add a quirk (or autodetection) to enable it. > The code currently doesn't bother asking drives that claim to be SCSI-2 > for long position information. (Because that feature was added in the > SSC spec, which came after SCSI-2.) > > Issue a READ POSITION with the short form specified: > > camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > > Issue a READ POSITION with the vendor-specific block numbers: > > camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > > Issue a READ POSITION with the long form data: > > camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > > If it supports the last one, then I can put a quirk (or autodetection) in > the driver and Bacula will get the hardware filemarks. You should try this > on your SDLT as well. It may well support it. Sadly, no: [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 |....| 00000014 [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 |....| 00000014 [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd camcontrol: error sending command (pass2:ahc0:0:2:0): READ POSITION. CDB: 34 06 00 00 00 00 00 00 00 00 (pass2:ahc0:0:2:0): CAM status: SCSI Status Error (pass2:ahc0:0:2:0): SCSI status: Check Condition (pass2:ahc0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) (pass2:ahc0:0:2:0): Command byte 1 bit 2 is invalid [root_at_cuppy:~] # The SDLT server is on 9.3 though: [root_at_knew:/usr/home/dan] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed cam_lookup_pass: No such file or directory cam_lookup_pass: either the pass driver isn't in your kernel cam_lookup_pass: or sa1 doesn't exist [root_at_knew:/usr/home/dan] # uname -a FreeBSD knew.unixathome.org 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015 root_at_amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 [root_at_knew:/usr/home/dan] # It took me a while to figure that out... there is no sa1 on *this* system. But, my SDLT: [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 |....| 00000014 [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 |....| 00000014 [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000020 [root_at_knew:/usr/home/dan] # > > Google didn't quickly produce a SCSI manual for the DEC drive, but the > Quantum SDLT manual indicates that it supports long position data, despite > identifying itself as a SCSI-2 drive. > >>> === Append files test === >>>> >>>> This test is essential to Bacula. >>>> >>>> I'm going to write one record in file 0, >>>> two records in file 1, >>>> and three records in file 2 >>>> >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1420-0 Now moving to end of medium. >>>> btape: btape.c:625-0 Moved to end of medium. >>>> We should be in file 3. I am at file 3. This is correct! >>>> >>>> Now the important part, I am going to attempt to append to the tape. >>>> >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> Done appending, there should be no I/O errors >>>> >>>> Doing Bacula scan of blocks: >>>> 1 block of 64448 bytes in file 1 >>>> End of File mark. >>>> 2 blocks of 64448 bytes in file 2 >>>> End of File mark. >>>> 3 blocks of 64448 bytes in file 3 >>>> End of File mark. >>>> 1 block of 64448 bytes in file 4 >>>> End of File mark. >>>> Total files=4, blocks=7, bytes = 451,136 >>>> End scanning the tape. >>>> We should be in file 4. I am at file 4. This is correct! >>>> >>>> >>>> It looks like the test worked this time, please add: >>>> >>>> Hardware End of Medium = No >>>> >>>> Fast Forward Space File = No >>>> to your Device resource in the Storage conf file. >>>> >>>> The above Bacula scan should have output identical to what follows. >>>> Please double check it ... >>>> === Sample correct output === >>>> 1 block of 64448 bytes in file 1 >>>> End of File mark. >>>> 2 blocks of 64448 bytes in file 2 >>>> End of File mark. >>>> 3 blocks of 64448 bytes in file 3 >>>> End of File mark. >>>> 1 block of 64448 bytes in file 4 >>>> End of File mark. >>>> Total files=4, blocks=7, bytes = 451,136 >>>> === End sample correct output === >>>> >>>> If the above scan output is not identical to the >>>> sample output, you MUST correct the problem >>>> or Bacula will not be able to write multiple Jobs to >>>> the tape. >>>> >>>> Skipping read backwards test because BSR turned off. >>>> >>>> >>>> === Forward space files test === >>>> >>>> This test is essential to Bacula. >>>> >>>> I'm going to write five files then test forward spacing >>>> >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. >>>> btape: btape.c:1909-0 Wrote block to device. >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1634-0 Now forward spacing 1 file. >>>> We should be in file 1. I am at file 1. This is correct! >>>> btape: btape.c:1646-0 Now forward spacing 2 files. >>>> We should be in file 3. I am at file 3. This is correct! >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1659-0 Now forward spacing 4 files. >>>> We should be in file 4. I am at file 4. This is correct! >>>> >>>> btape: btape.c:1677-0 Now forward spacing 1 more file. >>>> We should be in file 5. I am at file 5. This is correct! >>>> >>>> === End Forward space files test === >>>> >>>> >>>> Ah, I see you have an autochanger configured. >>>> To test the autochanger you must have a blank tape >>>> that I can write on in Slot 1. >>>> >>>> Do you wish to continue with the Autochanger test? (y/n): y >>>> >>>> >>>> === Autochanger test === >>>> >>>> 3301 Issuing autochanger "loaded" command. >>>> Nothing loaded in the drive. OK. >>>> 3303 Issuing autochanger "load 1 0" command. >>>> 3303 Autochanger "load 1 0" status is OK. >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK >>>> btape: btape.c:1564-0 Rewound "DLT" (/dev/nsa1) >>>> btape: btape.c:1571-0 Wrote EOF to "DLT" (/dev/nsa1) >>>> >>>> The test autochanger worked!! >>> >>> Great, thanks for running the test! Looks like things are working as well >>> as they were before. >>> >>>>> >>>>>> I'll let the other Bacula devs know about this. They deal with the hardware. I work on PostgreSQL. >>>>>> >>>>> >>>>> Thanks! If there are additional features they would like out of the tape >>>>> driver, I'm happy to talk about it. (Or help if they'd like to use the new >>>>> status reporting ioctl, MTIOCEXTGET or any of the other new ioctls.) >>>> >>>> Errors are interesting to me. Especially corrected errors. They are a good indicator of tape quality. >>>> >>> >>> Yes. At least on modern drives, there is a good bit available in the log >>> pages. You can try seeing what logs your drive supports by installing the >>> sg3_utils package and running 'sg_logs sa1 --all'. >>> >>> Given the large amount of data available on some drives, and the difficulty >>> distilling it down to a clear good/bad, I probably won't stick that in the >>> 'mt status' output. >>> >>> But you can certainly take a look at it and have an idea of whether your >>> particular tape/drive are experiencing issues. >> >> That's a lot of output: https://gist.github.com/dlangille/0e15a7fbf7acab56fd32 > > That is a lot. > >> I will run some Bacula jobs soon. I'm still setting up config files. > > Thanks for all the testing, I really appreciate it! > > Ken > -- > Kenneth Merry > ken_at_FreeBSD.ORG — Dan Langille http://langille.org/Received on Mon Mar 02 2015 - 15:10:00 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:56 UTC