On Mon, Mar 02, 2015 at 11:09:57 -0500, Dan Langille wrote: > > > On Mar 1, 2015, at 9:29 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > > > > On Sun, Mar 01, 2015 at 19:41:07 -0500, Dan Langille wrote: > >> > >>> On Mar 1, 2015, at 7:31 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > >>> > >>> On Sun, Mar 01, 2015 at 19:15:05 -0500, Dan Langille wrote: > >>>> > >>>>> On Feb 17, 2015, at 1:36 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: > >>>>> > >>>>> On Sat, Feb 14, 2015 at 18:22:43 -0500, Dan Langille wrote: > >>>>>> > >>>>>>> On Feb 13, 2015, at 7:32 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: > >>>>>>> > >>>>>>> > >>>>>>> I have a fairly large set of changes to the sa(4) driver and mt(1) driver > >>>>>>> that I'm planning to commit in the near future. > >>>>>>> > >>>>>>> A description of the changes is here and below in this message. > >>>>>>> > >>>>>>> If you have tape hardware and the inclination, I'd appreciate testing and > >>>>>>> feedback. > >>>>>> > >>>>>> I have a DLT 8000 and an SDLT 220. > >>>>>> > >>>>>> I don't have anything running current, but I have a spare machine which I could use for testing. > >>>>>> > >>>>>> Do you see any value is tests with that hardware? I'd be testing it via Bacula. > >>>>>> > >>>>>> disclosure: I'm the sysutils/bacula-* maintainer and a Bacula committer. > >>>>>> > >>>>> > >>>>> Actually, yes. Bacula is a bit tricky to configure, so your trying it out > >>>>> would be helpful if you have the time. > >>>>> > >>>>> In looking at the manuals for both the SDLT 220 and the DLT 8000, they both > >>>>> claim to support long position information for the SCSI READ POSITION > >>>>> command. > >>>>> > >>>>> You can see what I'm talking about by doing: > >>>>> > >>>>> mt eod > >>>>> mt status > >>>>> > >>>>> On my DDS-4 tape drive, this shows: > >>>>> > >>>>> # mt -f /dev/nsa3 status > >>>>> Drive: sa3: <SEAGATE DAT 06240-XXX 8071> Serial Number: HJ00YWY > >>>>> --------------------------------- > >>>>> Mode Density Blocksize bpi Compression > >>>>> Current: 0x26:DDS-4 1024 bytes 97000 enabled (DCLZ) > >>>>> --------------------------------- > >>>>> Current Driver State: at rest. > >>>>> --------------------------------- > >>>>> Partition: 0 Calc File Number: -1 Calc Record Number: -1 > >>>>> Residual: 0 Reported File Number: -1 Reported Record Number: -1 > >>>>> Flags: None > >>>>> > >>>>> But on an LTO-5, which will give long position information, I get: > >>>>> > >>>>> [root_at_doc ~]# mt status > >>>>> Drive: sa0: <IBM ULTRIUM-HH5 E4J1> > >>>>> --------------------------------- > >>>>> Mode Density Blocksize bpi Compression > >>>>> Current: 0x58:LTO-5 variable 384607 enabled (0x1) > >>>>> --------------------------------- > >>>>> Current Driver State: at rest. > >>>>> --------------------------------- > >>>>> Partition: 0 Calc File Number: 2 Calc Record Number: -1 > >>>>> Residual: 0 Reported File Number: 2 Reported Record Number: 32373 > >>>>> Flags: None > >>>>> > >>>>> That, in combination with the changes I made to the position information > >>>>> code in the driver, mean that even the old MTIOCGET ioctl should return an > >>>>> accurate file number at end of data. e.g., on the LTO-5: > >>>>> > >>>>> [root_at_doc ~]# mt ostatus > >>>>> Mode Density Blocksize bpi Compression > >>>>> Current: 0x58:LTO-5 variable 384607 0x1 > >>>>> ---------available modes--------- > >>>>> 0: 0x58:LTO-5 variable 384607 0x1 > >>>>> 1: 0x58:LTO-5 variable 384607 0x1 > >>>>> 2: 0x58:LTO-5 variable 384607 0x1 > >>>>> 3: 0x58:LTO-5 variable 384607 0x1 > >>>>> --------------------------------- > >>>>> Current Driver State: at rest. > >>>>> --------------------------------- > >>>>> File Number: 2 Record Number: -1 Residual Count -1 > >>>>> > >>>>> So the thing to try, in addition to just making sure that Bacula continues > >>>>> to work properly, is to try setting this for the tape drive in > >>>>> bacula-sd.conf: > >>>>> > >>>>> Hardware End of Medium = yes > >>>>> > >>>>> It looks like the Bacula tape program (btape) has a test mode, and it would > >>>>> be good to run through the tests on one of the tape drives and see whether > >>>>> they work, and whether the results are different before and after the > >>>>> changes. I'm not sure how to enable the test mode. > >>>> > >>>> I have this in /usr/local/etc/bacula/bacula-sd.conf > >>>> > >>>> Device { > >>>> Name = DLT > >>>> Description = "QUANTUM DLT7000 1624" > >>>> Media Type = DLT > >>>> Archive Device = /dev/nsa1 > >>>> > >>>> Autochanger = YES > >>>> Drive Index = 0 > >>>> > >>>> Offline On Unmount = no > >>>> Hardware End of Medium = yes > >>>> BSF at EOM = yes > >>>> Backward Space Record = no > >>>> Fast Forward Space File = no > >>>> TWO EOF = yes > >>>> } > >>>> > >>>> FYI, http://www.freebsddiary.org/digital-tl891.php (from 2006) has a btape test on this same model. > >>>> > >>>> Here's the test I ran tonight: > >>>> > >>>> [root_at_cuppy:/usr/home/dan] # btape -c /usr/local/etc/bacula/bacula-sd.conf /dev/nsa1 > >>>> Tape block granularity is 1024 bytes. > >>>> btape: butil.c:287-0 Using device: "/dev/nsa1" for writing. > >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>> *test > >>>> > >>>> === Write, rewind, and re-read test === > >>>> > >>>> I'm going to write 10000 records and an EOF > >>>> then write 10000 records and an EOF, then rewind, > >>>> and re-read the data to verify that it is correct. > >>>> > >>>> This is an *essential* feature ... > >>>> > >>>> btape: btape.c:1152-0 Wrote 10000 blocks of 64412 bytes. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1168-0 Wrote 10000 blocks of 64412 bytes. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1210-0 Rewind OK. > >>>> 10000 blocks re-read correctly. > >>>> Got EOF on tape. > >>>> 10000 blocks re-read correctly. > >>>> === Test Succeeded. End Write, rewind, and re-read test === > >>>> > >>>> btape: btape.c:1277-0 Block position test > >>>> btape: btape.c:1289-0 Rewind OK. > >>>> Reposition to file:block 0:4 > >>>> Block 5 re-read correctly. > >>>> Reposition to file:block 0:200 > >>>> Block 201 re-read correctly. > >>>> Reposition to file:block 0:9999 > >>>> Block 10000 re-read correctly. > >>>> Reposition to file:block 1:0 > >>>> Block 10001 re-read correctly. > >>>> Reposition to file:block 1:600 > >>>> Block 10601 re-read correctly. > >>>> Reposition to file:block 1:9999 > >>>> Block 20000 re-read correctly. > >>>> === Test Succeeded. End Write, rewind, and re-read test === > >>>> > >>>> > >>>> > >>>> === Append files test === > >>>> > >>>> This test is essential to Bacula. > >>>> > >>>> I'm going to write one record in file 0, > >>>> two records in file 1, > >>>> and three records in file 2 > >>>> > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1420-0 Now moving to end of medium. > >>> > >>> This is the critical piece. The test moves the tape to the end of the > >>> medium. With hardware position information, you can tell what filemark > >>> you're on. Without it, you can't. > >>> > >>>> btape: btape.c:622-0 tape_dev.c:345 ioctl MTIOCGET error on "DLT" (/dev/nsa1). ERR=No error: 0. > >>>> We should be in file 3. I am at file 0. This is NOT correct!!!! > >>>> > >>>> Append test failed. Attempting again. > >>>> Setting "Hardware End of Medium = no > >>>> and "Fast Forward Space File = no > >>>> and retrying append test. > >>> > >>> This is not surprsing, given that the drive doesn't support long read > >>> position data. (It's a SCSI-2 device.) So that means that Bacula will > >>> need to do it manually. > >> > >> Yes, I have nothing newer than SCSI-2. Even my SDLT is SCSI-2 but that > >> tape library is hooked up to a different computer and was doing backups today. > > > > So, here is one thing that we can try to see whether these drives support > > long position information, even though they only claim to be SCSI-2. If > > they do, we can potentially add a quirk (or autodetection) to enable it. > > The code currently doesn't bother asking drives that claim to be SCSI-2 > > for long position information. (Because that feature was added in the > > SSC spec, which came after SCSI-2.) > > > > Issue a READ POSITION with the short form specified: > > > > camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > > > > Issue a READ POSITION with the vendor-specific block numbers: > > > > camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > > > > Issue a READ POSITION with the long form data: > > > > camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > > > > If it supports the last one, then I can put a quirk (or autodetection) in > > the driver and Bacula will get the hardware filemarks. You should try this > > on your SDLT as well. It may well support it. > > Sadly, no: > > [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000010 00 00 00 00 |....| > 00000014 > [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000010 00 00 00 00 |....| > 00000014 > [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > camcontrol: error sending command > (pass2:ahc0:0:2:0): READ POSITION. CDB: 34 06 00 00 00 00 00 00 00 00 > (pass2:ahc0:0:2:0): CAM status: SCSI Status Error > (pass2:ahc0:0:2:0): SCSI status: Check Condition > (pass2:ahc0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) > (pass2:ahc0:0:2:0): Command byte 1 bit 2 is invalid > [root_at_cuppy:~] # Okay. Not too surprising I suppose. > The SDLT server is on 9.3 though: > > [root_at_knew:/usr/home/dan] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > camcontrol: cam_lookup_pass: CAMGETPASSTHRU ioctl failed > cam_lookup_pass: No such file or directory > cam_lookup_pass: either the pass driver isn't in your kernel > cam_lookup_pass: or sa1 doesn't exist > [root_at_knew:/usr/home/dan] # uname -a > FreeBSD knew.unixathome.org 9.3-RELEASE-p10 FreeBSD 9.3-RELEASE-p10 #0: Tue Feb 24 21:28:03 UTC 2015 root_at_amd64-builder.daemonology.net:/usr/obj/usr/src/sys/GENERIC amd64 > [root_at_knew:/usr/home/dan] # > > > It took me a while to figure that out... there is no sa1 on *this* system. > > But, my SDLT: > > [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000010 00 00 00 00 |....| > 00000014 > [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000010 00 00 00 00 |....| > 00000014 > [root_at_knew:/usr/home/dan] # camcontrol cmd sa0 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > 00000020 > [root_at_knew:/usr/home/dan] # Just to confirm, can you send the output of: camcontrol inquiry sa0 -v I want to make certain it reports that it is SCSI-2. If so, I'll change the check in the driver to try asking for long position information on SCSI-2 devices. If it fails, it'll fall back to the regular method. > > > > Google didn't quickly produce a SCSI manual for the DEC drive, but the > > Quantum SDLT manual indicates that it supports long position data, despite > > identifying itself as a SCSI-2 drive. > > > >>> === Append files test === > >>>> > >>>> This test is essential to Bacula. > >>>> > >>>> I'm going to write one record in file 0, > >>>> two records in file 1, > >>>> and three records in file 2 > >>>> > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1420-0 Now moving to end of medium. > >>>> btape: btape.c:625-0 Moved to end of medium. > >>>> We should be in file 3. I am at file 3. This is correct! > >>>> > >>>> Now the important part, I am going to attempt to append to the tape. > >>>> > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> Done appending, there should be no I/O errors > >>>> > >>>> Doing Bacula scan of blocks: > >>>> 1 block of 64448 bytes in file 1 > >>>> End of File mark. > >>>> 2 blocks of 64448 bytes in file 2 > >>>> End of File mark. > >>>> 3 blocks of 64448 bytes in file 3 > >>>> End of File mark. > >>>> 1 block of 64448 bytes in file 4 > >>>> End of File mark. > >>>> Total files=4, blocks=7, bytes = 451,136 > >>>> End scanning the tape. > >>>> We should be in file 4. I am at file 4. This is correct! > >>>> > >>>> > >>>> It looks like the test worked this time, please add: > >>>> > >>>> Hardware End of Medium = No > >>>> > >>>> Fast Forward Space File = No > >>>> to your Device resource in the Storage conf file. > >>>> > >>>> The above Bacula scan should have output identical to what follows. > >>>> Please double check it ... > >>>> === Sample correct output === > >>>> 1 block of 64448 bytes in file 1 > >>>> End of File mark. > >>>> 2 blocks of 64448 bytes in file 2 > >>>> End of File mark. > >>>> 3 blocks of 64448 bytes in file 3 > >>>> End of File mark. > >>>> 1 block of 64448 bytes in file 4 > >>>> End of File mark. > >>>> Total files=4, blocks=7, bytes = 451,136 > >>>> === End sample correct output === > >>>> > >>>> If the above scan output is not identical to the > >>>> sample output, you MUST correct the problem > >>>> or Bacula will not be able to write multiple Jobs to > >>>> the tape. > >>>> > >>>> Skipping read backwards test because BSR turned off. > >>>> > >>>> > >>>> === Forward space files test === > >>>> > >>>> This test is essential to Bacula. > >>>> > >>>> I'm going to write five files then test forward spacing > >>>> > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>> btape: btape.c:1909-0 Wrote block to device. > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1634-0 Now forward spacing 1 file. > >>>> We should be in file 1. I am at file 1. This is correct! > >>>> btape: btape.c:1646-0 Now forward spacing 2 files. > >>>> We should be in file 3. I am at file 3. This is correct! > >>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1659-0 Now forward spacing 4 files. > >>>> We should be in file 4. I am at file 4. This is correct! > >>>> > >>>> btape: btape.c:1677-0 Now forward spacing 1 more file. > >>>> We should be in file 5. I am at file 5. This is correct! > >>>> > >>>> === End Forward space files test === > >>>> > >>>> > >>>> Ah, I see you have an autochanger configured. > >>>> To test the autochanger you must have a blank tape > >>>> that I can write on in Slot 1. > >>>> > >>>> Do you wish to continue with the Autochanger test? (y/n): y > >>>> > >>>> > >>>> === Autochanger test === > >>>> > >>>> 3301 Issuing autochanger "loaded" command. > >>>> Nothing loaded in the drive. OK. > >>>> 3303 Issuing autochanger "load 1 0" command. > >>>> 3303 Autochanger "load 1 0" status is OK. > >>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>> btape: btape.c:1564-0 Rewound "DLT" (/dev/nsa1) > >>>> btape: btape.c:1571-0 Wrote EOF to "DLT" (/dev/nsa1) > >>>> > >>>> The test autochanger worked!! > >>> > >>> Great, thanks for running the test! Looks like things are working as well > >>> as they were before. > >>> > >>>>> > >>>>>> I'll let the other Bacula devs know about this. They deal with the hardware. I work on PostgreSQL. > >>>>>> > >>>>> > >>>>> Thanks! If there are additional features they would like out of the tape > >>>>> driver, I'm happy to talk about it. (Or help if they'd like to use the new > >>>>> status reporting ioctl, MTIOCEXTGET or any of the other new ioctls.) > >>>> > >>>> Errors are interesting to me. Especially corrected errors. They are a good indicator of tape quality. > >>>> > >>> > >>> Yes. At least on modern drives, there is a good bit available in the log > >>> pages. You can try seeing what logs your drive supports by installing the > >>> sg3_utils package and running 'sg_logs sa1 --all'. > >>> > >>> Given the large amount of data available on some drives, and the difficulty > >>> distilling it down to a clear good/bad, I probably won't stick that in the > >>> 'mt status' output. > >>> > >>> But you can certainly take a look at it and have an idea of whether your > >>> particular tape/drive are experiencing issues. > >> > >> That's a lot of output: https://gist.github.com/dlangille/0e15a7fbf7acab56fd32 > > > > That is a lot. > > > >> I will run some Bacula jobs soon. I'm still setting up config files. > > > > Thanks for all the testing, I really appreciate it! > > > > Ken > > -- > > Kenneth Merry > > ken_at_FreeBSD.ORG > > ? > Dan Langille > http://langille.org/ > > > > > Ken -- Kenneth Merry ken_at_FreeBSD.ORGReceived on Mon Mar 02 2015 - 15:31:52 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:56 UTC