On Mon, Mar 02, 2015 at 16:34:34 -0500, Dan Langille wrote: > > > On Mar 2, 2015, at 2:47 PM, Dan Langille <dan_at_langille.org> wrote: > > > > > >> On Mar 2, 2015, at 2:07 PM, Dan Langille <dan_at_langille.org> wrote: > >> > >> > >>> On Mar 2, 2015, at 12:28 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > >>> > >>> On Mon, Mar 02, 2015 at 11:44:09 -0500, Dan Langille wrote: > >>>> > >>>>> On Mar 2, 2015, at 11:31 AM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > >>>>> > >>>>> On Mon, Mar 02, 2015 at 11:09:57 -0500, Dan Langille wrote: > >>>>>> > >>>>>>> On Mar 1, 2015, at 9:29 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > >>>>>>> > >>>>>>> On Sun, Mar 01, 2015 at 19:41:07 -0500, Dan Langille wrote: > >>>>>>>> > >>>>>>>>> On Mar 1, 2015, at 7:31 PM, Kenneth D. Merry <ken_at_FreeBSD.ORG> wrote: > >>>>>>>>> > >>>>>>>>> On Sun, Mar 01, 2015 at 19:15:05 -0500, Dan Langille wrote: > >>>>>>>>>> > >>>>>>>>>>> On Feb 17, 2015, at 1:36 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: > >>>>>>>>>>> > >>>>>>>>>>> On Sat, Feb 14, 2015 at 18:22:43 -0500, Dan Langille wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> On Feb 13, 2015, at 7:32 PM, Kenneth D. Merry <ken_at_freebsd.org> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> I have a fairly large set of changes to the sa(4) driver and mt(1) driver > >>>>>>>>>>>>> that I'm planning to commit in the near future. > >>>>>>>>>>>>> > >>>>>>>>>>>>> A description of the changes is here and below in this message. > >>>>>>>>>>>>> > >>>>>>>>>>>>> If you have tape hardware and the inclination, I'd appreciate testing and > >>>>>>>>>>>>> feedback. > >>>>>>>>>>>> > >>>>>>>>>>>> I have a DLT 8000 and an SDLT 220. > >>>>>>>>>>>> > >>>>>>>>>>>> I don't have anything running current, but I have a spare machine which I could use for testing. > >>>>>>>>>>>> > >>>>>>>>>>>> Do you see any value is tests with that hardware? I'd be testing it via Bacula. > >>>>>>>>>>>> > >>>>>>>>>>>> disclosure: I'm the sysutils/bacula-* maintainer and a Bacula committer. > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Actually, yes. Bacula is a bit tricky to configure, so your trying it out > >>>>>>>>>>> would be helpful if you have the time. > >>>>>>>>>>> > >>>>>>>>>>> In looking at the manuals for both the SDLT 220 and the DLT 8000, they both > >>>>>>>>>>> claim to support long position information for the SCSI READ POSITION > >>>>>>>>>>> command. > >>>>>>>>>>> > >>>>>>>>>>> You can see what I'm talking about by doing: > >>>>>>>>>>> > >>>>>>>>>>> mt eod > >>>>>>>>>>> mt status > >>>>>>>>>>> > >>>>>>>>>>> On my DDS-4 tape drive, this shows: > >>>>>>>>>>> > >>>>>>>>>>> # mt -f /dev/nsa3 status > >>>>>>>>>>> Drive: sa3: <SEAGATE DAT 06240-XXX 8071> Serial Number: HJ00YWY > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Mode Density Blocksize bpi Compression > >>>>>>>>>>> Current: 0x26:DDS-4 1024 bytes 97000 enabled (DCLZ) > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Current Driver State: at rest. > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Partition: 0 Calc File Number: -1 Calc Record Number: -1 > >>>>>>>>>>> Residual: 0 Reported File Number: -1 Reported Record Number: -1 > >>>>>>>>>>> Flags: None > >>>>>>>>>>> > >>>>>>>>>>> But on an LTO-5, which will give long position information, I get: > >>>>>>>>>>> > >>>>>>>>>>> [root_at_doc ~]# mt status > >>>>>>>>>>> Drive: sa0: <IBM ULTRIUM-HH5 E4J1> > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Mode Density Blocksize bpi Compression > >>>>>>>>>>> Current: 0x58:LTO-5 variable 384607 enabled (0x1) > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Current Driver State: at rest. > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Partition: 0 Calc File Number: 2 Calc Record Number: -1 > >>>>>>>>>>> Residual: 0 Reported File Number: 2 Reported Record Number: 32373 > >>>>>>>>>>> Flags: None > >>>>>>>>>>> > >>>>>>>>>>> That, in combination with the changes I made to the position information > >>>>>>>>>>> code in the driver, mean that even the old MTIOCGET ioctl should return an > >>>>>>>>>>> accurate file number at end of data. e.g., on the LTO-5: > >>>>>>>>>>> > >>>>>>>>>>> [root_at_doc ~]# mt ostatus > >>>>>>>>>>> Mode Density Blocksize bpi Compression > >>>>>>>>>>> Current: 0x58:LTO-5 variable 384607 0x1 > >>>>>>>>>>> ---------available modes--------- > >>>>>>>>>>> 0: 0x58:LTO-5 variable 384607 0x1 > >>>>>>>>>>> 1: 0x58:LTO-5 variable 384607 0x1 > >>>>>>>>>>> 2: 0x58:LTO-5 variable 384607 0x1 > >>>>>>>>>>> 3: 0x58:LTO-5 variable 384607 0x1 > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> Current Driver State: at rest. > >>>>>>>>>>> --------------------------------- > >>>>>>>>>>> File Number: 2 Record Number: -1 Residual Count -1 > >>>>>>>>>>> > >>>>>>>>>>> So the thing to try, in addition to just making sure that Bacula continues > >>>>>>>>>>> to work properly, is to try setting this for the tape drive in > >>>>>>>>>>> bacula-sd.conf: > >>>>>>>>>>> > >>>>>>>>>>> Hardware End of Medium = yes > >>>>>>>>>>> > >>>>>>>>>>> It looks like the Bacula tape program (btape) has a test mode, and it would > >>>>>>>>>>> be good to run through the tests on one of the tape drives and see whether > >>>>>>>>>>> they work, and whether the results are different before and after the > >>>>>>>>>>> changes. I'm not sure how to enable the test mode. > >>>>>>>>>> > >>>>>>>>>> I have this in /usr/local/etc/bacula/bacula-sd.conf > >>>>>>>>>> > >>>>>>>>>> Device { > >>>>>>>>>> Name = DLT > >>>>>>>>>> Description = "QUANTUM DLT7000 1624" > >>>>>>>>>> Media Type = DLT > >>>>>>>>>> Archive Device = /dev/nsa1 > >>>>>>>>>> > >>>>>>>>>> Autochanger = YES > >>>>>>>>>> Drive Index = 0 > >>>>>>>>>> > >>>>>>>>>> Offline On Unmount = no > >>>>>>>>>> Hardware End of Medium = yes > >>>>>>>>>> BSF at EOM = yes > >>>>>>>>>> Backward Space Record = no > >>>>>>>>>> Fast Forward Space File = no > >>>>>>>>>> TWO EOF = yes > >>>>>>>>>> } > >>>>>>>>>> > >>>>>>>>>> FYI, http://www.freebsddiary.org/digital-tl891.php (from 2006) has a btape test on this same model. > >>>>>>>>>> > >>>>>>>>>> Here's the test I ran tonight: > >>>>>>>>>> > >>>>>>>>>> [root_at_cuppy:/usr/home/dan] # btape -c /usr/local/etc/bacula/bacula-sd.conf /dev/nsa1 > >>>>>>>>>> Tape block granularity is 1024 bytes. > >>>>>>>>>> btape: butil.c:287-0 Using device: "/dev/nsa1" for writing. > >>>>>>>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>>>>>>>> *test > >>>>>>>>>> > >>>>>>>>>> === Write, rewind, and re-read test === > >>>>>>>>>> > >>>>>>>>>> I'm going to write 10000 records and an EOF > >>>>>>>>>> then write 10000 records and an EOF, then rewind, > >>>>>>>>>> and re-read the data to verify that it is correct. > >>>>>>>>>> > >>>>>>>>>> This is an *essential* feature ... > >>>>>>>>>> > >>>>>>>>>> btape: btape.c:1152-0 Wrote 10000 blocks of 64412 bytes. > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1168-0 Wrote 10000 blocks of 64412 bytes. > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1210-0 Rewind OK. > >>>>>>>>>> 10000 blocks re-read correctly. > >>>>>>>>>> Got EOF on tape. > >>>>>>>>>> 10000 blocks re-read correctly. > >>>>>>>>>> === Test Succeeded. End Write, rewind, and re-read test === > >>>>>>>>>> > >>>>>>>>>> btape: btape.c:1277-0 Block position test > >>>>>>>>>> btape: btape.c:1289-0 Rewind OK. > >>>>>>>>>> Reposition to file:block 0:4 > >>>>>>>>>> Block 5 re-read correctly. > >>>>>>>>>> Reposition to file:block 0:200 > >>>>>>>>>> Block 201 re-read correctly. > >>>>>>>>>> Reposition to file:block 0:9999 > >>>>>>>>>> Block 10000 re-read correctly. > >>>>>>>>>> Reposition to file:block 1:0 > >>>>>>>>>> Block 10001 re-read correctly. > >>>>>>>>>> Reposition to file:block 1:600 > >>>>>>>>>> Block 10601 re-read correctly. > >>>>>>>>>> Reposition to file:block 1:9999 > >>>>>>>>>> Block 20000 re-read correctly. > >>>>>>>>>> === Test Succeeded. End Write, rewind, and re-read test === > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> === Append files test === > >>>>>>>>>> > >>>>>>>>>> This test is essential to Bacula. > >>>>>>>>>> > >>>>>>>>>> I'm going to write one record in file 0, > >>>>>>>>>> two records in file 1, > >>>>>>>>>> and three records in file 2 > >>>>>>>>>> > >>>>>>>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:1907-0 Wrote one record of 64412 bytes. > >>>>>>>>>> btape: btape.c:1909-0 Wrote block to device. > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:604-0 Wrote 1 EOF to "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:469-0 open device "DLT" (/dev/nsa1): OK > >>>>>>>>>> btape: btape.c:574-0 Rewound "DLT" (/dev/nsa1) > >>>>>>>>>> btape: btape.c:1420-0 Now moving to end of medium. > >>>>>>>>> > >>>>>>>>> This is the critical piece. The test moves the tape to the end of the > >>>>>>>>> medium. With hardware position information, you can tell what filemark > >>>>>>>>> you're on. Without it, you can't. > >>>>>>>>> > >>>>>>>>>> btape: btape.c:622-0 tape_dev.c:345 ioctl MTIOCGET error on "DLT" (/dev/nsa1). ERR=No error: 0. > >>>>>>>>>> We should be in file 3. I am at file 0. This is NOT correct!!!! > >>>>>>>>>> > >>>>>>>>>> Append test failed. Attempting again. > >>>>>>>>>> Setting "Hardware End of Medium = no > >>>>>>>>>> and "Fast Forward Space File = no > >>>>>>>>>> and retrying append test. > >>>>>>>>> > >>>>>>>>> This is not surprsing, given that the drive doesn't support long read > >>>>>>>>> position data. (It's a SCSI-2 device.) So that means that Bacula will > >>>>>>>>> need to do it manually. > >>>>>>>> > >>>>>>>> Yes, I have nothing newer than SCSI-2. Even my SDLT is SCSI-2 but that > >>>>>>>> tape library is hooked up to a different computer and was doing backups today. > >>>>>>> > >>>>>>> So, here is one thing that we can try to see whether these drives support > >>>>>>> long position information, even though they only claim to be SCSI-2. If > >>>>>>> they do, we can potentially add a quirk (or autodetection) to enable it. > >>>>>>> The code currently doesn't bother asking drives that claim to be SCSI-2 > >>>>>>> for long position information. (Because that feature was added in the > >>>>>>> SSC spec, which came after SCSI-2.) > >>>>>>> > >>>>>>> Issue a READ POSITION with the short form specified: > >>>>>>> > >>>>>>> camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > >>>>>>> > >>>>>>> Issue a READ POSITION with the vendor-specific block numbers: > >>>>>>> > >>>>>>> camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > >>>>>>> > >>>>>>> Issue a READ POSITION with the long form data: > >>>>>>> > >>>>>>> camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > >>>>>>> > >>>>>>> If it supports the last one, then I can put a quirk (or autodetection) in > >>>>>>> the driver and Bacula will get the hardware filemarks. You should try this > >>>>>>> on your SDLT as well. It may well support it. > >>>>>> > >>>>>> Sadly, no: > >>>>>> > >>>>>> [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 0 0 0 0 0 0 0 0 0" -i 20 - |hd > >>>>>> 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > >>>>>> 00000010 00 00 00 00 |....| > >>>>>> 00000014 > >>>>>> [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 1 0 0 0 0 0 0 0 0" -i 20 - |hd > >>>>>> 00000000 80 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| > >>>>>> 00000010 00 00 00 00 |....| > >>>>>> 00000014 > >>>>>> [root_at_cuppy:~] # camcontrol cmd sa1 -v -c "34 6 0 0 0 0 0 0 0 0" -i 32 - |hd > >>>>>> camcontrol: error sending command > >>>>>> (pass2:ahc0:0:2:0): READ POSITION. CDB: 34 06 00 00 00 00 00 00 00 00 > >>>>>> (pass2:ahc0:0:2:0): CAM status: SCSI Status Error > >>>>>> (pass2:ahc0:0:2:0): SCSI status: Check Condition > >>>>>> (pass2:ahc0:0:2:0): SCSI sense: ILLEGAL REQUEST asc:24,0 (Invalid field in CDB) > >>>>>> (pass2:ahc0:0:2:0): Command byte 1 bit 2 is invalid > >>>>>> [root_at_cuppy:~] # > >>>>> > >>>>> Okay. Not too surprising I suppose. > >> > >> > >> Does this mean much to you? > >> > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Sending SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Received SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: Filtered to period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Sending SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Received SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: Filtered to period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Sending SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: (ahc0:A:2:0): Received SDTR period 19, offset f > >> Mar 2 19:42:59 cuppy kernel: Filtered to period 19, offset f > >> > >> > >> I'm having trouble with labeling barcodes from Bacula and the above is seen in /var/log/messages > > > > The barcodes issue is resolved. I changed to using drive 0 instead of drive 1, now that we have both drives. > > > > *label barcodes slots=3,4 > > The defined Storage resources are: > > 1: File1 > > 2: OldTL891 > > Select Storage resource (1-2): 2 > > Connecting to Storage daemon OldTL891 at 10.55.0.18:9103 ... > > 3306 Issuing autochanger "slots" command. > > Device "DTL03" has 10 slots. > > Connecting to Storage daemon OldTL891 at 10.55.0.18:9103 ... > > 3306 Issuing autochanger "list" command. > > The following Volumes will be labeled: > > Slot Volume > > ============== > > 3 FAI261 > > 4 FAI262 > > Do you want to label these Volumes? (yes|no): yes > > Defined Pools: > > 1: Default > > 2: File > > 3: MYDLT > > 4: Scratch > > Select the Pool (1-4): 4 > > Connecting to Storage daemon OldTL891 at 10.55.0.18:9103 ... > > Sending label command for Volume "FAI261" Slot 3 ... > > 3000 OK label. VolBytes=64512 DVD=0 Volume="FAI261" Device="DTL03" (/dev/nsa0) > > Catalog record for Volume "FAI261", Slot 3 successfully created. > > Sending label command for Volume "FAI262" Slot 4 ... > > 3307 Issuing autochanger "unload slot 3, drive 0" command. > > 3304 Issuing autochanger "load slot 4, drive 0" command. > > 3305 Autochanger "load slot 4, drive 0", status is OK. > > 3000 OK label. VolBytes=64512 DVD=0 Volume="FAI262" Device="DTL03" (/dev/nsa0) > > Catalog record for Volume "FAI262", Slot 4 successfully created. > > *list volumes > > Pool: Default > > No results to list. > > Pool: File > > +---------+------------+-----------+---------+------------+----------+--------------+---------+------+-----------+-----------+---------------------+ > > | mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten | > > +---------+------------+-----------+---------+------------+----------+--------------+---------+------+-----------+-----------+---------------------+ > > | 1 | Vol-0001 | Append | 1 | 19,835,982 | 0 | 31,536,000 | 1 | 0 | 0 | File1 | 2015-03-02 17:50:40 | > > | 2 | Vol-0002 | Append | 1 | 0 | 0 | 31,536,000 | 1 | 0 | 0 | DLT | | > > +---------+------------+-----------+---------+------------+----------+--------------+---------+------+-----------+-----------+---------------------+ > > Pool: MYDLT > > No results to list. > > Pool: Scratch > > +---------+------------+-----------+---------+----------+----------+--------------+---------+------+-----------+-----------+-------------+ > > | mediaid | volumename | volstatus | enabled | volbytes | volfiles | volretention | recycle | slot | inchanger | mediatype | lastwritten | > > +---------+------------+-----------+---------+----------+----------+--------------+---------+------+-----------+-----------+-------------+ > > | 4 | FAI260 | Append | 1 | 64,512 | 0 | 31,536,000 | 1 | 1 | 1 | DLT | | > > | 5 | FAI263 | Append | 1 | 64,512 | 0 | 31,536,000 | 1 | 2 | 1 | DLT | | > > | 7 | FAI261 | Append | 1 | 64,512 | 0 | 31,536,000 | 1 | 3 | 1 | DLT | | > > | 8 | FAI262 | Append | 1 | 64,512 | 0 | 31,536,000 | 1 | 4 | 1 | DLT | | > > +---------+------------+-----------+---------+----------+----------+--------------+---------+------+-----------+-----------+-------------+ > > * > > > > But this is what arrives in /var/log/messages during the above: > > > > Mar 2 20:29:06 cuppy kernel: (ahc0:A:1:0): Sending SDTR period 19, offset f > > Mar 2 20:29:06 cuppy kernel: (ahc0:A:1:0): Received SDTR period 19, offset f > > Mar 2 20:29:06 cuppy kernel: Filtered to period 19, offset f > > Mar 2 20:30:22 cuppy kernel: (ahc0:A:1:0): Sending SDTR period 19, offset f > > Mar 2 20:30:22 cuppy kernel: (ahc0:A:1:0): Received SDTR period 19, offset f > > Mar 2 20:30:22 cuppy kernel: Filtered to period 19, offset f > > Mar 2 20:30:22 cuppy kernel: (ahc0:A:1:0): Sending SDTR period 19, offset f > > Mar 2 20:30:22 cuppy kernel: (ahc0:A:1:0): Received SDTR period 19, offset f > > Mar 2 20:30:22 cuppy kernel: Filtered to period 19, offset f > > Mar 2 20:30:59 cuppy kernel: (ahc0:A:1:0): Sending SDTR period 19, offset f > > Mar 2 20:30:59 cuppy kernel: (ahc0:A:1:0): Received SDTR period 19, offset f > > Mar 2 20:30:59 cuppy kernel: Filtered to period 19, offset f > > Mar 2 20:30:59 cuppy kernel: xpt_release_devq(): requested 1 > present 0 > > Mar 2 20:31:07 cuppy kernel: (ahc0:A:1:0): Sending SDTR period 19, offset f > > Mar 2 20:31:07 cuppy kernel: (ahc0:A:1:0): Received SDTR period 19, offset f > > Mar 2 20:31:07 cuppy kernel: Filtered to period 19, offset f > > > > Anything to be concerned about? > > When adding a 2nd job to a tape: > > 02-Mar 21:29 cuppy-sd JobId 19: Error: Unable to position to end of data on tape device "DTL03" (/dev/nsa0): ERR=tape_dev.c:345 ioctl MTIOCGET error on "DTL03" (/dev/nsa0). ERR=No error: 0. > > I've gotten this on three consecutive tapes. NOTE: These *are* tapes I no longer use because of higher rates of corrected errors. > > The problem seems to be locating the end of the data. Do you have 'Hardware End of Medium = no' in the configuration file? In looking at the code, it may be set to yes, and that could be the issue here. > > I can run a 4 or 5 jobs in a row, but if I restore a job, then add a job, it fails with the above message. > I would expect that on these particular tape drives because they don't support long position information. Ken -- Kenneth Merry ken_at_FreeBSD.ORGReceived on Mon Mar 02 2015 - 20:42:44 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:56 UTC