Re: zfs send/recv: STILL invalid Backup Stream

On 2014-07-24 16:11, Larry Rosenman wrote:
> On 2014-07-24 15:07, Allan Jude wrote:
>> On 2014-07-24 15:57, Larry Rosenman wrote:
>>> On 2014-07-24 14:53, Mark Martinec wrote:
>>>> 2014-07-24 21:31, Larry Rosenman wrote:
>>>>> borg.lerctr.org /home/ler # zxfer -dFkPvs -g 376 -O
>>>>> root_at_tbh.lerctr.org -R zroot  zroot/backups/TBH
>>>>> Creating recursive snapshot zroot_at_zxfer_26699_20140724135840.
>>>>> Checking grandfather status of all snapshots marked for deletion...
>>>>> Grandfather check passed.
>>>>> Sending zroot_at_zxfer_26699_20140724135840 to zroot/backups/TBH/zroot.
>>>>> Sending zroot/ROOT_at_zxfer_26699_20140724135840 to
>>>>> zroot/backups/TBH/zroot/ROOT.
>>>>> Sending zroot/ROOT/default_at_zxfer_23699_20140724134435 to
>>>>> zroot/backups/TBH/zroot/ROOT/default.
>>>>> Sending zroot/ROOT/default_at_zxfer_26699_20140724135840 to
>>>>> zroot/backups/TBH/zroot/ROOT/default.
>>>>>   (incremental to zroot/ROOT/default_at_zxfer_23699_20140724134435.)
>>>>> Sending zroot/home_at_zxfer_26699_20140724135840 to
>>>>> zroot/backups/TBH/zroot/home.
>>>>
>>>>> Write failed: Cannot allocate memory
>>>>   ====================================
>>>>
>>>>> cannot receive new filesystem stream: invalid backup stream
>>>>> Error when zfs send/receiving.
>>>>> borg.lerctr.org /home/ler #
>>>>>
>>>>> well that's different.......
>>>>
>>>> Sounds familiar, check my posting of today and links therein:
>>>>
>>>>   http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html
>>>>
>>>> Mark
>>> I'm not using netgraph to the best of my knowledge....
>>> and the only fails on the SENDING host are:
>>> 8 Bucket:                64,      0,      41,    3555,  257774,  11,   0
>>> 12 Bucket:               96,      0,      96,    2569,  123653,   0,   0
>>> 16 Bucket:              128,      0,   17195,     506,  215573,   0,   0
>>> 32 Bucket:              256,      0,     340,    4670,  900638,  50,   0
>>> 64 Bucket:              512,      0,   10691,     365, 
>>> 546888,185232,   0
>>> 128 Bucket:            1024,      0,    3563,     905,  348419,   0,   0
>>> 256 Bucket:            2048,      0,    2872,     162, 
>>> 249995,59834,   0
>>> vmem btag:               56,      0,  192811,   51500,  502264,1723,   0
>>>
>>>
>>
>> I regularly use zxfer to transfer 500+ GiB datasets over the internet.
>> This week I actually replicated a 2.1 TiB dataset with zxfer without
>> issue.
>>
>> I wonder which thing is running out of memory. Is there a delay while it
>> is 'running out of memory', or does it fail immediately? Does running
>> top while it is working on running out of memory reveal anything?
>>
>> I would expect to use up a lot of memory while doing deduplication, but
>> not otherwise.
>>
>> Note: I most often use openssh-portable rather than base ssh for
>> replication, as I enable the nonecipher to reduce CPU usage, and adjust
>> the TcpRcvBuf upwards to actually saturate a gigabit over the internet.
> 
> I wasn't watching exactly what it was doing, but the sending box has 16G
> and 18G Swap and swap
> has NOT been touched.
> 
> last pid: 74288;  load averages:  4.70,  5.61,  5.91    up 1+03:14:18 
> 15:10:44
> 115 processes: 3 running, 112 sleeping
> CPU:  0.6% user, 33.3% nice,  0.6% system,  0.1% interrupt, 65.4% idle
> Mem: 847M Active, 761M Inact, 14G Wired, 4616K Cache, 357M Free
> ARC: 12G Total, 6028M MFU, 5281M MRU, 3152K Anon, 120M Header, 688M Other
> Swap: 18G Total, 18G Free
> 
> so I have zero idea where to go here.
> 
> 

Most ZFS memory usage is 'wired' and so cannot be swapped, so lack of
swap activity isn't a good indicator.

-- 
Allan Jude