On 2014-07-24 18:56, Allan Jude wrote: > On 2014-07-24 16:11, Larry Rosenman wrote: >> On 2014-07-24 15:07, Allan Jude wrote: >>> On 2014-07-24 15:57, Larry Rosenman wrote: >>>> On 2014-07-24 14:53, Mark Martinec wrote: >>>>> 2014-07-24 21:31, Larry Rosenman wrote: >>>>>> borg.lerctr.org /home/ler # zxfer -dFkPvs -g 376 -O >>>>>> root_at_tbh.lerctr.org -R zroot zroot/backups/TBH >>>>>> Creating recursive snapshot zroot_at_zxfer_26699_20140724135840. >>>>>> Checking grandfather status of all snapshots marked for >>>>>> deletion... >>>>>> Grandfather check passed. >>>>>> Sending zroot_at_zxfer_26699_20140724135840 to >>>>>> zroot/backups/TBH/zroot. >>>>>> Sending zroot/ROOT_at_zxfer_26699_20140724135840 to >>>>>> zroot/backups/TBH/zroot/ROOT. >>>>>> Sending zroot/ROOT/default_at_zxfer_23699_20140724134435 to >>>>>> zroot/backups/TBH/zroot/ROOT/default. >>>>>> Sending zroot/ROOT/default_at_zxfer_26699_20140724135840 to >>>>>> zroot/backups/TBH/zroot/ROOT/default. >>>>>> (incremental to zroot/ROOT/default_at_zxfer_23699_20140724134435.) >>>>>> Sending zroot/home_at_zxfer_26699_20140724135840 to >>>>>> zroot/backups/TBH/zroot/home. >>>>> >>>>>> Write failed: Cannot allocate memory >>>>> ==================================== >>>>> >>>>>> cannot receive new filesystem stream: invalid backup stream >>>>>> Error when zfs send/receiving. >>>>>> borg.lerctr.org /home/ler # >>>>>> >>>>>> well that's different....... >>>>> >>>>> Sounds familiar, check my posting of today and links therein: >>>>> >>>>> >>>>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html >>>>> >>>>> Mark >>>> I'm not using netgraph to the best of my knowledge.... >>>> and the only fails on the SENDING host are: >>>> 8 Bucket: 64, 0, 41, 3555, 257774, 11, >>>> 0 >>>> 12 Bucket: 96, 0, 96, 2569, 123653, 0, >>>> 0 >>>> 16 Bucket: 128, 0, 17195, 506, 215573, 0, >>>> 0 >>>> 32 Bucket: 256, 0, 340, 4670, 900638, 50, >>>> 0 >>>> 64 Bucket: 512, 0, 10691, 365, >>>> 546888,185232, 0 >>>> 128 Bucket: 1024, 0, 3563, 905, 348419, 0, >>>> 0 >>>> 256 Bucket: 2048, 0, 2872, 162, >>>> 249995,59834, 0 >>>> vmem btag: 56, 0, 192811, 51500, 502264,1723, >>>> 0 >>>> >>>> >>> >>> I regularly use zxfer to transfer 500+ GiB datasets over the >>> internet. >>> This week I actually replicated a 2.1 TiB dataset with zxfer without >>> issue. >>> >>> I wonder which thing is running out of memory. Is there a delay while >>> it >>> is 'running out of memory', or does it fail immediately? Does running >>> top while it is working on running out of memory reveal anything? >>> >>> I would expect to use up a lot of memory while doing deduplication, >>> but >>> not otherwise. >>> >>> Note: I most often use openssh-portable rather than base ssh for >>> replication, as I enable the nonecipher to reduce CPU usage, and >>> adjust >>> the TcpRcvBuf upwards to actually saturate a gigabit over the >>> internet. >> >> I wasn't watching exactly what it was doing, but the sending box has >> 16G >> and 18G Swap and swap >> has NOT been touched. >> >> last pid: 74288; load averages: 4.70, 5.61, 5.91 up 1+03:14:18 >> 15:10:44 >> 115 processes: 3 running, 112 sleeping >> CPU: 0.6% user, 33.3% nice, 0.6% system, 0.1% interrupt, 65.4% idle >> Mem: 847M Active, 761M Inact, 14G Wired, 4616K Cache, 357M Free >> ARC: 12G Total, 6028M MFU, 5281M MRU, 3152K Anon, 120M Header, 688M >> Other >> Swap: 18G Total, 18G Free >> >> so I have zero idea where to go here. >> >> > > Most ZFS memory usage is 'wired' and so cannot be swapped, so lack of > swap activity isn't a good indicator. I would expect ZFS to give up ARC when it needed memory and couldn't get it.... I also am running Karl Denninger's Arc patch that makes the arc MUCH more responsive to freeing ARC when the system needs memory. -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler_at_lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688Received on Thu Jul 24 2014 - 22:14:58 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:51 UTC