Re: zfs send/recv: STILL invalid Backup Stream

From: Larry Rosenman <ler_at_lerctr.org>
Date: Thu, 24 Jul 2014 23:43:10 -0500
On 2014-07-24 19:56, Allan Jude wrote:
> On 2014-07-24 20:46, Mark Martinec wrote:
>> 2014-07-25 01:36 Larry Rosenman wrote:
>> 
>>> #!/bin/sh
>>> DATE=`date "+%Y-%m-%d"`
>>> #DATE2=2013-03-24
>>> #DATE2=`date -v "-1d" "+%Y-%m-%d"`
>>> # snap the source
>>> ssh root_at_tbh.lerctr.org zfs snapshot -r zroot_at_${DATE}
>>> # zfs copy the source to here.
>>> ssh root_at_tbh.lerctr.org "zfs send  -v -R zroot_at_${DATE} | \
>>>      ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH2\""
>> 
>> Btw, this double-ssh looks awkward, why not just:
>> 
>>   ssh root_at_tbh.lerctr.org "zfs send ..." | zfs recv ...
>> 
>> or better yet:
>> 
>>   ssh root_at_tbh.lerctr.org "zfs send ..." | mbuffer -m 16M | zfs recv 
>> ...
>> 
>> (The misc/mbuffer compensates for bursty zfs reads and writes.
>>  A note to myself: I should suggest to Allan to add mbuffer
>>  in a pipe as used in sysutils/zxfer, instead of patching zxfer
>>  for our local use :)
>> 
>> Mark
>> _______________________________________________
>> freebsd-current_at_freebsd.org mailing list
>> http://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to 
>> "freebsd-current-unsubscribe_at_freebsd.org"
> 
> zxfer can already do this, with the -D option
> I actually use misc/clpbar and get a progress bar as well
> 
> -D 'bar -s %%size%% -bl 1m -bs 128m'
> 
> or in your case: -D 'mbuffer -m 16M'


Ok, I did the mbuffer trick, and the SEND side is where the memory issue 
is:
borg.lerctr.org /home/ler/bin $ tail zfs-send.log
23:28:12   15.7G   zroot/home_at_2014-07-24_22:56
23:28:13   15.7G   zroot/home_at_2014-07-24_22:56
23:28:14   15.7G   zroot/home_at_2014-07-24_22:56
23:28:15   15.7G   zroot/home_at_2014-07-24_22:56
23:28:16   15.7G   zroot/home_at_2014-07-24_22:56
23:28:17   15.7G   zroot/home_at_2014-07-24_22:56
23:28:18   15.7G   zroot/home_at_2014-07-24_22:56
23:28:19   15.7G   zroot/home_at_2014-07-24_22:56
23:28:20   15.8G   zroot/home_at_2014-07-24_22:56
Write failed: Cannot allocate memory
borg.lerctr.org /home/ler/bin $

borg.lerctr.org /home/ler/bin $ tail zfs-recv.log
cannot receive new filesystem stream: invalid backup stream
borg.lerctr.org /home/ler/bin $

borg.lerctr.org /home/ler/bin $  cat backup-TBH-ZFS-initial.sh
#!/bin/sh
DATE=`date "+%Y-%m-%d_%H:%M"`
#DATE2=2013-03-24
#DATE2=`date -v "-1d" "+%Y-%m-%d"`
# snap the source
ssh root_at_tbh.lerctr.org zfs snapshot -r zroot_at_${DATE}
# zfs copy the source to here.
ssh root_at_tbh.lerctr.org 2>zfs-send.log "zfs send  -v -R zroot_at_${DATE} " 
| \
      mbuffer -m 16M 2>mbuffer.log | \
      zfs recv -F -u -v -d zroot/backups/TBH3 2>zfs-recv.log
# make sure we NEVER allow the backup stuff to automount.
/sbin/zfs list -H -t filesystem -r zroot/backups/TBH3| \
     awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh
borg.lerctr.org /home/ler/bin $

borg.lerctr.org /home/ler/bin $ ssh tbh vmstat -z
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP

UMA Kegs:               384,      0,     202,       8,     202,   0,   0
UMA Zones:             1664,      0,     202,       0,     202,   0,   0
UMA Slabs:               80,      0,  363320,   44080, 2348572,   0,   0
UMA RCntSlabs:           88,      0,    4484,      16,    4484,   0,   0
UMA Hash:               256,      0,       7,      83,      82,   0,   0
4 Bucket:                32,      0,    1911,   36589, 5255345,   0,   0
6 Bucket:                48,      0,    9406,    3542,  878903,   0,   0
8 Bucket:                64,      0,      42,    3554,  298443,  11,   0
12 Bucket:               96,      0,      93,    2572,  166067,   0,   0
16 Bucket:              128,      0,   30447,     987,  301403,   0,   0
32 Bucket:              256,      0,     352,    4658, 1157489,  50,   0
64 Bucket:              512,      0,   13669,     995, 1113780,268080,   
0
128 Bucket:            1024,      0,    3646,     822,  524977,   0,   0
256 Bucket:            2048,      0,    3648,     114,  482627,59834,   
0
vmem btag:               56,      0,  208448,   49779,  758362,1821,   0
VM OBJECT:              256,      0,   98960,    1570, 4440323,   0,   0
RADIX NODE:             144,      0,  235166,   29650,22669417,   0,   0
MAP:                    240,      0,       3,      61,       3,   0,   0
KMAP ENTRY:             128,      0,      10,     269,      10,   0,   0
MAP ENTRY:              128,      0,   11828,   24442,12463199,   0,   0
VMSPACE:                448,      0,     103,     311,   96786,   0,   0
fakepg:                 104,      0,       0,       0,       0,   0,   0
mt_zone:               4112,      0,     368,       0,     368,   0,   0
16:                      16,      0,  264961,    9382,56032463,   0,   0
32:                      32,      0,  155626,    2874,55177821,   0,   0
64:                      64,      0,  123597,  672111,53838666,   0,   0
128:                    128,      0,  159107,   58792,82084329,   0,   0
256:                    256,      0,   97004,  223276,48661927,   0,   0
512:                    512,      0,     737,    3991,33323191,   0,   0
1024:                  1024,      0,    1367,    1389, 2330023,   0,   0
2048:                  2048,      0,     271,     449,16353342,   0,   0
4096:                  4096,      0,   28044,   27448,  997566,   0,   0
SLEEPQUEUE:              80,      0,     613,     658,     613,   0,   0
64 pcpu:                  8,      0,    1884,    1188,    1884,   0,   0
Files:                   80,      0,     861,    1639, 5410116,   0,   0
rl_entry:                40,      0,     211,    1889,     211,   0,   0
TURNSTILE:              136,      0,     613,     307,     613,   0,   0
umtx pi:                 96,      0,       0,       0,       0,   0,   0
MAC labels:              40,      0,       0,       0,       0,   0,   0
PROC:                  1208,      0,     138,     114,   97778,   0,   0
THREAD:                1168,      0,     517,      95,    2429,   0,   0
cpuset:                  72,      0,     270,     280,     426,   0,   0
cyclic_id_cache:         64,      0,       0,       0,       0,   0,   0
audit_record:          1248,      0,       0,       0,       0,   0,   0
mbuf_packet:            256, 6519810,    4080,    3257, 9746764,   0,   
0
mbuf:                   256, 6519810,    1025,    3278,259201769,   0,   
0
mbuf_cluster:          2048, 1018718,    7337,      13,    7337,   0,   
0
mbuf_jumbo_page:       4096, 509359,       0,     809,45525783,   0,   0
mbuf_jumbo_9k:         9216, 150921,       0,       0,       0,   0,   0
mbuf_jumbo_16k:       16384,  84893,       0,       0,       0,   0,   0
mbuf_ext_refcnt:          4,      0,       0,    2510,  116647,   0,   0
dtrace_state_cache:    4096,      0,       0,       0,       0,   0,   0
g_bio:                  248,      0,       0,    1680,48127406,   0,   0
ttyinq:                 160,      0,     120,     630,    2460,   0,   0
DMAR_MAP_ENTRY:         120,      0,       0,       0,       0,   0,   0
ttyoutq:                256,      0,      64,     671,    1279,   0,   0
ata_request:            336,      0,       0,     242,   42892,   0,   0
vtnet_tx_hdr:            24,      0,       0,       0,       0,   0,   0
cryptop:                 88,      0,       0,       0,       0,   0,   0
cryptodesc:              72,      0,       0,       0,       0,   0,   0
FPU_save_area:          512,      0,       0,       0,       0,   0,   0
taskq_zone:              48,      0,       0,    3071,  250215,   0,   0
taskq_zone:              48,      0,       0,       0,       0,   0,   0
VNODE:                  472,      0,  113296,   62392, 2709794,   0,   0
VNODEPOLL:              112,      0,     355,     660,     357,   0,   0
BUF TRIE:               144,      0,       0,  105354,       0,   0,   0
S VFS Cache:            108,      0,  118644,   56881, 2428046,   0,   0
STS VFS Cache:          148,      0,       0,       0,       0,   0,   0
L VFS Cache:            328,      0,    1110,  144894,  804385,   0,   0
LTS VFS Cache:          368,      0,       0,       0,       0,   0,   0
NAMEI:                 1024,      0,       0,     172,30294048,   0,   0
DIRHASH:               1024,      0,       0,       0,       0,   0,   0
NCLNODE:                528,      0,       0,       0,       0,   0,   0
Mountpoints:            816,      0,      11,      19,      11,   0,   0
range_seg_cache:         64,      0,   59688,    6094, 4438296,   0,   0
zio_cache:              920,      0,       1,   42591,138793685,   0,   
0
zio_link_cache:          48,      0,       0,   44073,97697152,   0,   0
zio_buf_512:            512,      0,  411934,  265850, 1785999,   0,   0
zio_data_buf_512:       512,      0,  154726,   20594,  797483,   0,   0
zio_buf_1024:          1024,      0,    3061,     175,   74166,   0,   0
zio_data_buf_1024:     1024,      0,   69831,    9029,  333451,   0,   0
zio_buf_1536:          1536,      0,    1372,      96,   93985,   0,   0
zio_data_buf_1536:     1536,      0,   29821,    3103,  190819,   0,   0
zio_buf_2048:          2048,      0,     523,      47,  281620,   0,   0
zio_data_buf_2048:     2048,      0,   17707,    2389,  128398,   0,   0
zio_buf_2560:          2560,      0,     390,      28,   22269,   0,   0
zio_data_buf_2560:     2560,      0,   12713,      28,  108546,   0,   0
zio_buf_3072:          3072,      0,     182,      22,    7255,   0,   0
zio_data_buf_3072:     3072,      0,    8798,      34,   79372,   0,   0
zio_buf_3584:          3584,      0,     206,      11,   50516,   0,   0
zio_data_buf_3584:     3584,      0,    6275,      11,   64145,   0,   0
zio_buf_4096:          4096,      0,    1552,     416,18817937,   0,   0
zio_data_buf_4096:     4096,      0,    4625,       6,   59910,   0,   0
zio_buf_5120:          5120,      0,     133,       9,    8048,   0,   0
zio_data_buf_5120:     5120,      0,    6326,      11,   77387,   0,   0
zio_buf_6144:          6144,      0,      86,      11,    4503,   0,   0
zio_data_buf_6144:     6144,      0,    4632,      13,   58064,   0,   0
zio_buf_7168:          7168,      0,      61,      13,   48012,   0,   0
zio_data_buf_7168:     7168,      0,    3090,      13,   44010,   0,   0
zio_buf_8192:          8192,      0,      24,      29, 1857848,   0,   0
zio_data_buf_8192:     8192,      0,    2418,      11,   42210,   0,   0
zio_buf_10240:        10240,      0,      49,       7,    3311,   0,   0
zio_data_buf_10240:   10240,      0,    3611,      15,   59310,   0,   0
zio_buf_12288:        12288,      0,      44,      14,  999494,   0,   0
zio_data_buf_12288:   12288,      0,    2340,      13,   53847,   0,   0
zio_buf_14336:        14336,      0,      23,      10,    4184,   0,   0
zio_data_buf_14336:   14336,      0,    1852,      15,   36058,   0,   0
zio_buf_16384:        16384,      0,   72595,     164, 3891311,   0,   0
zio_data_buf_16384:   16384,      0,    1707,      11,   33236,   0,   0
zio_buf_20480:        20480,      0,     201,     261,  832993,   0,   0
zio_data_buf_20480:   20480,      0,    1932,      12,   43873,   0,   0
zio_buf_24576:        24576,      0,      16,      11,  365481,   0,   0
zio_data_buf_24576:   24576,      0,    1852,      13,   31773,   0,   0
zio_buf_28672:        28672,      0,       9,      25,  731759,   0,   0
zio_data_buf_28672:   28672,      0,    1169,      10,   27692,   0,   0
zio_buf_32768:        32768,      0,       9,      14,  112680,   0,   0
zio_data_buf_32768:   32768,      0,     962,      13,   20775,   0,   0
zio_buf_36864:        36864,      0,       5,      14,  112387,   0,   0
zio_data_buf_36864:   36864,      0,     701,      13,   15139,   0,   0
zio_buf_40960:        40960,      0,       4,      11,  115602,   0,   0
zio_data_buf_40960:   40960,      0,     720,      10,   12360,   0,   0
zio_buf_45056:        45056,      0,      48,      12,  102919,   0,   0
zio_data_buf_45056:   45056,      0,     434,      14,   10533,   0,   0
zio_buf_49152:        49152,      0,     188,      12,  113099,   0,   0
zio_data_buf_49152:   49152,      0,     373,      10,   11797,   0,   0
zio_buf_53248:        53248,      0,      24,      13,  157215,   0,   0
zio_data_buf_53248:   53248,      0,     347,      15,    7265,   0,   0
zio_buf_57344:        57344,      0,       1,      11,  154125,   0,   0
zio_data_buf_57344:   57344,      0,     274,      14,    7110,   0,   0
zio_buf_61440:        61440,      0,       2,      10,  163599,   0,   0
zio_data_buf_61440:   61440,      0,     180,      11,    5565,   0,   0
zio_buf_65536:        65536,      0,       1,      11,  186705,   0,   0
zio_data_buf_65536:   65536,      0,     213,      11,    7273,   0,   0
zio_buf_69632:        69632,      0,       2,      14,  145850,   0,   0
zio_data_buf_69632:   69632,      0,     176,       8,    5232,   0,   0
zio_buf_73728:        73728,      0,       1,      13,  136031,   0,   0
zio_data_buf_73728:   73728,      0,     127,      12,    4816,   0,   0
zio_buf_77824:        77824,      0,       1,      22,  162862,   0,   0
zio_data_buf_77824:   77824,      0,     136,      14,    4207,   0,   0
zio_buf_81920:        81920,      0,       0,     149,  195810,   0,   0
zio_data_buf_81920:   81920,      0,      99,      15,    4936,   0,   0
zio_buf_86016:        86016,      0,       1,     216,  150971,   0,   0
zio_data_buf_86016:   86016,      0,     103,      14,   33989,   0,   0
zio_buf_90112:        90112,      0,       1,     120,   75823,   0,   0
zio_data_buf_90112:   90112,      0,     145,      11,    3517,   0,   0
zio_buf_94208:        94208,      0,       0,      20,   72296,   0,   0
zio_data_buf_94208:   94208,      0,     149,      11,    3315,   0,   0
zio_buf_98304:        98304,      0,       0,      19,   68257,   0,   0
zio_data_buf_98304:   98304,      0,      79,      11,    4092,   0,   0
zio_buf_102400:      102400,      0,       1,      19,   62463,   0,   0
zio_data_buf_102400: 102400,      0,      92,      12,    2940,   0,   0
zio_buf_106496:      106496,      0,       0,      15,  203444,   0,   0
zio_data_buf_106496: 106496,      0,      62,      14,    2869,   0,   0
zio_buf_110592:      110592,      0,       0,      10,  184072,   0,   0
zio_data_buf_110592: 110592,      0,     152,      12,    2463,   0,   0
zio_buf_114688:      114688,      0,       1,      12,   65538,   0,   0
zio_data_buf_114688: 114688,      0,     115,      15,    3609,   0,   0
zio_buf_118784:      118784,      0,       0,      13,   73524,   0,   0
zio_data_buf_118784: 118784,      0,      57,      12,    2092,   0,   0
zio_buf_122880:      122880,      0,       0,      23,  101816,   0,   0
zio_data_buf_122880: 122880,      0,      46,      15,    2131,   0,   0
zio_buf_126976:      126976,      0,       0,      13,   99774,   0,   0
zio_data_buf_126976: 126976,      0,      46,      15,    1927,   0,   0
zio_buf_131072:      131072,      0,      21,      97,  715685,   0,   0
zio_data_buf_131072: 131072,      0,   64623,       8, 1133068,   0,   0
lz4_ctx:              16384,      0,       0,      38, 2874956,   0,   0
sa_cache:                80,      0,  113220,   62680, 2719738,   0,   0
dnode_t:                744,      0,  369347,  391598,  903612,   0,   0
dmu_buf_impl_t:         224,      0,  459185,  519692, 3494852,   0,   0
arc_buf_hdr_t:          216,      0,  697777,  122573, 4738762,   0,   0
arc_buf_t:               72,      0,  541440,  121695, 5738782,   0,   0
zil_lwb_cache:          192,      0,       7,    5073,   90526,   0,   0
zfs_znode_cache:        368,      0,  113220,   62340, 2709584,   0,   0
pipe:                   744,      0,     113,     177,   84427,   0,   0
procdesc:               128,      0,       0,       0,       0,   0,   0
ksiginfo:               112,      0,    1133,    2507,10649838,   0,   0
itimer:                 352,      0,       1,      32,       1,   0,   0
KNOTE:                  128,      0,     379,    1388,26313314,   0,   0
socket:                 696, 523340,     243,     317,  827216,   0,   0
unpcb:                  240, 523344,     152,     648,  171731,   0,   0
ipq:                     56,  31879,       0,    2059,     385,   0,   0
udp_inpcb:              392, 523340,      19,     421,  567701,   0,   0
udpcb:                   16, 523586,      19,    2993,  567701,   0,   0
tcp_inpcb:              392, 523340,      73,     457,   87773,   0,   0
tcpcb:                 1024, 523340,      67,     245,   87773,   0,   0
tcptw:                   88,  27810,       6,    1254,   29354,   0,   0
syncache:               160,  15375,       0,     800,   57377,   0,   0
hostcache:              136,  15370,     134,     620,    4951,   0,   0
tcpreass:                40,  63700,       0,    3100,   13421,   0,   0
sackhole:                32,      0,       0,    2125,   18903,   0,   0
sctp_ep:               1408, 523340,       0,       0,       0,   0,   0
sctp_asoc:             2352,  40000,       0,       0,       0,   0,   0
sctp_laddr:              48,  80012,       0,    1328,       8,   0,   0
sctp_raddr:             728,  80000,       0,       0,       0,   0,   0
sctp_chunk:             136, 400026,       0,       0,       0,   0,   0
sctp_readq:             104, 400026,       0,       0,       0,   0,   0
sctp_stream_msg_out:    104, 400026,       0,       0,       0,   0,   0
sctp_asconf:             40, 400000,       0,       0,       0,   0,   0
sctp_asconf_ack:         48, 400060,       0,       0,       0,   0,   0
udplite_inpcb:          392, 523340,       0,       0,       0,   0,   0
ripcb:                  392, 523340,       2,      58,       3,   0,   0
rtentry:                200,      0,      32,     408,      34,   0,   0
selfd:                   56,      0,     356,    2200,137062686,   0,   
0
SWAPMETA:               288, 2037438,       0,       0,       0,   0,   
0

borg.lerctr.org /home/ler/bin $

last pid: 97813;  load averages:  7.79,  8.41,  8.24  up 1+11:45:45    
23:42:11
118 processes: 7 running, 109 sleeping, 2 stopped

Mem: 982M Active, 1172M Inact, 13G Wired, 148M Cache, 637M Free
ARC: 11G Total, 4394M MFU, 5813M MRU, 1029K Anon, 181M Header, 469M 
Other
Swap: 18G Total, 18G Free


   PID USERNAME     THR PRI NICE   SIZE    RES STATE   C   TIME    WCPU 
COMMAND
97782 boinc          1 155  i31 91992K 26008K CPU1    1   0:40 100.00% 
wcgrid_f
96977 boinc          1 155  i31   375M   165M STOP    4  11:30  61.87% 
wcgrid_c
89981 boinc          1 155  i31 99308K 63248K CPU5    5  91:25  51.76% 
wcgrid_f
92270 boinc          1 155  i31    98M 64556K CPU3    3  62:55  50.88% 
wcgrid_f
92269 boinc          1 155  i31 99372K 63324K CPU4    4  63:49  50.20% 
wcgrid_f
92271 boinc          1 155  i31 99340K 63288K CPU0    0  64:00  48.19% 
wcgrid_f
89772 boinc          1 155  i31 99888K 63832K CPU7    7  92:52  41.36% 
wcgrid_f
89781 boinc          1 155  i31 99732K 63588K nanslp  5  94:54  38.18% 
wcgrid_f
85619 root           1  52    0   181M 97232K select  3   2:56  21.19% 
perl5
97806 root           1  28    0 44600K  8028K kqread  0   0:00   0.78% 
indexer-
   837 clamav         3  20    0   486M   399M uwait   5  12:42   0.20% 
clamd
70563 boinc          2 155  i31   101M 51436K nanslp  7 202:02   0.00% 
setiatho
68437 boinc          2 155  i31   105M 52516K nanslp  3 185:45   0.00% 
setiatho
83114 boinc          2 155  i31   105M 51880K nanslp  4 103:15   0.00% 
setiatho
85300 boinc          2 155  i31   101M 51460K nanslp  4  78:55   0.00% 
setiatho
86148 boinc          2 155  i31   105M 51492K nanslp  3  71:54   0.00% 
setiatho
87150 boinc          2 155  i31 95112K 46940K nanslp  4  30:37   0.00% 
setiatho
87451 boinc          2 155  i31 95112K 46224K nanslp  3  28:07   0.00% 
setiatho

borg.lerctr.org /home/ler/bin $

Where to now?


-- 
Larry Rosenman                     http://www.lerctr.org/~ler
Phone: +1 214-642-9640 (c)     E-Mail: ler_at_lerctr.org
US Mail: 108 Turvey Cove, Hutto, TX 78634-5688
Received on Fri Jul 25 2014 - 02:43:17 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:51 UTC