On 2014-07-24 15:44, Larry Rosenman wrote: > On 2014-07-24 15:26, Mark Martinec wrote: >> 2014-07-24 21:57, Larry Rosenman wrote: >> >>>>>> Sending zroot/home_at_zxfer_26699_20140724135840 to >>>>>> zroot/backups/TBH/zroot/home. >>>>> Write failed: Cannot allocate memory >>>> ==================================== >>>>> cannot receive new filesystem stream: invalid backup stream >>>>> Error when zfs send/receiving. >>>>> borg.lerctr.org /home/ler # >>>>> >>>>> well that's different....... >>>> >>>> Sounds familiar, check my posting of today and links therein: >>>> >>>> http://lists.freebsd.org/pipermail/freebsd-net/2014-July/039347.html >> >>> I'm not using netgraph to the best of my knowledge.... >> >> Check to make sure: >> vmstat -z | fgrep NetGraph >> >>> and the only fails on the SENDING host are: >>> 8 Bucket: 64, 0, 41, 3555, 257774, 11, >>> 0 >>> 12 Bucket: 96, 0, 96, 2569, 123653, 0, >>> 0 >>> 16 Bucket: 128, 0, 17195, 506, 215573, 0, >>> 0 >>> 32 Bucket: 256, 0, 340, 4670, 900638, 50, >>> 0 >>> 64 Bucket: 512, 0, 10691, 365, >>> 546888,185232, >>> 0 >>> 128 Bucket: 1024, 0, 3563, 905, 348419, 0, >>> 0 >>> 256 Bucket: 2048, 0, 2872, 162, 249995,59834, >>> 0 >>> vmem btag: 56, 0, 192811, 51500, 502264,1723, >>> 0 >> >> >> Adam Vande More gave other suggestions on that thread from 2011: >> >> >> http://lists.freebsd.org/pipermail/freebsd-emulation/2011-July/008971.html >> >> >> Mark > > > thebighonker.lerctr.org /home/ler $ vmstat -z|fgrep NetGraph > thebighonker.lerctr.org /home/ler $ > > This on physical hardware, FWIW. > > thebighonker.lerctr.org /home/ler $ vmstat -z > ITEM SIZE LIMIT USED FREE REQ FAIL > SLEEP > > UMA Kegs: 384, 0, 202, 8, 202, 0, > 0 > UMA Zones: 1664, 0, 202, 0, 202, 0, > 0 > UMA Slabs: 80, 0, 361152, 38448, 1561588, 0, > 0 > UMA RCntSlabs: 88, 0, 4360, 5, 4360, 0, > 0 > UMA Hash: 256, 0, 8, 82, 82, 0, > 0 > 4 Bucket: 32, 0, 4816, 33684, 2974043, 0, > 0 > 6 Bucket: 48, 0, 6118, 2514, 624434, 0, > 0 > 8 Bucket: 64, 0, 161, 3435, 265443, 11, > 0 > 12 Bucket: 96, 0, 177, 2488, 130226, 0, > 0 > 16 Bucket: 128, 0, 20301, 531, 224788, 0, > 0 > 32 Bucket: 256, 0, 433, 4577, 945047, 50, > 0 > 64 Bucket: 512, 0, 11137, 575, 579209,211292, > 0 > 128 Bucket: 1024, 0, 3509, 959, 369839, 0, > 0 > 256 Bucket: 2048, 0, 2847, 187, 269044,59834, > 0 > vmem btag: 56, 0, 209393, 34918, 557576,1723, > 0 > VM OBJECT: 256, 0, 91384, 791, 3480613, 0, > 0 > RADIX NODE: 144, 0, 227133, 37683,16029821, 0, > 0 > MAP: 240, 0, 3, 61, 3, 0, > 0 > KMAP ENTRY: 128, 0, 10, 269, 10, 0, > 0 > MAP ENTRY: 128, 0, 12666, 23604, 9357932, 0, > 0 > VMSPACE: 448, 0, 108, 306, 75357, 0, > 0 > fakepg: 104, 0, 0, 0, 0, 0, > 0 > mt_zone: 4112, 0, 368, 0, 368, 0, > 0 > 16: 16, 0, 267839, 2488,45160196, 0, > 0 > 32: 32, 0, 148139, 1486,39174301, 0, > 0 > 64: 64, 0, 291630, 504078,40379996, 0, > 0 > 128: 128, 0, 182409, 35490,63455920, 0, > 0 > 256: 256, 0, 89368, 230912,35841036, 0, > 0 > 512: 512, 0, 741, 3987,24215854, 0, > 0 > 1024: 1024, 0, 1371, 1385, 1726440, 0, > 0 > 2048: 2048, 0, 277, 417,10240207, 0, > 0 > 4096: 4096, 0, 38306, 17186, 913638, 0, > 0 > SLEEPQUEUE: 80, 0, 613, 658, 613, 0, > 0 > 64 pcpu: 8, 0, 1884, 1188, 1884, 0, > 0 > Files: 80, 0, 829, 1571, 4279893, 0, > 0 > rl_entry: 40, 0, 209, 1891, 209, 0, > 0 > TURNSTILE: 136, 0, 613, 307, 613, 0, > 0 > umtx pi: 96, 0, 0, 0, 0, 0, > 0 > MAC labels: 40, 0, 0, 0, 0, 0, > 0 > PROC: 1208, 0, 131, 121, 76219, 0, > 0 > THREAD: 1168, 0, 519, 93, 2039, 0, > 0 > cpuset: 72, 0, 270, 280, 426, 0, > 0 > cyclic_id_cache: 64, 0, 0, 0, 0, 0, > 0 > audit_record: 1248, 0, 0, 0, 0, 0, > 0 > mbuf_packet: 256, 6519810, 4081, 3256, 4910227, 0, > 0 > mbuf: 256, 6519810, 1052, 3251,93444045, 0, > 0 > mbuf_cluster: 2048, 1018718, 7337, 13, 7337, 0, > 0 > mbuf_jumbo_page: 4096, 509359, 12, 673,15015378, 0, > 0 > mbuf_jumbo_9k: 9216, 150921, 0, 0, 0, 0, > 0 > mbuf_jumbo_16k: 16384, 84893, 0, 0, 0, 0, > 0 > mbuf_ext_refcnt: 4, 0, 0, 2510, 85094, 0, > 0 > dtrace_state_cache: 4096, 0, 0, 0, 0, 0, > 0 > g_bio: 248, 0, 0, 1600,29986631, 0, > 0 > ttyinq: 160, 0, 180, 570, 2100, 0, > 0 > DMAR_MAP_ENTRY: 120, 0, 0, 0, 0, 0, > 0 > ttyoutq: 256, 0, 95, 640, 1093, 0, > 0 > ata_request: 336, 0, 0, 242, 33344, 0, > 0 > vtnet_tx_hdr: 24, 0, 0, 0, 0, 0, > 0 > cryptop: 88, 0, 0, 0, 0, 0, > 0 > cryptodesc: 72, 0, 0, 0, 0, 0, > 0 > FPU_save_area: 512, 0, 0, 0, 0, 0, > 0 > taskq_zone: 48, 0, 1, 3070, 194184, 0, > 0 > taskq_zone: 48, 0, 0, 0, 0, 0, > 0 > VNODE: 472, 0, 109527, 66161, 2663226, 0, > 0 > VNODEPOLL: 112, 0, 8, 762, 10, 0, > 0 > BUF TRIE: 144, 0, 0, 105354, 0, 0, > 0 > S VFS Cache: 108, 0, 114973, 60552, 2293717, 0, > 0 > STS VFS Cache: 148, 0, 0, 0, 0, 0, > 0 > L VFS Cache: 328, 0, 468, 145536, 800279, 0, > 0 > LTS VFS Cache: 368, 0, 0, 0, 0, 0, > 0 > NAMEI: 1024, 0, 0, 156,23968745, 0, > 0 > DIRHASH: 1024, 0, 0, 0, 0, 0, > 0 > NCLNODE: 528, 0, 0, 0, 0, 0, > 0 > Mountpoints: 816, 0, 11, 19, 11, 0, > 0 > range_seg_cache: 64, 0, 58377, 7157, 3500156, 0, > 0 > zio_cache: 920, 0, 1, 42591,104595517, 0, > 0 > zio_link_cache: 48, 0, 0, 44073,72482492, 0, > 0 > zio_buf_512: 512, 0, 552060, 193836, 1557459, 0, > 0 > zio_data_buf_512: 512, 0, 114904, 2056, 526943, 0, > 0 > zio_buf_1024: 1024, 0, 4246, 58, 56140, 0, > 0 > zio_data_buf_1024: 1024, 0, 51117, 1255, 211875, 0, > 0 > zio_buf_1536: 1536, 0, 1957, 59, 72179, 0, > 0 > zio_data_buf_1536: 1536, 0, 22199, 719, 127817, 0, > 0 > zio_buf_2048: 2048, 0, 691, 43, 217643, 0, > 0 > zio_data_buf_2048: 2048, 0, 13419, 1147, 87149, 0, > 0 > zio_buf_2560: 2560, 0, 455, 20, 16265, 0, > 0 > zio_data_buf_2560: 2560, 0, 9335, 493, 73967, 0, > 0 > zio_buf_3072: 3072, 0, 245, 16, 4658, 0, > 0 > zio_data_buf_3072: 3072, 0, 6940, 512, 53932, 0, > 0 > zio_buf_3584: 3584, 0, 248, 10, 38558, 0, > 0 > zio_data_buf_3584: 3584, 0, 5076, 352, 43162, 0, > 0 > zio_buf_4096: 4096, 0, 464, 729,14217630, 0, > 0 > zio_data_buf_4096: 4096, 0, 3857, 288, 41192, 0, > 0 > zio_buf_5120: 5120, 0, 167, 9, 6442, 0, > 0 > zio_data_buf_5120: 5120, 0, 6025, 461, 51297, 0, > 0 > zio_buf_6144: 6144, 0, 115, 6, 3196, 0, > 0 > zio_data_buf_6144: 6144, 0, 4453, 309, 38753, 0, > 0 > zio_buf_7168: 7168, 0, 85, 12, 36945, 0, > 0 > zio_data_buf_7168: 7168, 0, 3373, 273, 29007, 0, > 0 > zio_buf_8192: 8192, 0, 44, 39, 1373975, 0, > 0 > zio_data_buf_8192: 8192, 0, 2543, 205, 28736, 0, > 0 > zio_buf_10240: 10240, 0, 66, 7, 2534, 0, > 0 > zio_data_buf_10240: 10240, 0, 3986, 275, 39568, 0, > 0 > zio_buf_12288: 12288, 0, 71, 15, 654156, 0, > 0 > zio_data_buf_12288: 12288, 0, 2937, 209, 37077, 0, > 0 > zio_buf_14336: 14336, 0, 32, 9, 3164, 0, > 0 > zio_data_buf_14336: 14336, 0, 2236, 194, 23959, 0, > 0 > zio_buf_16384: 16384, 0, 72627, 226, 2784131, 0, > 0 > zio_data_buf_16384: 16384, 0, 1778, 155, 22372, 0, > 0 > zio_buf_20480: 20480, 0, 210, 14, 598092, 0, > 0 > zio_data_buf_20480: 20480, 0, 2724, 216, 29228, 0, > 0 > zio_buf_24576: 24576, 0, 18, 14, 270172, 0, > 0 > zio_data_buf_24576: 24576, 0, 2101, 173, 21241, 0, > 0 > zio_buf_28672: 28672, 0, 10, 54, 503029, 0, > 0 > zio_data_buf_28672: 28672, 0, 1406, 113, 18667, 0, > 0 > zio_buf_32768: 32768, 0, 10, 11, 77998, 0, > 0 > zio_data_buf_32768: 32768, 0, 1088, 96, 14116, 0, > 0 > zio_buf_36864: 36864, 0, 6, 11, 79009, 0, > 0 > zio_data_buf_36864: 36864, 0, 871, 87, 10066, 0, > 0 > zio_buf_40960: 40960, 0, 8, 10, 80622, 0, > 0 > zio_data_buf_40960: 40960, 0, 785, 57, 8149, 0, > 0 > zio_buf_45056: 45056, 0, 50, 10, 70143, 0, > 0 > zio_data_buf_45056: 45056, 0, 659, 61, 6889, 0, > 0 > zio_buf_49152: 49152, 0, 190, 9, 73508, 0, > 0 > zio_data_buf_49152: 49152, 0, 512, 42, 8420, 0, > 0 > zio_buf_53248: 53248, 0, 25, 10, 99339, 0, > 0 > zio_data_buf_53248: 53248, 0, 465, 30, 4838, 0, > 0 > zio_buf_57344: 57344, 0, 2, 15, 95818, 0, > 0 > zio_data_buf_57344: 57344, 0, 337, 33, 4753, 0, > 0 > zio_buf_61440: 61440, 0, 1, 23, 98972, 0, > 0 > zio_data_buf_61440: 61440, 0, 294, 33, 3792, 0, > 0 > zio_buf_65536: 65536, 0, 1, 9, 114664, 0, > 0 > zio_data_buf_65536: 65536, 0, 291, 29, 5152, 0, > 0 > zio_buf_69632: 69632, 0, 2, 10, 90765, 0, > 0 > zio_data_buf_69632: 69632, 0, 291, 22, 3555, 0, > 0 > zio_buf_73728: 73728, 0, 1, 12, 87316, 0, > 0 > zio_data_buf_73728: 73728, 0, 219, 23, 3361, 0, > 0 > zio_buf_77824: 77824, 0, 2, 9, 101758, 0, > 0 > zio_data_buf_77824: 77824, 0, 196, 25, 2907, 0, > 0 > zio_buf_81920: 81920, 0, 0, 15, 129645, 0, > 0 > zio_data_buf_81920: 81920, 0, 187, 18, 3578, 0, > 0 > zio_buf_86016: 86016, 0, 1, 7, 99241, 0, > 0 > zio_data_buf_86016: 86016, 0, 180, 22, 26051, 0, > 0 > zio_buf_90112: 90112, 0, 1, 12, 51491, 0, > 0 > zio_data_buf_90112: 90112, 0, 163, 18, 2438, 0, > 0 > zio_buf_94208: 94208, 0, 0, 9, 51043, 0, > 0 > zio_data_buf_94208: 94208, 0, 149, 12, 2288, 0, > 0 > zio_buf_98304: 98304, 0, 2, 8, 50322, 0, > 0 > zio_data_buf_98304: 98304, 0, 170, 25, 3002, 0, > 0 > zio_buf_102400: 102400, 0, 2, 10, 44642, 0, > 0 > zio_data_buf_102400: 102400, 0, 145, 19, 2017, 0, > 0 > zio_buf_106496: 106496, 0, 0, 15, 161697, 0, > 0 > zio_data_buf_106496: 106496, 0, 161, 16, 1944, 0, > 0 > zio_buf_110592: 110592, 0, 0, 19, 146777, 0, > 0 > zio_data_buf_110592: 110592, 0, 97, 15, 1636, 0, > 0 > zio_buf_114688: 114688, 0, 2, 8, 51698, 0, > 0 > zio_data_buf_114688: 114688, 0, 115, 20, 2691, 0, > 0 > zio_buf_118784: 118784, 0, 0, 8, 62915, 0, > 0 > zio_data_buf_118784: 118784, 0, 96, 11, 1440, 0, > 0 > zio_buf_122880: 122880, 0, 0, 12, 85924, 0, > 0 > zio_data_buf_122880: 122880, 0, 73, 17, 1510, 0, > 0 > zio_buf_126976: 126976, 0, 1, 11, 86133, 0, > 0 > zio_data_buf_126976: 126976, 0, 80, 9, 1351, 0, > 0 > zio_buf_131072: 131072, 0, 43, 43, 548063, 0, > 0 > zio_data_buf_131072: 131072, 0, 67545, 2, 646567, 0, > 0 > lz4_ctx: 16384, 0, 0, 16, 2249605, 0, > 0 > sa_cache: 80, 0, 109455, 66445, 2663034, 0, > 0 > dnode_t: 744, 0, 501793, 259152, 873895, 0, > 0 > dmu_buf_impl_t: 224, 0, 635273, 343604, 3042828, 0, > 0 > arc_buf_hdr_t: 216, 0, 643049, 141229, 3158205, 0, > 0 > arc_buf_t: 72, 0, 481443, 119872, 3705908, 0, > 0 > zil_lwb_cache: 192, 0, 8, 5072, 66838, 0, > 0 > zfs_znode_cache: 368, 0, 109455, 66105, 2663034, 0, > 0 > pipe: 744, 0, 113, 177, 66135, 0, > 0 > procdesc: 128, 0, 0, 0, 0, 0, > 0 > ksiginfo: 112, 0, 237, 2528, 8311580, 0, > 0 > itimer: 352, 0, 1, 32, 1, 0, > 0 > KNOTE: 128, 0, 376, 1391,17307485, 0, > 0 > socket: 696, 523340, 258, 302, 668458, 0, > 0 > unpcb: 240, 523344, 151, 649, 130961, 0, > 0 > ipq: 56, 31879, 0, 2059, 317, 0, > 0 > udp_inpcb: 392, 523340, 19, 421, 469503, 0, > 0 > udpcb: 16, 523586, 19, 2993, 469503, 0, > 0 > tcp_inpcb: 392, 523340, 100, 370, 67983, 0, > 0 > tcpcb: 1024, 523340, 78, 234, 67983, 0, > 0 > tcptw: 88, 27810, 22, 1013, 22066, 0, > 0 > syncache: 160, 15375, 0, 675, 45270, 0, > 0 > hostcache: 136, 15370, 255, 499, 3892, 0, > 0 > tcpreass: 40, 63700, 0, 2600, 7616, 0, > 0 > sackhole: 32, 0, 0, 1875, 7184, 0, > 0 > sctp_ep: 1408, 523340, 0, 0, 0, 0, > 0 > sctp_asoc: 2352, 40000, 0, 0, 0, 0, > 0 > sctp_laddr: 48, 80012, 0, 1328, 8, 0, > 0 > sctp_raddr: 728, 80000, 0, 0, 0, 0, > 0 > sctp_chunk: 136, 400026, 0, 0, 0, 0, > 0 > sctp_readq: 104, 400026, 0, 0, 0, 0, > 0 > sctp_stream_msg_out: 104, 400026, 0, 0, 0, 0, > 0 > sctp_asconf: 40, 400000, 0, 0, 0, 0, > 0 > sctp_asconf_ack: 48, 400060, 0, 0, 0, 0, > 0 > udplite_inpcb: 392, 523340, 0, 0, 0, 0, > 0 > ripcb: 392, 523340, 2, 58, 3, 0, > 0 > rtentry: 200, 0, 32, 408, 34, 0, > 0 > selfd: 56, 0, 333, 2223,29658712, 0, > 0 > SWAPMETA: 288, 2037438, 0, 0, 0, 0, > 0 > > thebighonker.lerctr.org /home/ler and I ran the script again.... It got further: 18:20:52 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:53 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:54 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:55 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:56 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:57 25.1G zroot/home_at_zxfer_26699_20140724135840 18:20:58 25.1G zroot/home_at_zxfer_26699_20140724135840 Connection to home.lerctr.org closed by remote host. warning: cannot send 'zroot/home_at_zxfer_26699_20140724135840': Broken pipe TIME SENT SNAPSHOT warning: cannot send 'zroot/home_at_2014-07-24': Broken pipe cannot open 'zroot/backups/TBH2/home': dataset does not exist borg.lerctr.org /home/ler/bin # BUT we still fail. #!/bin/sh DATE=`date "+%Y-%m-%d"` #DATE2=2013-03-24 #DATE2=`date -v "-1d" "+%Y-%m-%d"` # snap the source ssh root_at_tbh.lerctr.org zfs snapshot -r zroot_at_${DATE} # zfs copy the source to here. ssh root_at_tbh.lerctr.org "zfs send -v -R zroot_at_${DATE} | \ ssh home.lerctr.org \"zfs recv -F -u -v -d zroot/backups/TBH2\"" # make sure we NEVER allow the backup stuff to automount. /sbin/zfs list -H -t filesystem -r zroot/backups/TBH2| \ awk '{printf "/sbin/zfs set canmount=noauto %s\n",$1}' | sh borg.lerctr.org /home/ler/bin # this was to a FRESH zroot/backups/TBH2 -- Larry Rosenman http://www.lerctr.org/~ler Phone: +1 214-642-9640 (c) E-Mail: ler_at_lerctr.org US Mail: 108 Turvey Cove, Hutto, TX 78634-5688Received on Thu Jul 24 2014 - 21:36:40 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:51 UTC