Re: Help ZFS FreeBSD 8.0 RC2 Write performance issue

From: Sam Fourman Jr. <sfourman_at_gmail.com>
Date: Wed, 11 Nov 2009 15:26:16 -0600
On Wed, Nov 11, 2009 at 2:49 PM, Dan Nelson <dnelson_at_allantgroup.com> wrote:
> In the last episode (Nov 11), Ivan Voras said:
>> Sam Fourman Jr. wrote:
>> > I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is
>> > acting weird on writes.  I get ~150mbit writes idk if this is good or
>> > not?  but it paused for a few seconds every once and awhile.
>>
>> You didn't give any "iostat" statistics - I suspect that if you
>> correlate ifstat and iostat output that you will see that network
>> "pauses" happen during spikes in IO. You should check for this and post
>> your results.
>
> Yes, iostat would be useful here.  "iostat -zxC 2" will give you per-disk
> stats plus CPU usage every 2 seconds (CPU may be a factor if you have
> compression enabled).
>
> On a Solaris box I admin, setting zfs_write_limit_override helped stuttering
> while doing heavy writes.  It's not exported on FreeBSD, but it should be
> easy to add it as a RW sysctl; it lives in dsl_pool.c and can be tweaked at
> runtime.  Start big and tune it down so each write burst takes under a
> second; it looks like you're writing solid for around 6-8 seconds now.  The
> number will vary depending on your disk speed and how much ARC you have.


here are some iostats for you. I do not believe I have compression enabled
am I mistaken? isn'y SATA2 300MB/s? and I am doing ~6MB/s per disk?
I built this machine with 4GB of memory because I thought ZFS would like it.
now maybe a re(4) interface isnt the best choice. if that is the
problem here I can change it.
We spent ~$800 on a hardware RAID card thinking that it would help performance

Why is it that with sftp we do not see the pauses in Network transfer?


# iostat -zxC 1 (with NFS and ZFS)
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.1   0.4     2.0     6.1    0   3.1   0   0  0  1  1 98
da0        2.2   7.2   127.0   246.7    0 120.3   6
da1        2.0   7.5   117.3   246.2    0 120.4   6
da2        2.2   7.3   127.1   246.7    0 119.8   6
da3        2.0   7.6   115.5   246.2    0 119.8   5
da4        2.2   7.3   126.1   246.7    0 119.9   5
da5        2.0   7.5   115.6   246.3    0 119.8   5
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0  11.0     0.0   175.6    0   1.4   0   2  0  3  0 95
da0        0.0  47.9     0.0    94.3    0   1.0   1
da1        0.0  54.9     0.0    84.8    0   1.4   2
da2        0.0  44.9     0.0    92.8    0   1.0   1
da3        0.0  48.9     0.0    83.3    0   1.4   2
da4        0.0  47.9     0.0    95.3    0   1.0   1
da5        0.0  53.9     0.0    87.8    0   1.2   2
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
                                                      0  0  0  0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0  28.9     0.0   462.8    0   3.5   1   0  0  0  0 99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0      131.7   0.0  8392.0     0.0   35  73.6  60   0  0  4  3 93
da1      118.8   0.0  7561.7     0.0   35  74.9  62
da2      127.7   0.0  8136.5     0.0   35  77.7  63
da3      103.8   0.0  6603.7     0.0   29 137.6  79
da4      106.8   0.0  6795.8     0.0   33 112.9  74
da5      123.7   0.0  7919.4     0.0   35 126.9  83
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0       51.9  89.9  3322.5  1487.5    4  77.4  38   0  0  6  5 89
da1       55.9 109.8  3578.1  1469.6    4  76.4  49
da2       58.9  88.9  3769.8  1604.8    1  75.3  46
da3       69.9 105.8  4472.6  1652.8    0 117.0  82
da4       66.9  85.9  4280.9  1490.0    3 112.4  70
da5       66.9 108.8  4280.9  1469.6    3 105.3  69
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.7     0.0  5069.1   10  53.0  80   0  0  9  8 83
da1        0.0 107.7     0.0  5069.1   10  54.4  81
da2        0.0 103.7     0.0  4953.9   10  50.4  73
da3        0.0 102.7     0.0  4890.1   10  51.7  74
da4        0.0 108.7     0.0  5299.5    6  50.0  74
da5        0.0 108.7     0.0  5299.5    6  51.6  76
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.7     0.0  5196.4    4  50.5  76   0  0  8  8 85
da1        0.0 106.7     0.0  5196.4    4  52.1  78
da2        0.0 103.7     0.0  5196.4    4  49.3  73
da3        0.0 103.7     0.0  5196.4    4  52.1  75
da4        0.0 103.7     0.0  4991.5    3  51.7  78
da5        0.0 103.7     0.0  4991.5    5  52.8  80
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 114.8     0.0  5404.5    0  47.5  81   0  0  9  7 84
da1        0.0 111.8     0.0  5250.9    3  49.4  78
da2        0.0 113.8     0.0  5404.5    0  48.3  73
da3        0.0 113.8     0.0  5404.5    0  50.1  75
da4        0.0 111.8     0.0  5379.1    0  48.0  78
da5        0.0 111.8     0.0  5379.1    0  49.1  81
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 102.7     0.0  4890.1   13  50.2  70   0  0  8  9 83
da1        0.0 105.7     0.0  5043.7   13  53.0  77
da2        0.0 105.7     0.0  5120.5    9  47.7  73
da3        0.0 105.7     0.0  5120.5    9  50.5  75
da4        0.0 107.7     0.0  5120.1    9  49.7  77
da5        0.0 107.7     0.0  5120.1    9  51.6  79
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  5353.4    0  52.6  81   0  0  7  7 85
da1        0.0 109.8     0.0  5353.4    0  53.9  81
da2        0.0 103.8     0.0  5122.3    0  53.0  78
da3        0.0 103.8     0.0  5122.3    0  54.5  78
da4        0.0 105.8     0.0  5122.8    0  52.8  80
da5        0.0 105.8     0.0  5122.8    0  54.7  81
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.8     0.0  5150.4    8  50.0  77   0  0  9  8 83
da1        0.0 107.8     0.0  5150.4    8  52.3  79
da2        0.0 107.8     0.0  5176.4    8  49.2  77
da3        0.0 107.8     0.0  5176.4    8  51.5  78
da4        0.0 108.8     0.0  5175.9    8  49.9  77
da5        0.0 108.8     0.0  5175.9    8  52.7  77
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.7     0.0  5299.6    0  53.5  78   0  0  7  7 86
da1        0.0 107.7     0.0  5299.6    0  56.1  78
da2        0.0 105.7     0.0  5146.0    2  54.0  82
da3        0.0 103.7     0.0  5043.3    4  54.1  77
da4        0.0 107.7     0.0  5248.7    1  54.1  78
da5        0.0 107.7     0.0  5248.7    1  56.2  78
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5124.5    8  48.7  77   0  0  9  9 82
da1        0.0 102.8     0.0  4893.9   12  52.0  76
da2        0.0 105.8     0.0  5022.2   13  51.7  75
da3        0.0 107.8     0.0  5125.0   13  54.9  81
da4        0.0 107.8     0.0  5150.5    8  50.2  78
da5        0.0 107.8     0.0  5150.5    8  50.7  79
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 105.7     0.0  5171.9    1  53.0  78   0  0  9  7 84
da1        0.0 109.7     0.0  5402.3    1  54.8  81
da2        0.0 107.7     0.0  5274.1    3  51.5  81
da3        0.0 105.7     0.0  5171.4    5  52.0  77
da4        0.0 105.7     0.0  5146.0    1  53.8  76
da5        0.0 105.7     0.0  5146.0    1  55.1  78
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.8     0.0  5124.6    9  49.8  79   0  0  9 11 80
da1        0.0 105.8     0.0  4996.8   11  52.0  78
da2        0.0 105.8     0.0  5022.3   13  51.3  73
da3        0.0 107.8     0.0  5125.1   13  54.5  79
da4        0.0 107.8     0.0  5150.6    9  49.4  76
da5        0.0 107.8     0.0  5150.6    9  51.3  77
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 105.8     0.0  5148.1    0  53.2  79   0  0  6  7 87
da1        0.0 107.8     0.0  5275.8    0  54.7  79
da2        0.0 108.8     0.0  5378.6    0  52.2  81
da3        0.0 108.8     0.0  5378.6    0  54.1  81
da4        0.0 106.8     0.0  5148.1    0  53.9  82
da5        0.0 106.8     0.0  5148.1    0  55.6  82
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0   4.0     0.0    12.0    0   0.5   0   0  0 16  6 77
da0        0.0 108.8     0.0  4988.2   64  69.9  65
da1        0.0 109.8     0.0  5091.5   64  91.7  73
da2        0.0 107.8     0.0  5014.1   64  48.3  57
da3        0.0  86.8     0.0  3733.3   43  71.4  73
da4        0.0 109.8     0.0  5088.0   64  70.6  71
da5        0.0 104.8     0.0  4578.0   62  59.0  61
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0  98.8     0.0  5724.0   69 490.3 109   0  0  2  1 97
da1        0.0  86.8     0.0  5007.5   69 562.7 102
da2        0.0 125.7     0.0  7222.9   69 465.5 114
da3        0.0  92.8     0.0  5303.4   70 538.2 100
da4        0.0 113.8     0.0  6557.3   70 486.6 104
da5        0.0 110.8     0.0  6339.2   69 468.6 114
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 104.8     0.0  6032.5   68 706.3 100   0  0  2  0 98
da1        0.0 103.8     0.0  5993.6   62 654.4  99
da2        0.0 104.8     0.0  6057.4   67 657.3 100
da3        0.0 104.8     0.0  6057.4   62 654.0 100
da4        0.0 104.8     0.0  6057.4   67 644.2 100
da5        0.0 103.8     0.0  5993.6   61 683.7  99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 104.8     0.0  6057.3   67 624.9 100   0  0  1  1 98
da1        0.0 104.8     0.0  6057.3   63 531.2 100
da2        0.0 104.8     0.0  6057.3   68 625.4 100
da3        0.0 104.8     0.0  6032.3   63 551.6 100
da4        0.0 104.8     0.0  6057.3   68 625.7 100
da5        0.0 105.8     0.0  6121.1   61 510.1 101
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  6160.2   68 616.8 100   0  0  2  1 97
da1        0.0 106.8     0.0  6160.2   62 522.8 100
da2        0.0 106.8     0.0  6160.2   67 617.1 100
da3        0.0 106.8     0.0  6185.1   62 522.8 100
da4        0.0 106.8     0.0  6160.2   67 617.4 100
da5        0.0 106.8     0.0  6160.2   62 503.4 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.7     0.0  6221.3   68 607.9 100   0  0  0  2 98
da1        0.0 108.7     0.0  6285.1   63 515.2 100
da2        0.0 108.7     0.0  6285.1   68 608.3 101
da3        0.0 108.7     0.0  6260.2   63 515.3 100
da4        0.0 107.7     0.0  6221.3   67 608.6 100
da5        0.0 107.7     0.0  6221.3   62 496.0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6354.5   68 599.2 101   0  0  1  1 98
da1        0.0 109.8     0.0  6329.5   63 507.8 100
da2        0.0 108.8     0.0  6290.6   65 599.4  99
da3        0.0 108.8     0.0  6290.6   62 507.9  99
da4        0.0 109.8     0.0  6354.5   66 599.7 100
da5        0.0 109.8     0.0  6329.5   62 489.0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 112.8     0.0  6518.3   31 586.1 100   0  0  1  1 98
da1        0.0 112.8     0.0  6518.3   53 480.6 100
da2        0.0 112.8     0.0  6493.4    5 573.1 100
da3        0.0 112.8     0.0  6174.5   63 677.5 100
da4        0.0 112.8     0.0  6493.4   15 576.6 100
da5        0.0 111.8     0.0  5690.1   42 846.4 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 159.6     0.0  5581.4    1 208.5  86   0  0  7  7 86
da1        0.0 188.5     0.0  6181.4    1 346.6  87
da2        0.0 131.7     0.0  4023.3    2 152.0  76
da3        0.0 191.5     0.0  7381.9    5 244.5  86
da4        0.0 140.6     0.0  4442.3    5 177.2  80
da5        0.0 164.6     0.0  5976.9    5 149.8  84
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 103.8     0.0  4945.3   13  50.8  78   0  0 12  7 81
da1        0.0 103.8     0.0  4945.3   13  52.6  79
da2        0.0 105.8     0.0  4983.7   13  52.0  77
da3        0.0 108.8     0.0  5150.5   13  54.2  80
da4        0.0 107.8     0.0  5150.5   13  52.5  79
da5        0.0 107.8     0.0  5150.5   13  54.9  79
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5250.5    4  51.8  80   0  0  7  7 86
da1        0.0 106.8     0.0  5250.5    4  53.5  79
da2        0.0 107.8     0.0  5225.1    5  52.8  81
da3        0.0 107.8     0.0  5225.1    5  54.9  82
da4        0.0 102.8     0.0  5020.5    8  51.7  74
da5        0.0 102.8     0.0  5020.5    8  53.2  74
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 102.7     0.0  5069.2    0  53.3  77   0  0  7  7 85
da1        0.0 102.7     0.0  5069.2    0  56.2  80
da2        0.0 104.7     0.0  5094.7    0  54.2  80
da3        0.0 104.7     0.0  5094.7    0  55.7  80
da4        0.0 108.7     0.0  5299.6    0  55.0  83
da5        0.0 108.7     0.0  5299.6    0  57.2  83
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  5304.2    5  48.8  77   0  0  7  8 85
da1        0.0 109.8     0.0  5304.2    5  50.9  79
da2        0.0 105.8     0.0  5048.1    9  49.2  75
da3        0.0 105.8     0.0  5048.1    9  50.7  77
da4        0.0 106.8     0.0  5073.6    9  49.7  74
da5        0.0 106.8     0.0  5073.6    9  50.4  74
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 105.7     0.0  5095.2    1  55.4  80   0  0 10  8 83
da1        0.0 105.7     0.0  5095.2    1  57.4  80
da2        0.0 108.7     0.0  5376.5    1  54.4  81
da3        0.0 108.7     0.0  5376.5    1  56.5  82
da4        0.0 104.7     0.0  5120.1    5  53.7  79
da5        0.0 108.7     0.0  5350.5    1  56.6  86
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0  13.0     0.0   207.7    0   1.8   0   0  0 10 10 80
da0        0.0 107.8     0.0  5124.5    6  49.7  78
da1        0.0 107.8     0.0  5124.5    6  51.6  79
da2        0.0 104.8     0.0  4970.7    8  49.4  73
da3        0.0 102.8     0.0  4868.4   10  52.2  70
da4        0.0 110.8     0.0  5330.1    6  50.6  80
da5        0.0 106.8     0.0  5099.5    6  51.1  75
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0   1.0     0.0     2.0    0   0.2   0   0  0  8 10 82
da0        0.0 102.7     0.0  5069.0    0  51.9  78
da1        0.0 102.7     0.0  5069.0    0  53.1  78
da2        0.0 105.7     0.0  5197.2    0  52.0  77
da3        0.0 107.7     0.0  5299.4    0  53.1  81
da4        0.0 105.7     0.0  5069.0    0  54.6  79
da5        0.0 105.7     0.0  5069.0    0  55.1  81
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 105.8     0.0  5099.8    7  49.0  74   0  0  9  7 84
da1        0.0 105.8     0.0  5099.8    7  49.6  75
da2        0.0 110.8     0.0  5329.9    3  49.2  80
da3        0.0 110.8     0.0  5329.9    3  51.2  80
da4        0.0 109.8     0.0  5240.5    4  50.6  78
da5        0.0 106.8     0.0  5073.8    7  51.5  77
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5198.3    0  53.9  80   0  0  6  7 87
da1        0.0 106.8     0.0  5198.3    0  56.4  82
da2        0.0 101.8     0.0  4968.3    0  53.5  77
da3        0.0 101.8     0.0  4968.3    0  54.4  80
da4        0.0 103.8     0.0  5057.6    0  53.3  75
da5        0.0 106.8     0.0  5224.3    0  55.6  77
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5098.2    7  48.8  73   0  0  7  9 84
da1        0.0 106.8     0.0  5098.2    7  50.7  76
da2        0.0 110.8     0.0  5328.2    3  49.2  78
da3        0.0 110.8     0.0  5328.2    3  51.2  80
da4        0.0 108.8     0.0  5238.9    4  49.8  75
da5        0.0 105.8     0.0  5072.2    7  50.7  73
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 137.7     0.0  7561.1   70 223.8  94   0  0 13  2 85
da1        0.0 116.8     0.0  6087.7   70 241.2  94
da2        0.0  92.8     0.0  4976.1   70 297.9  91
da3        0.0 111.8     0.0  5906.1   70 277.4  72
da4        0.0  80.8     0.0  4320.9   70 279.3  88
da5        0.0 106.8     0.0  5625.7   70 316.1  91
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 111.7     0.0  6451.7   70 643.3 100   0  0  1  1 98
da1        0.0 111.7     0.0  6451.7   70 643.6 100
da2        0.0 111.7     0.0  6451.7   70 595.3 100
da3        0.0 104.7     0.0  6054.7   69 641.7 118
da4        0.0 112.7     0.0  6515.5   69 620.6 101
da5        0.0 103.7     0.0  5990.8   70 602.5 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 108.8     0.0  6287.9   69 635.6 100   0  0  2  2 97
da1        0.0 109.8     0.0  6326.8   70 636.0 100
da2        0.0 109.8     0.0  6326.8   70 635.7 100
da3        0.0 109.8     0.0  6351.7   69 635.9 101
da4        0.0 108.8     0.0  6262.9   70 635.4 100
da5        0.0 109.8     0.0  6326.8   70 635.3 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0   8.0     0.0   127.7    0   1.1   0   0  0  3  0 96
da0        0.0 109.8     0.0  6326.8   65 616.5 100
da1        0.0 108.8     0.0  6287.9   62 630.3 100
da2        0.0 108.8     0.0  6287.9   66 635.8  99
da3        0.0 109.8     0.0  6326.8   60 634.5 100
da4        0.0 109.8     0.0  6351.7   66 635.5 100
da5        0.0 108.8     0.0  6287.9   61 635.4  99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6351.7   65 562.5 100   0  0  4  1 95
da1        0.0 109.8     0.0  6351.7   62 507.9 100
da2        0.0 109.8     0.0  6351.7   66 586.2 100
da3        0.0 108.8     0.0  6287.9   61 471.3  99
da4        0.0 108.8     0.0  6287.9   65 576.8  99
da5        0.0 109.8     0.0  6351.7   61 503.6 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 108.8     0.0  6263.0   66 562.1  99   0  0  2  0 97
da1        0.0 108.8     0.0  6263.0   63 507.6  99
da2        0.0 109.8     0.0  6326.8   66 580.3 100
da3        0.0 109.8     0.0  6326.8   61 471.1 100
da4        0.0 109.8     0.0  6326.8   65 561.7 100
da5        0.0 109.8     0.0  6326.8   61 488.7 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 129.8     0.0  4390.7    0 380.0  70   0  0  1  4 95
da1        0.0 154.7     0.0  5853.3    0 440.6  91
da2        0.0 166.7     0.0  6683.4    0 410.9  96
da3        0.0 169.7     0.0  6133.9    0 439.4  95
da4        0.0 176.7     0.0  7427.7    0 424.0  98
da5        0.0 170.7     0.0  6736.9    0 449.1  97
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 111.8     0.0  5148.2    8  44.7  75   0  0  7  7 85
da1        0.0 111.8     0.0  5148.2    8  47.3  75
da2        0.0 109.8     0.0  5148.2    8  45.2  75
da3        0.0 109.8     0.0  5148.2    8  47.2  76
da4        0.0 107.8     0.0  4917.7   13  46.1  77
da5        0.0 107.8     0.0  4917.7   13  48.2  76
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 105.7     0.0  5196.9    0  52.8  80   0  0  8  9 83
da1        0.0 105.7     0.0  5196.9    0  55.7  83
da2        0.0 106.7     0.0  5171.9    1  53.8  78
da3        0.0 106.7     0.0  5171.9    1  55.9  78
da4        0.0 109.7     0.0  5427.8    0  52.2  81
da5        0.0 109.7     0.0  5427.8    0  54.8  82
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5227.4    4  47.3  75   0  0 12  8 80
da1        0.0 106.8     0.0  5227.4    4  48.9  75
da2        0.0 107.8     0.0  4996.7    9  49.3  80
da3        0.0 107.8     0.0  4996.7    9  52.0  81
da4        0.0 104.8     0.0  4996.7    8  47.2  65
da5        0.0 104.8     0.0  4996.7    8  50.0  68
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 102.7     0.0  4966.8    0  53.2  76   0  0  8  9 83
da1        0.0 102.7     0.0  4966.8    0  56.1  77
da2        0.0 105.7     0.0  5222.6    0  52.4  76
da3        0.0 105.7     0.0  5222.6    0  55.4  78
da4        0.0 106.7     0.0  5196.7    0  53.6  81
da5        0.0 106.7     0.0  5196.7    0  56.1  81
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 111.8     0.0  5279.0    6  50.9  82   0  0  6 10 84
da1        0.0 113.8     0.0  5406.7    4  52.7  82
da2        0.0 106.8     0.0  5150.7    9  49.9  76
da3        0.0 106.8     0.0  5150.7    9  51.4  77
da4        0.0 106.8     0.0  5176.6    8  49.1  76
da5        0.0 106.8     0.0  5176.6    8  50.2  78
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 103.7     0.0  5146.0    0  53.0  77   0  0  7  9 84
da1        0.0 101.7     0.0  5018.3    0  55.6  79
da2        0.0 108.7     0.0  5273.7    0  54.7  82
da3        0.0 108.7     0.0  5273.7    0  55.2  82
da4        0.0 107.7     0.0  5248.2    0  54.3  80
da5        0.0 107.7     0.0  5248.2    0  56.9  80
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 106.8     0.0  5073.6    9  49.1  74   0  0 10  9 82
da1        0.0 106.8     0.0  5073.6    9  51.1  74
da2        0.0 108.8     0.0  5304.2    5  48.1  76
da3        0.0 108.8     0.0  5304.2    5  50.3  78
da4        0.0 106.8     0.0  5048.1    9  49.3  78
da5        0.0 106.8     0.0  5048.1    9  51.2  78
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 107.8     0.0  5122.3   11  50.3  79   0  0  8  8 83
da1        0.0 107.8     0.0  5122.3   11  52.7  83
da2        0.0 103.8     0.0  4892.3   11  50.4  71
da3        0.0 103.8     0.0  4892.3   11  51.7  71
da4        0.0 109.8     0.0  5378.8    6  48.5  74
da5        0.0 109.8     0.0  5378.8    6  50.0  77
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 108.5     0.0  5188.6    4  51.0  78   0  0  7  9 84
da1        0.0 108.5     0.0  5188.6    4  53.0  80
da2        0.0 106.6     0.0  5188.6    4  51.6  72
da3        0.0 106.6     0.0  5188.6    4  52.6  73
da4        0.0  99.6     0.0  4958.1    4  50.5  76
da5        0.0  99.6     0.0  4958.1    4  52.4  77
^C
#




Here is a quick sample with sftp and ZFS (keep in mind ssh has
encryption NFS does not)

# ifstat -b -i re0
      re0
 Kbps in  Kbps out
260931.9   8590.13
205778.7   6825.03
285794.0   9565.88
242095.0   8016.92
238060.4   7924.14
262288.7   8660.99
260094.6   8660.25
268791.7   8869.08
303607.5  10010.25
263964.4   8736.56
261743.6   8659.45
246390.6   8191.83
300105.3  10305.06
255019.9   8291.85
283355.3   9390.99
277390.9   9198.74
279070.0   9218.81
278103.5   9339.65
258649.8   8492.39
251343.6   8320.49
283036.9   9346.05
283302.4   9277.87
278886.2   9384.52
300472.5   9772.91
292509.8   9669.22
299081.4   9816.50
313643.0  10299.40
289208.9   9747.82
313738.8  10228.35
207670.6   6837.77
248762.1   8224.13
194959.1   6457.88
265009.2   9495.61
249181.9   8228.59
272940.6   8945.51
226037.8   7485.62
273701.2   9048.08
236890.4   7834.88

# iostat -zxC 1
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.1   0.4     1.8     6.1    0   3.0   0   1  0  1  1 97
da0        2.0  13.4   117.8   551.1   16 203.8  10
da1        1.9  13.7   108.8   550.6   16 200.5  10
da2        2.1  13.4   117.8   551.1   20 203.0  10
da3        1.9  13.8   107.1   550.6   20 200.1   9
da4        2.0  13.4   116.8   551.1   16 203.4   9
da5        1.9  13.8   107.3   550.6   16 199.8   9
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 108.8     0.0  6289.0   69 315.8 101  21  0 18 13 48
da1        0.0 107.8     0.0  6225.1   70 321.8 102
da2        0.0 108.8     0.0  6289.0   69 331.0 100
da3        0.0 111.8     0.0  6455.7   70 322.7 101
da4        0.0 104.8     0.0  6058.4   69 326.7 101
da5        0.0 108.8     0.0  6289.0   69 327.1 101
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6326.8   69 633.3 100  15  0  8  9 68
da1        0.0 109.8     0.0  6326.8   70 627.8 100
da2        0.0 109.8     0.0  6351.8   69 624.3 100
da3        0.0 108.8     0.0  6287.9   69 629.0  99
da4        0.0 109.8     0.0  6351.8   69 640.0 100
da5        0.0 109.8     0.0  6351.8   69 625.0 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6351.9   44 634.9 100   3  0  3  3 91
da1        0.0 108.8     0.0  6288.0   46 634.5  99
da2        0.0 109.8     0.0  6326.9   41 634.8 100
da3        0.0 109.8     0.0  6351.9   39 634.6 100
da4        0.0 109.8     0.0  6326.9   48 635.2 100
da5        0.0 108.8     0.0  6263.0   45 635.2  99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 111.8     0.0  6163.1   68 356.9  98   9  0 18  6 67
da1        0.0 127.7     0.0  7109.6   68 354.4  98
da2        0.0 106.8     0.0  5829.3   68 399.2  91
da3        0.0 104.8     0.0  5701.1   67 349.2  84
da4        0.0 113.8     0.0  6291.3   68 377.4  96
da5        0.0 110.8     0.0  6124.2   68 353.0  79
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 129.8     0.0  6805.8   70 519.5 100  19  0 14 19 49
da1        0.0 119.8     0.0  5371.1   70 502.2 100
da2        0.0  93.8     0.0  5408.1   58 607.6 104
da3        0.0 139.8     0.0  6706.4   70 457.6 111
da4        0.0 132.8     0.0  7582.5   60 476.8 102
da5        0.0 130.8     0.0  6156.3   69 511.3 117
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6326.8   70 535.4 100  17  0 12 12 59
da1        0.0 109.8     0.0  6351.8   70 523.0 100
da2        0.0 109.8     0.0  6351.8   58 396.3 100
da3        0.0 109.8     0.0  6326.8   70 521.1 100
da4        0.0 109.8     0.0  6351.8   60 431.1 100
da5        0.0 109.8     0.0  6326.8   69 515.6 101
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6351.7   70 632.7 100  18  0 11 15 56
da1        0.0 109.8     0.0  6326.8   70 632.7 100
da2        0.0 109.8     0.0  6326.8   58 414.1 100
da3        0.0 110.8     0.0  6415.6   69 632.4 101
da4        0.0 109.8     0.0  6326.8   60 468.3 100
da5        0.0 109.8     0.0  6351.7   69 632.5 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 104.8     0.0  5996.4    0 631.6  93  22  0  8 14 57
da1        0.0 123.7     0.0  6598.7    0 616.9  99
da2        0.0 135.7     0.0  7264.3    5 668.5 101
da3        0.0 115.8     0.0  6149.6    0 623.4  97
da4        0.0  99.8     0.0  5323.8    0 746.0  89
da5        0.0 125.7     0.0  6735.4    0 615.4  99
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 126.7     0.0  4006.1   70 115.8  43  17  0 18 12 54
da1        0.0 126.7     0.0  3634.9   70 115.3  51
da2        0.0 109.8     0.0  3090.1   70 100.1  52
da3        0.0 109.8     0.0  2708.9   70 107.4  48
da4        0.0 115.8     0.0  3163.9   70  92.1  52
da5        0.0 118.8     0.0  3196.3   70 111.5  52
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 103.8     0.0  5993.5   67 592.8 108  20  0 12 15 53
da1        0.0 114.8     0.0  6646.1   62 547.2 100
da2        0.0 117.8     0.0  6812.8   67 554.3 100
da3        0.0 108.8     0.0  6287.9   64 566.6 103
da4        0.0 108.8     0.0  6287.9   66 595.2 100
da5        0.0 104.8     0.0  6057.4   62 597.1 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.8     0.0  6326.6   67 597.7 100  23  0 10 15 52
da1        0.0 109.8     0.0  6326.6   62 546.1 100
da2        0.0 109.8     0.0  6326.6   67 601.6 100
da3        0.0 110.8     0.0  6390.5   62 563.9 101
da4        0.0 109.8     0.0  6351.5   66 603.0 100
da5        0.0 109.8     0.0  6326.6   62 566.4 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
ad4        0.0   1.0     0.0     6.0    0   0.4   0  18  0 10 13 59
da0        0.0 109.7     0.0  6349.1   67 578.3 100
da1        0.0 109.7     0.0  6349.1   62 506.0 100
da2        0.0 109.7     0.0  6349.1   67 578.2 100
da3        0.0 109.7     0.0  6324.2   62 486.9 100
da4        0.0 109.7     0.0  6324.2   66 578.2 100
da5        0.0 109.7     0.0  6349.1   61 505.5 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 110.8     0.0  6367.5   37 577.3 101  18  0 10 12 60
da1        0.0 109.8     0.0  6303.6   40 505.1 100
da2        0.0 109.8     0.0  6303.6   44 577.4 100
da3        0.0 109.8     0.0  6303.6   60 486.3 100
da4        0.0 109.8     0.0  6303.6   47 577.3 100
da5        0.0 109.8     0.0  6316.6   58 500.2 100
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 145.7     0.0  5188.2   70 280.0  85  19  0 16 10 55
da1        0.0 146.7     0.0  4827.9   70 360.9  89
da2        0.0 154.7     0.0  6099.2   70 300.0  83
da3        0.0 173.6     0.0  6462.0   70 371.0  84
da4        0.0 163.7     0.0  6266.4   70 297.1  91
da5        0.0 176.6     0.0  6327.8   70 352.1  89
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 122.7     0.0  7082.2   69 507.1 102  25  0 11 13 51
da1        0.0 117.8     0.0  6812.8   70 514.6  99
da2        0.0 104.8     0.0  6057.3   69 589.2 108
da3        0.0 102.8     0.0  5929.6   69 591.3 109
da4        0.0 107.8     0.0  6224.0   70 561.0 101
da5        0.0 105.8     0.0  6096.3   70 566.3 104
                        extended device statistics             cpu
device     r/s   w/s    kr/s    kw/s wait svc_t  %b  us ni sy in id
da0        0.0 109.7     0.0  6348.9   69 632.2 100  20  0 12 11 57
da1        0.0 109.7     0.0  6324.0   70 633.1 100
da2        0.0 109.7     0.0  6324.0   69 632.3 100
da3        0.0 109.7     0.0  6348.9   69 633.1 100
da4        0.0 109.7     0.0  6348.9   70 633.1 100
da5        0.0 109.7     0.0  6348.9   70 632.3 100
^C
#
Received on Wed Nov 11 2009 - 20:26:17 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:57 UTC