Help ZFS FreeBSD 8.0 RC2 Write performance issue

From: Sam Fourman Jr. <sfourman_at_gmail.com>
Date: Wed, 11 Nov 2009 12:50:43 -0600
Hello list,

I am running FreeBSD 8.0RC2 and I dont understand why my ZFS/NFS is
acting weird on writes.
I get ~150mbit writes idk if this is good or not? but it paused for a
few seconds every once and awhile.

I have cut and pasted a bunch of stuff below. does anyone have any insight?
this is a recent machine with a AMD processor and 4GB of memory.
the switch in the middle is a Dell 5224 gigabit switch if that maters.

and the Clients performing the writes are also FreeBSD 8RC2

I will say one thing... While I am performing writes.. if I was to do
a Read from a different client
I only can get like ~3mbit reads. if I do nothing but a read on a
single client I can get ~300mbit reads


any help would be appreciated

Sam Fourman Jr.
Fourman Networks.

# zpool status
  pool: Network
 state: ONLINE
 scrub: none requested
config:

	NAME        STATE     READ WRITE CKSUM
	Network     ONLINE       0     0     0
	  raidz1    ONLINE       0     0     0
	    da0     ONLINE       0     0     0
	    da1     ONLINE       0     0     0
	    da2     ONLINE       0     0     0
	    da3     ONLINE       0     0     0
	    da4     ONLINE       0     0     0
	    da5     ONLINE       0     0     0

errors: No known data errors



# ifconfig
re0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
	options=389b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC>
	ether 90:e6:ba:10:0e:2d
	inet 192.168.12.10 netmask 0xffffff00 broadcast 192.168.12.255
	inet 192.168.12.11 netmask 0xffffff00 broadcast 192.168.12.255
	inet 192.168.12.12 netmask 0xffffff00 broadcast 192.168.12.255
	inet 192.168.12.13 netmask 0xffffff00 broadcast 192.168.12.255
	media: Ethernet autoselect (1000baseT <full-duplex>)
	status: active
plip0: flags=8810<POINTOPOINT,SIMPLEX,MULTICAST> metric 0 mtu 1500
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
	options=3<RXCSUM,TXCSUM>
	inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
	inet6 ::1 prefixlen 128
	inet 127.0.0.1 netmask 0xff000000
# ifstat -b -i re0
       re0
 Kbps in  Kbps out
    3.37      5.18
    3.37      5.05
    3.37      5.05
   42.21     48.22
    5.89      5.05
    3.37      5.05
    3.37      5.05
   41.68     48.20
  110.44    101.74
    3.37      5.05
    3.37      5.05
    5.03      6.37
   10.83     12.84
   62.05     62.50
    3.37      5.05
    3.37      5.05
    5.21      6.47
    3.37      5.05
56521.85   2052.38
133949.5   4723.58
141345.1   4968.34
154383.0   5433.63
154244.1   5423.14
121558.2   4306.39
155455.9   5475.89
142424.3   5015.22
159326.7   5609.79
152141.6   5281.99
146702.9   7199.11
151088.5   5312.14
151621.7   5346.30
112848.5   3939.65
    4.80      5.05
   46.20     49.68
    5.21      6.47
    3.37      5.05
    3.37      5.05
    5.21      6.47
   44.72     48.20
132888.9   4652.04
153707.7   5402.98
152234.6   5343.49
136510.0   4814.00
124268.0   4437.97
158612.9   5590.24
156464.4   5514.69
142010.5   5010.10
148333.6   5211.41
       re0
 Kbps in  Kbps out
155707.6   5525.03
156187.8   5504.36
121838.5   4282.46
16874.25    621.48
    5.21      6.47
   46.17     49.61
    3.37      5.05
    5.21      6.47
    3.37      5.05
30120.71   1058.03
156991.7   5639.63
147261.1   5311.55
131024.8   4725.30
151883.2   5353.18
155146.7   5456.85
161816.2   5727.37
144066.7   5138.94
155767.4   5507.38
142959.7   5043.90
152714.3   5428.75
110144.8   3987.22
    3.37      5.05
    3.37      5.05
    5.03      6.37
    3.37      5.05
   64.62     68.86
11302.01    367.31
154841.7   5489.48
160238.3   5635.34
151150.4   5323.96
160726.1   5696.35
152421.8   5381.29
161405.3   5751.44
152095.4   5349.11
116161.9   4104.15
158146.7   5573.70
147790.0   5247.66
12551.27    462.41
    3.37      5.05
    3.37      5.05
    3.37      5.05
 2160.03   2117.73
11297.74    376.93
158567.7   5610.60
147388.6   5186.73
151555.5   5349.12
157579.2   5587.67
149001.2   5268.48
       re0
 Kbps in  Kbps out
154172.6   5507.73
162795.7   5753.79
151039.5   5315.01
153744.4   5478.71
88269.23   3137.44
    3.37      5.05
    3.37      5.05
   25.07     24.70
   44.72     48.20
    3.37      5.05
107780.3   3813.35
151123.3   5326.76
153103.2   5440.45
149461.4   5318.25
154987.6   5465.26
147789.8   5223.30
140811.6   4981.23
158819.1   5658.08
156550.1   5542.69
117199.0   4096.97
    3.37      5.05
    3.37      5.05
    3.37      5.05
   44.70     48.17
    3.37      5.05
116172.3   4084.82
159684.2   5595.67
155388.3   5470.82
153778.8   5471.12
160333.3   5650.18
151555.5   5346.59
153912.8   5427.27
146753.1   5234.99
151624.0   5337.10
41923.68   1478.39
    3.37      5.05
    3.37      5.05
   66.24     67.47
    3.37      5.05
61305.44   2143.38
157087.6   5549.02
155647.7   5553.67
157510.1   5605.24
146786.6   5180.34
135555.9   4780.34
158633.3   5578.98
157735.7   5538.46
151445.5   5390.60
       re0
 Kbps in  Kbps out
88655.49   3138.99
    5.21      7.61
    3.37      5.05
    3.37      5.05
  314.11    309.87
35855.92   3019.12
126837.6   4492.57
136117.2   4806.87
147924.3   5229.35
150553.5   5315.17
158044.2   5609.08
159151.6   5627.26
147855.5   5178.02
156533.7   5495.67
125278.2   4390.51
   44.72     48.20
    3.37      5.05
    3.37      5.05
    3.37      5.05
12400.47    420.22
141994.2   5045.35
157535.1   5537.08
154312.9   5412.69
159218.2   5633.57
152698.8   5365.41
155403.4   5563.36
150104.5   5317.83
156397.1   5517.41
86013.25   3048.44
    3.37      5.05
   44.72     48.20
    3.37      5.05
    3.37      5.05
47334.93   1647.06
149320.0   5347.42
149435.6   5308.92
156021.1   5474.01
152563.6   5375.73
155523.0   5473.40
152630.9   5385.65
152203.3   5394.63
142484.9   5030.69
56701.45   2052.13
    3.37      5.05
    3.37      5.05
   46.17     49.62
    5.03      6.37
94137.63   3297.45
       re0
 Kbps in  Kbps out
164023.3   5785.23
147958.4   5279.17
152165.2   5356.31
147259.5   5208.91
155024.8   5456.01
158034.7   5552.35
147619.9   5249.18
135962.4   4821.04
    3.37      5.05
    3.37      5.05
    3.37      5.05
   44.54     48.00
24643.59    808.53
155887.8   5499.69
152622.6   5374.52
90909.52   3216.71
150315.9   6516.74
149293.3   6100.74
157711.7   5644.70
142586.9   5036.40
146279.9   5149.50
126335.6   4510.90
    3.37      5.05
    3.37      5.05
    3.37      5.05
    3.37      5.05
31785.91   1137.64
152113.2   5361.44
149366.4   5262.94
141578.1   4984.64
147796.4   5205.79
154105.2   5549.08
161407.3   5713.83
149983.9   5294.44
145774.1   5148.62
58054.58   2079.92
    3.37      5.05
   45.64     48.75
    3.37      5.05
   10.38     41.29
115633.4   4057.17
145172.9   5109.34
160267.6   5680.67
154851.5   5455.03
157873.1   5621.18
150019.6   5295.47
160156.1   5655.63
146290.4   5200.17
       re0
 Kbps in  Kbps out
97345.63   3477.19
    3.37      6.19
    3.37      5.05
    3.37      5.05
   46.56     49.61
73127.16   2544.00
150076.1   5293.10
152638.9   5381.02
153774.1   5412.67
155966.5   5522.56
161909.6   5662.65
158143.3   5553.83
150476.0   5284.16
115083.2   4048.69
11872.56    459.17
    3.37      5.05
    3.37      5.05
    5.10      6.69
48749.13   1788.02
160387.1   5730.22
151622.1   5301.40
158541.1   5553.69
163521.1   5719.97
155853.6   5478.11
159261.4   5626.13
^C
#

/boot/loader.conf

kern.ipc.semaem=32767
kern.ipc.semvmx=65534
kern.ipc.semusz=184
kern.ipc.semume=80
kern.ipc.semopm=200
kern.ipc.semmsl=120
kern.ipc.semmnu=4096
kern.ipc.semmns=8192
kern.ipc.semmni=32767
kern.ipc.semmap=60


/etc/sysctl.conf

# Jail Settings
security.jail.jailed: 0
security.jail.mount_allowed: 0
security.jail.chflags_allowed: 0
security.jail.allow_raw_sockets: 1
security.jail.enforce_statfs: 2
security.jail.sysvipc_allowed: 1
security.jail.socket_unixiproute_only: 1
security.jail.set_hostname_allowed: 1

# More Shared memory for jails/PostgreSQL

kern.ipc.shmall=65536
kern.ipc.shmmax=134217728
kern.ipc.semmap=4096


# dmesg
Copyright (c) 1992-2009 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 8.0-RC2 #2: Tue Nov 10 21:05:03 CST 2009
    sfourman_at_.PuffyBSD.Com:/usr/obj/usr/src/sys/GENERIC amd64
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: AMD Phenom(tm) II X2 550 Processor (3100.02-MHz K8-class CPU)
  Origin = "AuthenticAMD"  Id = 0x100f42  Stepping = 2
  Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SSE2,HTT>
  Features2=0x802009<SSE3,MON,CX16,POPCNT>
  AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
  AMD Features2=0x37ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS,SKINIT,WDT>
  TSC: P-state invariant
real memory  = 4294967296 (4096 MB)
avail memory = 3847766016 (3669 MB)
ACPI APIC Table: <092809 APIC1459>
FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
FreeBSD/SMP: 1 package(s) x 2 core(s)
 cpu0 (BSP): APIC ID:  0
 cpu1 (AP): APIC ID:  1
ACPI Warning: Optional field Pm2ControlBlock has zero address or
length:        0       0/1 20090521 tbfadt-655
ioapic0 <Version 2.1> irqs 0-23 on motherboard
kbd1 at kbdmux0
acpi0: <092809 XSDT1459> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
acpi0: reservation of fee00000, 1000 (3) failed
acpi0: reservation of ffb80000, 80000 (3) failed
acpi0: reservation of fec10000, 20 (3) failed
acpi0: reservation of fed40000, 5000 (3) failed
acpi0: reservation of 0, a0000 (3) failed
acpi0: reservation of 100000, cff00000 (3) failed
ACPI HPET table warning: Sequence is non-zero (2)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <32-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
acpi_hpet0: HPET never increments, disabling
device_attach: acpi_hpet0 attach returned 6
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
vgapci0: <VGA-compatible display> port 0xd000-0xd0ff mem
0xd0000000-0xdfffffff,0xfe9f0000-0xfe9fffff,0xfe800000-0xfe8fffff irq
18 at device 5.0 on pci1
pci1: <multimedia, HDA> at device 5.1 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
pci2: <ACPI PCI bus> on pcib2
re0: <RealTek 8168/8168B/8168C/8168CP/8168D/8168DP/8111B/8111C/8111CP/8111DP
PCIe Gigabit Ethernet> port 0xe800-0xe8ff mem
0xfdbff000-0xfdbfffff,0xfdbf8000-0xfdbfbfff irq 18 at device 0.0 on
pci2
re0: Using 1 MSI messages
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
re0: Ethernet address: 90:e6:ba:10:0e:2d
re0: [FILTER]
atapci0: <ATI IXP700/800 SATA300 controller> port
0xc000-0xc007,0xb000-0xb003,0xa000-0xa007,0x9000-0x9003,0x8000-0x800f
mem 0xfe7ffc00-0xfe7fffff irq 22 at device 17.0 on pci0
atapci0: [ITHREAD]
atapci0: AHCI v1.10 controller with 4 3Gbps ports, PM supported
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
ata4: <ATA channel 2> on atapci0
ata4: [ITHREAD]
ata5: <ATA channel 3> on atapci0
ata5: [ITHREAD]
ohci0: <OHCI (generic) USB controller> mem 0xfe7fe000-0xfe7fefff irq
16 at device 18.0 on pci0
ohci0: [ITHREAD]
usbus0: <OHCI (generic) USB controller> on ohci0
ohci1: <OHCI (generic) USB controller> mem 0xfe7fd000-0xfe7fdfff irq
16 at device 18.1 on pci0
ohci1: [ITHREAD]
usbus1: <OHCI (generic) USB controller> on ohci1
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xfe7ff800-0xfe7ff8ff
irq 17 at device 18.2 on pci0
ehci0: [ITHREAD]
usbus2: EHCI version 1.0
usbus2: <EHCI (generic) USB 2.0 controller> on ehci0
ohci2: <OHCI (generic) USB controller> mem 0xfe7fc000-0xfe7fcfff irq
18 at device 19.0 on pci0
ohci2: [ITHREAD]
usbus3: <OHCI (generic) USB controller> on ohci2
ohci3: <OHCI (generic) USB controller> mem 0xfe7fb000-0xfe7fbfff irq
18 at device 19.1 on pci0
ohci3: [ITHREAD]
usbus4: <OHCI (generic) USB controller> on ohci3
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xfe7ff400-0xfe7ff4ff
irq 19 at device 19.2 on pci0
ehci1: [ITHREAD]
usbus5: EHCI version 1.0
usbus5: <EHCI (generic) USB 2.0 controller> on ehci1
pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
atapci1: <ATI IXP700/800 UDMA133 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 20.1 on
pci0
ata0: <ATA channel 0> on atapci1
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci1
ata1: [ITHREAD]
pci0: <multimedia, HDA> at device 20.2 (no driver attached)
isab0: <PCI-ISA bridge> at device 20.3 on pci0
isa0: <ISA bus> on isab0
pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 5.0 on pci3
pci4: <PCI bus> on pcib4
arcmsr0: <Areca SATA Host Adapter RAID Controller (RAID6 capable)
> mem 0xfebff000-0xfebfffff,0xfdc00000-0xfdffffff irq 22 at device 14.0 on pci4
ARECA RAID ADAPTER0: Driver Version 1.20.00.16 2009-10-10
ARECA RAID ADAPTER0: FIRMWARE VERSION V1.47 2009-06-25
arcmsr0: [ITHREAD]
ohci4: <OHCI (generic) USB controller> mem 0xfe7fa000-0xfe7fafff irq
18 at device 20.5 on pci0
ohci4: [ITHREAD]
usbus6: <OHCI (generic) USB controller> on ohci4
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppc0: [ITHREAD]
ppbus0: <Parallel port bus> on ppc0
plip0: <PLIP network interface> on ppbus0
plip0: [ITHREAD]
lpt0: <Printer> on ppbus0
lpt0: [ITHREAD]
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
cpu0: <ACPI CPU> on acpi0
acpi_throttle0: <ACPI CPU Throttling> on cpu0
hwpstate0: <Cool`n'Quiet 2.0> on cpu0
cpu1: <ACPI CPU> on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
orm0: <ISA Option ROM> at iomem 0xcf000-0xcffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 480Mbps High Speed USB v2.0
usbus3: 12Mbps Full Speed USB v1.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 480Mbps High Speed USB v2.0
usbus6: 12Mbps Full Speed USB v1.0
ad4: 38146MB <Seagate ST340014AS 8.05> at ata2-master SATA150
ugen0.1: <ATI> at usbus0
uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <ATI> at usbus1
uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <ATI> at usbus2
uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
ugen3.1: <ATI> at usbus3
uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
ugen4.1: <ATI> at usbus4
uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen5.1: <ATI> at usbus5
uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
ugen6.1: <ATI> at usbus6
uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
Waiting 5 seconds for SCSI devices to settle
uhub6: 2 ports with 2 removable, self powered
uhub0: 3 ports with 3 removable, self powered
uhub1: 3 ports with 3 removable, self powered
uhub3: 3 ports with 3 removable, self powered
uhub4: 3 ports with 3 removable, self powered
uhub2: 6 ports with 6 removable, self powered
uhub5: 6 ports with 6 removable, self powered
(probe16:arcmsr0:0:16:0): inquiry data fails comparison at DV1 step
da0 at arcmsr0 bus 0 target 0 lun 0
da0: <SAMSUNG HD103SJ R001> Fixed Direct Access SCSI-5 device
da0: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da0: Command Queueing enabled
da0: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da1 at arcmsr0 bus 0 target 0 lun 1
da1: <SAMSUNG HD103SJ R001> Fixed Direct Access SCSI-5 device
da1: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da1: Command Queueing enabled
da1: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da2 at arcmsr0 bus 0 target 0 lun 2
da2: <SAMSUNG HD103SJ R001> Fixed Direct Access SCSI-5 device
da2: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da2: Command Queueing enabled
da2: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da3 at arcmsr0 bus 0 target 0 lun 3
da3: <Hitachi HDT721010SLA360 R001> Fixed Direct Access SCSI-5 device
da3: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da3: Command Queueing enabled
da3: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da4 at arcmsr0 bus 0 target 0 lun 4
da4: <Hitachi HDT721010SLA360 R001> Fixed Direct Access SCSI-5 device
da4: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da4: Command Queueing enabled
da4: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
da5 at arcmsr0 bus 0 target 0 lun 5
da5: <Hitachi HDT721010SLA360 R001> Fixed Direct Access SCSI-5 device
da5: 166.666MB/s transfers (83.333MHz, offset 32, 16bit)
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors: 255H 63S/T 121601C)
SMP: AP CPU #1 Launched!
Trying to mount root from ufs:/dev/ad4s1a
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version 13
ZFS storage pool version 13
re0: link state changed to UP
#
Received on Wed Nov 11 2009 - 17:50:43 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:57 UTC