Hello, i'm testing the zfs since 7.0-Beta 1. First, i had only access to an 32 Bit Machine (P4/3GHz with 2GB RAM, 2xHD for RAID1 and 2xHD for ZFS Raid 0). While running iozone with the following call: iozone -R -a -z -b file.wks -g 4G -f testile (This is inspired by Dominic Kay from Sun, see http://blogs.sun.com/dom/entry/zfs_v_vxfs_iozone for details). the well known "kmem_malloc" error occured and stopped the system. (panic: kmem_malloc (131072): kmem_map too small: 398491648 total allocated cpuid=1) I tested several optimizations as suggested in the ZFS Tuning Guide and several postings on this list. The problem stayed mainly the same, it stopped with a "kmem_malloc" or rebooted without warning. This depends on the configuration, if i raised the vm.kmem_-sizes or only the KVA_PAGES or both. But it never ever made the benchmark. With more memory in vm.kmem_size and vm.kmem_size_max, the problem came later. But ok, the main target for the ZFS is to use amd64, not i386. Now i have access to an Intel Woodcrest-System, it's a Xeon 5160 with 4GB RAM and 1xHD. It has UFS for the System and Home and one ZFS only for data (for the iozone-Benchmark). It has a vanilla kernel, i haven't touched it. I've tested the default settings from Beta 3 and applied the tuning tips from the Tuning Guide. It shows the same behaviour as on the 32 Bit machine. One major difference: it makes always a reboot. There's no kmem_malloc error message (which made the system hang). The problem is the "-z" option in the iozone-Benchmark. Without it, the benchmark works (on the i386 and on the amd64-Machine). This option makes iozone testing small record sizes for large files. On an UFS-Filesystem, iozone works with the "-z" option. Though, it seems to me, that this is a problem with ZFS. Here are some more informations (from the amd64-System): 1. The captured iozone output [root_at_zfs /tank/iozone]# iozone -R -a -z -b filez-512M.wks -g 4G -f testile Iozone: Performance Test of File I/O Version $Revision: 3.283 $ Compiled for 64 bit mode. Build: freebsd Contributors:William Norcott, Don Capps, Isom Crawford, Kirby Collins Al Slater, Scott Rhine, Mike Wisner, Ken Goss Steve Landherr, Brad Smith, Mark Kelly, Dr. Alain CYR, Randy Dunlap, Mark Montague, Dan Million, Jean-Marc Zucconi, Jeff Blomberg, Benny Halevy, Erik Habbinga, Kris Strecker, Walter Wong. Run began: Thu Nov 29 09:38:45 2007 Excel chart generation enabled Auto Mode Cross over of record size disabled. Using maximum file size of 4194304 kilobytes. Command line used: iozone -R -a -z -b filez-512M.wks -g 4G -f testile Output is in Kbytes/sec Time Resolution = 0.000001 seconds. Processor cache size set to 1024 Kbytes. Processor cache line size set to 32 bytes. File stride size set to 17 * record size. random random bkwd record stride KB reclen write rewrite read reread read write read rewrite read fwrite frewrite fread freread 64 4 141594 426897 811459 874936 1105556 555990 712397 551422 780776 269837 376589 674790 780776 64 8 220788 647135 1421755 1492919 1279447 659861 1188789 666414 1279447 477808 653436 1017549 1204796 64 16 290890 1066042 2052169 2203800 1879725 1087638 1679761 1124074 1892980 457452 1124074 1210227 1638743 64 32 500060 1780008 2892445 3203069 2772930 2662899 3057153 2467108 2662899 587635 1734015 1210227 2067979 64 64 842003 2067979 3791156 4274062 3541098 2203800 2923952 2278628 2772930 1049372 1879725 1049372 2467108 128 4 92551 465304 841746 1291158 1165070 554157 1123617 559353 895074 348661 465304 739689 776042 128 8 161425 700139 1421023 2238774 1997245 1032829 1266785 1022989 1391557 449337 684954 1543594 1885042 128 16 271722 1152564 2211113 2286447 2098744 1684006 1852520 1123617 2843510 516318 1152564 1618028 2464907 128 32 403721 1749872 3036502 3277486 2905056 1852520 2511022 1885042 4267461 656477 1878447 1905110 3199360 128 64 702889 2420455 4104338 6406138 5603747 3759450 3445772 2202044 2969325 907174 2286447 2090572 2511022 128 128 1281909 3124872 4889281 3657016 6727225 4407601 3982553 4557257 3445772 927549 3277486 1243315 4267461 256 4 147637 437582 913973 1300235 1128110 528987 773140 534517 1117543 383232 487664 797843 1112909 256 8 281614 894929 1523457 2228578 2015259 977189 1339154 980760 1361224 523570 680960 1262029 1332507 256 16 442086 1347557 2419396 3465855 3123106 1673001 2984226 1660068 2783115 868157 1142514 1619994 1924938 256 32 656399 1897721 3087188 3236056 3654598 2754556 2880164 1868008 3043436 924200 2015259 2045979 2325094 256 64 948698 2441400 4197489 4422226 4496299 3605511 5569035 4009406 4009406 1157291 2533571 2371308 2330140 256 128 1455317 3285566 5117791 5217259 4755158 5117791 4264168 5142301 4819184 1274008 3454704 2435861 2726577 256 256 1790149 3368013 5428265 5717301 5455847 3557725 4929815 3667079 5117791 1598292 3557725 1152323 4477549 512 4 212984 535558 962218 1319250 1190529 535024 977103 535024 1007827 391390 550523 1225169 1245780 512 8 382604 853596 1438555 2381315 2016784 992456 1383861 1019791 1647088 660600 927716 1289930 2016784 512 16 560875 1518938 2235086 3220553 3304808 1803387 2124526 1807941 2625909 759416 1190529 2370799 2622702 512 32 748563 1961520 3239989 3634839 4784885 2958758 2910635 2942541 3239989 1064266 2081286 2089386 2625909 512 64 876593 2572435 6319739 6395018 6246213 3877684 4061007 3325278 6395018 1150351 2708713 3504346 3941742 512 128 1580419 3504346 7540170 7871843 7211022 4701087 4494470 5624545 4660280 1861217 3459188 2922519 4340054 512 256 1662389 3634839 7871843 7758090 8109658 5566231 7410080 5509113 5509113 2089386 3678421 2722449 4457157 512 512 1809465 4237291 5551840 5278892 8528336 5067142 5453155 4262523 5744919 1854787 4532414 1467055 4296637 1024 4 278265 570168 1190795 1395010 1257748 551995 1145689 582541 1027060 420382 555637 1162433 1261072 1024 8 457164 948995 1909888 2375790 2183727 1049138 1851434 1064479 1548960 660276 1056883 1701108 2010935 1024 16 626288 1506051 2908281 3544270 3294177 1799462 2688041 1808555 2694787 839384 1508167 2069060 2617596 1024 32 782353 2094282 3458646 5096034 4904018 2995501 3314514 2968586 2960401 992412 2950233 2782064 3314514 1024 64 992412 3210456 4265934 5789881 6519323 3999762 3506651 4300102 4356809 1082454 3121783 3411938 3923040 1024 128 1790460 3735385 4949226 7644933 7577494 5479632 4898425 6102490 5251818 2178189 4594502 2592318 4339202 1024 256 1871603 3941039 5120336 6016997 8000971 5303701 5950309 5983467 8542002 2353657 4654248 2679655 4488860 1024 512 1920993 3863055 7590887 8525047 8457895 5885083 6102490 5885083 5853003 2077065 5169641 2844715 4215688 1024 1024 1899750 4162573 6650556 8457895 8457895 5251818 6559147 5330028 6244448 2260740 5144870 939446 3239514 2048 4 319497 562949 1158327 1331606 1236708 562764 1068425 573817 1063399 417193 560634 1045410 1210394 2048 8 492434 1031724 1712231 2265394 2153510 1038584 1738569 1055170 1616531 618167 969758 1550304 1923059 2048 16 658494 489934 1943069 2800938 2921939 1681397 2115848 1790388 2131071 761059 1552545 1880950 2245848 2048 32 726249 2585923 3122697 4423683 4643695 2925920 2722809 2984890 2790021 946360 2322353 2278010 2980747 2048 64 938605 3334853 4432815 6261772 6397004 4088876 4562305 4540600 4321315 1062741 3459769 2929912 3766184 2048 128 1838281 3093459 5159722 6939647 7065219 5277002 5135046 6097316 6284678 1901771 4509611 3319389 4031308 2048 256 1851755 4330028 6340344 7368238 7761014 5900467 5095447 6280083 8097557 2365196 4581773 3065856 3953383 2048 512 1915768 4086930 5816563 7262337 7529708 5596757 5475461 6243566 8498106 2090107 4112366 2925920 3555722 2048 1024 1820362 4231899 5836323 6845629 7616502 5596757 5596757 5689430 6583305 2011785 3917325 1446164 2800938 2048 2048 1635615 3261415 4356379 4686767 4581773 3799501 4146110 3723735 3423913 1590196 3103517 781698 1279252 4096 4 348300 542656 1089070 1193062 1166017 549724 1025553 568816 1010116 409231 533954 1021467 1096997 4096 8 530295 934714 1647663 1875571 1920223 1031774 1577900 1051347 1548041 593785 920836 1445897 1639018 4096 16 685418 1507031 2268027 2630589 2830848 1752009 2109303 1790165 2118929 799368 1515406 1809206 2077169 4096 32 821342 2265336 2767462 3387995 3875619 2870101 2705147 2974454 2528382 986962 2179963 2154808 2470570 4096 64 918817 3131124 3740604 4214051 4780411 3938705 3937802 4467170 4218190 969255 3209507 2698349 2921346 4096 128 1891055 3897601 4132949 4669958 5447345 5409609 4438318 6041154 4417775 2526895 4133944 2880206 3126565 4096 256 1824383 3586757 4546375 4869845 5565581 5369035 4526013 6254490 6930704 2739658 4087714 2907010 3160500 4096 512 1795779 3789282 4581535 4813898 5519094 5713661 3337967 6234064 8569343 2220250 3605576 2801763 3082251 4096 1024 1693965 3100608 4136930 4481152 4612286 4361695 4075109 5980170 7877762 1618939 2996764 1609384 2385825 4096 2048 1480920 2303609 3040789 3086681 3640724 3965067 2858639 3647681 4471821 1701009 2344476 899243 1219398 4096 4096 1356290 1889183 2216526 2127852 2062457 1809206 2133401 1827100 2164855 1360263 1898998 570725 816735 8192 4 362720 511809 1046346 1121884 1097236 536654 1000347 576623 969141 387694 506207 961735 1019552 8192 8 529851 860611 1538050 1706315 1751188 966360 1507812 1061442 1460823 548222 854129 1367834 1505896 8192 16 680110 1301810 2051106 2317406 2473559 1581735 2028702 1805195 779294 697903 1244385 1693281 1842957 8192 32 806286 1757908 2422291 2876301 3207351 2443306 2427254 3014063 2382815 845844 1732994 1907835 2160336 8192 64 886112 2266801 3145975 3417925 3860295 3124235 3390605 4533661 3402019 949352 2239469 2267998 2455002 8192 128 1767039 2645131 3547798 3707029 4237859 3685162 3671773 6135880 3813593 2034829 2607196 2509694 2596361 8192 256 1773240 2545768 3481651 3668245 4348340 3588182 3897073 6389174 4051471 2038814 2623119 2480166 2502201 8192 512 1750386 2479450 3394960 3433293 4293461 3367015 3713841 6384425 8723390 1883674 2530022 2119034 2222376 8192 1024 1595914 2238156 2951916 2975692 3776706 3239099 3121964 6068689 8208631 1786703 2223671 1584215 1788283 8192 2048 1352223 1702763 2300494 2292665 2480166 1865977 2287019 3397646 4500997 1420011 1706315 976633 1147014 8192 4096 1250681 1486739 1787911 1793323 1843352 1695370 1775164 1680445 1997565 1232600 1503129 652596 803139 8192 8192 1154452 1319761 1455069 1472846 1459768 1300726 1423954 1339884 1442546 1144950 1322606 516293 714951 16384 4 369841 508235 1056202 1068120 1023817 500000 981134 571149 960831 375350 495344 975231 980267 16384 8 528567 842369 1551236 1585889 1588860 866654 835923 1025467 1222770 526494 825927 1381440 1385199 16384 16 671778 1233547 2023910 2080466 2164065 1329664 1982341 1833262 1936425 686928 1213979 1683359 1726108 16384 32 795614 1641187 2464582 2569259 2783088 1883199 2440685 3050429 2367285 812932 1624581 1933700 1986180 16384 64 867705 1939595 2763835 2113807 1956771 2085833 2820903 4603454 2861427 859382 1918744 2167751 2213771 16384 128 1719585 2121180 3007310 3073348 3550145 2588323 3098988 6316241 3112463 1756641 2145887 2265815 2307276 16384 256 1722128 2133363 3064166 3132182 3589833 2558450 3070464 6529907 3223269 1739654 2140873 2258666 2300479 16384 512 1689235 2088178 2980054 3036814 3449287 2429726 3206275 6657699 3178248 1701153 2101332 2128341 2176057 16384 1024 1541873 1916711 2721153 2731536 3165365 2128077 2387269 6350680 7992676 1505460 1894779 1600479 1707578 16384 2048 1334415 1602345 2174336 2166316 2351006 1734691 2177574 4147771 5501604 1350597 1589595 1048915 1163715 16384 4096 1199778 1398192 1705968 1703093 1701532 1415588 1697833 1668604 1915589 1200826 1407441 710919 794106 16384 8192 1116376 1282922 1406001 1413637 1414743 1292428 1398675 1309246 1406462 1112653 1282994 592119 704657 16384 16384 1110172 582399 1385311 1386009 1385059 1224055 1377950 1274333 1382857 1054031 1285321 508156 699244 32768 4 372304 508607 1047874 1063243 981315 484274 989881 578644 946589 371098 495336 975292 978590 32768 8 532024 834704 1568959 1579035 1492298 807249 1477571 1067355 1408592 530837 819182 1373622 1398972 32768 16 672109 1209106 2028744 2071679 2027338 1219135 1953262 1812107 729735 658122 1189743 1698878 1728709 32768 32 789743 1607683 2498126 2564790 2598787 1656873 2436307 3058153 2385355 799004 1593775 1410819 742348 32768 64 866533 1862775 2837314 2581798 2910127 1960981 2813903 4636659 2847955 870236 1863407 2177395 2212553 32768 128 1708891 2054553 3026769 3087280 3257634 2203931 3012637 6378919 3046156 1713984 2064521 2109033 2259067 32768 256 1694480 2053264 3051364 3117179 3342646 2218087 3093952 6639946 3129672 1689335 2054799 2274921 2298202 32768 512 1643892 1996707 3001584 3030439 3223254 2150144 2976685 6695584 3054144 1637331 2000980 2140866 2148866 32768 1024 1521119 1736506 2723623 2726378 2848428 1921556 2731363 6594388 2805746 1489323 1837890 1644895 1709677 32768 2048 1310567 1545094 2139467 2138069 2243322 1598428 2137105 4349911 5918020 1306853 1560090 1099161 1159237 32768 4096 1176372 1369161 1248686 1602584 1621606 1388653 1632935 1680412 1628716 1160911 1354494 748656 790561 32768 8192 1099592 1267526 1391958 1392663 1393850 1265810 1387952 1302235 1392960 1101557 1260493 640750 701055 32768 16384 1095516 1262496 1117400 1346928 1349878 1245642 1354334 1267526 1364147 1092416 1257932 584203 693268 65536 4 374405 494913 1038635 1044365 931862 459414 637268 560846 785907 362103 483264 960655 963808 65536 8 531517 493245 1529676 1568635 1422065 771057 1481240 1068072 1369385 522986 483967 1344739 1368983 65536 16 677528 1189897 2036301 2062164 1953273 1159172 1958338 1279973 1662844 657470 1173935 1689904 1709962 65536 32 677710 1483951 2498803 2550034 2483923 1587129 2454249 3053153 822778 772112 1578988 1945613 1974445 65536 64 860618 1838257 1849512 2164055 2711474 1838933 2770002 4654229 2742093 858666 1814246 2172521 2185546 65536 128 1691558 2017914 3038741 3054714 3106568 2084796 2811854 6463790 2934102 1676240 2025095 2281583 2296355 65536 256 1686980 2006366 3069587 3102886 3177198 2089932 3059985 6673607 3077250 1674820 1061986 2252787 2267917 65536 512 1630241 1963388 2983496 3011904 3099772 2015931 2989661 6775096 3012036 1627085 1979635 1538511 2035276 65536 1024 1517113 1815660 2698961 2716968 2767910 1861121 2697292 6627907 2731249 1502362 1591375 1661749 1691204 65536 2048 1295690 1505093 2130426 2130691 2158566 1555937 2138897 4386059 2156940 1282834 1526439 1124057 1149449 65536 4096 1162127 1353992 1590104 1596569 1611206 1366071 1618377 1681881 1883218 1150984 1342840 766762 739432 65536 8192 1095258 1252834 1378519 1381713 1190458 970628 1366125 1292115 1381769 1084781 1245720 667019 698067 65536 16384 1035829 1241943 1353046 1355281 1354026 1240928 1324892 1264784 1346399 1086706 1245342 631520 660618 131072 4 374708 67502 947641 965305 720576 374085 989050 582278 908399 360387 493141 977615 980182 131072 8 107107 812985 1515790 1520911 1311678 676420 1466602 1055262 1330194 525966 804448 743734 1229788 131072 16 680264 1195290 2008586 2022307 1832587 1093223 1964270 1813042 706252 669749 1175662 1705579 1707311 131072 32 106148 1544419 2494662 2518525 2378291 1515352 2421961 3085797 809141 785097 1394916 1955453 1967779 131072 64 108041 1808414 2856277 2873176 2831912 1806316 2789947 4656863 2746515 842576 1811339 2087034 2122627 131072 128 1687636 2022902 2993533 3017108 3052591 2038701 2179056 6404066 2887102 1666377 2018046 2264664 2276102 131072 256 1681077 1183528 3038436 3058365 3117805 2045155 3062573 6717216 3052438 1658970 1990701 2154124 2178434 131072 512 1623202 1952682 2776715 2841426 2960008 1995463 2958399 6791652 2946175 1619735 1950271 2023558 2044645 131072 1024 1509788 1815425 2671118 2679320 2730039 1840637 2695399 6713360 2091592 1483838 1792879 1673392 1704400 131072 2048 1293005 1427567 2099604 2103042 2134445 1545274 1916675 4336543 2107702 1278240 1507407 1133370 1141036 131072 4096 1161459 1328375 1529054 1562983 1591549 1342043 1506957 1703624 1596666 1152211 1330938 768354 781609 131072 8192 1088076 1248103 1364849 1366498 1372509 1206579 1337196 1287050 1381157 1086238 1244132 679073 696251 131072 16384 1087366 1170869 1341231 1341424 1346082 1244524 1344215 1266383 1348158 1083553 1242425 656150 685018 262144 4 78348 82750 984433 984719 662446 4241 904116 566042 689919 59843 65426 854493 849495 262144 8 44825 60678 1383797 1385143 986093 6057 1314960 1057283 18183 64572 41608 328454 323455 262144 16 69581 39378 353862 345430 86179 10961 1824828 1819974 32383 71422 40011 256305 130413 262144 32 86580 39847 229804 178580 73543 21165 125643 3057751 23598 89668 36895 140787 151135 262144 64 82459 39821 137519 196416 604973 48252 269840 4688173 398886 93792 43000 1968134 2080805 262144 128 93319 94096 2914784 2927964 2947121 2023013 2801283 6448339 2824556 95276 94054 2194792 2204283 262144 256 93024 94814 2941240 2947161 2983961 1953878 3020579 6723533 2987772 94706 98237 2228507 2242309 262144 512 91277 95183 2867973 2874511 2895342 1972593 2947785 6776199 2905288 96609 96759 2102936 2092785 262144 1024 93220 98179 2566213 2570995 2563031 1793344 2630833 6774613 2612691 94218 97061 1630360 1649826 262144 2048 89142 92368 2033034 2035168 1996357 1517818 2080439 4445977 2070304 92457 96821 1115501 1122734 262144 4096 91395 95861 1572258 1572568 1548187 1332683 1598634 1696296 1586599 95099 96081 761970 778332 262144 8192 91109 91966 1339855 1339938 1315509 1254440 1356151 1282504 1357239 93755 94517 683781 690001 262144 16384 88165 92226 1308019 1319966 1312347 1243714 1335521 1260659 1348019 92305 93163 666334 681539 524288 4 77062 34259 138554 66888 2398 1076 76285 565360 1379 64676 31640 86615 111837 524288 8 72310 32074 95857 70668 3878 2024 46995 2. The last output from "systat -vmstat" before the crash 3 users Load 0.00 0.01 0.00 Nov 29 10:19 Mem:KB REAL VIRTUAL VN PAGER SWAP PAGER Tot Share Tot Share Free in out in out Act 37284 6676 94000 7940 3260300 count All 480780 8300 4347280 14540 pages Proc: Interrupts r p d s w Csw Trp Sys Int Sof Flt cow 4489 total 30 3325 2 6311 487 142 13 zfod atkbd0 1 ozfod fdc0 irq6 2.0%Sys 0.5%Intr 0.1%User 0.0%Nice 97.4%Idle %ozfod ata0 irq14 | | | | | | | | | | | daefr 484 atapci1 20 = prcfr 2001 cpu0: time dtbuf 25 totfr 3 bge0 256 Namei Name-cache Dir-cache 100000 desvn react 2001 cpu1: time Calls hits % hits % 57187 numvn pdwak 24998 frevn pdpgs intrn Disks ad4 629456 wire KB/t 126 30648 act tps 232 104672 inact MB/s 28.50 16 cache %busy 100 3260284 free 219632 buf 3. Last output from top before the crash last pid: 6793; load averages: 0.00, 0.01, 0.00 up 1+20:16:06 10:19:40 31 processes: 2 running, 29 sleeping CPU states: 0.0% user, 0.0% nice, 9.4% system, 0.4% interrupt, 90.2% idle Mem: 30M Active, 102M Inact, 615M Wired, 16K Cache, 214M Buf, 3183M Free Swap: 4096M Total, 4096M Free PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND 6710 root 1 98 0 24984K 18416K CPU1 0 2:05 1.56% iozone 6709 root 1 5 0 8340K 2128K ttyin 0 0:04 0.00% systat 746 root 1 96 0 10568K 3992K select 1 0:01 0.00% sendmail 756 root 1 8 0 5736K 1392K nanslp 0 0:00 0.00% cron 611 root 1 96 0 4684K 1348K select 1 0:00 0.00% syslogd 750 smmsp 1 20 0 10568K 3804K pause 0 0:00 0.00% sendmail 369 _dhcp 1 96 0 3724K 1408K select 1 0:00 0.00% dhclient 6678 root 1 4 0 32928K 4528K sbwait 0 0:00 0.00% sshd 6694 root 1 4 0 32928K 4552K sbwait 0 0:00 0.00% sshd 6700 root 1 4 0 32928K 4552K sbwait 0 0:00 0.00% sshd 6705 root 1 8 0 8660K 2696K wait 1 0:00 0.00% bash 6696 root 1 8 0 8660K 2692K wait 1 0:00 0.00% bash 739 root 1 96 0 20952K 3520K select 1 0:00 0.00% sshd 798 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 802 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 799 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 805 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 804 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 801 root 1 5 0 4668K 1196K ttyin 0 0:00 0.00% getty 4. The applied kernel settings kern.maxvnodes="400000" vm.kmem_size_max="512M" vm.kmem_size="512M" 5. Output from zpool [root_at_zfs /root]# zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 ad4s1g ONLINE 0 0 0 errors: No known data errors Can anyone confirm the issue? Any solutions around? If further data or information is required, i'll try to deliver it. Michael -- Die Erde ist die Irrenanstalt des Universums. Public Key: http://sks.keyserver.penguin.de:11371/pks/lookup?op=get&search=0x5D0A2BC3CEB 3F472Received on Fri Nov 30 2007 - 16:00:13 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:23 UTC