On 29 Sep, Poul-Henning Kamp wrote: > In message <200509290214.j8T2EogR006609_at_gw.catspoiler.org>, Don Lewis writes: > >>Also, g_io_check() can only return ENOMEM if that is the provider error >>status, but the provider seems to be happy: > > ENOMEM can also be returned by the code in the GEOM class. Ok, that was obscure enough. It looks like the uma_zalloc(biozone, M_NOWAIT | M_ZERO) call in g_clone_bio() is the likely culprit. Lots of allocation failures for this zone: (kgdb) print *biozone $2 = {uz_name = 0xc086938c "g_bio", uz_lock = 0xc10295a8, uz_keg = 0xc10295a0, uz_link = {le_next = 0x0, le_prev = 0xc10295d8}, uz_full_bucket = { lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0}, uz_ctor = 0, uz_dtor = 0, uz_init = 0, uz_fini = 0, uz_allocs = 22457718, uz_frees = 22456210, uz_fails = 6603729, uz_fills = 1, uz_count = 128, uz_cpu = {{uc_freebucket = 0x0, uc_allocbucket = 0x0, uc_allocs = 0, uc_frees = 0}}} Any easy way to figure out why the memory allocation is repeatedly failing? Here's the output of vmstat -z: ITEM SIZE LIMIT USED FREE REQUESTS UMA Kegs: 140, 0, 62, 10, 62 UMA Zones: 120, 0, 62, 28, 62 UMA Slabs: 64, 0, 2169, 132, 51890 UMA RCntSlabs: 104, 0, 104, 266, 37549 UMA Hash: 128, 0, 1, 29, 3 16 Bucket: 76, 0, 12, 38, 45 32 Bucket: 140, 0, 30, 26, 69 64 Bucket: 268, 0, 14, 14, 72 128 Bucket: 524, 0, 2409, 244, 53217 VM OBJECT: 132, 0, 5249, 17255, 2323529 MAP: 192, 0, 7, 33, 7 KMAP ENTRY: 68, 56616, 867, 1317, 797654 MAP ENTRY: 68, 0, 1881, 3271, 615413 PV ENTRY: 24, 1494950, 186575, 68335, 77309488 DP fakepg: 72, 0, 0, 0, 0 mt_zone: 64, 0, 194, 101, 194 16: 16, 0, 2357, 282, 2618883 32: 32, 0, 36199, 19962, 9905089 64: 64, 0, 5020, 1883, 3448783 128: 128, 0, 29552, 4078, 5969903 256: 256, 0, 2556, 309, 6529179 512: 512, 0, 71, 33, 5456 1024: 1024, 0, 139, 65, 67843 2048: 2048, 0, 3123, 57, 775663 4096: 4096, 0, 216, 97, 22551 Files: 72, 0, 151, 326, 2846128 PROC: 524, 0, 143, 130, 14242 THREAD: 372, 0, 273, 77, 5308 KSEGRP: 88, 0, 273, 47, 707 UPCALL: 44, 0, 0, 156, 866 VMSPACE: 300, 0, 92, 142, 14191 mbuf_packet: 256, 0, 66, 51, 82560936 mbuf: 256, 0, 3, 540, 231234746 mbuf_cluster: 2048, 25600, 117, 91, 12760461 ACL UMA zone: 388, 0, 0, 0, 0 g_bio: 132, 0, 1508, 0, 22457718 ata_request: 200, 0, 0, 0, 0 ata_composite: 192, 0, 0, 0, 0 VNODE: 272, 0, 29373, 6523, 4183368 VNODEPOLL: 76, 0, 0, 0, 0 S VFS Cache: 68, 0, 11060, 2828, 844011 L VFS Cache: 291, 0, 0, 0, 0 NAMEI: 1024, 0, 24, 52, 7276418 NFSMOUNT: 480, 0, 2, 14, 2 NFSNODE: 424, 0, 25, 902, 7316 DIRHASH: 1024, 0, 491, 97, 56851 PIPE: 408, 0, 4, 59, 4381 KNOTE: 68, 0, 0, 112, 80 socket: 356, 25608, 66, 55, 807098 unpcb: 140, 25620, 17, 39, 217 udpcb: 180, 25608, 25, 41, 805728 inpcb: 180, 25608, 24, 64, 1152 tcpcb: 460, 25600, 24, 32, 1152 tcptw: 48, 5148, 0, 156, 237 syncache: 100, 15366, 0, 78, 348 hostcache: 76, 15400, 1, 99, 2 tcpreass: 20, 1690, 0, 0, 0 sackhole: 20, 0, 0, 0, 0 ripcb: 180, 25608, 0, 0, 0 rtentry: 132, 0, 14, 44, 14 SWAPMETA: 276, 121576, 2620, 5542, 354141 FFS inode: 132, 0, 29314, 4123, 4175811 FFS1 dinode: 128, 0, 183, 1497, 13155 FFS2 dinode: 256, 0, 29124, 3441, 4162649 It looks like the problem might be caused by having too much I/O in progress. (kgdb) print runningbufspace $3 = 1585152 (kgdb) print lorunningspace $4 = 524288 print hirunningspace $5 = 1048576 so runningbufspace >> hirunningspace The bufdaemon and the syncer both ignore the runningbufspace limit. Maybe they should obey the limit, but each be guaranteed a minimum quota.Received on Thu Sep 29 2005 - 05:31:10 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:44 UTC