Re: you are in an fs with millions of small files

From: Poul-Henning Kamp <phk_at_phk.freebsd.dk>
Date: Thu, 09 Jun 2005 21:15:36 +0200
In message <200506091911.j59JBGQn096385_at_gate.bitblocks.com>, Bakul Shah writes:

>Start with a small array.  When it gets full double it (or
>grow by half if you want to waste less memory).

If I might make an observation...

You do not waste malloc'ed memory until you access it.  You waste some
page table entries etc in the kernel, but the actual pages do not
become part of the memory-pressure mob-rule until you touch it the
first time.

This is a very important point which people still (after 20 years of
virtual memory systems and 10 years of phkmalloc) still do not seem
to have fully understood.

I would advice a simple doubling and a terminal realloc to cut down
to actual size.


-- 
Poul-Henning Kamp       | UNIX since Zilog Zeus 3.20
phk_at_FreeBSD.ORG         | TCP/IP since RFC 956
FreeBSD committer       | BSD since 4.3-tahoe    
Never attribute to malice what can adequately be explained by incompetence.
Received on Thu Jun 09 2005 - 17:15:57 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:36 UTC