Re: Storing a lot of little files

Eugene wrote:
>   I need to store a lot (hundreds of millions) of very little files (from 8 bytes
>   to 50K) in my filesystem.  What's the best way to optimize it? Which newfs options can you
>   recommend me?

Eric already provided an answer to this.

The type of problem you are considering is similiar to the way a Usenet news 
server using tradspool works, or the way a large Squid proxy server would want 
to layout it's cache.  The techniques and advice used by INN and Squid for 
configuring filesystems and laying out datafiles using multi-layer hierarchy 
with perhaps a few hundred top-level directories, each containing a few 
hundred second-level directories, etc are topics worth your consideration.

>   Can UFS2 optiomize storage of little files? For example, put some
>   files in one cluster - so don't let the 8 byte file to occupy a
>   whole cluster of about 16K?

Good question.  UFS breaks up 8K "clusters" into 1K fragments, so the amount 
of wasted space due to instrinsic fragmentation is significantly less.  That 
being said, 8-byte files are too small to be kept individually with much 
efficiency in any block-sized filesystem available to you, given the common 
512-byte sector size of devices tends to constrain the minimum "cluster" size 
one can set.

There are some useful ideas about how to deal with that from the IronDoc 
project (Dave McCuster? [sp?] from Apple), but your requirements are difficult 
to meet without taking some complex issues into consideration.

>   Is it possible to allocate inodes dynamically (like in ReiserFS?),
>   bacause I'm not sure I can predict accuratly number of little files,
>   and inodes not to finish before disk get full.

No.  Allocate many more inodes that you need if you can't guess accurately, or 
else be prepared to backup, newfs using different parameters, and restore your 
  data when you run out.

>   Does FreeBSD support file systems except of UFS?

FreeBSD supports FAT, ISO-9660, NTFS, ext2, probably others, although the 
degree of support decreases in order of my listing.

>   What about ReiserFS, XFS, JFS?

Nope.  Partially licensing, partially lack of developer resources.

-- 
-Chuck