Re: comments in the page coloring options in /sys/conf/NOTES

On Mon, Jun 07, 2004 at 01:04:30PM +0200, Alexander Leidinger wrote:
> On Sun, 6 Jun 2004 15:12:49 -0500
> Alan Cox <alc_at_cs.rice.edu> wrote:
> 
> > My recent commit fixed a "syntax" error in the comments, specifically,
> > a reference to a missing macro.  The comments are, however, still
> > "semantically" broken: 
> > 
> > 1. Cache size alone does not correctly determine the number of colors,
> > except for direct map caches.  The correct formula is
> > 
> >  (cache size in bytes / page size in bytes) / degree of cache associativity
> > 
> > Thus, the comments would lead one to configure his/her system with too
> > many colors.  (Relative to configuring a system with too few colors,
> > this is not so bad.)
> 
> vm_page.h contains:
> ---snip---
> #if PQ_CACHESIZE >= 1024
> #define PQ_PRIME1 31    /* Prime number somewhat less than PQ_HASH_SIZE */
> #define PQ_PRIME2 23    /* Prime number somewhat less than PQ_HASH_SIZE */
> #define PQ_L2_SIZE 256  /* A number of colors opt for 1M cache */
> ---snip---
> 
> The three defines seem to be the tunables for the page coloring, but
> neither of them seem to be near cache_size/page_size. So even for the
> direct mapped case this doesn't seem to fit your explanation.

Actually, it does.  Divide 1MB (the cache size) by 4KB (the page size)
and you get 256, the appropriate number of colors for a 1MB direct-
mapped cache.  Despite its name, PQ_L2_SIZE, is not the L2 cache size.
As the comment suggests, it is the number of colors.

> ... Is it as
> easy as using appropriate values for those defines at boot time or is
> there more work involved for auto-tuning version?

At boot time.
 
> Is there a way to determine the size of the L2 cache and the
> associativity at run time in the kernel or are these properties which we
> have to obtain from a table (which we have to write and maintain for
> automatic tuning)?

On modern amd64 and i386 CPUs, yes.  I would be surprised if ia64
didn't provide the information.  Run a "boot -v" and look carefully
at the information.  It describes the cache structure.

> > 2. The references to L1 should be removed.  They are historical
> > leftovers.
> 
> cut&paste:
> ---snip---
> Index: sys/conf/NOTES
> ===================================================================
> RCS file: /big/FreeBSD-CVS/src/sys/conf/NOTES,v
> retrieving revision 1.1227
> diff -u -r1.1227 NOTES
> --- sys/conf/NOTES      1 Jun 2004 06:22:56 -0000       1.1227
> +++ sys/conf/NOTES      7 Jun 2004 10:07:16 -0000
> _at__at_ -103,13 +103,13 _at__at_
>  
>  # Options for the VM subsystem
>  # L2 cache size (in KB) can be specified in PQ_CACHESIZE
> -options        PQ_CACHESIZE=512        # color for 512k/16k cache
> +options        PQ_CACHESIZE=512        # color for 512k cache
>  # Deprecated options supported for backwards compatibility
>  #options       PQ_NOOPT                # No coloring
> -#options       PQ_LARGECACHE           # color for 512k/16k cache
> -#options       PQ_HUGECACHE            # color for 1024k/16k cache
> -#options       PQ_MEDIUMCACHE          # color for 256k/16k cache
> -#options       PQ_NORMALCACHE          # color for 64k/16k cache
> +#options       PQ_LARGECACHE           # color for 512k cache
> +#options       PQ_HUGECACHE            # color for 1024k cache
> +#options       PQ_MEDIUMCACHE          # color for 256k cache
> +#options       PQ_NORMALCACHE          # color for 64k cache
>  
>  # This allows you to actually store this configuration file into
>  # the kernel binary itself, where it may be later read by saying:
> Index: sys/vm/vm_pageq.c
> ===================================================================
> RCS file: /big/FreeBSD-CVS/src/sys/vm/vm_pageq.c,v
> retrieving revision 1.13
> diff -u -r1.13 vm_pageq.c
> --- sys/vm/vm_pageq.c   30 May 2004 00:42:38 -0000      1.13
> +++ sys/vm/vm_pageq.c   7 Jun 2004 10:38:31 -0000
> _at__at_ -176,8 +176,8 _at__at_
>   *
>   *     The page coloring optimization attempts to locate a page
>   *     that does not overload other nearby pages in the object in
> - *     the cpu's L1 or L2 caches.  We need this optimization because 
> - *     cpu caches tend to be physical caches, while object spaces tend 
> + *     the cpu's L2 cache.  We need this optimization because cpu
> + *     caches tend to be physical caches, while object spaces tend 
>   *     to be virtual.
>   *
>   *     This routine must be called at splvm().
> ---snip---
> 
> I think the comment with the reference to L1 in sys/vm/vm_zeroidle.c is
> ok.

Correct.  (In fact, on newer generation CPUs, we use non-temporal, i.e.,
non-cached, stores to zero pages in the idle loop.  So, there is no
inteference with the cache.)
 
> If this is ok, do I have your approval to commit it?

Yes, please commit it.

Regards,
Alan