heavy threads users, test?

Hi, If you have a system that is using threads heavily
(either libthr or libpthreads) or have a heavy (possibly varying)
load (the heavier the better) (and the more varying and different
(threaded or not) apps you run on it the better), then I would like to 
ask you to try get the following information for me..


The basic command is:

julian_at_jules:sysctl vm.zone | grep -E "THREAD|PROC|REQUESTS"
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
THREAD:          328,        0,    162,     30,  1061173
PROC:            440,        0,    130,     32,    92919

What I need to see is the variation in the total number
of threads and processes available and in use as a function of time..

basically,
"In a real world scenario, how much does the total of "USED and FREE"
change over time?"

This is because, for both processes and threads, there are significant
resources (e.g. kernel stack) allocated as these entities are created
and freed when they are destroyed and I'm trying to work out 
how much "thrashing" we are getting (or avoiding) by caching them
in a system that is under varying load. 
Should we add more hysteresis or less?

Theoretically when a system reaches "steady state (if such  thing
exists) then threads should hopefully move between the thread cache 
and active use but hopefully we should not be creating and destroying
many (if any).

The following hacked up script reports whenever threads have been
recently created or destroyed:

#!/bin/sh
PREV="0"
while :
do
        LINE=`sysctl vm.zone | grep  "THREAD:"`
        IFS=' ,'
        set $LINE
        NEW=`expr $4 + $5`
        if [ $PREV != $NEW ]
        then
                echo `date`": total threads changed from $PREV to $NEW"
                sysctl vm.zone | grep  -E "THREAD:|PROC:|REQUESTS"
        fi
        PREV=$NEW
        sleep 60
done

If you have an "interesting' load (under -current) I would like 
to see teh output of the above script
run over the course of a day or two (or longer).
(sample output below)



julian_at_jules:sh /tmp/xx.sh
Mon Apr 12 00:32:22 PDT 2004: total threads changed from 0 to 192
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
THREAD:          328,        0,    162,     30,  1061173
PROC:            440,        0,    130,     32,    92919
Mon Apr 12 00:34:23 PDT 2004: total threads changed from 192 to 264
ITEM            SIZE     LIMIT     USED    FREE  REQUESTS
THREAD:          328,        0,    243,     21,  1061896
PROC:            440,        0,    191,     52,    93640


Thanks!

Julian