Re: ixgbe and fast interrupts

On Mon, Nov 21, 2011 at 11:29:29AM -0500, John Baldwin wrote:
> On Friday, November 18, 2011 5:04:58 pm Luigi Rizzo wrote:
> > On Fri, Nov 18, 2011 at 11:16:00AM -0800, Doug Barton wrote:
> > > On 11/18/2011 09:54, Luigi Rizzo wrote:
> > > > One more thing (i am mentioning it here for archival purposes,
> > > > as i keep forgetting to test it). Is entropy harvesting expensive ?
> > > 
> > > No. It was designed to be inexpensive on purpose. :)
> > 
> > hmmm....
> > unfortunately I don't have a chance to test it until monday
> > (probably one could see if the ping times change by modifying
> > the value of kern.random.sys.harvest.* ).
> > 
> > But in the code i see the following:
> > 
> > - the harvest routine is this:
> > 
> >     void
> >     random_harvest(void *entropy, u_int count, u_int bits, u_int frac,
> > 	enum esource origin)
> >     {
> >         if (reap_func)
> >                 (*reap_func)(get_cyclecount(), entropy, count, bits, frac,
> >                     origin);
> >     }
> > 
> > - the reap_func seems to be bound to
> > 
> >     dev/random/randomdev_soft.c::random_harvest_internal()
> > 
> >   which internally uses a spinlock and then moves entries between
> >   two lists.
> > 
> > I am concerned that the get_cyclecount() might end up querying an
> > expensive device (is it using kern.timecounter.hardware ?)
> 
> On modern x86 it just does rdtsc().
> 
> > So between the indirect function call, spinlock, list manipulation
> > and the cyclecounter i wouldn't be surprised it the whole thing
> > takes a microsecond or so.
> 
> I suspect it is not quite that expensive.
> 
> > Anyways, on monday i'll know better. in the meantime, if someone
> > wants to give it a try... in our tests between two machines and
> > ixgbe (10G) interfaces, an unmodified 9.0 kernel has a median ping
> > time of 30us with "slow" pings (say -i 0.01 or larger) and 17us with
> > a ping -f .
> 
> Did you time it with harvest.interrupt disabled?

yes, thanks for reminding me to post the results.

Using unmodified ping (which has 1us resolution on the reports),
there is no measurable difference irrespective
of the setting of kern.random.sys.harvest.ethernet,
kern.random.sys.harvest.interrupt and kern.timecounter.hardware.
Have tried to set hw mitigation to 0 on the NIC (ixgbe on both
sides) but there is no visible effect either.

However I don't trust my measurements because i cannot explain them.
Response times have a min of 20us (about 50 out of 5000 samples)
and a median of 27us, and i really don't understand if the low
readings are real or the result of some races.
Ping does a gettimeofday() for the initial timestamp, and relies
on in-kernel timestamp for the response.

cheers
luigi