Re: RFC: use EM_LEGACY_IRQ in if_lem.c ?

On Thu, Jul 26, 2012 at 10:52:08PM -0700, Adrian Chadd wrote:
> On 25 July 2012 08:14, Luigi Rizzo <rizzo_at_iet.unipi.it> wrote:
> 
> >> I suggest doing some digging to understand why. I bet we all know the
> >> answer, but it would be nice to have it documented and investigated. I
> >> bet em(4) isn't the only device that would benefit from this?
> >
> > I am not so sure i know the answer on bare iron (and my take is that the
> > difference is more or less irrelevant there), but in the virtualized case
> > the improvement is almost surely because the code used in FAST_INTR
> > has a couple of MMIO accesses to disable/enable interrupts on the
> > card while the taskqueue runs.  These are expensive in a VM
> > (such accesses cost ~10K cycles each, even with hw support)
> 
> Hm, really? Doing these register accesses to a virtualised em NIC in a
> VM is that expensive, or is there something else going on I don't
> understand?

access to emulated registers is expensive because you need to
exit from the hardware VM before you can run the emulation code,
and the exit is incredibly expensive as it needs to save a ton
of state. Things are marginally better in qemu, but there
everything is more expensive so you still see a measurable
performance gain if you save the accesses.

There is a recent paper/talk at usenix that discusses briefly
the problem

https://www.usenix.org/conference/usenixfederatedconferencesweek/software-techniques-avoiding-hardware-virtualization-exits

The ~10k cycles per access were measured in kvm on recent i5 and i7 processors.

For qemu, the change moves the packet rate from 7.5 to 8.3 Kpps,
if you do the math the difference is about 13us per packet
so we are in a similar ballpark.
(the numbers are low because the default emulation does not
support interrupt mitigation, i have patches for that, see the
qemu-devel mailing list

http://lists.nongnu.org/archive/html/qemu-devel/2012-07/msg03195.html

which bring the rate up to some 50kpps.


cheers
luigi