This patch implements two features for the 'lem' driver that greatly improve the throughput under proper hypervisor. This is joint work with Vincenzo Maffione and Giuseppe Lettieri, I am posting it here for review, will then commit it if there are no objections. The first change is to implement a sysctl to access the 'itr' interrupt moderation register for the devices supported by this driver. It is little more than adding a struct into the device descriptor, and one line to create the dynamic sysctl entry, same as it is done for the other mitigation registers. The second change is more interesting and has huge benefits on througput. Under virtualization, "VM exits" (which happen every time there is an access to a register of the emulated peripheral) are extremely expensive. In the tx path of the 'lem' driver, there is a write to the TDT register on every packet sent. The patch we propose, if enabled through a sysctl (defaults off, so no change from current behaviour) defers writes to the TDT register when there is a pending transmit interrupt. This means that, together with proper emulation of interrupt mitigation on the hypervisor side, the number of VM exits is dramatically reduced. To give you an idea, on a modern system with qemu-kvm and companion patches, UDP throughput is KVM QEMU standard KVM, standard driver 20 Kpps 6.3 Kpps modified KVM, standard driver 37 Kpps 28 Kpps modified KVM, modified driver 200 Kpps 34 Kpps As you can see, on kvm this change gives a 5x speedup to the tx path, which combines nicely with the 2x speedup that comes from supporting interrupt mitigation alone in the hypervisor. Without kvm (or kqemu ?) the benefits are much lower, as the guest becomes too slow. Patch follows. It would be good if people with real hardware using the 'lem' driver could test it to make sure it does no harm on their devices (in any case the sysctl variable dev.em.0.mit_enable must be set to explicitly enable it at runtime). (for those curious to test it under kvm, i am also attaching a patch that you need to apply to qemu in order to exploit the effect of interrupt mitigation; it is a followup of a similar patch i posted in july to the qemu mailing list, and i will post it the update there as well, shortly. Unfortunately we do not have kvm on freebsd..) cheers luigi
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:33 UTC