[I'm copying freebsd-current_at_FreeBSD.org because ppl there might know more about this...] qemu on FreeBSD hosts used to be able to run a (FreeBSD at least) guest with the same HZ as the host (like, 1000) with (mostly) proper timing once, but no longer. :( It seems there are two problems involved: a) use of apic seems to cause the clock irq rate to be doubled to 2 * HZ (can anyone explain why?), i.e. a FreeBSD 7 guest on a FreeBSD 7 host only gets proper timing after setting hint.apic.0.disabled=1 via the loader. (as can be verified by `vmstat -i' and `time sleep 2' in an installed guest or via the fixit->cdrom/dvd shell on a FreeBSD livefs or dvd1 iso.) b) qemu running on FreeBSD 8 hosts (and most likely head) has the additional problem of running its timers only at HZ/2 when using setitimer(2) (called `-clock unix' in qemu), as seen below. (as also seen below, timer_settime(2) aka `-clock dynticks' in qemu behaves even worse, but that is similarly true on FreeBSD 7 which is why I removed the patch that enabled that from our qemu port(s) a few days ago.) And the only reason FreeBSD 8 guests are usually less affected by these problems is they now reduce their HZ to 100 when they detect being run in a VM. (which makes sense for other reasons as well, don't get me wrong... but of course doesn't help when the host is running with HZ=100 too.) On Tue, Sep 01, 2009 at 10:12:48PM +0200, Juergen Lock wrote: > On Mon, Aug 31, 2009 at 11:27:23PM +0200, Juergen Lock wrote: > > On Mon, Aug 31, 2009 at 09:47:27AM +0200, Jan Kiszka wrote: > > > Juergen Lock wrote: > > > > On Thu, Aug 27, 2009 at 07:56:41PM +0200, Jan Kiszka wrote: > > > >> Juergen Lock wrote: > > > >>> On Tue, Aug 25, 2009 at 12:38:04PM +0200, Jan Kiszka wrote: > > > >>>> Mohammed Gamal wrote: > > > >>>>> On Tue, Aug 25, 2009 at 1:16 PM, Mohammed Gamal<m.gamal005_at_gmail.com> wrote: > > > >>>>>> On Tue, Aug 25, 2009 at 12:33 PM, Jan Kiszka<jan.kiszka_at_web.de> wrote: > > > >>>>>>> Mohammed Gamal wrote: > > > >>>>>>>> qemu-system-x86_64 -hda /dev/null -cdrom <path_to_ubuntu_iso_image> > > > >>>>>>>> > > > >>>>>>> I only have kubuntu-9.04-alternate-amd64.iso at hand ATM, and with that > > > >>>>>>> image I'm unable to reproduce. Will download and check standard ubuntu > > > >>>>>>> later today. > > > >>>>>>> > > > >>>>>>>> I was using qemu-kvm, but I assume that using -no-kvm would be > > > >>>>>>>> equivalent to using plain qemu, no? > > > >>>>>>> Generally yes, but not necessarily (e.g. the BIOSes are different). So > > > >>>>>>> it's better to check such issues also against "clean" qemu, specifically > > > >>>>>>> as we are on qemu-devel here. > > > >>>>>>> > > > >>>>>>> Jan > > > >>>>>>> > > > >>>>>>> > > > >>>>>> Just tested this now on a vanilla qemu, I am still able to reproduce > > > >>>>>> the same issue. > > > >>>>>> > > > >>>>> This bug might be related to the same problem > > > >>>>> https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/379000 > > > >>>> I think at least those issues should be solved with recent qemu and > > > >>>> bioses. Note that this report refers to a fairly old qemu version > > > >>>> (0.10.0-derived). > > > >>> Btw I had reported the same symptom as in that ubuntu ticket for FreeBSD 8 > > > >>> hosts both with 0.10.6 and 0.11.0rc1 before: > > > >>> http://lists.gnu.org/archive/html/qemu-devel/2009-08/msg00396.html > > > >>> As mentioned in that post I was able to work around the issue by passig > > > >>> the linux guest kernels `no_timer_check' after which they seemed to boot up > > > >>> just fine, so I suspect in that case its not actually an apic routing > > > >>> problem but just guest timer irqs arriving late/irregularly which cause > > > >>> the guest kernel timer checks to time out and fail. > > > >> Does this happen with git head and its corresponding bios, too? I cannot > > > >> imagine that the FreeBSD platform is so irregularly generating timer > > > >> events for qemu that Linux gets unhappy during this test loop (I think > > > >> to remember it needs 3 out of 10 ticks or so to be satisfied). > > > > > > > > Alright so I testwise updated the FreeBSD qemu-devel port to git head > > > > (db3c9e08e0d6eaf83f9d7a2c87dc45af3ac8f4dd, update at > > > > http://people.freebsd.org/~nox/qemu/qemu-devel-20090829.patch > > > > > > You will have to help me isolating the reason as I don't have any BSD > > > host. And running recent qemu/-kvm on Linux hosts does not trigger > > > problems around booting ubuntu here. > > > > Well I don't quite know where to start looking, of course I'd be > > happy to test patches... :) > > Ok I now found the #if 0 in vl.c:host_alarm_handler(), enabled it and got > this with a FreeBSD 7.2 iso (which defaults to HZ=1000) and -clock dynticks: > ..and this was on a FreeBSD 8 host: > timer: min=2555 us max=32004 us avg=5150 us avg_freq=194.149 Hz > timer: min=2451 us max=7205 us avg=3012 us avg_freq=331.960 Hz > timer: min=2855 us max=7994 us avg=3019 us avg_freq=331.191 Hz > timer: min=2099 us max=20003 us avg=3057 us avg_freq=327.073 Hz > timer: min=2143 us max=11897 us avg=3081 us avg_freq=324.527 Hz > timer: min=2110 us max=80014 us avg=3165 us avg_freq=315.913 Hz > timer: min=2088 us max=51999 us avg=3206 us avg_freq=311.873 Hz > > and this with the same guest and -clock unix: > > timer: min=1614 us max=7121 us avg=2009 us avg_freq=497.692 Hz > timer: min=876 us max=11134 us avg=2016 us avg_freq=495.967 Hz > timer: min=753 us max=9245 us avg=2014 us avg_freq=496.455 Hz > timer: min=1798 us max=6211 us avg=2008 us avg_freq=497.941 Hz > timer: min=1772 us max=6142 us avg=2008 us avg_freq=497.943 Hz > timer: min=1767 us max=6157 us avg=2008 us avg_freq=497.942 Hz > timer: min=14 us max=6139 us avg=2008 us avg_freq=497.939 Hz > timer: min=736 us max=9969 us avg=2016 us avg_freq=495.962 Hz > timer: min=176 us max=11427 us avg=2076 us avg_freq=481.631 Hz > timer: min=229 us max=9827 us avg=2030 us avg_freq=492.546 Hz > timer: min=430 us max=11082 us avg=2078 us avg_freq=481.166 Hz > timer: min=226 us max=9871 us avg=2026 us avg_freq=493.517 Hz > timer: min=605 us max=10829 us avg=2082 us avg_freq=480.243 Hz > timer: min=57 us max=10976 us avg=2028 us avg_freq=493.031 Hz > timer: min=398 us max=11036 us avg=2080 us avg_freq=480.704 Hz > > `vmstat -i' in the guest (available e.g. from the fixit->cdrom/dvd shell > on a livefs or dvd1 iso) gives similar values as avg_freq above for the > `cpu0: timer' irq rate. (Btw, FreeBSD 8 reduces its HZ to 100 when > running in a vm so the `vmstat -i' values will be a little lower if you > try that as a guest.) > > And `vmstat -i' on the host (also running with HZ=1000) reports a > rate of 1999 for each cpu's timer irq, so there are clearly quite a few > irqs missing there... > > (Btw I'm wondering if the poor usb performance I reported a long time > ago already could be related to this also...) And here is the same guest with -clock unix on a FreeBSD 7 host: (same with and without apic, except of course that only without apic timing in the guest is ok.) timer: min=847 us max=6146 us avg=1005 us avg_freq=994.950 Hz timer: min=866 us max=6130 us avg=1005 us avg_freq=994.960 Hz timer: min=868 us max=6133 us avg=1005 us avg_freq=994.958 Hz timer: min=287 us max=35199 us avg=1089 us avg_freq=918.216 Hz timer: min=201 us max=13551 us avg=1029 us avg_freq=971.757 Hz timer: min=792 us max=6211 us avg=1005 us avg_freq=994.961 Hz timer: min=145 us max=6185 us avg=1007 us avg_freq=992.984 Hz timer: min=852 us max=6152 us avg=1005 us avg_freq=994.961 Hz timer: min=820 us max=6175 us avg=1005 us avg_freq=994.960 Hz timer: min=654 us max=6242 us avg=1005 us avg_freq=994.966 Hz timer: min=675 us max=6196 us avg=1005 us avg_freq=994.952 Hz timer: min=829 us max=6162 us avg=1005 us avg_freq=994.960 Hz timer: min=503 us max=6136 us avg=1005 us avg_freq=994.960 Hz timer: min=859 us max=6138 us avg=1005 us avg_freq=994.960 Hz timer: min=856 us max=6149 us avg=1005 us avg_freq=994.960 Hz timer: min=832 us max=6164 us avg=1005 us avg_freq=994.961 Hz timer: min=829 us max=6174 us avg=1005 us avg_freq=994.958 Hz timer: min=826 us max=6173 us avg=1005 us avg_freq=994.960 Hz timer: min=828 us max=6168 us avg=1005 us avg_freq=994.961 Hz timer: min=841 us max=6160 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6135 us avg=1005 us avg_freq=994.959 Hz timer: min=857 us max=6139 us avg=1005 us avg_freq=994.959 Hz timer: min=845 us max=6147 us avg=1005 us avg_freq=994.960 Hz timer: min=854 us max=6143 us avg=1005 us avg_freq=994.960 Hz timer: min=854 us max=6142 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6142 us avg=1005 us avg_freq=994.960 Hz timer: min=859 us max=6141 us avg=1005 us avg_freq=994.961 Hz timer: min=849 us max=6141 us avg=1005 us avg_freq=994.962 Hz timer: min=857 us max=6143 us avg=1005 us avg_freq=994.962 Hz timer: min=864 us max=6138 us avg=1005 us avg_freq=994.958 Hz timer: min=863 us max=6139 us avg=1005 us avg_freq=994.961 Hz timer: min=869 us max=6136 us avg=1005 us avg_freq=994.960 Hz timer: min=857 us max=6143 us avg=1005 us avg_freq=994.959 Hz timer: min=865 us max=6133 us avg=1005 us avg_freq=994.960 Hz And the same guest/host with -clock dynticks: timer: min=1657 us max=32164 us avg=6732 us avg_freq=148.534 Hz timer: min=1675 us max=7158 us avg=2010 us avg_freq=497.480 Hz timer: min=1773 us max=7200 us avg=2010 us avg_freq=497.480 Hz timer: min=1798 us max=7148 us avg=2010 us avg_freq=497.480 Hz timer: min=1851 us max=7144 us avg=2013 us avg_freq=496.739 Hz timer: min=1821 us max=7136 us avg=2020 us avg_freq=495.017 Hz timer: min=1823 us max=7173 us avg=2010 us avg_freq=497.480 Hz timer: min=1825 us max=39002 us avg=2076 us avg_freq=481.664 Hz timer: min=1851 us max=7145 us avg=2012 us avg_freq=496.985 Hz timer: min=1855 us max=16998 us avg=2106 us avg_freq=474.803 Hz timer: min=1855 us max=7138 us avg=2010 us avg_freq=497.480 Hz timer: min=1858 us max=7136 us avg=2010 us avg_freq=497.480 Hz timer: min=1856 us max=7138 us avg=2010 us avg_freq=497.480 Hz timer: min=1855 us max=7138 us avg=2017 us avg_freq=495.754 Hz timer: min=1818 us max=7143 us avg=2009 us avg_freq=497.728 Hz timer: min=1851 us max=7141 us avg=2010 us avg_freq=497.480 Hz timer: min=1859 us max=7140 us avg=2010 us avg_freq=497.480 Hz timer: min=1851 us max=7138 us avg=2010 us avg_freq=497.480 Hz For people wanting to experiment, you can escape to the loader prompt from the FreeBSD boot menu and then type: set hint.apic.0.disabled=1 and optionally: set kern.hz="1000" and then: boot -v (for verbose dmesg, otherwise `boot' without the -v.) Any enlightnig comments appreciated... :) JuergenReceived on Mon Sep 07 2009 - 19:06:19 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:55 UTC