Re: Wow... (<-- blown away at performance)

From: Alexander Best <arundel_at_freebsd.org>
Date: Wed, 23 Feb 2011 01:31:10 +0000
On Tue Feb 22 11, Alexander Best wrote:
> On Tue Feb 22 11, Alexander Best wrote:
> > On Tue Feb 22 11, Brandon Gooch wrote:
> > > On Tue, Feb 22, 2011 at 2:31 PM, Alexander Best <arundel_at_freebsd.org> wrote:
> > > > On Tue Feb 22 11, Garrett Cooper wrote:
> > > >> On Tue, Feb 22, 2011 at 2:10 AM, Eir Nym <eirnym_at_gmail.com> wrote:
> > > >> > On 22 February 2011 11:15, Garrett Cooper <yanegomi_at_gmail.com> wrote:
> > > >> >>    I don't know what to say, but r218938 screams with flash videos
> > > >> >> (native Linux speed). Not sure if it's the new binutils or if it's the
> > > >> >> new linuxulator patches, but I can run multiple instances of youtube
> > > >> >> in parallel (5 total with other miscellaneous flash animation) without
> > > >> >> it totally lagging out Firefox/X11, and it appears to close the
> > > >> >> instances of firefox properly now. Hopefully this version fares better
> > > >> >> than r218113 did (I think I hit a kernel bug after 2 weeks uptime,
> > > >> >> where my system just hardlocked for no apparent reason).
> > > >> >>    Anyhow, hope others have similar results.
> > > >> >> Cheers!
> > > >> >> -Garrett
> > > >> >>
> > > >> >> $ uname -a
> > > >> >> FreeBSD bayonetta.local 9.0-CURRENT FreeBSD 9.0-CURRENT #0 r218938M:
> > > >> >> Mon Feb 21 23:10:51 PST 2011
> > > >> >> gcooper_at_bayonetta.local:/usr/obj/usr/src/sys/BAYONETTA  amd64
> > > >> >
> > > >> > Which FlashPlayer version do you test? Adobe has made significant
> > > >> > performance changes in 10.2 (from 10.1). You can search for StageVideo
> > > >> > performance to learn more about. Youtube already use them since 10.2
> > > >> > beta
> > > >>
> > > >>     linux-f10-flashplugin-10.1r102.65 . The performance increases are
> > > >> claimed to be "up to 85%" on the Stage Video site, but I'm seeing a
> > > >> more than 200% increase (now it actually scales between multiple
> > > >> instances, instead of croaks with one instance, tiling up and down the
> > > >> screen when moving the window slider for instance or switching tabs).
> > > >> Besides, it seems like it needs external support from the video
> > > >> driver, and I'm not sure that that bridge exists in the linuxulator.
> > > >>     Also, it looks like npviewer.bin still hangs to resources on until
> > > >> Firefox closes (or I kill it :)..), so something still needs to be
> > > >> resolved there, but that isn't a regression (it's acted that way for
> > > >> ages), and shouldn't be too hard to do.
> > > >
> > > > i think the problem with npviewer.bin having to be killed by hand is a futex
> > > > issue in combination with a high number of threads. this is the output of a
> > > > test from darren hart's futex test suite under freebsd 9.0:
> > > >
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=1
> > > > Result: 18622 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=2
> > > > Result: 15469 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=3
> > > > Result: 12713 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=4
> > > > Result: 12247 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=5
> > > > Result: 11814 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=6
> > > > Result: 13468 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=8
> > > > Result: 12061 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=10
> > > > Result: 12854 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=12
> > > > Result: 12999 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=16
> > > > Result: 12402 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=24
> > > > Result: 9815 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=32
> > > > Result: 8518 Kiter/s
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=64
> > > >        ERROR: Resource temporarily unavailable: pthread_create
> > > > Result: ERROR
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=128
> > > >        ERROR: Resource temporarily unavailable: pthread_create
> > > > Result: ERROR
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=256
> > > >        ERROR: Resource temporarily unavailable: pthread_create
> > > > Result: ERROR
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=512
> > > >        ERROR: Resource temporarily unavailable: pthread_create
> > > > Result: ERROR
> > > > futex_wait: Measure FUTEX_WAIT operations per second
> > > >        Arguments: iterations=100000000 threads=1024
> > > >        ERROR: Resource temporarily unavailable: pthread_create
> > > > Result: ERROR
> > > >
> > > > cheers.
> > > > alex
> > > 
> > > Have you found any method or workaround to mitigate this issue?
> 
> i did a ktrace -i and linux_kdump run and the results are quite interesting.
> with only 1 thread linux_sys_futex() *never* failed with EAGAIN. starting
> with 2 threads however i'm seeing an increasing number of linux_sys_futex()
> failures (all with EAGAIN).

it seems however that the cause for the tests with 64-1024 threads to fail is
linux_mmap():

 10070 futex_wait CALL  linux_clone(0x3d0f00,0xfc20d4b4,0xfc20dbd8,0xffffd9bc,0xfc20dbd8)
 10122 futex_wait RET   linux_fork 0
 10122 futex_wait CALL  linux_set_robust_list(0xf820dbe0,0xc)
 10122 futex_wait RET   linux_set_robust_list 0
 10122 futex_wait CALL  linux_sys_futex(0xffffdb48,0x80,0,0,0,0)
 10070 futex_wait RET   linux_clone 10123/0x278b
 10123 futex_wait RET   linux_fork 0
 10070 futex_wait CALL  linux_mmap2(0,0x4000000,0x3,0x20022,0xffffffff,0)
 10123 futex_wait CALL  linux_set_robust_list(0xfc20dbe0,0xc)
 10123 futex_wait RET   linux_set_robust_list 0
 10123 futex_wait CALL  linux_sys_futex(0xffffdb48,0x80,0,0,0,0)
 10070 futex_wait RET   linux_mmap2 -1 errno 12 Cannot allocate memory
 10070 futex_wait CALL  write(0x2,0xffffb2c8,0x44)
 10070 futex_wait GIO   fd 2 wrote 68 bytes
       "        \^[[1;33mERROR\^[[0m: Resource temporarily unavailable: pthread_create
       "
 10070 futex_wait RET   write 68/0x44
 10070 futex_wait CALL  linux_sys_futex(0xffffdb48,0x81,0x7fffffff,0,0,0)

cheers.
alex

> 
> the overhead is quite massive. linux_kdump > /tmp/futex-failure produced 941
> megs of output. doing
> grep -v "futex_wait" futex-failure > futex-grep
> produces an output of 192K!! so it's quite obvious that the futex_wait stuff
> is responsible for causing massive activity, due to the constant failures and
> retries.
> 
> here's a bit of output:
> 
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32125 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32125 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
>  32125 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex 0
>  32126 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x80,0x2,0,0,0)
>  32125 futex_wait CALL  linux_sys_futex(0xffffdb5c,0x81,0x1,0,0,0)
>  32126 futex_wait RET   linux_sys_futex -1 errno 35 Resource temporarily unavailable
> 
> cheers.
> alex
> 
> > 
> > no. i've increased the kern.threads.max_threads_per_proc value, but that had
> > no effect. so it seems to be a bug in the linuxulator's futex implementation.
> > 
> > cheers.
> > alex
> > 
> > > 
> > > -Brandon
> > 
> > -- 
> > a13x
> 
> -- 
> a13x

-- 
a13x
Received on Wed Feb 23 2011 - 00:31:10 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:11 UTC