Re: Possible Threading problem with -CURRENT / MySQL?

On Wed, 16 Jun 2004, Julian Elischer wrote:

> Robert, can you try see if HZ=5000 (up from 100) changes the
> performance? 

I haven't explored the HZ=BIGGER case very thoroughly, but ran a couple of
test cases with HZ at 1000 instead of 100, and found that generally it
resulted in a slight (1.5%) performance drop in the smattering of cases I
looked at.  The cases I looked at were NETPERF-SMP-MPSAFE-ADMTX-HZ=1000,
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT-HZ=1000. 

In general, I found that the biggest variables improving performance
relative to out-of-box SMP configuration were: 

- Removing Giant from UNIX domain sockets
- Running with SCHED_4BSD instead of SCHED_ULE
- Disabling HTT
- Using ADAPTIVE_MUTEXES

By doing these things, I went from an out-of-box queries/sec for the
simple select "smack" with 11 clients from 2667 q/s to 6955 q/s, or a bit
over doubling the transaction rate.  That as compared to 4939 q/s, which
was the best UP result I got (Giant-free UNIX domain sockets, SCHED_4BSD):

20040615-UP				4898.71 q/s (22.39 stddev)
20040615-SMP				2666.93 q/s (32.01 stddev)
NETPERF-UP-MPSAFE-4BSD			4939.89 q/s (74.06 stddev)
NETPERF-SMP-MPSAFE-ADMTX-4BSD-!HTT	6955.18 q/s (156.91 stddev)

Some of my 4BSD performance numbers were also slightly pessimized with
what appears to be a property of the 4BSD scheduling period being longer
than ULE.  With ULE, the results settled on one loop of the benchmark. 
With 4BSD, it sometimes took 2-3 loops of the benchmark to settle.  As a
result, 4BSD stddev's are also generally higher.  I haven't tried
regenerating the results dropping those outliers.

Using out-of-box -CURRENT without the netperf patches necessary to run
UNIX domain sockets safely, I got best results using:

20040615-UP-4BSD			4886.84 q/s (46.03 stddev)
20040615-SMP-ADMTX-4BSD-!HTT		5838.76 q/s (45.11 stddev)

In my environment, I couldn't easily test ULE without HTT, since the HTT
disabling sysctl/tunable with ULE results in a hang. 

(Obviously, in all of the above, WITNESS and INVARIANTS are disabled; I
didn't disable the userspace malloc debugging flags). 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research


> 
> 
> On Wed, 16 Jun 2004, Robert Watson wrote:
> 
> > 
> > On Tue, 15 Jun 2004, Julian Elischer wrote:
> > 
> > > On Mon, 14 Jun 2004, mike wrote:
> > > 
> > > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> > > > as well. it sounds like scheduler/threading is to blame but we were not
> > > > able to give sufficient or proper motivation to the folks who could
> > > > examine this deeper - we even offered $500 cash to whomever stepped up to
> > > > help resolve this.
> > > > 
> > > > linux runs almost 2x as fast on the same hardware with no configuring -
> > > > and we get nearly the same results running in single CPU mode vs. dual CPU
> > > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > > > combination.
> > >  
> > > You complained about this some time ago and you have still not responded 
> > > with the information I suggest..
> > 
> > I sent this to Jeremy privately, since it was just some preliminary
> > measurements, but figured I'd send it publically since the results were
> > interesting (if tentative, I need to do a lot more work to make them
> > useful.  There are a number of variables I need to look at including:
> > 
> > - Disabling HTT.  A chat with Scott Long this evening suggests that HTT
> >   may be substantially hurting the test cases given increased IPIs, etc.
> >   Unfortunately, it looks like I can't easily twiddle HTT without being
> >   local to the machine, and I'm at home right now (it being 1:30am and
> >   all).   Removing HTT may help substantially with the dip in performance
> >   in the SMP configuration.
> > 
> > - I'd like to compare against RELENG_4 and a recent Linux kernel.
> >   Unfortunately, the box is configured for neither right now.
> > 
> > - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd
> >   like to try SCHED_4BSD.
> > 
> > - This was without adaptive mutexes, which seem to be helpful for others,
> >   so I should give them a try.
> > 
> > I don't have any amd64 hardware, so I don't know what if any role it will
> > play in the results.  The performance drop observed in the report appears
> > to be on amd64 (I may have misread).
> > 
> > Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
> > robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research
> > 
> > 
> > ----
> > Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT)
> > From: Robert Watson <rwatson_at_FreeBSD.org>
> > To: JG <amd64list_at_jpgsworld.com>
> > Subject: Re: Possible Threading problem with -CURRENT / MySQL?
> > 
> > 
> > On Tue, 15 Jun 2004, JG wrote:
> > 
> > > Fwiw, it has to be something that was committed between May 18th and
> > > yesterday.  ~May 18th was the last time I built -CURRENT during my last
> > > round of testing and I did not have any of these problems.  Then someone
> > > emailed me recently and said there were some commits that might effect
> > > the outcome of the mysql benchmarks. 
> > 
> > Ok, so these results are on a dual-processor XEON + hyperthreads, so four
> > logical processors.  I used two dates off CVS, 20040515 and 20040615.  I
> > also benchmarked my netperf branch.  I don't have RELENG_4 on the box, but
> > might be able to load RELENG_4 on it later this week.  In each case, I
> > took ten samples, dropped the first value as getting into the cache, and
> > took the mean of the rest.  For this test, I used the select test; I'll
> > try the other smack query set tomorrow.  In each case, I ran with "10
> > 1000" as the arguments to the test.  I used the default threading
> > configuration in -CURRENT, which is libpthread (libkse). 
> > 
> >                                 Mean       Stdev
> > 20040515-UP                     4752.27    14.63
> > 20040515-SMP                    2550.35    19.23
> > 
> > 20040615-UP                     4898.71    22.39
> > 20040615-SMP                    2666.93    32.01
> > 
> > Netperf-UP-giant                4902.41    14.3
> > Netperf-SMP-giant               2566.18    16.83
> > Netperf-UP-mpsafe               4799.35    22.04
> > Netperf-SMP-mpsafe              3022.51    18.06
> > 
> > Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages
> > the SMP numbers a fair amount due to additional IPIs without benefit. 
> > However, the numbers basically suggest that on my hardware, the UP
> > configuration is marginally faster than it was last month, and that if you
> > throw in the netperf branch, the SMP case is a moderate amount faster. 
> > This suggests that either I'm just lucky, or that the performance loss
> > might be specific to the amd64 version of FreeBSD.  I'm going to run some
> > more numbers tomorrow and try to post something more rigorous to the
> > -threads list. 
> > 
> > I don't have RELENG_4 on the box or Linux on the box, but I may get a
> > chance to later this week. 
> > 
> > Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
> > robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research
> > 
> > 
> > 
> 
>