Re: Possible Threading problem with -CURRENT / MySQL?

Robert, can you try see if HZ=5000 (up from 100) changes
the performance?


On Wed, 16 Jun 2004, Robert Watson wrote:

> 
> On Tue, 15 Jun 2004, Julian Elischer wrote:
> 
> > On Mon, 14 Jun 2004, mike wrote:
> > 
> > > welcome to our hell. we've been experiencing mysql problems on freebsd 5.x
> > > as well. it sounds like scheduler/threading is to blame but we were not
> > > able to give sufficient or proper motivation to the folks who could
> > > examine this deeper - we even offered $500 cash to whomever stepped up to
> > > help resolve this.
> > > 
> > > linux runs almost 2x as fast on the same hardware with no configuring -
> > > and we get nearly the same results running in single CPU mode vs. dual CPU
> > > mode on fbsd... something is definately fubar with the mysql+fbsd5.x
> > > combination.
> >  
> > You complained about this some time ago and you have still not responded 
> > with the information I suggest..
> 
> I sent this to Jeremy privately, since it was just some preliminary
> measurements, but figured I'd send it publically since the results were
> interesting (if tentative, I need to do a lot more work to make them
> useful.  There are a number of variables I need to look at including:
> 
> - Disabling HTT.  A chat with Scott Long this evening suggests that HTT
>   may be substantially hurting the test cases given increased IPIs, etc.
>   Unfortunately, it looks like I can't easily twiddle HTT without being
>   local to the machine, and I'm at home right now (it being 1:30am and
>   all).   Removing HTT may help substantially with the dip in performance
>   in the SMP configuration.
> 
> - I'd like to compare against RELENG_4 and a recent Linux kernel.
>   Unfortunately, the box is configured for neither right now.
> 
> - I need to try twiddling schedulers -- this was with SCHED_ULE, and I'd
>   like to try SCHED_4BSD.
> 
> - This was without adaptive mutexes, which seem to be helpful for others,
>   so I should give them a try.
> 
> I don't have any amd64 hardware, so I don't know what if any role it will
> play in the results.  The performance drop observed in the report appears
> to be on amd64 (I may have misread).
> 
> Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
> robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research
> 
> 
> ----
> Date: Wed, 16 Jun 2004 01:15:39 -0400 (EDT)
> From: Robert Watson <rwatson_at_FreeBSD.org>
> To: JG <amd64list_at_jpgsworld.com>
> Subject: Re: Possible Threading problem with -CURRENT / MySQL?
> 
> 
> On Tue, 15 Jun 2004, JG wrote:
> 
> > Fwiw, it has to be something that was committed between May 18th and
> > yesterday.  ~May 18th was the last time I built -CURRENT during my last
> > round of testing and I did not have any of these problems.  Then someone
> > emailed me recently and said there were some commits that might effect
> > the outcome of the mysql benchmarks. 
> 
> Ok, so these results are on a dual-processor XEON + hyperthreads, so four
> logical processors.  I used two dates off CVS, 20040515 and 20040615.  I
> also benchmarked my netperf branch.  I don't have RELENG_4 on the box, but
> might be able to load RELENG_4 on it later this week.  In each case, I
> took ten samples, dropped the first value as getting into the cache, and
> took the mean of the rest.  For this test, I used the select test; I'll
> try the other smack query set tomorrow.  In each case, I ran with "10
> 1000" as the arguments to the test.  I used the default threading
> configuration in -CURRENT, which is libpthread (libkse). 
> 
>                                 Mean       Stdev
> 20040515-UP                     4752.27    14.63
> 20040515-SMP                    2550.35    19.23
> 
> 20040615-UP                     4898.71    22.39
> 20040615-SMP                    2666.93    32.01
> 
> Netperf-UP-giant                4902.41    14.3
> Netperf-SMP-giant               2566.18    16.83
> Netperf-UP-mpsafe               4799.35    22.04
> Netperf-SMP-mpsafe              3022.51    18.06
> 
> Unfortunately, I can't turn off HTT remotely, and I'm guessing it damages
> the SMP numbers a fair amount due to additional IPIs without benefit. 
> However, the numbers basically suggest that on my hardware, the UP
> configuration is marginally faster than it was last month, and that if you
> throw in the netperf branch, the SMP case is a moderate amount faster. 
> This suggests that either I'm just lucky, or that the performance loss
> might be specific to the amd64 version of FreeBSD.  I'm going to run some
> more numbers tomorrow and try to post something more rigorous to the
> -threads list. 
> 
> I don't have RELENG_4 on the box or Linux on the box, but I may get a
> chance to later this week. 
> 
> Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
> robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research
> 
> 
>