Running the network stack without Giant -- what to try and when

As many of you have seen from status reports, e-mails, bug reports, etc,
the FreeBSD Project has been working for some time on getting the network
stack to run in parallel on multiple CPUs.  We're now at a point where a
substantial amount of functionality appears to run pretty successfully
without the Giant lock, and we're ready for more people to start running
it that way so we can find and fix problems.  Let me start by enumerating
a few caveats: 

- While we've been doing pretty heavy testing in MPSAFE configurations,
  the nature of multiprocessor development and adapting code for MP safety
  means that it's unlikely this will "just work" for all takers.  However,
  it may well work for many.

- We've focussed primarily on getting mainstream network configurations
  to run without Giant: this means that less mainstream subsystems (parts
  of IPv6, some netgraph nodes, IPX, etc) are currently unsafe without the
  Giant lock turned on.  Some less mainstream network devices are also not
  currently able to operate without Giant.  There is active work in all of
  these area to remedy this issue.

- You may run into hard to diagnose problems.  We'd like to try to
  diagnose them anyway, but if you start to experience new problems,
  you'll want to go read the Handbook chapter on preparing kernel bug
  reports and diagnosing problems.  You'll also want to be prepared to run
  the system with INVARIANTS and WITNESS turned on.

- Not all workloads will experience a performance benefit -- some, for
  various reasons, will get worse.  However, several interesting
  performance loads get measurably better.  If you don't see an
  improvement, or you see things get worse, please don't be surprised --
  you may want to look at some of the suggestions I make below on ways to
  make the results more predictable.  Generally, you shouldn't see
  substantial performance degradation, if any, but it can't be ruled out,
  especially due to scheduling issues, etc.

- We can and will destroy your data.  We don't mean to, because we like
  your data, and we try not to, but this is, after all, operating system
  development.

With all that in mind, now is in fact a good time to start experimenting
with things, as these changes appear to be relatively stable in our
initial testing.  Note that there is some current instability in the CVS
HEAD, and so I'd ask for some caution in reporting problems as being
caused by debug.mpsafenet -- it may or may not be our fault :-).  I've
disabled PREEMPTION locally for thread centric testing, but haven't needed
to for other testing.

Here's some technical information on how to get started:

(1) Determine if all of the stack components you will operate with are
    MPsafe.  For common configurations, answering the following questions
    will help you decide this:

	- Are you using IPv6, IPX, ATM, or KAME IPSEC?  If you answered
	  yes to any of these questions, it is not yet safe for you to run
	  without Giant.

	- Are your using Netgraph?  If yes, it may be that you are not yet
	  able to run without Giant.  It is worth giving it a try, but you
	  may experience panics, etc, especially in MP configurations.

	- Are you using SLIP or kernel PPP (not to be confused with user
	  ppp, which is what most FreeBSD users use with modems).

	- Are you using any physical network interfaces other than the
	  following: bge, dc, em, ep, fxp, rl, sis, xl, wi.

	  The following may well work: en, gx, pcn, sf.  However, they
	  have not been marked MPSAFE by the driver maintainer.

	  NOTE: Do you maintain a network interface driver?  Is it not on
	  this list?  Shame on you!  Or maybe shame on me for not listing
	  it, even though it should work.  Drop me a private e-mail with
	  and questions or comments.  Please update the busdma driver
	  status web page with your driver's status.

(2) If you are comfortable that you are using an MPSAFE-supported
    configuration, then you can use the following tunable in loader.conf
    to disable the Giant lock over the network stack on your system:

	debug.mpsafenet="1"

    Note that this is a boot-time only flag; you can inspect the setting
    with a sysctl, but it cannot currently be changed at runtime.

    Do a dmesg and confirm that all your probed network interfaces are
    marked as MPSAFE or not GIANT LOCKED (or whatever we call it now).  If
    you have a network interface that is still GIANT LOCKED, it may not be
    able to function correctly with debug.mpsafenet=1.  However, if you're
    not actively using it, it probably won't cause a problem.  For
    example, firewire network interfaces can't currently be used with
    debug.mpsafenet=1.  However, if idle, they shouldn't cause any
    problems.  We're currently working to improve compatibility with
    device drivers that aren't mpsafe, and hope to have a prototype soon.

Some notes:

On SMP-centric performance measurements, such as local UNIX domain socket
use by MySQL on MP systems, I've observed 30%-40% performance improvements
by disabling Giant (some details below).  My recommended configuration for
testing out the impact of disabling Giant on MP systems is: 

	- Set "options ADAPTIVE_MUTEXES" -- this seems to help a lot with
	  contention and load.

	- Disable HTT.  In my workloads, which tended to pound the kernel,
	  this hurt quite a bit.  Obviously, the effectiveness of HTT
	  depends on the instruction mix, so this may not be for you.

	- Pick one of ULE and 4BSD, and then try the other.  I found 4BSD
	  helped a lot for MySQL, but I've seen other benchmarks with quite
	  different results.

	- For stability purposes with MySQL, I currently have to disable
	  PREEMPTION, as the MySQL benchmarks I use are pretty
	  thread-centric and trigger preemption-related bugs with the kernel
	  threading bits.

	- If you want to measure performance, make sure to disable
	  INVARIANTS, INVARIANTS_SUPPORT, WITNESS, etc.

Some notes on bug reporting:

	- Make sure to identify that you are running with debug.mpsafenet.
	  If the problem is reproduceable, make sure to indicate if it goes
	  away or persists when you disable debug.mpsafenet.

	- If you appear to be experiencing a hang/deadlock, please try
	  running with WITNESS.  I'd actually like to see most people
	  running with WITNESS for a bit to shake out lock order issues, as
	  I've introduced a lot of orders.  If experiencing lock order
	  reversals, please include the full console warning including
	  stack trace.

	- INVARIANTS also considered good.  Even if you aren't running
	  with WITNESS, do run with INVARIANTS.

	- If you experience a hang, see if you can get into DDB -- if you
	  are having problems getting in using a console break, try a serial
	  console.  When debugging, at minimum DDB 'ps' output, along with
	  traces of interesting processes.  Typically interesting will be
	  processes that appear to be involved in the hang, etc.
	  Obviously, this requires some intuition about what causes the
	  hang and I can't offer hard and fast rules here.

	- Experimenting with debug.mpsafenet=1 and UP is also interesting,
	  not just SMP.  With PREEMPTION turned on, it may result in lower
	  latency and/or lower throughput.  Or not.  Regardless, it's
	  interesting -- you don't have to have SMP to give it a spin.

FYI, while results can and will vary, I was pleased to observe moving from
a UP->MP speedup of 1.07 on a dual-processor box to a speedup of 1.42 with
the supersmack benchmark using 11 workers and 1000 select transactions
with MySQL.  For reference, that was with the 4BSD scheduler and adaptive
mutexes.  For loopback netperf with TCP and UDP, I observed no change in
performance (well, 1% better for UDP RR, but basically no change).  Note
that the MySQL benchmark here is basically a UNIX domain socket IPC test,
and so real world databases will give pretty different results since they
won't be pure IPC.  The results appear to be very sensitive to the choice
of scheduler, and for a variety of reasons I've preferred 4BSD during
recent testing (not least, better results in terms of throughput). 

There are a lot of people who have been working on this for quite some
time -- I can't thank them all here, but I will point at the netperf web
page as a place to look for ongoing patches, change logs, and some
credits:

    http://www.watson.org/~robert/freebsd/netperf/

I try to keep it up to date about once a week or so as I drop new patch
sets.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research