HEADS UP: Giant-free networking now the default in CVS HEAD (6.x)

HEADS UP!

The default setting for debug.mpsafenet has changed from 0 to 1, which
means that by default, a number of kernel elements will now run without
Giant by default, and often in parallel or preemptively.  This includes: 

- Network interface ithreads for supported (MPSAFE) network device
  drivers.

- The netisr thread when processing MPSAFE network protocols.

- Many network-related system calls, including the send and receive paths.

- Many network related timeout and callout routines happening with clock
  ticks.

As mentioned in the commit message, there are now a number of seatbelts
and ways to change the setting to restore more conservative operation, as
this change may expose existing bugs or introduce new bugs.  In fact, that
is highly likely to be the case.  If you experience new network related
problems, as a first debugging step, please try putting the Giant lock
back over the network stack.  This can be done by setting the loader
tunable "debug.mpsafenet" to 0, or by compiling "options NET_WITH_GIANT" 
into your kernel configuration.  If the problem goes away, that doesn't
necessarily mean it's a bug in MPSAFE operation, as we could just be
encouraging an existing bug to expose itself, but it's also fairly likely
it is. 

If you want to run in an environment with even more parallelism, you can
set net.isr.enable to 1 to run network processing to completion in
interrupt threads, which has the effect of deferring less work to the
netisr.

If you'd like to read more about the on-going work, including seeing
earlier posts about the work, precautions to take, etc, you can read about
it at: 

    http://www.watson.org/~robert/freebsd/netperf/

I do anticipate a slightly bumpy ride with this setting now as the
default, but we have easy ways to back off the change at run-time and
compile-time should it get too bumpy.  There will likely be some serious
nits we pick up and fix in the next three to four days, in particular, so
some caution might need to be exercised. 

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Principal Research Scientist, McAfee Research

---------- Forwarded message ----------
Date: Sat, 28 Aug 2004 15:11:13 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/conf options src/sys/sys kernel.h src/sys/net         netisr.c

rwatson     2004-08-28 15:11:13 UTC

  FreeBSD src repository

  Modified files:
    sys/conf             options 
    sys/sys              kernel.h 
    sys/net              netisr.c 
  Log:
  Change the default disposition of debug.mpsafenet from 0 to 1, which
  will cause the network stack to operate without the Giant lock by
  default.  This change has the potential to improve performance by
  increasing parallelism and decreasing latency in network processing.
  
  Due to the potential exposure of existing or new bugs, the following
  compatibility functionality is maintained:
  
  - It is still possible to disable Giant-free operation by setting
    debug.mpsafenet to 0 in loader.conf.
  
  - Add "options NET_WITH_GIANT", which will restore the default value of
    debug.mpsafenet to 0, and is intended for use on systems compiled with
    known unsafe components, or where a more conservative configuration is
    desired.
  
  - Add a new declaration, NET_NEEDS_GIANT("componentname"), which permits
    kernel components to declare dependence on Giant over the network
    stack.  If the declaration is made by a preloaded module or a compiled
    in component, the disposition of debug.mpsafenet will be set to 0 and
    a warning concerning performance degraded operation printed to the
    console.  If it is declared by a loadable kernel module after boot, a
    warning is displayed but the disposition cannot be changed.  This is
    implemented by defining a new SYSINIT() value, SI_SUB_SETTINGS, which
    is intended for the processing of configuration choices after tunables
    are read in and the console is available to generate errors, but
    before much else gets going.
  
  This compatibility behavior will go away when we've finished the last
  of the locking work and are confident that operation is correct.
  
  Revision  Changes    Path
  1.478     +1 -0      src/sys/conf/options
  1.12      +84 -3     src/sys/net/netisr.c
  1.118     +6 -0      src/sys/sys/kernel.h