HEADS UP: More socket buffer locking merged

I've just merged another section of the rwatson_netperf patch to CVS.
This introduced new locking, but does not remove existing locking.  This
means, hopefully, we avoid introducing races, but we are exposed to
additional locking assertions and checking to test lock coverage and
correctness.  The changes in the tree are not yet sufficient to run with
debug.mpsafenet=1.  While I've been running with variations on these
changes for six months, I recenlty did some rotilling and cleanup on the
patches before merging, so some of this has not seen a lot of exposure in
its current form.  When reporting assertion failures, WITNESS warnings,
etc, please make sure to include all the pertinent output (stack traces,
etc), and if possible, functions and line numbers in files.  As always,
the developer's handbook includes useful information on generating dumps,
setting up serial consoles to make it easier to copy output, etc.

Thanks,

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Senior Research Scientist, McAfee Research

---------- Forwarded message ----------
Date: Mon, 21 Jun 2004 00:20:44 +0000 (UTC)
From: Robert Watson <rwatson_at_FreeBSD.org>
To: src-committers_at_FreeBSD.org, cvs-src_at_FreeBSD.org, cvs-all_at_FreeBSD.org
Subject: cvs commit: src/sys/kern uipc_socket.c uipc_socket2.c         uipc_usrreq.c src/sys/netkey keysock.c src/sys/sys socketvar.h

rwatson     2004-06-21 00:20:44 UTC

  FreeBSD src repository

  Modified files:
    sys/kern             uipc_socket.c uipc_socket2.c 
                         uipc_usrreq.c 
    sys/netkey           keysock.c 
    sys/sys              socketvar.h 
  Log:
  Merge next step in socket buffer locking:
  
  - sowakeup() now asserts the socket buffer lock on entry.  Move
    the call to KNOTE higher in sowakeup() so that it is made with
    the socket buffer lock held for consistency with other calls.
    Release the socket buffer lock prior to calling into pgsigio(),
    so_upcall(), or aio_swake().  Locking for this event management
    will need revisiting in the future, but this model avoids lock
    order reversals when upcalls into other subsystems result in
    socket/socket buffer operations.  Assert that the socket buffer
    lock is not held at the end of the function.
  
  - Wrapper macros for sowakeup(), sorwakeup() and sowwakeup(), now
    have _locked versions which assert the socket buffer lock on
    entry.  If a wakeup is required by sb_notify(), invoke
    sowakeup(); otherwise, unconditionally release the socket buffer
    lock.  This results in the socket buffer lock being released
    whether a wakeup is required or not.
  
  - Break out socantsendmore() into socantsendmore_locked() that
    asserts the socket buffer lock.  socantsendmore()
    unconditionally locks the socket buffer before calling
    socantsendmore_locked().  Note that both functions return with
    the socket buffer unlocked as socantsendmore_locked() calls
    sowwakeup_locked() which has the same properties.  Assert that
    the socket buffer is unlocked on return.
  
  - Break out socantrcvmore() into socantrcvmore_locked() that
    asserts the socket buffer lock.  socantrcvmore() unconditionally
    locks the socket buffer before calling socantrcvmore_locked().
    Note that both functions return with the socket buffer unlocked
    as socantrcvmore_locked() calls sorwakeup_locked() which has
    similar properties.  Assert that the socket buffer is unlocked
    on return.
  
  - Break out sbrelease() into a sbrelease_locked() that asserts the
    socket buffer lock.  sbrelease() unconditionally locks the
    socket buffer before calling sbrelease_locked().
    sbrelease_locked() now invokes sbflush_locked() instead of
    sbflush().
  
  - Assert the socket buffer lock in socket buffer sanity check
    functions sblastrecordchk(), sblastmbufchk().
  
  - Assert the socket buffer lock in SBLINKRECORD().
  
  - Break out various sbappend() functions into sbappend_locked()
    (and variations on that name) that assert the socket buffer
    lock.  The !_locked() variations unconditionally lock the socket
    buffer before calling their _locked counterparts.  Internally,
    make sure to call _locked() support routines, etc, if already
    holding the socket buffer lock.
  
  - Break out sbinsertoob() into sbinsertoob_locked() that asserts
    the socket buffer lock.  sbinsertoob() unconditionally locks the
    socket buffer before calling sbinsertoob_locked().
  
  - Break out sbflush() into sbflush_locked() that asserts the
    socket buffer lock.  sbflush() unconditionally locks the socket
    buffer before calling sbflush_locked().  Update panic strings
    for new function names.
  
  - Break out sbdrop() into sbdrop_locked() that asserts the socket
    buffer lock.  sbdrop() unconditionally locks the socket buffer
    before calling sbdrop_locked().
  
  - Break out sbdroprecord() into sbdroprecord_locked() that asserts
    the socket buffer lock.  sbdroprecord() unconditionally locks
    the socket buffer before calling sbdroprecord_locked().
  
  - sofree() now calls socantsendmore_locked() and re-acquires the
    socket buffer lock on return.  It also now calls
    sbrelease_locked().
  
  - sorflush() now calls socantrcvmore_locked() and re-acquires the
    socket buffer lock on return.  Clean up/mess up other behavior
    in sorflush() relating to the temporary stack copy of the socket
    buffer used with dom_dispose by more properly initializing the
    temporary copy, and selectively bzeroing/copying more carefully
    to prevent WITNESS from getting confused by improperly
    initialized mutexes.  Annotate why that's necessary, or at
    least, needed.
  
  - soisconnected() now calls sbdrop_locked() before unlocking the
    socket buffer to avoid locking overhead.
  
  Some parts of this change were:
  
  Submitted by:   sam
  Sponsored by:   FreeBSD Foundation
  Obtained from:  BSD/OS
  
  Revision  Changes    Path
  1.186     +31 -7     src/sys/kern/uipc_socket.c
  1.134     +237 -27   src/sys/kern/uipc_socket2.c
  1.129     +8 -3      src/sys/kern/uipc_usrreq.c
  1.27      +2 -0      src/sys/netkey/keysock.c
  1.126     +43 -5     src/sys/sys/socketvar.h