HEADS UP: IPX/SPX now marked MPSAFE in 6.x-CURRENT

Over the last hour or so, I've merged a set of changes I've been working
on for a couple of weeks that introduce locking to the IPX/SPX
implementation covering several critical global variables, the raw and
regular IPX/SPX PCB lists, and the PCB data structures for IPX and
IPX/SPX.  This involved some additional cleanup of the SPX code.  The
result is that the IPX/SPX code is *largely* MPSAFE -- certainly enough so
to run without the Giant lock except under fairly unusual circumstances
(see below).  As such, I've removed the flag causing inclusion of IPX/SPX
in the kernel to force the Giant lock over the entire network stack.  It
can be restored, if problems are experienced, by setting debug.mpsafenet=0
in /boot/loader.conf.

There are a number of issues that remain necessary to address in the
IPX/SPX code, including:

(1) The protocol address lists (ifaddrs) are not locked down, so rapid
    changes in addresses during high volume input might result in a read
    of inconsistent address data from kernel memory.  This also needs to
    be addressed with several other protocols, but in practice is a very
    difficult race to exercise. 

(2) IPX raw sockets permit outbound packets to be "sniffed" as well as
    inbound packets, resulting in reentrance of the IPX stack in the
    output path.  This can result in some lock order problems as higher
    level locks are often held over the output path in a downward
    direction.  I will likely introduce redelivery of "sniffed" sent
    packets using a netisr, similar to the way we handle routing socket
    messages.  This is an edge case in how the stack is used -- neither
    the default behavior, nor used in current applications that I know of.

I hope to get the chance to dig into both of these issues in the next
couple of weeks.  I've set of an MFC after of 2-4 weeks for various
elements of the above changes.  Since IPX/SPX is not widely used in the
FreeBSD world, I suspect we won't get much testing of these changes, so if
you are a user of IPX/SPX, your testing would be much appreciated!

There are likely nits and bugs in this code -- if you experience crashes,
please make sure your kernel is built with INVARIANTS and WITNESS, as that
will make it much easier to debug locking problems.  When reporting bugs,
please include all relevant console and debugging output relating to
failed assertions, traps, etc.  I've attempted to test several forms of
normal socket I/O, but I'm not set up to exercise netncp/nwfs or IPX
routing.

Thanks,

Robert N M Watson