Re: [PATCH] Mantaining turnstile aligned to 128 bytes in i386 CPUs

:Does the same hold true with kernel threads in FreeBSD (e.g. two threads
:using FPU)?

    Preemption and pinning make the issue a bit more difficult for FreeBSD,
    but the basic idea remains valid.

    From the point of view of NPXTHREAD the situation is very simple:

    * NPXTHREAD = NULL

	nobody owns the FP, nobody is using the FP.  If the kernel wants
	to use the FP it just FNCLEX + CLTS and sets NPXTHREAD = curthread.
	When it is finished, it undoes that sequence (NPXTHREAD = NULL,
	set CR0_TS again).

	PLUSES: FP state does not need to be saved or restored

	ISSUES: due to cpu migration and preemption the setup and teardown
	sequence must be done with the cpu pinned, inside a critical section.
	But the actual use of the FP does not need to occur inside a 
	critical section or with the cpu pinned.

    * NPXTHREAD = other_thread

	Some other thread owns the FP, but it isn't our thread so we can
	safely save the FP state for the other thread without worrying
	about creating a situation where we thrash the T_DNA exception.
	Save FP state, FNCLEX, CLTS, set NPXTHREAD = curthread.
	When finished, NPXTHREAD = NULL, set CR0_TS, do *not* restore
	the 'other' thread's FP state.

	PLUSES: The FP state probably had to be saved anyway, it's no big
	deal or at least it is not as big a deal as the NPXTHREAD = curthread
	case.

	ISSUES: Same as above.

    * NPXTHREAD = curthread

	The current thread (either userland or a pushed kernel FP context)
	is using the FP.  If the kernel decides it needs the FP it
	must save the FP state, FNCLEX, CLTS, do its thing.  When finished
	it can decide to set NPXTHREAD = NULL and set CR0_TS, or it 
	can restore the previously saved state and leave NPXTHREAD = curthread.

	PLUSES: Very few

	ISSUES: Same as above, but here the kernel must decide whether it
	is worth stealing the FP or not, because it might get into a 
	thrashing situation with the T_DNA exception under certain
	userland loads.

	Note that there are many cases where userland may use the FP unit
	very occassionally.  In such cases you *DO* want to be able to steal
	it, so perhaps some heuristic is needed to determine the cost of
	stealing the FP unit dynamically.

    It is possible to abstract it even more... for example, one can set
    CR0_TS when going from userland to the kernel and completely abstract
    out the kernel's use of the FP unit at the cost of a higher entrance
    fee to get in and out of the kernel.  I decided NOT to do this in
    DragonFly.  If the DragonFly kernel wants to use the FP it has to 
    check and adjust the NPXTHREAD state.

    But, to be absolutely clear here, it costs virtually *nothing* to use
    the FP in the kernel for non-FP media instructions (i.e. movdq and
    friends) if userland has not used the FP recently.  You push a 
    temporary save area, set NPXTHREAD, FNCLEX, CLTS, use the FP, then
    pop the save area pointer, set NPXTHREAD to NULL, and set CR0_TS, and
    that's it.  It may seem like a lot of steps but those are all
    very fast instructions verses having to actually save and restore the
    512 byte FP state.  The biggest overhead would actually be the critical
    section and cpu pinning required to properly transition the NPXTHREAD
    state.

					-Matt
					Matthew Dillon 
					<dillon_at_backplane.com>