Re: best approximation of getcpu() ?

On Fri, Dec 16, 2016 at 09:29:15AM +0000, David Chisnall wrote:
> On 16 Dec 2016, at 03:10, Alan Somers <asomers_at_FreeBSD.org> wrote:
> > 
> > What about pthread_setaffinity(3) and friends?  You can use it to pin
> > a thread to a single CPU, and know that it will never migrate.
> 
> This is not a useable solution for anything that needs to live in a library and also doesn???t solve the problem.
> 
> The Linux get_cpu call() is used for caches that are somewhere between global and thread-local.  Accessing them still requires a lock, but it???s very likely to be uncontended (contention only happens when you???re context switched at exactly the wrong time, or if a thread is migrated between cores in between the get_cpu() call and usage) and so you can use the userspace fast path for the lock and not suffer from cache contention effects.  
> 
> One x86, you can use cpuid from userspace and get the current core ID.  I have some code that does this and re-checks every few hundred accesses, storing the current CPU ID in a thread-local variable.  Using the per-CPU caches is a lot faster than using the global cache (and reduces contention on the global cache).  It would be great if we could have a syscall to do this on FreeBSD (it would be even better if we could have specify a TLS variable that the kernel automatically updates for the userspace thread when the scheduler migrates the thread between cores).

indeed the following line seems to do the job for x86
	asm volatile("cpuid" : "=d"(curcpu), "=a"(tmp), "=b"(tmp), "=c"(tmp) : "a"(0xb) );
(there must be a better way to tell the compiler that eax, ebx, ecx, edx are
all clobbered).

0xb is the CPUID function that returns the current APIC id for the
core (not necessarily matching the OS core-id)

The only problem is that this instruction is serialising and slow,
seems to take some 70-100ns on several of my machines so you
cannot afford to call it at all times but need the value cached
somewhere. Exposing it as thread local storage, or a VDSO syscall,
would be nicer because the kernel knows when it is actually changing
value.

cheers
luigi