Re: libthr and main thread stack size

On Fri, Sep 19, 2014 at 03:27:25PM -0400, John Baldwin wrote:
> I suspect it was done out of reasons of being overly conservative in
> interpreting RLIMIT_STACK. I think it is quite surprising behavior
> though and would rather we make your option the default and implement
> what the Open Group says above.

Ok, below is the patch.  I felt bad about adding yet another magic and
undocumented tunable to our libthr.  Since there seems to be no
alternative than a tunable to enforce old behaviour, I documented
the quirks I am aware of.

Doc people, please review the man page in the patch.

diff --git a/lib/libthr/thread/thr_init.c b/lib/libthr/thread/thr_init.c
index 9bf0e29..72a067a 100644
--- a/lib/libthr/thread/thr_init.c
+++ b/lib/libthr/thread/thr_init.c
_at__at_ -445,7 +445,7 _at__at_ init_private(void)
 	struct rlimit rlim;
 	size_t len;
 	int mib[2];
-	char *env;
+	char *env, *env_bigstack, *env_splitstack;
 
 	_thr_umutex_init(&_mutex_static_lock);
 	_thr_umutex_init(&_cond_static_lock);
_at__at_ -473,8 +473,9 _at__at_ init_private(void)
 		len = sizeof (_usrstack);
 		if (sysctl(mib, 2, &_usrstack, &len, NULL, 0) == -1)
 			PANIC("Cannot get kern.usrstack from sysctl");
-		env = getenv("LIBPTHREAD_BIGSTACK_MAIN");
-		if (env != NULL) {
+		env_bigstack = getenv("LIBPTHREAD_BIGSTACK_MAIN");
+		env_splitstack = getenv("LIBPTHREAD_SPLITSTACK_MAIN");
+		if (bigstack != NULL || env_splitstack == NULL) {
 			if (getrlimit(RLIMIT_STACK, &rlim) == -1)
 				PANIC("Cannot get stack rlimit");
 			_thr_stack_initial = rlim.rlim_cur;
diff --git a/share/man/man7/libthr.7 b/share/man/man7/libthr.7
new file mode 100644
index 0000000..16d916f
--- /dev/null
+++ b/share/man/man7/libthr.7
_at__at_ -0,0 +1,254 _at__at_
+.\" Copyright (c) 2014 The FreeBSD Foundation, Inc.
+.\" All rights reserved.
+.\"
+.\" This documentation was written by Konstantin Belousov <kib_at_FreeBSD.org>
+.\" under sponsorship from the FreeBSD Foundation.
+.\"
+.\" Redistribution and use in source and binary forms, with or without
+.\" modification, are permitted provided that the following conditions
+.\" are met:
+.\" 1. Redistributions of source code must retain the above copyright
+.\"    notice, this list of conditions and the following disclaimer.
+.\" 2. Redistributions in binary form must reproduce the above copyright
+.\"    notice, this list of conditions and the following disclaimer in the
+.\"    documentation and/or other materials provided with the distribution.
+.\"
+.\" THIS SOFTWARE IS PROVIDED BY THE AUTHORS AND CONTRIBUTORS ``AS IS'' AND
+.\" ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+.\" IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+.\" ARE DISCLAIMED.  IN NO EVENT SHALL THE AUTHORS OR CONTRIBUTORS BE LIABLE
+.\" FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+.\" DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS
+.\" OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
+.\" HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT
+.\" LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY
+.\" OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
+.\" SUCH DAMAGE.
+.\"
+.\" $FreeBSD$
+.\"
+.Dd September 20, 2014
+.Dt libthr
+.Os
+.Sh NAME
+.Nm libthr
+.Nd FreeBSD implementation of the Posix threading library
+.Sh LIBRARY
+.Lb libpthread
+.Sh DESCRIPTION
+The man page documents the quirks and tunables of the
+.Fx
+implementation for the
+.Lb libpthread .
+When linking with the
+.Li -lpthread ,
+the run-time dependency
+.Dv libthr.so.3
+library is recorded in the produced object.
+.Pp
+The library is tigthly integrated with the Run-time Link-editor
+.Xr ld-elf.so.1 1
+and
+.Lb libc ,
+all three components must be built from the same source tree.
+Mixing
+.Dv libc.so
+and
+.Nm
+libraries from different versions of
+.Fx
+is not supported.
+The run-time linker
+.Li ld-elf.so.1
+has some code to ensure backward-compatibility with older
+.Nm .
+.Sh MUTEX ACQUISITION
+The locked mutex (see
+.Xr pthread_mutex_lock 3 )
+is represented by a volatile variable of type
+.Dv lwpid_t ,
+which records the global system identifier of the thread
+owning the lock.
+The
+.Nm
+performs a congested mutex acquisition in three stages, each of which
+is more resource-consuming than the previous.
+.Pp
+First, the
+.Li spin loop
+is performed, where the library attempts to acquire the lock by
+.Xr atomic 9
+operations.
+The loop count is controlled by the
+.Ev LIBPTHREAD_SPINLOOPS
+environment variable.
+.Pp
+If the
+.Li spin loop
+was unable to acquire the mutex, the
+.Li yield loop
+is executed, performing the same
+.Xr atomic 9
+acquisition attempts as
+.Li spin loop ,
+but each attempt is followed by yield of the CPU time of the thread by
+.Xr sched_yield 2
+syscall.
+By default, the
+.Li yield loop
+is not executed.
+This is controlled by
+.Ev LIBPTHREAD_YIELDLOOPS
+environment variable.
+.Pp
+If both
+.Li spin
+and
+.Li yield loops
+failed to acquire the lock, the thread is taken off the CPU and
+put to sleep in kernel with the
+.Xr umtx 2
+syscall.
+Kernel wakes up a thread and hands the ownership of the lock to
+the woken thread.
+.Sh THREADS STACKS
+Each thread is provided with the private stack area used by C runtime.
+The size of the main (initial) thread stack is set by kernel, and is
+controlled by the
+.Dv RLIMIT_STACK
+process resource limit (see
+.Xr getrlimit 2 ) .
+.Pp
+By default, the main thread size is equal to the value of resource
+.Dv RLIMIT_STACK
+for the process.
+If the
+.Dv LIBPTHREAD_SPLITSTACK_MAIN
+environment variable is present (its value does not matter),
+the main thread size if chomped to 4MB on 64bit architectures, and to
+2MB on 32bit architectures, on the threading library initialization.
+The rest of the address space area reserved by the kernel for initial
+process stack, is used for non-initial threads stack in this case.
+The presence of the
+.Dv LIBPTHREAD_BIGSTACK_MAIN
+environment variable overrides the
+.Dv LIBPTHREAD_SPLITSTACK_MAIN ,
+it is kept for backward-compatibility.
+.Pp
+The size of the stacks for threads created by the process at run-time
+with the
+.Xr pthread_create 3
+call, is controlled by thread attributes, see
+.Xr pthread_attr 3 ,
+in particular, the
+.Xr pthread_attr_setstacksize 3 ,
+.Xr pthread_attr_setguardsize 3
+and
+.Xr pthread_attr_setstackaddr 3 .
+If no attributes for the thread stack size are specified, the default
+non-initial thread stack size is 2MB for 64bit architectures, and 1MB
+for 32bit architectures.
+.Sh RUN-TIME SETTINGS
+The following environment variables are recognized by
+.Dv libthr
+and adjust the operation of the library at run-time:
+.Bl -tag -width LIBPTHREAD_SPLITSTACK_MAIN
+.It Ev LIBPTHREAD_BIGSTACK_MAIN
+Disables the chomp of the initial thread stack, enabled by
+.Ev LIBPTHREAD_SPLITSTACK_MAIN .
+.It Ev LIBPTHREAD_SPLITSTACK_MAIN
+Causes the chomp of the initial thread stack, as described in the
+section
+.Li THREAD_STACKS .
+This was the default behaviour of the
+.Nm
+before
+.Fx 11.0 .
+.It Ev LIBPTHREAD_SPINLOOPS
+The integer value of the variable overrides the default count of
+iterations in the
+.Li spin loop
+of the mutex acquisition.
+The default count is 2000, set by the
+.Dv MUTEX_ADAPTIVE_SPINS
+define in the
+.Nm
+sources.
+.It Ev LIBPTHREAD_YIELDLOOPS
+The non-zero integer value of the variable allows the
+.Li yield loop
+in the process of the mutex acquisition.
+The value is the counter of loop operations.
+.It Ev LIBPTHREAD_QUEUE_FIFO
+The integer value of the variable specifies how often the blocked
+threads are put into the head of the sleep queue, instead of it tail.
+The bigger value reduces the frequency of the FIFO discipline.
+The value must be between 0 and 255.
+.El
+.Sh INTERACTION WITH RUN-TIME LINKER.
+The
+.Nm
+library must appear before
+.Dv libc
+in the global order of depended objects.
+.Pp
+.Pp
+Loading the
+.Nm
+library with the
+.Xr dlopen 3
+call in the process after the program binary is activated,
+is not supported, and causes miscellaneous and hard to diagnose misbehaviour.
+This is due to
+.Nm
+interposing several important
+.Dv libc
+symbols to provide thread-safe services.
+In particular,
+.Dv errno
+and locking stubs from
+.Dv libc
+are affected.
+This requirement is currently not enforced.
+.Pp
+If the program loads the modules at run-time, and modules may require
+the threading services, the main program binary must be linked with
+.Dv libpthread ,
+even if it does not require any service from the library.
+.Pp
+The library cannot be unloaded, the
+.Xr dlclose 3
+function does not perform any action when called with a handle for
+.Dv libthr .
+One of the reason is that interposing of
+.Dv libc
+functions cannot be undone.
+.Sh SIGNALS
+The implementation also interposes the user-installed
+.Xr signal 3
+handlers.
+The interposing is done to postpone signal delivery to threads which
+entered the (libthr-internal) critical sections, where the calling
+of the signal handler is unsafe.
+Example of such situation is owning the internal library lock.
+When the signal is delivered while signal handler cannot be safely
+called, call is postponed and performed after the critical section
+is left.
+This should be taken into account when interpreting the
+.Xr ktrace 1
+logs.
+.Sh SEE ALSO
+.Xr atomic 9 ,
+.Xr dlclose 3 ,
+.Xr dlopen 3 ,
+.Xr errno 3 ,
+.Xr getenv 3 ,
+.Xr getrlimit 2 ,
+.Xr ktrace 1 ,
+.Xr ld-elf.so.1 1 ,
+.Xr libc 3 ,
+.Xr pthread_attr 3 ,
+.Xr pthread_attr_setstacksize 3 ,
+.Xr pthread_create 3 ,
+.Xr signal 3 ,
+.Xr umtx 2 .