Re: Unfortunate dynamic linking for everything

On Tue, 18 Nov 2003, David Schultz wrote:

> On Tue, Nov 18, 2003, Robert Watson wrote:
> > (2) Shells again, because they will be fork()d and exec()d frequently
> >     during heavily scripted activities, such as system boot, periodic
> >     events, large make jobs, etc.  And presumably the only shell of
> >     interest is sh, although some of the supporting non-builtin binaries
> >     may also be of interest. 
> 
> This is the only performance argument that truly seems compelling,
> particularly since fork()/exec() performance for dynamic binaries in
> FreeBSD really sucks as compared to Solaris and AIX (and presumably
> other systems that have gone dynamic long ago).  I think this is mostly
> an aspect of the dynamic linker and the lack of prebinding rather than
> performance problems in the VM system. 

Some of Matthew Dodd's exploration of prebinding seems to indicate it's a
big win for large applications with many libraries (XWindows, KDE,
OpenOffice, whathaveyou), but that the base cost of loading relocation
caches overwhelms the benefits of prebinding.  More refinement could
correct this, perhaps.  Part of the cost of excessive cost of prebinding
right now is that we almost always map libraries to different addresses
for different programs, which results in a very large relocation cache.  A
quick sample on my notebook reveals libc mapped at no less than 28
different base addresses.  On systems like Mac OS X, use of a "shared
region" permits not only use of prebinding, but assumptions of common load
addresses for common libraries.  In addition, the "shared region" approach
allows the reuse of page table and VM meta-data across many processes. 
Leaving aside special-purpose optimizations like that, using some simple
heuristics, we could probably easily reduce the number of load addressed
dramatically, making prebinding a far more feasible approach, and
something we should explore thoroughly.

My feeling is we should all go away for a day or two, and run our favorite
macro-benchmark on our favorite sensitive dynamic linking-sensitive
application. I.e., buildworld, system boot, periodic, or whatever. Attempt
to focus on the actual performance loss, both the cause, and potential
solutions.  Returning to a static /bin/sh would be one potential solution. 
John Dyson's suggestion of "mixed mode" linking, in which we statically
link known required parts of binaries, but permit dynamic linking of
"plug-in"  functionality is another quite reasonable solution, as long as
we take care of the potential issue of "plug-ins" relying on symbols from
a library that is statically linked to the application, but not in the set
of symbols depended on by the application.  Remaining with dynamic linking
is another possible "solution" if macrobenchmarks don't reveal an
interesting hit.  If I had to guess, I think it probably comes down to
comparing the costs of repeated fork/exec of /bin/sh, and that the "mixed
mode" solution may be the easiest way to address a possible performance
problem while retaining the dynamic linking benefits.  But I want to be
sure we address a real-world performance problem, not an anticipated one. 

For big applications like KDE, Open Office, et al, prebinding probably is
the right approach, and hopefully we have plenty of time before 5.3 to
refine Matthew's work and see if we can't get things up to speed.

Robert N M Watson             FreeBSD Core Team, TrustedBSD Projects
robert_at_fledge.watson.org      Network Associates Laboratories