Re: Unfortunate dynamic linking for everything

On Tue, 18 Nov 2003 dyson_at_iquest.net wrote:
> Scott Long said:
> > On Tue, 18 Nov 2003 dyson_at_iquest.net wrote:
> > > M. Warner Losh said:
> > > > In message: <200311181307.hAID7uHa032514_at_dyson.jdyson.com>
> > > >             dyson_at_iquest.net writes:
> > > > : 	It really doesn't make sense to arbitrarily cut-off a
> > > > : 	discussion especially when a decision might be incorrect.
> > > >
> > > > I'd say that good technical discussion about why this is wrong would
> > > > be good.  However, emotional ones should be left behind.  Except for
> > > > John's message, most of the earlier messages have been more emotional
> > > > than technical.
> > > >
> > > > John, do you have any good set of benchmarks that people can run to
> > > > illustrate your point?
> > > >
> > > I'll see if I can find some of my old benchmarks (from memory,
> > > not disk -- lots of stuff has rotted away.).  I had hoped
> > > to avoid doing much in the way of proof.  Pointing to the
> > > much more complex process map along with the implication
> > > for significantly higher overhead :-).  (The VM code generally
> > > gets slower with more complex process maps and more memory used.
> > > For smallish programs -- including those with a few shared libs,
> > > changing the VM code won't help much, because the cost is built
> > > in to the complexity of the process image.)
> >
> > I'm glad that real arguments are finally starting to surface =-)
> >
> > Our rationale for encouraging Gordon is as follows:
> >
> > 1.  4.x upgrade path:  As we approach 5-STABLE, a lot of users might want
> >     to upgrade from 4-STABLE.  Historically in 4.x, the / partition has
> >     been very modest in size.  One just simply cannot cram the bloat that
> >     has grown in 5.x into a 4.x partition scheme.  Of course there is the
> >     venerable 'dump - clean install - restore' scheme, but we were looking
> >     for something a little more user-friendly.
> >
> /bin and /sbin don't need to contain everything :-).  Adding a full
> featured /rescue should also help to counterbalance this (but maybe
> the upgrade would be selective?)  There are reasons for /usr/bin
> and /usr/sbin (along with /usr/local, /usr/share.)
>
> Disk space is CHEAP and grandfathering 20MB hard drives (or the
> yr2000 equivalent of 4GB hard drives) is nice -- but repartitioning
> them with more reasonable root allocations seems like a good
> idea anyway :-).  50MB root was silly when 2-4GB disks were common.
>
> Continuing to pay for a poorly made partitioning decision in the past
> seems to be unworthy (in my opinion.)

/boot has grown quite large too and threatens to be unbounded in size as
times go on.  Shaving off the 30-40MB of size in /bin and /sbin can
help alleviate this, even on system formatted in 5.x partition sized.

This argument wasn't the most compelling one by itself, but it played a
part in the decision so I included it here.

>
> >
> > 2.  NSS support out-of-the-box:  Again, this is a user-experience issue
> >     in that forcing NSS users to recompile world was seen as a potential
> >     roadblock to them.
> >
> The cool thing about properly implemented shared libs is that not everything
> has to be shared.  Even the kernel supports runtime loading/addition
> of modules, and the NSS sounds like a good candidate for a library
> feature.  Burdening everything with the sparse .data associated with
> libc seems to be a waste (but, perhaps the performance that was carefully
> crafted into the codebase is no longer important?)  Really -- it is
> possible that performance isnt' important anymore, and fully accepting
> the ongoing loss of performance for features (without careful tradeoffs)
> might be an appropriate strategy.  (In the case of the NSS stuff, it
> shouldn't require everything to be built dynamic.)

Unfortunately, our dynamic linker prevents dlopen() and friends from
working in a static binary.  There was some talk about having NSS
lookups happen through a proxy process, but our implementation just
plain wasn't done that way, and no one has stepped forward with
alternate code.

NSS modules can affect a surprisingly large number of utilities, including
the shells.  Selectively choosing which utilities to link static vs.
shared would have resulted in quite a patchy result that probably would
not have had a significant benefit.

>
> >
> > 3.  Binary security updates: there is a lot of interest in providing a
> >     binary update mechanism for doing security updates.  Having a dynamic
> >     root means that vulnerable libraries can be updated without having to
> >     update all of the static binaries that might use them.
> >
> The additional hole of exploiting the system through the shared libs
> is a negative tradeoff.

Exploits in libraries happen though.  The LD_LIBRARY_PATH attack is an old
one that most Unixes are hopefully hardened against.

>
> >
> > 4.  I've probably forgotten the other factors...
> >
> > As for performance, we felt that the typical use of FreeBSD did not fall
> > into the category of doing forkbomb performance tests.
> >
> Nope - even shell execution times (e.g. long shell scripts) will see a
> difference.  Prejudicial comments like 'fork-bombs' and
> 'microbenchmarks' aren't really fair.  Proper benchmarking is
> time consuming and analysis is tricky.  System loading conditions
> are difficult to control (for experiments.)

I apologize for the poor choice of words, I certainly did not mean to
insult you or anyone else.

>
> Note that alot of the lost performance is hidden by the VM code (e.g. one
> of the purposes for prezeroing on SMP box -- before SMP was fine grained.)
> The cost of sparse data and needless inheritance of modified pages
> has performance hits that are incredibly difficult to accurately (and
> honestly) measure.  Anyone can create 'preordained' benchmark results.
>
> This is one reason why I am loathe to get into the FreeBSD performance
> game again -- it is much more difficult than a few micro-benchmarks :-)
> that simply measure a fork exec time.
>
> John
>
>
>

I agree that reaching for improved performance is a beneficial goal, and
that tradeoffs of performance vs. features must be carefully considered
and balanced.  With FreeBSD's popularity as a server platform, things like
integrated NSS were too long in coming and were very compelling reasons to
go this route.  So it looks like the performance vs. features see-saw has
tipped from one side to the other.  Backing out dynamic root will not help
rebalance it though.  Again, I hope that this will inspire some graduate
student somewhere to experiment with optimizing the VM system in this
direction.

Scott