Re: CFT: FreeBSD Package Base

Cc: list trimmed to relevant.  Very long essey below, be warned.

On Sun, Apr 28, 2019 at 03:52:21PM -0400, kris_at_ixsystems.com wrote:
> FreeBSD Community,
> 
>  
> 
> I'm pleased to announce a CFT for builds of FreeBSD 12-stable and 13-current
> using "TrueOS-inspired" packaged base. These are stock FreeBSD images which
> will allow users to perform all updating via the 'pkg' command directly.
> Rather than trying to answer all questions in this announcement, we've
> created a FAQ page with more details. Please refer to this page, and let us
> know if you have additional questions that we can include on that page going
> forward.
> 
>  
> 
> Additionally, I will be hosting a Package Base working group at BSDCan 2019,
> and welcome user and developer attendance to discuss this and other ongoing
> package work:
> 
>  
> 
> https://wiki.freebsd.org/DevSummit/201905/PackageBase
> 
>  
> 
>  
> 
> FAQ
> 
> -------------------------------------------------------------
> 
> https://trueos.github.io/pkgbase-docs/
> 

I do not know what are design decisions for trueos pkgbase are, but I
do know something about in-tree split and why some packaging decisions
where made. I cannot attend your WG, but I believe the reasoning used
for the in-tree is important enough to represent it intact from the
source.  I have to start with some explanatory long text to put it into
the proper perspective.

There are two knots of interdependinces which are critical for correctness
of any upgrade where the target system cannot be simply discarded on failure:
1. C runtime
2. Minimal boot path to prompt.
Let me elaborate both, starting from point 1, which is typically very obscure
despite having the fundamental nature for anything related to upgrades.

The basic execution environment for any program executed by the FreeBSD
kernel is formed by combination of kernel' syscall interface and some
system userspace code which makes the expected environment over the
bare-bone image state after execve. The environment is typically named
C runtime environment since C language ABI is directly tied into it,
and normal C programs only get whatever is provided by the C runtime
unless additional libraries are linked in. Trully, it is not just C
runtime, any other execution environment on top of the OS is based on
this one, but since almost every 'advanced' language runtime is backed
by C language and its runtime, the name stuck.

FreeBSD C runtime, arguably, is provided by the following four objects:
	/libexec/ld-elf.so.1
	/lib/libc.so.7
	/lib/libthr.so.3
	/lib/libm.so.5
There, we do *guarantee* that the external ABI of the whole pack of
these four objects is backward compatible, i.e. if the binary was
compiled against set if base libraries at earlier date (may be also
on earlier branch), then the binary behaviour would be same when
executed on newer C runtime pack. This is not trivial to achieve,
besides technical measures that helps there, like backward-compatible
syscall interface, symbol versioning, providing fall-back code for
older interface, a lot of overhead in the development is enforced, like
carefull reviews of the changes, the policy and related discipline of
versioning, following published ABI standards, and so on.

But, internal ABI of the C runtime pack, i.e. interfaces which make rtld
work with libc and libthr, or way by which libthr, when loaded, makes
libc thread-aware, are not stable, and more, they are often changed
in backward-incompatible way. Requiring backward-compatibility there
would stop our ability to evolve the system. Answering some questions in
advance, yes, rtld delves into libc, libthr patches libc on load, libc
has hooks to control some libthr behaviour.

The only provision that we make is that ld-elf.so.1 is required to work
with older libc/libthr combination, but even then libc and libthr must
be built from the same sources with the same options set.

Now, returning to pkgbase, if you look at what libs are packed into clibs,
you see:
	ld-elf.so.1
	libc.so.7 (and modules like iconv tables or nss, if any)
	libthr.so.3
	libdl.so.1
	libgcc{, _eh, _s}.so.1
	libm.so.5
	libedit.so.7
	libncurses{, w}.so.8
	libc++.so.1
It adds very popular libs like libncurses/libedit, and C++ runtime. The
basic reasoning is that this package is small and chances of something
going wrong while installing it are small as result. Put it other way,
the small clibs package organization makes it highly probable that
system is left in the consistent state (either all new libs, or all old
libs) after the upgrade, whetever the outcome is.

If the C runtime pack is not split from the whole 700MB+ update blob, libthr
update has almost certain chance to occur long after or before libc update,
so failures do tend to leave inconsistent rtld/libc/libthr set.  At best,
it gives you strange glitches, at worst you get unusable system that cannot
be repaired without external media.

Now, the second item, the minimal boot path. By definition, it consists
of everything that is required to get bare-bone shell prompt in single
user mode, and where user can repair failed upgrade. Arguably, it should
also include the tools to configure the network and fix filesystems. So
it should consists of
	loader (including forth/lua scripts)
	kernel
	C runtime
	/sbin/init
	/bin/sh
	newfs/fsck/tunefs for UFS
	zfs/zfspool and libs for ZFS
	ifconfig/route/ping
In this set, zfs and network management tools must be synced with the kernel,
since ABI of the management syscalls is not guaranteed to be stable even
on stable branches.

The above brain dump is at least partial enumeration of things that were
discussed between me and Glen when Glen created the current in-tree
packaging code.