Packaging of base system

I'm not sure if this is the right place to post, so please point me
elsewhere if there's a better one.

Recent discussions here mentioned the installation/upgrading/packaging of
the FreeBSD base system.

Question: is there any fundamental reason why this could not now be moved to
using the 'package' system rather than just chunked tarballs, in time for
FreeBSD 6?

Jordan Hubbard noted in section 3.1 of
http://people.freebsd.org/~jkh/package-and-install.txt
(written nearly 5 years ago)

   "As I mentioned in the history section, one of the more annoying  
    problems with FreeBSD's current distribution format is the dividing
    line between distributions and packages.  There should really only be 
    one type of "distribution format" and, of course, it should be the    
    package (There Can Be Only One).  Achieving this means we're first   
    going to have to grapple with several problems, however:

    First, eliminating the distribution format means either teaching the
    package tools how to deal with a split archive format (they currently 
    do not) or divorcing ourselves forever from floppies as a distribution
    medium."

I think that today, pkg_add is quite capable of coping with split archives:

# cat /cdrom/base/base.?? | pkg_add -

(Replace 'cat' with sysinstall, which will prompt you when it's time to
change floppies)

   "Second, there's the issue of packages currently requiring temporary
    space as part of their extraction method.  If we're going to have
    things like "bin" be a package, even if we split it up into
    subcomponents and make "bin" simply a package which contains a list of
    dependencies and nothing more (which is desirable), there are still   
    going to be pieces which are non-extractable under the current scheme
    because the available disk space is too small to contain both the
    temporary copy and the final installed copy, which may not be on the
    same file system and cannot be simply moved into place."

Do the package tools still unpack into a temporary area before installing?

If they do, I think libarchive could solve this. AFAICS, the main problem is
that we need to extract files like +CONTENTS containing the _at_cwd directive
before we know where to unpack the rest of the files. At worst, we could
scan through the archive once, extract the +XXX files, rewind and scan again
to extract; at best we could require that these +XXX files always occur at
the start (as they seem to), in which case you can extract those files into
a temporary area, process them, and then continue to unpack the rest. It
would mean not having to switch to an alternative format, such as 'zip' as
Jordan suggests.

Why am I raising this? Using the pkg_add format for the base system could
deliver some immediate benefits.

(1) You could tell which choice of distribution sets you made at install
time, just by doing 'pkg_info'

freebsd-base-5.3	Binary base distribution (required)
freebsd-compat4x-5.3	FreeBSD 4.x binary compatibility
freebsd-dict-5.3	Spelling checker dictionary files
freebsd-doc-5.3		Miscellaneous FreeBSD online docs
freebsd-man-5.3		System manual pages - recommended
freebsd-src-5.3		Sources for everything

(2) The binary upgrade process can therefore choose automatically which
distribution sets need to be upgraded.

(3) Binary updates can *remove* files which were in the previous
distribution, but not the upgraded one. /etc/rc.d/lomac is an example of
this (a file which was in 5.3 but which needs to vanish for 5.4). Another
example is /usr/src: at the moment you can't safely just install the new
source package, because stale source files are left around. If the source
came as a package, then installing the new source package could also remove
files which belonged to the old source package only.

(4) An MD5 checksum of the 'old' version of every file would be maintained
in the package database, as _at_comment MD5:.......

Hence binary update of startup scripts (/etc/rc.d/*) and configuration files
(/etc/*) can safely and automatically replace those scripts if you have not
touched them.

If you *have* touched them but the RCS IDs are the same, they can be left
alone. Otherwise, a report can be made of differences which must be resolved
manually.

(5) It would allow the base distribution to be divided into more
fine-grained components easily. In particular, I would like to see the
development tools made a separate package (they account more more than half
the base system); separating out the text processing tools, bind and
sendmail would also be useful. This would mean you could easily have a
fully-featured "runtime" FreeBSD system for use on compact flash cards, for
example, without the overhead of things like the Fortran 77 compiler :-) It
could be made much more fine-grained than that, of course.

Some of the desirable facilities may not exist in the package system yet:
for example I don't know if you can tag a file as being a 'config file' such
that it can be automatically replaced if it was untouched. If not, this
could be added later. (If too many features like this need to be added, then
one might argue we should drop the FreeBSD pkg format and use an existing
one which has all the features required - even, dare I say it, RPM. But
that's a somewhat separate question)

There are of course remaining issues to do with interactive package
configuration which Jordan also raises. But would this not be a useful step
in the right direction?

The binary upgrade process is, IMO, the weakest part of FreeBSD at present.

Regards,

Brian.