Re: [CFT] packaging the base system with pkg(8)

From: Glen Barber <gjb_at_FreeBSD.org>
Date: Tue, 8 Mar 2016 13:18:47 +0000
On Tue, Mar 08, 2016 at 03:40:16PM +0300, Slawa Olhovchenkov wrote:
> On Wed, Mar 02, 2016 at 11:54:29PM +0000, Glen Barber wrote:
> 
> > To obtain the sources for testing, please use the projects/release-pkg
> > branch:
> > 
> >  # svn co svn://svn.freebsd.org/base/projects/release-pkg /usr/src
> > 
> > The projects/release-pkg branch is (at this time) in sync with head
> > revision r296327.
> > 
> > After checking out the project branch, build the userland and kernel as
> > normal with the 'buildworld' and 'buildkernel' targets.  Afterward,
> > packages can be created with the 'packages' target.
> > 
> >  # cd /usr/src
> >  # make [make flags] buildworld
> >  # make [make flags] buildkernel
> >  # make packages
> > 
> > At present, the base system consists of 755 packages with the default
> > build (empty src.conf(5) and make.conf(5)) for amd64.  The number of
> > packages depends on several factors, but for most cases a runtime binary
> > is split into several components.  In particular, most shared libraries
> > are individually packaged, in addition to debugging symbols, profiling
> > libraries, and 32-bit packaged separately.
> > 
> > The package repository will be created within /usr/obj/usr/src/repo by
> > default.
> 
> I am get snapshot .iso for install test setup in VirtualBox and using
> projects/release-pkg for sources. After make buildworld buildkernel
> packages and pkg install '*' I am have some words.
> 
> I am not developer, I am like maintenance services. I am do maintaing
> systems more ten 20 years, some systems maintainig more ten 10 year
> continuous, some systems got for maintenance after years unmaintening.
> All of this give some requirement and vision different from
> developers. Please, do not reject this!
> 
> First, you do collocal work, thanks!
> 
> I am don't check all, but already found some stranges:
> 
> Package FreeBSD-clibs-development contain /usr/lib/libthr.so,
> /usr/lib/libedit.so and etc (and same in other packages).
> 

Correct.  The libraries in the 'clibs' package must be installed before
the rest of the system can be safely upgraded, to ensure consistency
with the core runtime libraries.

> Misspeling FreeBSD-debug, FreeBSD-development and FreeBSD-profile as
> FreeBSD-runtime-debug, FreeBSD-runtime-development and FreeBSD-runtime-profile?
> 

I am not sure why this was originally named as they are, but this can be
fixed.

> I am reseach spliting to package and try some aggregation:
> 
> NumPkgs  tarSize(MB)  flatSize(MB)  Aggregation
> 1          30.7            102      FreeBSD-kernel-generic-release
> 1          57.1            331      FreeBSD-kernel-generic-debug
> 1           2.8              5.4    FreeBSD-clibs
> 1	    3.5		    24	    FreeBSD-clibs-development
> 1           2.4		    11	    FreeBSD-clibs-debug
> 1	    1.3		     9.8    FreeBSD-clibs-profile
> 1          20.7		   103	    FreeBSD-runtime
> 1	    5.9		    38.1    FreeBSD-development
> 1	    2.9		     2.8    FreeBSD-runtime-manuals
> 1	   14.9		    65	    FreeBSD-debug
> 1	    2.2		    12.5    FreeBSD-profile
> 1	   24.3		    93	    FreeBSD-clang
> 1	    8.7		    66	    FreeBSD-clang-debug
> 116	   19.0		    80	    FreeBSD-*
> 89	    3.2		    14	    FreeBSD-*-development
> 110	   12.5		    61	    FreeBSD-*-debug
> 85	    2.8		    13	    FreeBSD-*-profile
> 85	    6.0		    18	    FreeBSD-*-lib32-*
> 88	    7.4		    30	    FreeBSD-*-lib32-development
> 84	   11.6		    43	    FreeBSD-*-lib32-debug
> 85	    5.8		    24	    FreeBSD-*-lib32-profile
> 
> I.e -development is substantially less of main package and don't need
> separatly (and many .so incorrectly packaging into -development).
> Same as -profile vs -debug (and -profile useless w/o -debug).
> 

Regarding the .so files, I am not clear on the original intent behind
separating the actual shared library from the installed symbolic link to
the real shared library, but in my investigation into this, only the
symlinks are provided by the '-development' package.

For example:

 root_at_pkgbase:/ # file /usr/lib/libjail.so
 /usr/lib/libjail.so: symbolic link to ../../lib/libjail.so.1
 root_at_pkgbase:/ # pkg shlib libjail.so
 No packages provide libjail.so.
 No packages require libjail.so.
 root_at_pkgbase:/ # pkg shlib libjail.so.1
 libjail.so.1 is provided by the following packages:
 FreeBSD-jail-11.0.s20160303044402
 FreeBSD-jail-lib32-11.0.s20160303044402
 libjail.so.1 is linked to by the following packages:
 FreeBSD-runtime-11.0.s20160303044402

Moving them to the package that installs the shared library itself
should be fairly easy to do.

Regarding '-profile' and '-debug' package separation, it is possible to
install the debugging files without requiring the profiling libraries
now, so I think keeping them as separate packages is the best way to
achieve this.  (Note, profiling libraries will not be installed with
WITHOUT_PROFILE=1 in src.conf(5)).

> Manual must be installed always, IMHO (size is small and version of
> manual must matcj version of utility).
> 

This is similarly broken up by package, at least where it has been found
so far.  The jail.conf(5) manual is only installed if the 'FreeBSD-jail'
package is installed, for example.

 root_at_pkgbase:/ # pkg which /usr/share/man/man5/jail.conf.5.gz
 /usr/share/man/man5/jail.conf.5.gz was installed by package
   FreeBSD-jail-11.0.s20160303044402

> Packaging of individual utilites is useless (total 19MB vs
> 30.7+2.8+20.7+2.9) and incorrect (for example, WITHOUT_ACCT not only
> don't build accton/lastcomm/sa but also cut off accaunting code from
> kernel for space saving and perforamce).
> 

Packaging individual utilities is not useless, depending on who you ask.
One of the first replies I received when starting separating userland
utilities into separate packages was further splitting rwho(1) and
rwhod(8) into different packages, the use case being not necessarily
needing (or wanting) the rwho(1) utility on systems where rwhod(8) runs.

> I am propose don't distinct profile and debug, development and main
> package.
> 
> I am propose divide only to FreeBSD-kernel, FreeBSD-clibs (clibs,
> runtime and manuals), FreeBSD-clang, FreeBSD-lib32.
> 

This will make updates for SAs and ENs too large.  This is part of the
reason the packages were split up as they have been so far (and will
continue to be further split as progress is made).

If the argument is simply "there are too many packages", see one of the
previous replies in this thread that discusses the background on why
this decision was made.

But that aside, trying to make everyone happy will turn out to make no
one happy.

> Dividing to many packages is anoyning on install and maintancing (what
> exact keys of this utilites this version?! stupid admin don't install
> manuals!)
> 
> About use cases. I am try to imagine different use cases and don't
> found answer how do this:
> 
> 1. package building as `make packages` witch version as timestamp of
> start buildworld. I.e. on every buildworld every package will be
> rebuild, take new version and will be reinstaled. Where is profit of
> package spliting?
> 

This is the case for 11-CURRENT.  The PKG_VERSION evaluates the BRANCH
from Makefile.inc1 to determine if it should use the timestamp.  (Since
-CURRENT is a fast-moving target, and we recommend keeping the userland
and kernel in sync, this makes sense, at least to me.)

> 2. After src.conf change some package don't build. Where analog of
> `make delete-old delete-old-libs`?
> 

I believe 'pkg autoremove' should handle this, but I will check.

> 3. After src.conf chanege some (WITHOUT_ACCT for example) some
> packages can't be installed. How handle this?
> 

Have you run into this?  If so, it needs to be investigated.  But
otherwise, the generated packages respect make.conf(5) and src.conf(5),
so this should not happen in theory.

> 4. How install debug symbols after installing separately set of
> packages? Not all *-debug*- and don't selecting all 200 packages
> individualy?
> 

Could you clarify what you mean a bit more?  Specifically, I am unclear
of the order of events you mean.

> 5. Take system installed by unknow person (ex: ISP support 5 years
> ago). Try to write program. Don't find nothing for this. Version is
> 11.0-BETA3. How to install required packages? For system w/o inetrnet
> connectivity and with lost install media?
> 

The plan at the moment is to include pkg(8) in the repository with the
base system.  This will also be required for architectures we do not
have an upstream package repository (powerpc, for example).  This is
handled by the 'package-pkg' target at the moment.

> As for windows (please found in garbage CD with exact version and
> insert in lost CD-ROM)?
> 
> May be preserve for this all packages in some places on HDD?
> 

I am having trouble parsing these two sentences.  Could you please
clarify?

> 6.  pkg which /etc/ssh/ssh_config
> /etc/ssh/ssh_config was not found in the database
> 

There are may occurrences of this at the moment.  Configuration file
merging needs to be resolved in pkg(8) before we can safely add files
like sshd_config(8) to the 'FreeBSD-ssh' package.

> What?! How to handle this? What is instead of mergemaster?
> (etcupdate currently to dangerous, I am totaly destroed one host by
> etcupdate. Also, current database of etcupdate is very strange).

We use etcupdate(8) in the FreeBSD infrastructure, and have not run into
such problems.

Glen


Received on Tue Mar 08 2016 - 12:18:49 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:03 UTC