Re: [CFT] packaging the base system with pkg(8)

From: Slawa Olhovchenkov <slw_at_zxy.spb.ru>
Date: Tue, 8 Mar 2016 18:14:59 +0300
On Tue, Mar 08, 2016 at 01:18:47PM +0000, Glen Barber wrote:

> On Tue, Mar 08, 2016 at 03:40:16PM +0300, Slawa Olhovchenkov wrote:
> > On Wed, Mar 02, 2016 at 11:54:29PM +0000, Glen Barber wrote:
> > 
> > > To obtain the sources for testing, please use the projects/release-pkg
> > > branch:
> > > 
> > >  # svn co svn://svn.freebsd.org/base/projects/release-pkg /usr/src
> > > 
> > > The projects/release-pkg branch is (at this time) in sync with head
> > > revision r296327.
> > > 
> > > After checking out the project branch, build the userland and kernel as
> > > normal with the 'buildworld' and 'buildkernel' targets.  Afterward,
> > > packages can be created with the 'packages' target.
> > > 
> > >  # cd /usr/src
> > >  # make [make flags] buildworld
> > >  # make [make flags] buildkernel
> > >  # make packages
> > > 
> > > At present, the base system consists of 755 packages with the default
> > > build (empty src.conf(5) and make.conf(5)) for amd64.  The number of
> > > packages depends on several factors, but for most cases a runtime binary
> > > is split into several components.  In particular, most shared libraries
> > > are individually packaged, in addition to debugging symbols, profiling
> > > libraries, and 32-bit packaged separately.
> > > 
> > > The package repository will be created within /usr/obj/usr/src/repo by
> > > default.
> > 
> > I am get snapshot .iso for install test setup in VirtualBox and using
> > projects/release-pkg for sources. After make buildworld buildkernel
> > packages and pkg install '*' I am have some words.
> > 
> > I am not developer, I am like maintenance services. I am do maintaing
> > systems more ten 20 years, some systems maintainig more ten 10 year
> > continuous, some systems got for maintenance after years unmaintening.
> > All of this give some requirement and vision different from
> > developers. Please, do not reject this!
> > 
> > First, you do collocal work, thanks!
> > 
> > I am don't check all, but already found some stranges:
> > 
> > Package FreeBSD-clibs-development contain /usr/lib/libthr.so,
> > /usr/lib/libedit.so and etc (and same in other packages).
> > 
> 
> Correct.  The libraries in the 'clibs' package must be installed before
> the rest of the system can be safely upgraded, to ensure consistency
> with the core runtime libraries.

You are miss may point: FreeBSD-clibs-development named as optional
package, need for writing some programs (like in linuxes (and I am hate
this spliting)) but contain /usr/lib/libthr.so and /usr/lib/libedit.so
that must be part of runtime/clibs.

> > I am reseach spliting to package and try some aggregation:
> > 
> > NumPkgs  tarSize(MB)  flatSize(MB)  Aggregation
> > 1          30.7            102      FreeBSD-kernel-generic-release
> > 1          57.1            331      FreeBSD-kernel-generic-debug
> > 1           2.8              5.4    FreeBSD-clibs
> > 1	    3.5		    24	    FreeBSD-clibs-development
> > 1           2.4		    11	    FreeBSD-clibs-debug
> > 1	    1.3		     9.8    FreeBSD-clibs-profile
> > 1          20.7		   103	    FreeBSD-runtime
> > 1	    5.9		    38.1    FreeBSD-development
> > 1	    2.9		     2.8    FreeBSD-runtime-manuals
> > 1	   14.9		    65	    FreeBSD-debug
> > 1	    2.2		    12.5    FreeBSD-profile
> > 1	   24.3		    93	    FreeBSD-clang
> > 1	    8.7		    66	    FreeBSD-clang-debug
> > 116	   19.0		    80	    FreeBSD-*
> > 89	    3.2		    14	    FreeBSD-*-development
> > 110	   12.5		    61	    FreeBSD-*-debug
> > 85	    2.8		    13	    FreeBSD-*-profile
> > 85	    6.0		    18	    FreeBSD-*-lib32-*
> > 88	    7.4		    30	    FreeBSD-*-lib32-development
> > 84	   11.6		    43	    FreeBSD-*-lib32-debug
> > 85	    5.8		    24	    FreeBSD-*-lib32-profile
> > 
> > I.e -development is substantially less of main package and don't need
> > separatly (and many .so incorrectly packaging into -development).
> > Same as -profile vs -debug (and -profile useless w/o -debug).
> > 
> 
> Regarding the .so files, I am not clear on the original intent behind
> separating the actual shared library from the installed symbolic link to
> the real shared library, but in my investigation into this, only the
> symlinks are provided by the '-development' package.

OK, this is may fault about incorrect size estimation, but why we miss
this links w/o installing '-development' packages?

> Regarding '-profile' and '-debug' package separation, it is possible to
> install the debugging files without requiring the profiling libraries
> now, so I think keeping them as separate packages is the best way to
> achieve this.  (Note, profiling libraries will not be installed with
> WITHOUT_PROFILE=1 in src.conf(5)).

Yes, I undertund this. But what profit of this? Addtional size is
small, many small packages is bad. We already have expirense with
spliting Xorg to many small packages -- no profit of this.

> > Manual must be installed always, IMHO (size is small and version of
> > manual must matcj version of utility).
> > 
> 
> This is similarly broken up by package, at least where it has been found
> so far.  The jail.conf(5) manual is only installed if the 'FreeBSD-jail'
> package is installed, for example.

Also, no profit of this. We need long selection process from dozen
packages at install time and have (in world) many different
installations with random absenses different utilites. This is bad for
me. Take for maintance some random system I am expect to got standart
FreeBSD, w/o random cutting. I am don't need auditing for checking
installed or outdating 800 packages! When I am need starting jail I am
don't need to search where proper version of absent package and how
this package naming. For outdated system especially.

> > Packaging of individual utilites is useless (total 19MB vs
> > 30.7+2.8+20.7+2.9) and incorrect (for example, WITHOUT_ACCT not only
> > don't build accton/lastcomm/sa but also cut off accaunting code from
> > kernel for space saving and perforamce).
> > 
> 
> Packaging individual utilities is not useless, depending on who you ask.
> One of the first replies I received when starting separating userland
> utilities into separate packages was further splitting rwho(1) and
> rwhod(8) into different packages, the use case being not necessarily
> needing (or wanting) the rwho(1) utility on systems where rwhod(8) runs.

Best way for this -- custom repos builded from custom src.conf, IMHO.

> > I am propose don't distinct profile and debug, development and main
> > package.
> > 
> > I am propose divide only to FreeBSD-kernel, FreeBSD-clibs (clibs,
> > runtime and manuals), FreeBSD-clang, FreeBSD-lib32.
> > 
> 
> This will make updates for SAs and ENs too large.  This is part of the
> reason the packages were split up as they have been so far (and will
> continue to be further split as progress is made).

What you vision of mainline lifecycle of FreeBSD install system?
Install only from official RELEASE media, update only to -RELENG
branch from official repo?

> If the argument is simply "there are too many packages", see one of the
> previous replies in this thread that discusses the background on why
> this decision was made.

May be I miss this.

> But that aside, trying to make everyone happy will turn out to make no
> one happy.

Yes. We need to select some mainline lifecycle.

> > Dividing to many packages is anoyning on install and maintancing (what
> > exact keys of this utilites this version?! stupid admin don't install
> > manuals!)
> > 
> > About use cases. I am try to imagine different use cases and don't
> > found answer how do this:
> > 
> > 1. package building as `make packages` witch version as timestamp of
> > start buildworld. I.e. on every buildworld every package will be
> > rebuild, take new version and will be reinstaled. Where is profit of
> > package spliting?
> > 
> 
> This is the case for 11-CURRENT.  The PKG_VERSION evaluates the BRANCH
> from Makefile.inc1 to determine if it should use the timestamp.  (Since
> -CURRENT is a fast-moving target, and we recommend keeping the userland
> and kernel in sync, this makes sense, at least to me.)

What about -stable?

> > 2. After src.conf change some package don't build. Where analog of
> > `make delete-old delete-old-libs`?
> > 
> 
> I believe 'pkg autoremove' should handle this, but I will check.

No. This is only removing unneeding depends. Not manual installed packages.

> > 3. After src.conf chanege some (WITHOUT_ACCT for example) some
> > packages can't be installed. How handle this?
> > 
> 
> Have you run into this?  If so, it needs to be investigated.  But
> otherwise, the generated packages respect make.conf(5) and src.conf(5),
> so this should not happen in theory.

1. Build packages with empty src.conf
2. install all w/o FreeBSD-acct.
3. Rebuild witch WITHOUT_ACCT in src.conf
4. upgrade
5. try to install old FreeBSD-acct.

> > 4. How install debug symbols after installing separately set of
> > packages? Not all *-debug*- and don't selecting all 200 packages
> > individualy?
> > 
> 
> Could you clarify what you mean a bit more?  Specifically, I am unclear
> of the order of events you mean.

1. Install FreeBSD-acct and FreeBSD-runtime.
2. Try to install -debug only for installed packages (i.e. only
FreeBSD-acct-debug and FreeBSD-runtime-debug, w/o FreeBSD-hast-debug
and etc.

> > 5. Take system installed by unknow person (ex: ISP support 5 years
> > ago). Try to write program. Don't find nothing for this. Version is
> > 11.0-BETA3. How to install required packages? For system w/o inetrnet
> > connectivity and with lost install media?
> > 
> 
> The plan at the moment is to include pkg(8) in the repository with the
> base system.  This will also be required for architectures we do not
> have an upstream package repository (powerpc, for example).  This is
> handled by the 'package-pkg' target at the moment.

I am talk about preserving install packeges on install target, not
about source repos.

> > As for windows (please found in garbage CD with exact version and
> > insert in lost CD-ROM)?
> > 
> > May be preserve for this all packages in some places on HDD?
> > 
> 
> I am having trouble parsing these two sentences.  Could you please
> clarify?

Take for maintace some system, installed from lost install media.
You need adding some missed packages. Where you find it?
Some version don't presering on FreeBSD ftp (11.0-BETA3, for example).
Preserving all packaes on install target can be help.

> > 6.  pkg which /etc/ssh/ssh_config
> > /etc/ssh/ssh_config was not found in the database
> > 
> 
> There are may occurrences of this at the moment.  Configuration file
> merging needs to be resolved in pkg(8) before we can safely add files
> like sshd_config(8) to the 'FreeBSD-ssh' package.

> > What?! How to handle this? What is instead of mergemaster?
> > (etcupdate currently to dangerous, I am totaly destroed one host by
> > etcupdate. Also, current database of etcupdate is very strange).
> 
> We use etcupdate(8) in the FreeBSD infrastructure, and have not run into
> such problems.

You just don't fault in commandline. I am type etcupdate on fresh
system and lost all configs, /var/tmp and etc.
Received on Tue Mar 08 2016 - 14:15:04 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:03 UTC