Re: [HEADSUP] No more pkg_install on HEAD by default

On 14/07/2013 11:12, Teske, Devin wrote:

> Interesting. I notice that (while looking ahead to see a prefix: of
> /usr/local in the +MANIFEST), the tarball itself has files that include
> /usr/local in their path.

Yes -- we consider the $PREFIX to be 'baked into' the package at compile
time.  You can relocate a package to some extent at installation time --
so for instance you can install into a jail or chroot from outside it --
but the software would expect all the files to be arranged as shown in
the package at runtime.

> Does pkg handle packages where the prefix (in +MANIFEST) is
> "/usr/local" _but_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"?
> (e.g. pkg_install style) or is this the new way going forward?

No -- it always wants a fully qualified path there.

> So just to give you a better idea of how we manage packages here.
> 
> We've realized that if you want to package for FreeBSD (in 9 and
> older), you could get by alright if you knew how to create a
> +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now
> for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both
> in-fact YAML).
> 
> So rather than teach the people here how to use tools, I taught them
> what I think is more important... how to manage +CONTENTS, SPECFILE,
> and now up-and-coming, +MANIFEST.
> 
> However, I'd be lying if I said I taught them *all* the gory details.
> In reality, for +CONTENTS they edit a PLIST which is essentially
> +CONTENTS without the "_at_comment MD5:" entries. For SPECFILE, they
> manage the full thing except there's a small section that is the
> standard RPM bootstrap that is labeled as "do not touch".
> 
> From what you posted of +COMPACT_MANIFEST was nice, except it lacks
> the list of files.

There's actually two lists: files and directories.  The principle
difference being that the files list contains SHA256 checksums of each
file.  (Although the files list does also contain sym-links etc, which
don't have a checksum).

There are other data in the full manifest but not in the compact one --
pre- and post- installation or deinstallation scripts are the most
important bits not yet mentioned.

> The basic principle here is that if you have to maintain a list of
> files, and that list ends up being compiled into a +MANIFEST, why not
> just teach your peers how to read/write/maintain +MANIFEST files (in
> version control of course) so that there are never any mysteries as
> to how the package performs.
> 
> I know this sounds screwy... but (as with our other YAML files), it's
> really nice because (as is the case with SPECFILE), we
> version-control things as-close to the end-product as possible (take
> for example the case of "pre-install" or "post-install" et cetera
> scripts -- I'm sure a tool like "pkg create" or "pkg_create" or other
> could do a good job of assembling the pieces into the +MANIFEST, but
> then it's not being version-controlled as closely.

Actually, 'pkg create' basically takes the manifest and uses it to build
the package.  It has a special mode to combine data as presented during
ports compilation into a manifest -- again, this is a compatibility
thing.  If the ports didn't have to support pkg_tools still we could
generate package metadata more directly in the format needed by the
manifest.

> Before this workflow, mistakes were rampant and there wasn't much
> hope of wrangling the "oops, forgot to touch the PLIST" or "oops,
> forgot to update the post-install script" mistakes.
> 
> I see a nice bright future in re-working my "pkgbase" to be able to
> supplant metadata into a revision-controlled +MANIFEST.
> 
> Ideally, I don't want them to have to be burdened with maintaining
> both a +MANIFEST and +COMPACT_MANIFEST, so it seems most logical to
> generate one from the other (the latter being the revision-controlled
> copy sans-meta-data; the meta-data is added at "make" time before
> then running validations and generating the tarball).

Internally, the +COMPACT_MANIFEST is generated from the +MANIFEST --
probably best to think in terms of dividing the manifest into sections,
some of which (lists of files, install scripts) your dev people would
actively modify, some of which can be automatically generated by pkg(8)
(shlib provides/requires tracking[*]) and the rest as static boilerplate
you can just paste together as required to create a manifest.

[*] Actually, this is automatically generated at the time the package is
assembled into a tarball.  You don't need to fill in that section of the
manifest eplicitly.

> The problem of breakage to the system by API changes is less
> important, because we track security releng branches and use binaries
> so changes are very slow to creep in. But since *can* use a different
> framework for each/every major release branch, we can track API
> changes quite easily.

Ah.  Remember that pkg(8) isn't part of the base system and is developed
on a timetable independent of the timetable used by FreeBSD itself.
When we release a new version of pkg, it would be applied simultaneously
on all branches and supported releases.  Of course, if you have your own
private repository system you can control the version of pkg supplied by it.

> I'm aware that only slow moving dinosaurs like us are interested in
> such backward-thinking mechanics... but this stuff really works for
> us, and I'm certain that digesting +MANIFEST will be very beneficial
> to our growth.

One of our aims when developing pkg(8) has been to make it possible to
have as much or as little interaction with the official FreeBSD package
building clusters as you want.  You could use official packages
exclusively; or you could create a repo containing just the software
packages crucial to you with local customizations, and install the rest
from the official repos; or you can build everything yourself.  Finally,
it should be possible for companies or software projects to release just
their own software in pkg(8) format, and for an administrator to select
that for installation on their system rather than the default version
(if any) from ports.

	Cheers,

	Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey