Re: [HEADSUP] No more pkg_install on HEAD by default

On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote:

> On 14/07/2013 06:48, Teske, Devin wrote:
>> Question: Where can I learn more about the actual format of what's in
>> the new tarballs? This is going to be important not for bsdconfig,
>> but $work (we have our own build platform; I'm going to have to
>> rewrite it from mastering PLIST files to mastering YAML MANIFEST
>> files and I want to know all the gritty details).
> 
> We do need a pkg-manifest(5) man page, which can double as a
> pkg-tarball(5) page since the manifest file is where most of the
> interesting bits are.
> 
> A pkg tarball is a compressed tar archive like so:
> 
> lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz
> -rw-r--r--  0 root   wheel     530 Jan  1  1970 +COMPACT_MANIFEST
> -rw-r--r--  0 root   wheel    6385 Jan  1  1970 +MANIFEST
> -rw-r--r--  0 root   wheel   17567 Jan  1  1970 +MTREE_DIRS
> -r--r--r--  0 root   wheel   19453 Jul  7 12:26
> /usr/local/etc/bash_completion.d/_pkg.bash
> -r-xr-xr-x  0 root   wheel     629 Jul  7 12:26
> /usr/local/etc/periodic/daily/400.status-pkg
> -r-xr-xr-x  0 root   wheel     823 Jul  7 12:26
> /usr/local/etc/periodic/daily/411.pkg-backup
> -r-xr-xr-x  0 root   wheel     678 Jul  7 12:26
> /usr/local/etc/periodic/daily/490.status-pkg-changes
> -r-xr-xr-x  0 root   wheel    2558 Jul  7 12:26
> /usr/local/etc/periodic/security/410.pkg-audit
> -r-xr-xr-x  0 root   wheel     611 Jul  7 12:26
> /usr/local/etc/periodic/security/460.pkg-checksum
> -r--r--r--  0 root   wheel     839 Jul  7 12:26
> /usr/local/etc/pkg.conf.sample
> -r--r--r--  0 root   wheel   43432 Jul  7 12:26 /usr/local/include/pkg.h
> -r--r--r--  0 root   wheel  727558 Jul  7 12:26 /usr/local/lib/libpkg.a
> lrwxr-xr-x  0 root   wheel       0 Jul  7 12:26 /usr/local/lib/libpkg.so
> -> libpkg.so.1
> -r--r--r--  0 root   wheel 1227064 Jul  7 12:26 /usr/local/lib/libpkg.so.1
> -rw-r--r--  0 root   wheel     187 Jul  7 12:26
> /usr/local/libdata/pkgconfig/pkg.pc
> [... etc ...]
> 

Interesting. I notice that (while looking ahead to see a prefix: of /usr/local in the +MANIFEST), the tarball itself has files that include /usr/local in their path.

Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _but_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install style) or is this the new way going forward?



> There must at least be a +MANIFEST -- other meta data files are
> optional.  +COMPACT_MANIFEST is a subset of the full +MANIFEST.  They're
> both YAML documents.  +COMPACT_MANIFEST looks like this, for example:
> 
> ---
> name: pkg
> version: 1.1.4
> origin: ports-mgmt/pkg
> comment: New generation package manager
> arch: freebsd:9:x86:64
> www: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d
> maintainer: portmgr_at_FreeBSD.org
> prefix: /usr/local
> licenselogic: single
> licenses:
> - BSD
> flatsize: 6311507
> desc: |
>  New Generation package management tool for FreeBSD
> 
>  WWW: http://wiki.freebsd.org/pkgng
> categories:
> - ports-mgmt
> shlibs_required:
> - libpkg.so.1
> shlibs_provided:
> - libpkg.so.1
> message: |
>  If you are upgrading from the old package format, first run:
> 
>    # pkg2ng
> 

Nice. Very nice.



> +MTREE_DIRS is a compatibility thing with the old pkg_tools.  It's not
> needed in general as +MANIFEST can provide all that meta data itself.
> It isn't going to be deprecated for at least as long as the ports tree
> continues to support pkg_tools though.
> 
> Beyond that, the rest of the pkg tarball just contains a tar archive of
> all the files, directories, sym-links etc to be installed by the
> package.  Note that pkg(8) has no problem with creating an empty
> directory for a package, unlike pkg_tools.
> 

Excellent!


> Now, while you can grovel through the details of pkg tarballs and
> internal data formats like this, be aware: the format of the manifest
> and the details of the meta-data included in the pkg-tarballs is subject
> to change without warning as we develop pkg(8) further.  We only promise
> API stability through the pkg(8) commands or for the libpkg.so library
> functions; reports of breakage from usage outside those APIs will
> receive little sympathy.
> 

Understood.

So just to give you a better idea of how we manage packages here.

We've realized that if you want to package for FreeBSD (in 9 and older), you could get by alright if you knew how to create a +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both in-fact YAML).

So rather than teach the people here how to use tools, I taught them what I think is more important... how to manage +CONTENTS, SPECFILE, and now up-and-coming, +MANIFEST.

However, I'd be lying if I said I taught them *all* the gory details. In reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS without the "_at_comment MD5:" entries. For SPECFILE, they manage the full thing except there's a small section that is the standard RPM bootstrap that is labeled as "do not touch".

>From what you posted of +COMPACT_MANIFEST was nice, except it lacks the list of files.

The basic principle here is that if you have to maintain a list of files, and that list ends up being compiled into a +MANIFEST, why not just teach your peers how to read/write/maintain +MANIFEST files (in version control of course) so that there are never any mysteries as to how the package performs.

I know this sounds screwy... but (as with our other YAML files), it's really nice because (as is the case with SPECFILE), we version-control things as-close to the end-product as possible (take for example the case of "pre-install" or "post-install" et cetera scripts -- I'm sure a tool like "pkg create" or "pkg_create" or other could do a good job of assembling the pieces into the +MANIFEST, but then it's not being version-controlled as closely.

Before this workflow, mistakes were rampant and there wasn't much hope of wrangling the "oops, forgot to touch the PLIST" or "oops, forgot to update the post-install script" mistakes.

I see a nice bright future in re-working my "pkgbase" to be able to supplant metadata into a revision-controlled +MANIFEST.

Ideally, I don't want them to have to be burdened with maintaining both a +MANIFEST and +COMPACT_MANIFEST, so it seems most logical to generate one from the other (the latter being the revision-controlled copy sans-meta-data; the meta-data is added at "make" time before then running validations and generating the tarball).

The problem of breakage to the system by API changes is less important, because we track security releng branches and use binaries so changes are very slow to creep in. But since *can* use a different framework for each/every major release branch, we can track API changes quite easily.

I'm aware that only slow moving dinosaurs like us are interested in such backward-thinking mechanics... but this stuff really works for us, and I'm certain that digesting +MANIFEST will be very beneficial to our growth.
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.