On Jul 14, 2013, at 11:23 PM, Baptiste Daroussin wrote: On Sun, Jul 14, 2013 at 10:12:19AM +0000, Teske, Devin wrote: On Jul 14, 2013, at 2:30 AM, Matthew Seaman wrote: On 14/07/2013 06:48, Teske, Devin wrote: Question: Where can I learn more about the actual format of what's in the new tarballs? This is going to be important not for bsdconfig, but $work (we have our own build platform; I'm going to have to rewrite it from mastering PLIST files to mastering YAML MANIFEST files and I want to know all the gritty details). We do need a pkg-manifest(5) man page, which can double as a pkg-tarball(5) page since the manifest file is where most of the interesting bits are. A pkg tarball is a compressed tar archive like so: lucid-nonsense:...cache/pkg/All:% tar -tvf pkg-1.1.4.txz -rw-r--r-- 0 root wheel 530 Jan 1 1970 +COMPACT_MANIFEST -rw-r--r-- 0 root wheel 6385 Jan 1 1970 +MANIFEST -rw-r--r-- 0 root wheel 17567 Jan 1 1970 +MTREE_DIRS -r--r--r-- 0 root wheel 19453 Jul 7 12:26 /usr/local/etc/bash_completion.d/_pkg.bash -r-xr-xr-x 0 root wheel 629 Jul 7 12:26 /usr/local/etc/periodic/daily/400.status-pkg -r-xr-xr-x 0 root wheel 823 Jul 7 12:26 /usr/local/etc/periodic/daily/411.pkg-backup -r-xr-xr-x 0 root wheel 678 Jul 7 12:26 /usr/local/etc/periodic/daily/490.status-pkg-changes -r-xr-xr-x 0 root wheel 2558 Jul 7 12:26 /usr/local/etc/periodic/security/410.pkg-audit -r-xr-xr-x 0 root wheel 611 Jul 7 12:26 /usr/local/etc/periodic/security/460.pkg-checksum -r--r--r-- 0 root wheel 839 Jul 7 12:26 /usr/local/etc/pkg.conf.sample -r--r--r-- 0 root wheel 43432 Jul 7 12:26 /usr/local/include/pkg.h -r--r--r-- 0 root wheel 727558 Jul 7 12:26 /usr/local/lib/libpkg.a lrwxr-xr-x 0 root wheel 0 Jul 7 12:26 /usr/local/lib/libpkg.so -> libpkg.so.1 -r--r--r-- 0 root wheel 1227064 Jul 7 12:26 /usr/local/lib/libpkg.so.1 -rw-r--r-- 0 root wheel 187 Jul 7 12:26 /usr/local/libdata/pkgconfig/pkg.pc [... etc ...] Interesting. I notice that (while looking ahead to see a prefix: of /usr/local in the +MANIFEST), the tarball itself has files that include /usr/local in their path. Does pkg handle packages where the prefix (in +MANIFEST) is "/usr/local" _but_ "tar tf" of the unxz'd tarball is _not_ "/usr/local"? (e.g. pkg_install style) or is this the new way going forward? There must at least be a +MANIFEST -- other meta data files are optional. +COMPACT_MANIFEST is a subset of the full +MANIFEST. They're both YAML documents. +COMPACT_MANIFEST looks like this, for example: --- name: pkg version: 1.1.4 origin: ports-mgmt/pkg comment: New generation package manager arch: freebsd:9:x86:64 www: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=LPoff3ddLDoHE30AU6R5vgBklqxsjVAosXGjJop6uO4%3D%0A&s=30b27cf33d7e594144fc38f20bc75d84464788e715f7fc82b2495f813fd4be2d maintainer: portmgr_at_FreeBSD.org<mailto:portmgr_at_FreeBSD.org> prefix: /usr/local licenselogic: single licenses: - BSD flatsize: 6311507 desc: | New Generation package management tool for FreeBSD WWW: https://urldefense.proofpoint.com/v1/url?u=http://wiki.freebsd.org/pkgng&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=cLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0A&s=cef93a5fe98a3fa0fbc9df3e0d11c9bbeaee44290306834ddef30f2ba925446c categories: - ports-mgmt shlibs_required: - libpkg.so.1 shlibs_provided: - libpkg.so.1 message: | If you are upgrading from the old package format, first run: # pkg2ng Nice. Very nice. +MTREE_DIRS is a compatibility thing with the old pkg_tools. It's not needed in general as +MANIFEST can provide all that meta data itself. It isn't going to be deprecated for at least as long as the ports tree continues to support pkg_tools though. Beyond that, the rest of the pkg tarball just contains a tar archive of all the files, directories, sym-links etc to be installed by the package. Note that pkg(8) has no problem with creating an empty directory for a package, unlike pkg_tools. Excellent! Now, while you can grovel through the details of pkg tarballs and internal data formats like this, be aware: the format of the manifest and the details of the meta-data included in the pkg-tarballs is subject to change without warning as we develop pkg(8) further. We only promise API stability through the pkg(8) commands or for the libpkg.so library functions; reports of breakage from usage outside those APIs will receive little sympathy. Understood. So just to give you a better idea of how we manage packages here. We've realized that if you want to package for FreeBSD (in 9 and older), you could get by alright if you knew how to create a +CONTENTS file from scratch. For RedHat, it's the SPECFILE. And now for FreeBSD 10, it looks an awful lot like a SPECFILE (they are both in-fact YAML). The only thing you need if you really want to package is a small subset of the manifest written in YAML: Have a look at what the ports tree do to create the minimalistic manifest and what form it has and you will see how easy it is. It is so awful and complicated that there are already a load of custom scripts to create packages with pkg without using the ports tree. As an example: https://urldefense.proofpoint.com/v1/url?u=https://github.com/z0nt/pkg&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=Mrjs6vR4%2Faj2Ns9%2FssHJjg%3D%3D%0A&m=cLKrok7Cl%2BjTLEB024lmmMXfxb1fKAOZvvxLV8RjU%2FU%3D%0A&s=fa25d4b179a2b815c1bffab49adb850f44a6437f175aa93dd159887652935258 , but there is way more. I for example wrote in the past a plugin for pkg to be able to parse and USE RPM's specfiles as an input for pkg. Cool. Wonder if anyone will end up using that. It does sound rather neat for (perhaps) people more RedHat centric that may want to re-use the same SPECFILE of their RPM to produce a pkgng file. So rather than teach the people here how to use tools, I taught them what I think is more important... how to manage +CONTENTS, SPECFILE, and now up-and-coming, +MANIFEST. The problem is that you compare 2 things that cannot be compared, In normal light, yes... but when actually, we've essentially wiped the existence and knowledge of "source RPMs" from our minds as our build system imbues us with that luxury (the luxury of never needing a source RPM, never producing one, and never even thinking about one -- we go from SPECFILE directly to binary RPM and when we need to modify a binary RPM, we take a template SPECFILE and populate it with the metadata extracted from the rpm file itself (using "rpm" utility to extract said metadata). SPECFILE is the equivalent of pkg + the ports tree. In normal case, yes... in our case no. Every single SPECFILE in our build system uses the exact same assembly procedure to produce [directly] a binary RPM. In fact specfiles are the equivalent of a given "port" !!! Are you writing rpm files directly by hand? there is no SPECFILE inside RPM but rather a header container metadata in binary format and a body which is a cpio.(gz,bz2,xz). Correct, a binary RPM does not contain a SPECFILE. But, every piece of data that goes into the SPECFILE is contained in metadata extractable with "rpm -q -p RPMFILE --qf FORMAT" (iirc). This is how we turn a binary RPM into a build-directory with a SPECFILE that -- when we say "make" -- produces the same binary RPM file with-or-without any/all changes to the flat [unpacked] hierarchy: http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/unpack.sh?revision=1.1&content-type=text%2Fplain<http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/unpack.sh?revision=1.1&content-type=text/plain> You can take a look at an example SPECFILE that's been generated from a binary RPM using the above script: http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/rhel6-x86/System_Environment/Daemons/sup/SPECFILE?revision=1.1&view=markup NOTE: In the case of the above URL, it turned out that the "sup" package wasn't available for RedHat EL 6 -- so we slurped in some other simple package and replaced its guts with that of a sup package compiled specifically for RedHat EL 6 (with fixes). But the SPECFILE starts its life as a template for each/every unpack of a binary RPM (see below): http://druidbsd.cvs.sf.net/viewvc/druidbsd/pkgbase/redhat/Mk/template.spec?revision=1.1&view=markup You are not creating the RPM file directly but rather create a SPECFILE, the equivalent on FreeBSD is to create a port!!! Incorrect; as you can see from the above, we *do* treat SPECFILE as a PLIST (aka +CONTENTS). We do build the RPM from the SPECFILE. You're absolutely right that the SPECFILE *normally* contains logic on how to turn a vendor software X (in source form) into [first] a "source" RPM and then a "binary" RPM. But we've taken all that logic out of the SPECFILE and turned it into a dumb thing that just goes direct to binary RPM. the +MANIFEST as used to be the +CONTENTS are the equivalent of the RPM metadata. Right -- we use "rpm"'s --queryformat parameter to extract the metadata. We wrote them in YAML so that they are easy to debug and that we can use third party well used and tested parsers instead of writting our own. I'm very thankful that the compiled pkgng package isn't as opaque as RPM. I think it's a good thing for transparency. I also like that you're porting your tools. pkgng has indeed a bright future and you've been making great choices. (and there is no "but" -- just wanted to say "thank you") I hope you don't see my dissection attempt as a way to subvert what you're doing. Quite the opposite, I'm dissecting so that I can learn in an effort to integrate. However, I'd be lying if I said I taught them *all* the gory details. In reality, for +CONTENTS they edit a PLIST which is essentially +CONTENTS without the "_at_comment MD5:" entries. For SPECFILE, they manage the full thing except there's a small section that is the standard RPM bootstrap that is labeled as "do not touch". >From what you posted of +COMPACT_MANIFEST was nice, except it lacks the list of files. The basic principle here is that if you have to maintain a list of files, and that list ends up being compiled into a +MANIFEST, why not just teach your peers how to read/write/maintain +MANIFEST files (in version control of course) so that there are never any mysteries as to how the package performs. Have a closer look at pkg create, you will see that we are still able to use the plist or +CONTENTS as a possible input for pkg create, it is even the main way to create packages. Ooo, nice. That will smooth migration (however, we it will be a while before we get onto 10.x+ so we've got some time -- meaning, I might favor a route that is legacy free.. agree?) Once again, the main and most important requirement to build packages was to be compatible with the ports tree, and the ports tree is using plist/+CONTENTS. and pkg2ng does only works by using the +CONTENTS. I know this sounds screwy... but (as with our other YAML files), it's really nice because (as is the case with SPECFILE), we version-control things as-close to the end-product as possible (take for example the case of "pre-install" or "post-install" et cetera scripts -- I'm sure a tool like "pkg create" or "pkg_create" or other could do a good job of assembling the pieces into the +MANIFEST, but then it's not being version-controlled as closely. You can keep what you have already done. Once again pkg does not have the equivalent of the SPECFILE because we have the ports tree! In Linux, rpmbuild will take a SPECFILE in the manner you mention (acting as the pre-cursor to the SRPM), but it will also take a SPECFILE to build a binary RPM (and in that last manner, the SPECFILE is not like the ports-tree as it expects binaries, not source, to build the cpio-ball). In the manner we're using SPECFILE (the latter manner), it *is* like a PLIST or a MANIFEST. so we do not need to provide a custom package building system. We could, as it is really easy, but we do not need so it is not there yet, as said earlier I already wrote a pkg plugin that is able to build packages from a RPM specfile, I also modified archlinux's makepkg to be able to output pkgng files. I did the same on slackare, etc. Here the limitation is not technical at all. That's really nice and encouraging. As I do-say... it's remarkable and the future of pkgng looks bright ;D -- Devin _____________ The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.Received on Mon Jul 15 2013 - 14:06:52 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:39 UTC