Re: [HEADSUP] No more pkg_install on HEAD by default

On 14/07/2013 04:18, Teske, Devin wrote:

> ASIDE: For efficiency, I will actually need three things: (1) a list
> of all packages (2) their descriptions and (3) their run-time dependencies.

That would be the repository catalogue, which you can download by 'pkg
update' (or it will happen automatically before pkg operations.)  Or,
more precisely, the repository /catalogues/ for each of the package
repositories enabled in pkg.conf

> NOTE: I'll need run-time dependencies so that as they're checking
> boxes in the UI, I can update accordingly (has nothing to do with how
> the dependencies are handled when the packages get installed; I'll let
> pkg handle that when it comes time, but for the UI and for the "review"
> screen, I want the user to be informed and I can do that more
> efficiently if I have a master-list and don't have to make continuous
> queries).

In 1.1 the repository catalogue is just the collection of all the YAML
format metadata from all the packages in the repository, plus optional
cryptographic signatures.  While you could try and parse that directly,
there's little point in re-inventing that wheel -- pkg will load the
catalogue data into a sqlite database, and it provides 'pkg search' or
'pkg rquery' specifically for searching catalogue data. ('pkg rquery' is
aimed at scripting use, 'pkg search' is more for interactive use.)

> That being said, I was planning on doing a "pkg rquery" to get that
> master list of [minimally] 3-pieces of information.

Something like:

pkg rquery -a '%n-%v %o\n%e\n%dn-%dv %do\n---\n'

> What protocol does that "rquery" run on? What would one have to do
> to set up their own server that can accept an "rquery"? (our customers
> don't have Internet access)

It's built around SQL queries against the sqlite database built from the
repository catalogues.  Once you've got the repo catalgoue, it doesn't
need network access to just query the data.

> Last but not least...
> 
> Can you do an "rquery" on a local repository? (say, one that has
> been mounted via NFS or some other media, local or otherwise but looking like
> a local filesystem once-mounted). What would be required to get a local
> repository like that going?

You can certainly build a local repository -- essentially you just
assemble a set of packages in some sort of directory heirarchy, and then
just run 'pkg repo' to create the catalogue.

You can then access the repo by any of the protocols supported by
libfetch -- which includes file://, suitable for a repo on a local or
NFS mounted directory.  Beyond what libfetch provides, you can also use
ssh://.

You'ld usually use poudriere or similar to build your own set of
packages to populate your private repository, but there's nothing to
stop you eg. mirroring some selection of packages from a public package
repository and building your private repo from them.  Or configuring
your systems to use a local package repository /and/ a public
repository.  See pkg-repository(5).

>> I do suspect that HTTP_PROXY support is probably not available as I
>> recall seeing a post where someone was asking about that getting
>> implemented for fetch. I'll have to research that again, though.

> Thanks. Keep me up to date on that.

pkg will use the proxing capabilities of libfetch -- so it will abide by
any HTTP_PROXY or FTP_PROXY or any of a number of other environment
variables as described in fetch(3).  You can use the PKG_ENV setting in
pkg.conf(5) to set the environment use by pkg commands.

> Neither sysinstall nor bsdinstall really give HTTP_PROXY access much
> thought (they support it, but only minimally). They just construct a
> raw HTTP header with the FTP URL and send that directly to the proxy.
> One cute thing it does (when initializing the connection) is test for
> Squid and if-so, appends ";type=i" to the URL (to force binary
> download versus ascii).

Everything that pkg downloads is currently in the form of an xz(1)
compressed tar archive[*] so definitely needs to downloaded in binary mode.

[*] Even in some cases where the tarball contains only one file, which
is a bit of a nonsense, but not hugely so.  That may well be changed in
future versions.

> No support for proxy-server authentication (however user/pass
> authentication for the FTP server is passed through to the
> proxy-server).

pkg uses libfetch's proxy auth support.  You can also authenticate
access to a repository via FTP or HTTP or HTTPS by encoding a username
and password in the repository URL in the usual way.  For access over
SSH, use any of the authentication mechanisms provided by the ssh server
-- use of ssh-agent(1) and key-based auth is recommended, to avoid
having to type in passphrases repeatedly.

I think if you pkgng-ify a test system and have a play with the
capabilities pkg(8) provides (and maybe poudriere(8) too), you will be
pleased, maybe even pleasantly surprised, by what it can do.

	Cheers,

	Matthew

-- 
Dr Matthew J Seaman MA, D.Phil.
PGP: http://www.infracaninophile.co.uk/pgpkey