Re: pkgng default schedule... registering a few reasons for rethinking the final implementation...

On 08/23/2012 13:10, Jeremy Messenger wrote:
> On Thu, Aug 23, 2012 at 11:49 AM, Kris Moore <kris_at_pcbsd.org> wrote:
>> On 08/23/2012 12:26, Jeffrey Bouquet wrote:
>>> I am following with dread the planned implementation of the deprecation of /var/db/pkg as a package registry... I use each /var/db/pkg directory as a database into the port installation/status, using sed/grep/portmaster/portmanager/.sh scripts/find/pipes etc... to fix stuff.  For instance, an upgrade py26 > py27.
>>> cd /var/db/pkg
>>> ls -lac | grep py26
>>> ls -lac | grep python
>>> as the more simple example.
>>> ....
>>> With due respect to its developers and the persons who agree that
>>> the package tools could be upgraded, the mandatory
>>> usage of a front-end database to a file directory one
>>> is here viewd as mutt-only-mbox, registry-and-bsod rather
>>> than /etc/local/rc files, deprecation of sed/grep/find/locate/.sh/portmaster/portmanager as tools to fixup/upgrade the ports that are registered;
>>> ...
>>> I see concurrently too few tests on lower-end p2, p3 as to whether
>>> pkg can run with lesser memory machines (routers...) (pfsense)
>>> ...
>>> I suspect stalling of successful frontends to bsd (pc-bsd, ghostbsd,
>>> pfsense..) due to less-reliability, more-possibility of bugs..
>>>
>> This is of some concern to me as well. A number of our utilities /
>> scripts rely on checking /var/db/pkg as a means to test if a particular
>> package is installed. This is often much faster than running the pkg_*
>> commands, especially when we may be checking thousands of packages in a
>> single run. It will be some work to adjust our utilities to using the
>> various "pkg" commands now, but it can be done. What worries me is
>> performance. If this is significantly slower, it may cause some issues
>> on our end.
> Guys, please test it before you say anything. Otherwise it's going to
> be moved forward without you.
>
>

Well, it was about time I got to doing a benchmark of this anyway :)

I did quick benchmark of how one of our utilities parses through a list
of 1k packages on a newer i5 system:

First test, using /var/db/pkg/<pkg> check we have been doing:

0.178s 0:00.31 54.8%
0.123s 0:00.26 61.5%
0.099s 0:00.15 60.0%
 
Second test, using "pkg info <pkg>":

5.347s 0:11.41 91.7% 
5.444s 0:11.52 91.3%
5.878s 0:11.32 91.4%

The pkg info command is quite a bit slower in this case, but 5 seconds
isn't horrible.

Now I ran the same benchmark on a slower 1.66gz Atom system, checking
about 1200~ packages:

First test, using /var/db/pkg/<pkg> check we have been doing:

0.604s 0:00.76 86.8%
0.622s 0:00.77 84.4%
0.614s 0:00.73 90.4%
 
Second test, using "pkg info <pkg>":

28.507s 0:54.80 99.1%
28.282s 0:54.60 99.4%
28.302s 0:54.52 99.4%

Now this is what concerns me a bit. It took closer to 30 seconds, which
is quite a while to wait, especially if a utility like ours has to run
these checks when it starts up, to show the user whats installed / not
installed on the system.

The only way around It I've found is to do a quick "pkg info" on the
entire DB, dump that to a list, then begin to grep through that list for
each item, but it still takes 10~ seconds on the atom. That may be what
I end up having to do, but it still stinks to go from a half a second
startup, to 10 seconds each time. Any other ideas on how to do this
faster with the new pkgng?

-- 
Kris Moore
PC-BSD Software
iXsystems