Re: [HEADSUP & CFT] pkg 1.0rc1 and schedule

On 07/12/2012 03:02 PM, Baptiste Daroussin wrote:
> On Thu, Jul 12, 2012 at 11:48:41AM -0700, Doug Barton wrote:
>> I do not mean this e-mail to be in any way critical. I was told after
>> the new OPTIONS framework discussion that I should have asked questions
>> before the change, so I'm asking these questions now; in a genuine
>> attempt to get information.
>>
>> On 07/12/2012 03:01 AM, Baptiste Daroussin wrote:
>>
>> In the time that you have been working on this project I have asked
>> numerous times for you(pl.) to answer the following questions:
>>
>> 1. What are the goals for pkg?
> 
> The why part of this mail should reply this question, no?

Well clearly not, because if it did I wouldn't keep asking the same
questions over and over again. :)

> Anyway the goal is to have a decent package manager, providing modern features:
> repositories, decent dependency tracking, decent reverse dependency tracking,
> managing upgrade correctly (I'll explain this more later), provide a decent
> library for third party tools (desktop integration via PackageKit for example)

I don't know what "decent" means. I don't know what "modern features"
means (beyond the buzzwords that you've included).

> Providing easy package management for enterprise

Having set up package management systems for enterprises before, *this*
is actually a big-picture goal that I have a lot of sympathy for. But
again, what's missing is *details* about here is what large enterprises
need to make things work for them, here's why the existing tools don't
meet those needs, and here is how pkg does meet them.

> (who never got problems
> managing packages on a large set of freebsd servers, and how complicated it is
> on FreeBSD to have automated reliable puppet,salt,chef,cfengine like tools)
> One of the proof of this problem is how fast people integrated pkgng in those
> tools.

This gets to the heart of my biggest fear regarding this whole project.
It's new, it's shiny, and it looks like forward progress is being made.
Thus, it's attracted a lot of attention, input, time, etc. Heck, it may
even BE forward progress, but I don't know how to measure that because I
don't know what the goals of the project are. Thus, my fear is that
without *details* about what the project is, and what it's trying to
accomplish, we're going to put an exponentially larger volume of work
into the transition and end up no closer to the goal of having a mature
package management system.

And just to be clear, I am *not* saying that "pkg sucks" or that it's
bad, or wrong, or anything else. I'm saying that I don't know how to
evaluate it, because you haven't given us a criteria by which to measure
it.

So what's the problem? We *desperately* need a better system for ports
and packages. We only have so many person-hours we can devote to making
that happen. If we spend all of them on pkg, and it ends up not helping
us enough, we've burnt out our volunteers for no good reason.

>> 2. Why can't the existing tools fulfill those goals?
> 
> The existing tools can't fulfill those goals, because they are hardly
> maintainable, the code hasn't change much since when they were written, lot of
> people have tried over the year to improve them, but all of them gave up. The
> design of the tools, (I mean the code) is really imho not adapted to be
> improved, I spent a lot of time trying to work on it before starting a complete
> new project.

This paragraph really frightens me.

> For example they do not know what is a version, they do not know what are the
> reverse dependencies except through this ugly hack that is +REQUIRED_BY, the
> database is pretty fragile: who never got the package corrupted: empty _at_pkgdep
> line for example.

So these 2 are a lot closer to what I'd like to see ... *details* about
what isn't working now. I would tend to disagree with you that
+REQUIRED_BY is an ugly hack, it's no uglier than any of the other text
file based dependency tracking we have. But thank you for giving us more
information.

So taking your last example, how does pkg handle the situation where the
user wants to forcibly delete a package that is depended on by another
package?

>> 3. How does pkg fulfill them?
>>
>> You've put some of this in the various places where pkg is documented,
>> but I don't see any thorough treatment of these questions. You have some
>> of it below, which I'd like to see expanded on if you would be so kind. :)
> 
> It is true that, I'm not very good at documenting in general, and even more in
> english, hopefully, the documentation is improving a lot recently, there is the
> for usage:
> http://wiki.freebsd.org/PkgPrimer
> and for all other things:
> http://wiki.freebsd.org/pkgng
> 
> Lot of native english speakers have joined the project and help with
> documentation, if you find someting missing, do not hesitate to had the section
> in the apropriate wiki page, I often have a look at them, and try to fill all
> the blank section to answer user questions.

I'm not looking for "how to." I've explained to you numerous times that
I'm looking for a project description. And your English is fine, I
understand what you write perfectly, the problem is that you are not
willing to sit down and write out, in detail, what the project's goals are.

So I'm left to conclude that either A) you don't know because you've
never actually sat down and thought them through, or B) you don't think
you should have to explain yourself. I find both answers disturbing. Of
course there is always answer C) you think I'm a jerk and can't be
bothered to answer my questions. If that's the case, fair enough. Just
tell me that so I can stop trying to make sense of it.

>>> Why pkg?
>>> pkg_* tools have become hardly maintainable over the time,
>>
>> I agree on this point, but the right solution (as some of us have been
>> saying for years) is to move the pkg_* tools into the ports tree. You
>> are correctly handling that by keeping pkg in the ports tree, I'm simply
>> pointing out that this isn't a reason we need to switch to pkg.
>>
>>> it lacks lots of features most of people are expecting from a package manager:
>>>   - binary upgrade
>>
>> I'm not sure what you mean by this. We have the ability to create binary
>> packages now.
> 
> No we haven't :), I know we can mimic a binary upgrade using for example
> portmaster (I describe this in a poudriere howto) but this is not fully binary
> upgrade, it is deinstalling/reinstalling a package. Binary upgrade is much more
> complexe than that, for example one thing you can't handle now is a package that
> has been splitted into lib vs runtime will break with the current way we can do
> it. 

So what you're saying is that you want to do an in-place binary upgrade
of an already installed package? If so, that's an interesting goal, and
I'd like to hear more about it. Including but not limited to, what are
the advantages and disadvantages to doing that, vs. deinstall->install
of the "old" kind of packages? How do you handle the case where the new
version has less files than the old? What happens if the files in the
installed version have become modified/corrupted somehow?

>>>   - ability to search information about remote packages
>>
>> This is a good feature, certainly. However there is no reason we can't
>> create a tool to do this, or add the functionality to an existing tool.
> 
> Have a look at what pkgng can present you as information and you will see the
> difference.

Sorry, this response doesn't address what I wrote. You're saying that
you want this feature, so you created a tool that does it. That doesn't
change my point that moving to pkg isn't necessary to accomplish this.

>>>   - real reverse dependency tracking
>>>   - tracking leaves
>>
>> Can you expand on what these 2 mean?
> 
> Of course. The current reverse dependency tracking in pkg_install is a hack: a
> +REQUIRED_BY file trying to maintain the list of packages that may depend on the
> said dependency, a good way to see it is a hack is to see how often the file get
> broken (and on portmaster you added an options to fix them so you might know :))

Actually if you pkg_delete stuff in the right order it never gets
broken. But, users don't do that, so you're correct that one of the
features of portmaster is that it keeps +CONTENTS and +REQUIRED_BY up to
date when dependent packages are updated.

But the same logic I use in portmaster could easily be brought into the
ports framework itself, if that was something that people wanted.

>> What I'm looking for is compelling motivation to make this overwhelming
>> change to the ports infrastructure.
> 
> There is not much changes needed in the ports infrastructure. It now works ootb
> But there are tons of improvements pkgng will offers, like: ability to simply
> add new plist keyword without the need of modifying pkgng,

That's a feature that could be had just as easily by moving pkg_* to the
ports tree.

> native support for registering a package from a stagedir.

That's another set of changes that it would be nice to see a thorough
project plan for. :)

>> IMO it would be a very large mistake to switch the default in any branch
>> until the problem with the nvidia drivers is sorted out. We have a lot
>> of users (myself included) who use this port, and by switching the
>> default there's going to be 1 of 2 outcomes for those users. Either they
>> will opt-out, which means you won't get the level of testing you're
>> looking for; or you'll break their existing ports installation. Neither
>> outcome is desirable.
> 
> I proposed 3 differents way to fix the nvidia driver problem, no one really
> takle the task on it even if most agreed with at least one of the method.
> 
> So waiting for someone to provide a really clean way to provide nvidia driver,
> I'm now working on a small modification of the current way it works, that will
> bypass the pkgng strictness.
> 
> So nvidia-driver should be fixed on pkgng quite soon (if maintainer do accept
> the modification :)).

As long as the solution arrives in the ports tree *before* the default
is switched, it should be fine.

Doug