Re: [ECFT] pkgng 0.1-alpha1: a replacement for pkg_install

On Fri, Mar 25, 2011 at 1:14 PM, Alexander Leidinger
<Alexander_at_leidinger.net> wrote:
> On Fri, 25 Mar 2011 16:35:21 +0100 Pietro Cerutti <gahr_at_freebsd.org>
> wrote:
>
>> On 2011-Mar-25, 15:03, Julien Laffaye wrote:
>> > >>> What about DB corruption/loss? Do you keep
>> > >>> the /var/db/pkg/<package>/xxx files even with pkgng and only
>> > >>> use the DB as a way to speed up some work (so
>> > >>> the DB corruption just requires to run pkg2ng), or are you lost
>> > >>> of the DB is
>> > >>> lost?
>> > >>>
>> > >>
>> > >> Nothing is done about DB corruption/loss, I am not sure we need
>> > >> to do something.
>> > >> Maybe.
>> > >
>> > > I would say "for sure". Info: In Solaris 10 sqlite is used for
>> > > the service managenemt framework (SMF). It is possible that the
>> > > DB is corrupt in some bad situations. In this case you have to
>> > > rebuild the DB (script provided, been there, had to use it).
>> >
>> > If sqlite is properly used with transactions, it is very hard to
>> > corrupt the database. But if hardware lies to us and say that the
>
> And as I told above, I even had such a case (more than once), and the
> hardware was not buggy. What do you want to tell in this case, "life
> sucks, reinstall everything"?

If you use binary packages, pulling down everything should be trivial,
fast, and easy to install. If you're using ports, well then things are
going to be slow as expected.

>> > data is on disk whereas it isnt... what can we do?
>
> Sometimes you have to stay with broken hardware.

Sometimes you have to go buy new parts?

Playing with broken hardware is like playing with fire -- sometimes
you'll get burned if it goes out of commission during critical
operations. I would be more concerned about overall system operation
than having a packaging system that can handle all error conditions
that should be rightfully handled by various kernel subsystems. If the
kernel's doing it's job, then the packaging manager can do its job as
well.

>> > Another potential problem is fsync(), but if it is broken on FreeBSD
>> > we want to fix it!
>> >
>> > BTW, the goal is to only have the database and not the flat files.
>> > If you are paranoid about power outage, use something like zfs
>> > snapshots...
>
> There are more FS than only ZFS (personally I use ZFS, and I have
> snapshots, but this is not a good solution for this problem).

A lot of filesystems feature snapshot'ing, including UFS. If you
aren't smart enough to back up your data you're toast if the data is
gone.

I would be more concerned about the program getting killed, not
getting properly cleaned up, etc as this is something that the package
manager frontend (or whatever the official name is) should catch and
fail gracefully with. Things need to follow an ACID methodology and be
recoverable in the event that it can't be ACID, or it's no better than
pkg_install/ports currently is if it's caught in the middle of a
critical operation today installing or removing software.

If SQLite can't deliver this level of ACID-like capability, then
pkg_install needs to be redesigned.

> As I told already, if it isn't automatic, nearly nobody will use it.
> And the package management stuff has to be automatic, no freshman will
> think about setting up a snapshot script when he starts to use
> packages/ports.

I'd just provide an export command to print out a <blah> (JSON?)
version of the information, and move on. None of the other major
packaging systems out there that I know of use flat files for this
data, and I would rather not make it automatic because it's an
unnecessary performance hit. If the user feels the need for backing up
his/her data they will. If not, they're SoL in the event of a crash.

>> No need to look for strange scenarios, I'm surely going to sudo rm -f
>> the file more sooner than later, so... maybe just save a copy?
>
> A copy or two would be enough, but it has to be done automatically, and
> once a day is not enough. A copy after each X modifications maybe
> (for suitable definitions of X and 'modifications').

Please see my comment above. There's no reason why this belongs in a
packaging system (you can add it as an external tool, but the point is
to avoid architectural mistakes that leaked into the old pkg_install
over the period of 10 years or so).

Thanks,
-Garrett

PS Sorry for being so hardnosed on this, but I want something that's
fast and correct, instead of something bloated, slow, half-baked,
harder to test, etc. pkg_install gets executed enough times during a
port upgrade that having something more streamlined for most usecases
is the only way to go, and there's enough code that doesn't get
executed on a regular basis that has no business being in pkg_install.