Re: [ECFT] pkgng 0.1-alpha1: a replacement for pkg_install

On Fri, 25 Mar 2011 13:47:13 -0700 Garrett Cooper <yanegomi_at_gmail.com>
wrote:

> On Fri, Mar 25, 2011 at 1:14 PM, Alexander Leidinger
> <Alexander_at_leidinger.net> wrote:
> > On Fri, 25 Mar 2011 16:35:21 +0100 Pietro Cerutti <gahr_at_freebsd.org>
> > wrote:
> >
> >> On 2011-Mar-25, 15:03, Julien Laffaye wrote:
> >> > >>> What about DB corruption/loss? Do you keep
> >> > >>> the /var/db/pkg/<package>/xxx files even with pkgng and only
> >> > >>> use the DB as a way to speed up some work (so
> >> > >>> the DB corruption just requires to run pkg2ng), or are you
> >> > >>> lost of the DB is
> >> > >>> lost?
> >> > >>>
> >> > >>
> >> > >> Nothing is done about DB corruption/loss, I am not sure we
> >> > >> need to do something.
> >> > >> Maybe.
> >> > >
> >> > > I would say "for sure". Info: In Solaris 10 sqlite is used for
> >> > > the service managenemt framework (SMF). It is possible that the
> >> > > DB is corrupt in some bad situations. In this case you have to
> >> > > rebuild the DB (script provided, been there, had to use it).
> >> >
> >> > If sqlite is properly used with transactions, it is very hard to
> >> > corrupt the database. But if hardware lies to us and say that the
> >
> > And as I told above, I even had such a case (more than once), and
> > the hardware was not buggy. What do you want to tell in this case,
> > "life sucks, reinstall everything"?
> 
> If you use binary packages, pulling down everything should be trivial,
> fast, and easy to install. If you're using ports, well then things are
> going to be slow as expected.

And if there is a fast way to cut down the slow part... why not?

> >> > data is on disk whereas it isnt... what can we do?
> >
> > Sometimes you have to stay with broken hardware.
> 
> Sometimes you have to go buy new parts?

Yes, but if we talk e.g. about aging hardware, having the luck to get
hit directly in the parts which hurt is not nice. You want to have the
time to find a suitable replacement.

> Playing with broken hardware is like playing with fire -- sometimes
> you'll get burned if it goes out of commission during critical
> operations. I would be more concerned about overall system operation
> than having a packaging system that can handle all error conditions
> that should be rightfully handled by various kernel subsystems. If the
> kernel's doing it's job, then the packaging manager can do its job as
> well.

You know that the world is not an ideal one. Shit happens and Murphy
visits you.

> >> > Another potential problem is fsync(), but if it is broken on
> >> > FreeBSD we want to fix it!
> >> >
> >> > BTW, the goal is to only have the database and not the flat
> >> > files. If you are paranoid about power outage, use something
> >> > like zfs snapshots...
> >
> > There are more FS than only ZFS (personally I use ZFS, and I have
> > snapshots, but this is not a good solution for this problem).
> 
> A lot of filesystems feature snapshot'ing, including UFS. If you
> aren't smart enough to back up your data you're toast if the data is
> gone.

So... why do we have /var/backups/master.passwd.bak then? To make life
easy.

> I would be more concerned about the program getting killed, not
> getting properly cleaned up, etc as this is something that the package
> manager frontend (or whatever the official name is) should catch and
> fail gracefully with. Things need to follow an ACID methodology and be
> recoverable in the event that it can't be ACID, or it's no better than
> pkg_install/ports currently is if it's caught in the middle of a
> critical operation today installing or removing software.

I agree.

> If SQLite can't deliver this level of ACID-like capability, then
> pkg_install needs to be redesigned.

AFAIK it can.

> > As I told already, if it isn't automatic, nearly nobody will use it.
> > And the package management stuff has to be automatic, no freshman
> > will think about setting up a snapshot script when he starts to use
> > packages/ports.
> 
> I'd just provide an export command to print out a <blah> (JSON?)
> version of the information, and move on. None of the other major
> packaging systems out there that I know of use flat files for this
> data, and I would rather not make it automatic because it's an
> unnecessary performance hit. If the user feels the need for backing up
> his/her data they will. If not, they're SoL in the event of a crash.

- It does not need to be done with every change.
- You do not know if it is a performance hit or not, we do not have
  numbers.
- If making an automatic export after X modifications is not expensive,
  I say: why not? It would make more easy in case of fire.

> >> No need to look for strange scenarios, I'm surely going to sudo rm
> >> -f the file more sooner than later, so... maybe just save a copy?
> >
> > A copy or two would be enough, but it has to be done automatically,
> > and once a day is not enough. A copy after each X modifications
> > maybe (for suitable definitions of X and 'modifications').
> 
> Please see my comment above. There's no reason why this belongs in a
> packaging system (you can add it as an external tool, but the point is
> to avoid architectural mistakes that leaked into the old pkg_install
> over the period of 10 years or so).

Backups are not for architectural mistakes, they are for situations
where something went wrong. Something may go wrong even without
architectural mistakes. Safety nets are good. They are even better if
they do not cost anything. We do not know how much this would cost, but
does this mean we are not even allowed to talk about it? If the penalty
is too big, sure, ditch the idea of doing it often, but as long as we
do not know the numbers, please, don't tell the end of the world is
near.

> Thanks,
> -Garrett
> 
> PS Sorry for being so hardnosed on this, but I want something that's
> fast and correct, instead of something bloated, slow, half-baked,
> harder to test, etc. pkg_install gets executed enough times during a
> port upgrade that having something more streamlined for most usecases
> is the only way to go, and there's enough code that doesn't get
> executed on a regular basis that has no business being in pkg_install.

I agree fully with you, I also hope for something very fast, but as
long as we do not have numbers, please ...

Bye,
Alexander.

-- 
http://www.Leidinger.net    Alexander _at_ Leidinger.net: PGP ID = B0063FE7
http://www.FreeBSD.org       netchild _at_ FreeBSD.org  : PGP ID = 72077137