Re: SUMMARY: Fast releases demand binary updates..

On Thu, 12 Jan 2006, Jo Rhett wrote:

<much unhappiness elided>

> For now we'll have to see if Colin can find ways around the problems without 
> the tools he needs to do it right.

It seems fairly clear to me that Colin's tools provide useful building blocks 
that can be used to create something like what you're looking for: it is an 
efficient way to generate, manage, and distribute difference sets between 
snapshots of a system.  What Colin's pieces explicitly don't provide is a 
administrative infrastructure for imposing heavy weight packaging semantics 
and version information -- specifically, an explicit version database, 
roll-back, and a mechanism for configuration merges.  I'm sure many sites 
would like to have the heavier weight mechanism, but it's certainly a 
non-trivial project that will take a lot of time for someone to get right.

Someone(tm) will need to spend several months on it to get it really working 
well, but I think it would be well worth the effort.  I do suspect, however, 
that the starting point for this is to discard the formal project releases, 
and instead simply provide infrastructure for someone with their own notion of 
what releases are, and what versions make sense.  That way if a site chooses 
to deploy our stock releases or patches, that's fine, but they can also manage 
their own custom snapshots.

Presumably a first starting point is to take the assumption that the build 
system will produce a series of install snapshots, presumably using 
installworld/installkernel to a specific DESTDIR, and that all updates will be 
generated between them, with version names and relevant difference 
combinations to be assigned to them by the admin.

The second part will be handling what's normally done by mergemaster in some 
way -- for some environments, simply generating a second tarball that contains 
the output of distrib-dirs (etc) and then providing that to bootstrap 
mergemaster rather than a source tree would be sufficient.  Other environments 
might wish to simply splat down replacement files based on having run 
mergemaster in advance on the build machine.  A bit of minimal but 
well-crafted glue is presumably required here, such as the ability to point 
mergemaster at the update blob and tell it "Yeah, so, I want my / to be merged 
from here, not /usr/src/etc".

The final part is the meta-data for these snapshots, both carried with them, 
and on the target system.  If using Colin's binary difference pieces, they 
will often be quite compact, and it's easy to imagine storing the necessary 
information to roll back or forwards by several revisions in a relatively 
small amount of space.  Tricky things include how to accomplish relative 
atomicity in updates.  If we assume that the snapshot itself, along with 
possibly a version name/number (from, to), is basically sufficient meta-data, 
that's quite advantageous.  It becomes the responsibility of the admin to not 
screw up by assigning the same name to multiple versions, etc.

But as always, the tricky thing is someone actually doing the work.  It's a 
non-trivial and time-consuming task, and isn't just a weekend's hacking. 
There will almost certainly be a number of "Oh damn, I didn't think of 
that!"'s along the way.  I don't think anyone is opposed to it happening, but 
getting someone to do all that work with little recompense (or even 
significant recompense) requires some amount of convincing!

Robert N M Watson