Re: HEADS-UP: installworld on r299292 through r299317 will replace master.passwd, passwd, and group files

On Tuesday, May 10, 2016 06:30:53 PM Glen Barber wrote:
> On Tue, May 10, 2016 at 10:22:04AM -0700, John Baldwin wrote:
> > On Tuesday, May 10, 2016 05:12:28 PM Glen Barber wrote:
> > > On Tue, May 10, 2016 at 10:04:47AM -0700, John Baldwin wrote:
> > > > On Tuesday, May 10, 2016 06:32:29 AM Glen Barber wrote:
> > > > > On Tue, May 10, 2016 at 08:25:22AM +0200, Thomas Zander wrote:
> > > > > > On 10 May 2016 at 08:18, O. Hartmann <ohartman_at_zedat.fu-berlin.de> wrote:
> > > > > > 
> > > > > > > I haven't figured out so far how far this goes. Lucky for those having
> > > > > > > recent /etc/ backups. A pity FreeBSD doens't backup this by default.
> > > > > > 
> > > > > > After having shot myself in the foot some time ago, "zfs snapshot" has
> > > > > > become a part of my standard upgrade procedures :-)
> > > > > > 
> > > > > 
> > > > > No argument that this is valuable, but we cannot rely on filesystem
> > > > > specific solutions.  Similar topic came up a few days ago following
> > > > > lunch.  It got me thinking of a better way to ensure this kind of thing
> > > > > does not require home-grown foot protection from cannons.
> > > > > 
> > > > > It should be fairly trivial to automatically backup /etc (and related)
> > > > > when 'distribution' is run, either intentionally or accidentally (or by
> > > > > commit mistakes, such as this).
> > > > 
> > > > Saving the output of 'etcupdate diff' nightly might not be a bad first step.
> > > > 
> > > 
> > > This is also a good way to alleviate such things, however I am unsure
> > > how to handle cases where 'etcupdate' would inadvertently run into
> > > a conflict.  This was my concern with implementing an "automatic"
> > > etcupdate run in the runtime package.
> > 
> > I mean as part of the nightly jobs we could add one that stores
> > 'etcupdate diff' in /var the same as we do with backups of the master.passwd,
> > group, and aliases files in /var/backups.  You can then at least use that to
> > reconstruct altered /etc files by applying the diffs.  This isn't meant to be
> > an automated update run, but just saving a diff as part of the nightly jobs.
> > 
> 
> To be clear, I do not disagree with the idea as a whole.  However,
> I think we should considering making this part of the 'installworld' (or
> a dependent step of installworld) so we don't need to rely on periodic
> script execution.  (Consider cases where one may shut down their laptop
> before going to sleep, and the job never runs).

On many machines config changes to /etc happen more often than installworlds
(e.g. adding a new user, etc.).  What we could do is to add a suggestion of
saving an etcupdate snapshot after installworld to the list of world steps in
the handbook.  Warren is working on adding etcupdate to the world instructions
in the handbook anyway so this could perhaps be done after that.  However, I
don't think 'make installworld' itself should do this.

> > As far as what to do in runtime packages, presumably there isn't a single
> > package with all of etc, but etc files can be split up (ppp.conf in the ppp
> > package, etc.) and pkg needs to do its own 3-way merge of changes to conf
> > files when upgrading.
> 
> As the state of things are now, /etc is not included in any package
> (config files exempted), but pkg(8) does have merge ability now.  The
> problem with the initial implementation of "packaging /etc" was that it
> broke etcupdate(8) and presumably mergemaster(8), as the files were now
> part of an 'install'-style target, not 'distribute'-style target.
> 
> But, I think you gave me an idea on how to handle this.  (I'll test what
> I have in mind, but I'm not sure if it will continue to break the merge
> tools again as a side effect.)

My expectation is that if pkg managed /etc files you wouldn't need etcupdate
or mergemaster anymore with a packaged base.  People who are using source-based
upgrades could continue to do that, but people using packages would just use
'pkg upgrade' and have /etc files handled as part of that.

> > (This would be nice for conf files for ports in
> > /usr/local/etc as well.)  You still need to figure out how to handle
> > conflicts, but if pkg manages /etc files as config files and does a 3-way
> > merge of the old package and new package then that will serve to reimplement
> > etcupdate as part of 'pkg upgrade'.
> 
> Merge conflicts where human intervention is required is why I am
> hesitant to add an implicit 'etcupdate' invocation as a post-install
> script, as it breaks automated, non-interactive upgrades.

If pkg is going to really handle config files it has to handle this case
anyway.  Note that etcupdate already punts if this happens as well, but in my
experience such conflicts are quite rare.  A similar model as to what
happens with etcupdate (leave existing file in place until conflicts are
resolved but provide a command to resolve conflicts as well as a way to
query the system for any pending conflicts).

> > Having a 'pkg confdiff' or some such to
> > output diffs made to conf files would be the equivalent of 'etcupdate diff'
> > in that case (and would be nice as it would apply to conf files in ports as
> > well).
> > 
> 
> Agreed.
> 
> But, to be honest, I'd like to use etcupdate for this if it comes down
> to it.  We use it in the cluster, and have never run into an issue
> (until I introduced the change mentioned above, which removed things
> like /etc/auto_master (part of the autofs package).

I kind of think that etcupdate should only be for non-packaged base.
For packaged base I think having pkg manage this makes the most sense
and that etcupdate wouldn't be used in that case.  In particular, if
you get config files working well in pkg then things like 'pkg confdiff'
and 'pkg config resolve --check' (or whatever it is called) would
handle both "base" /etc files and config files from 3rd party packages
in one tool giving a consistent UI for users.1

-- 
John Baldwin