Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

On Oct 8, 2013, at 2:28 PM, Nathan Whitehorn wrote:

> On 10/08/13 23:21, Kurt Lidl wrote:
>> 
>> On 10/8/2013, Nathan Whitehorn wrote:
>>> On 10/07/13 21:59, Allan Jude wrote:
>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>> implement installing on a ZFS pool. It supports both GPT and MBR,
>>>> the 4k
>>>> sector gnop trick, and optional GELI encryption. We would like to
>>>> commit
>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>> obvious bugs before we send it off to re_at_ to get it committed.
>>>> 
>>>> It includes a single configuration menu that allows you to select
>>>> all of
>>>> the required details, including which drives to use (gets details from
>>>> camcontrol, also includes an inspection utility that presents the
>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>> ZFS RAID level to use (taking in to consideration the selected
>>>> number of
>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>> 
>>>> 
>>>> Additional, it includes some other changes to bsdinstall:
>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt
>>>> to no
>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with
>>>> just 1
>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>> feature has been combined into the regular 'services to enable' dialog
>>>> and enabled by default
>>>> 
>>>> 
>>>> You can browse the patches here:
>>>> http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/
>>>> 
>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>> 
>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz
>>>> 
>>>> http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso
>>>> 
>>>> 
>>>> We look forward to your feedback
>>>> 
>>> 
>>> Thanks for doing this! I had a few comments:
>>> 1. ZFS is not bootable on all architectures. Could you adjust that menu
>>> item to only display for i386, amd64, and (I think?) sparc64. Use uname
>>> -m, not -p, for this.
>>> 1a. The script is broken on sparc64 in any case, which uses VTOC8
>>> instead of GPT.
>>> 2. Why are you using camcontrol? That is guaranteed not to work on
>>> non-CAM systems. You should use the GEOM ident string if you need an ID.
>>> 3. Any plans to integrate this into the regular partition editor? ZFS
>>> support is important enough that I will definitely not get in the way,
>>> even as a bolt-on, but it would be a shame for it to stay that way. The
>>> editor is also designed for ZFS to be added.
>>> 4. What is this gnop stuff for?
>>> 5. I think some substantial part of the MBR code will blow up if you are
>>> reinitalizing a previously formatted disk (the bsdlabel will be retasted
>>> and come back from the dead).
>>> -Nathan
>> 
>> I wrote some support for adding ZFS to the partition editor a couple of
>> months ago, around the time that 9.2 was branched.  One of the biggest
>> things that I have not done is integrate setting up a zfs mirror between
>> any disks before creating the zpool.  Nor did I do the gnop hack to
>> create 4K disk blocks before creating the pool.
>> 
>> I more or less changed the partedit program to take an argument when
>> invoked with "ufs" (same as current behavior) or "zfs", which knows
>> about zfs setup stuff.  The hookup to the actual scripts was, shall
>> we say, much less intrusive than what has been published here.
>> 
>> I guess it's time to dig out the patches and make them available to
>> others.
> 
> That would be great! The original idea was to have a "zfs_ops.c" to go
> with "gpart_ops.c" and have some fields in the disk item struct that
> said which to use. Not sure if that's ultimately workable, though. I can
> hack on it once the patches are there in any case.

One of the options presented was to integrate ZFS knowledge into the
partedit module.

However, another approach that is not being spoken about is in the
other direction: rewriting partedit to be in sh(1).

What I'd like to see is the --treeview widget of dialog(1) "fixed" before
such a transition is made, so the former option may be the best bet for
now. For anyone that hasn't run the treeview or treeview2 samples in
head/contrib/dialog/samples ... it's not anything we'd want to use to
replace dialog(3) with. I've seen much nicer treeview implementations
than what is in [c]dialog (currently we're using dialog-1.2-20130925 and
the treeview for some reason incorporates a radio-button column which
we do _not_ want for a disk partition layout menu).

Of course, right now you're all looking at me like I'm crazy.

Like a fox.

There are real tangible benefits to rewriting the C code of partedit into
shell syntax. The value-add is not just perceived. Some of the values
obtained are:

+ Conflated namespace with singular API rather than having to learn
how to script each individual blocking-utility which implements a unique
method for scripting its input.

+ Top-down integration of new functionality. We're integrating ZFS right
now and that's great that we have a fully-functional libzfs et. al. for doing
this stuff in C, but usually the command-line utility functionality precedes
programatic functionality; it is in this spirit that we can integrate new
functionalities into the installer purely from a base utility standpoint (if a
utility exists to do something, we can integrate support for it rather than
[potentially] having to wait until a C library surfaces).

+ Easier patching for custom deployments. The more code we write in C
the more code has to be custom-built for in-house deployments (versus
shell in-which you can patch-and-deploy).

+ More robust data. While it's true that most data (such as hardware
descriptions of disk devices) is accessible from C libraries, we stand to
gain more robust access to data if we are forced to get the information
from a userland tool. Why? because then we make sure that the data is
in some-form available from some tool (and not *only* available from a
library requiring you to write a C program to get at that data). In other
words, the shell approach promises to expose more central methods of
accessing informative data *while* at the same time making it easier to
bring that data into the installer.

I could go on, but I digress. I think I'm making good arguments.

I think also we should be mindful that arguments such as:

"What if dialog(1) goes away?" (A: dialog(3) and dialog(1) have always
been provided by the same vendor -- for multiple distributions from at
least two generations of dialog functionality -- regardless of C-based or
utility-based)

and

"But shell is nasty; slow; and not as powerful as C" (it depends in what
context; the first is rhetoric, the second is only true for poor implement-
ations, and the third may be true in some contexts, but I consider the
answer to "how maintainable is it" to be a factor in the "power" of a
language, so don't necessarily consider C to be more powerful than
shell as the latter is as-or-more maintainable with fewer LoC and a
higher return on investment; see previous [above] arguments).

And to that, I say...

bsdconfig(8) is 100% shell. Run it; Inspect it; .... Love it?
-- 
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.