Re: [CFT] Patch to bsdinstall to support root-on-ZFS and GELI

On 2013-10-09 14:14, Teske, Devin wrote:
> On Oct 9, 2013, at 10:47 AM, Allan Jude wrote:
>
>> On 2013-10-09 13:21, Teske, Devin wrote:
>>> On Oct 8, 2013, at 8:49 PM, Allan Jude wrote:
>>>
>>>> On 2013-10-07 15:59, Allan Jude wrote:
>>>>> Devin Teske and I have been working on a big patch to bsdinstall to
>>>>> implement installing on a ZFS pool. It supports both GPT and MBR, the 4k
>>>>> sector gnop trick, and optional GELI encryption. We would like to commit
>>>>> this in time for 10.0-BETA1 so it needs some testing to work out any
>>>>> obvious bugs before we send it off to re_at_ to get it committed.
>>>>>
>>>>> It includes a single configuration menu that allows you to select all of
>>>>> the required details, including which drives to use (gets details from
>>>>> camcontrol, also includes an inspection utility that presents the
>>>>> detailed output of camcontrol inquiry/identify, and gpart show), what
>>>>> ZFS RAID level to use (taking in to consideration the selected number of
>>>>> drives), GPT/mbr, 4k YES/no, GELI yes/NO, pool name, etc.
>>>>>
>>>>>
>>>>> Additional, it includes some other changes to bsdinstall:
>>>>> 1. Change the default to the 'non-standard keyboard mapping' prompt to no
>>>>> 2. Replace the 3 separate dialogs to configure an ipv4 address with just 1
>>>>> 3. Remove the dialog asking if you wish to enable crash dumps, this
>>>>> feature has been combined into the regular 'services to enable' dialog
>>>>> and enabled by default
>>>>>
>>>>>
>>>>> You can browse the patches here:
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://druidbsd.cvs.sf.net/viewvc/druidbsd/bsdinstall_zfs/&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=32f69a33d9c98c63ecf3962b89eea0af631f5a2fd6e82b55afb6ec139f3f803d
>>>>>
>>>>> I've built a bootonly.iso (10.0-ALPHA4) to make testing easier,
>>>>> available compressed (48 MB) or uncompressed (211 MB):
>>>>>
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso.xz&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=fd0f3ed623f12c8cb5eee607cf82122523a0800ab9f5624cb3f77ae2646e1b1b
>>>>>
>>>>> https://urldefense.proofpoint.com/v1/url?u=http://www.allanjude.com/bsd/zfsbootonly_2013-10-06.iso&k=%2FbkpAUdJWZuiTILCq%2FFnQg%3D%3D%0A&r=LTzUWWrRnz2iN3PtHDubWRSAh9itVJ%2BMUcNBCQ4tyeo%3D%0A&m=oPEdrpFUFqEutq%2B7DHGyBIA0gTt5%2BZ6FWvbN6q4bjm4%3D%0A&s=fa49aaa528513a591ac40a2f0c5666abae2d7269bd66ac70d7100475a9574300
>>>>>
>>>>>
>>>>> We look forward to your feedback
>>>>>
>>>> We've made more improvements, including corporating most all of the
>>>> feedback we've gotten so far
>>>>
>>>>
>>>> Outstanding items:
>>>> 1. Apply the changes to ipv6 config the way we did ipv4
>>>> 2. improve disk identification (model info and serial # instead of one
>>>> or the other)
>>>> 3. Include a helpful message before the GELI step where you have to
>>>> enter your password many times, the user will be less confused if it is
>>>> explained why they have to enter their password 3 * number of disks times
>>> I'm hopeful that we can script the application of a password that we
>>> first prompt for.
>>>
>>> What tool is prompting for a password? Can we not just provide an answer
>>> on stdin? (e.g., echo "$pass" | tool_that_needs_pass)
>>>
>> It is 'geli create' and 'geli attach'. I am not sure if we want to have
>> the password show up in the process list (obviously in the installer
>> this is less of an issue, but)
>>
> It won't.
>
> echo is a shell built-in.
>
> If we're uber paranoid... we prefix the word "builtin"; e.g.,...
>
> builtin echo "$pass" | tool_that_needs_pass
>
> You'll see the tool in ps, but you won't see the echo (that's part of the
> shell invocation -- all you see in ps is an instance of sh).
>
I have implemented this, 'geli init' takes the -J <file> flag, which can
be set to - to mean stdin. 'geli attach' is the same except the flag is -j

I first prompt for the password with a dialog --passwordbox, so the user
only has to enter it once

Do we think it makes sense to prompt the user twice, and confirm the two
entered passphrases are the same?


>
>>>> 4. Validate vdev type choice inside the vdev type menu, and warn the
>>>> user if they have made an invalid selection, so they can add more disks
>>>> or chance their selection, without having to try to start the
>>>> installation first
>>> This will be done with fanciness ;D (read: ... --and-widget --infobox ... and
>>> sundry smartness; retaining as much as possible the ability to do things
>>> out of order but never arise at a point of astonishment).
>>>
>> I don't think we need --and-widget, just in the function where we apply
>> the results of the menu selection,
> The purpose of --and-widget with an --infobox is to let the user know that
> validation is occurring each/every time they make a selection.
>
> Seeing the infobox before being returned to the previous menu (in the case
> of selecting a valid vdev_type) is to cement in the mind of the user that their
> selection was validated. Of course, in the case of an invalid selection, they
> get a message box. What the message box says depends on:
>
> 1. Are they trying to select a vdev_type for which they don't have enough
> devices? Tell them that they don't have enough devices, and bring them
> back to the vdev_type menu (to select a different [valid] vdev type *or*
> cancel and go back to the ZFS menu where they can optionally Rescan
> for more devices -- allowing that vdev_type to be selected without issue).
> In the prompt that tells them that they don't have enough disks in their
> system to select that vdev_type, we sould hint that they can choose cancel
> and go back to use the "Rescan" option to scan for more devices.
>
> 2. Are they trying to select a vdev_type for which they have enough devices
> but have not yet made enough selections? Drop them back to the ZFS menu
> so they can go select the appropriate number of devices.
>
> Good?
>

I kind of shortened it a bit. When you make a selection in the vdev
submenu, if your selection is invalid you get a --yesno box explaining
that 'type <blah> requires <X> disks'. And are prompted to either
'choose again' (back to the vdev menu) or 'return to the zfs menu' (to
select more disks)

>> we can add a regular --msgbox telling
>> them that their config won't work, and they need to either select more
>> drives or a different vdev type
>>
> I agree on the msgbox -- and I still think the temporaneous infobox
> injected via --and-widget would be value-add.
>
> Question is what to do after the msgbox.
>
> I posit that if the vdev_type is valid but they don't have enough disks
> currently selected, that they be tossed back at the ZFS menu. However,
> if they instead select a vdev_type which couldn't possible be satisfied
> given the number of devices available to the hardware... keep them
> in the vdev_type menu. 
>
>
>>>> 5. Whatever else you guys find wrong tonight
>>>>
>>>> I generated new test images, and attached the patch (which got REALLY
>>>> big when Devin Teske decided to fix "all of the things":
>>>>
>>> And then I merged "all of the things" into HEAD, so the patch-set shrunk
>>> back to its normal size. Now we have global exit codes which will make
>>> merging of code that is based off of Thomas Dickey's samples easier.
>> I am glad to see all of the good ideas, and plans to make everything
>> wonderful, but my biggest concern is getting this over to re_at_ so it can
>> get in to 10.0-BETA1, the deadline for which is looming (like, tomorrow
>> I think).
>>
> I hate to say it... but it was the netconfig and keymap changes that put us
> out of our way. If anything, I'd like to see those get dropped and have us
> focus on ZFS.
>
The netconfig changes other than Warren's wireless patch have been
backed out

The keymap change has been simplied down to just Warren's test system,
but implemented as a menu instead of a yes/no/other box

This can be redone better later

>> As such, I have rolled back the patches to netconfig and netconfig_ipv4
>> (my stuff to reduce the number of dialogs to configure ipv4, it posed
>> some problems with the possible usage of xdialog, and didn't actually
>> offer an option to 'cancel'). I kept Warren's netconfig wireless patch
>>
> Cool. We're thinking along the same lines.
>
>
>> This leaves the only real outstanding problem the keymap thing. I
>> propose changing it from a yes/no/other to a --menu, and hopefully we
>> can find the bug with the display name, or just make it show the keymap
>> name instead.
>>
> Last night I started doing what I knew "had to be done".
>
> Just as was the case with "bsdconfig timezone" ... I literally went into
> tzsetup(8) and converted the C code line-by-line to shell. (don't take
> that *too* literally... swaths of optimization were applied int he process;
> point being that the shell quite-ably reads the ISO3166 tables and zone
> files in the same *exact* manner as tzetup).
tzsetup says 'if you are not sure if your CMOS clock is in UTC, choose
no', but the default in the dialog menu is not 'no'. This is wrong. I
couldn't fix this because tzsetup is C not SH

> I've gone into kbdmap and looked at how it parses things.
>
> No biggie... looks like INDEX.keymaps (whose suffix matches the directory
> he lives in) is nothing more than a colon-delimited 3-field syntax with leading
> whitespace chopped and a comment-character of "#". Nothing a trival awk(1)
> script couldn't handle.
>
> To make things cute... I've got the code parsing into a shell structs (the same
> way I parse dhclient.leases and other file formats). By having an awk script
> read the file and generate shell statements that in-turn load the data into the
> exigent namespace.
>
> I'll see what I can get in ASAP.
If you can work something out, great. I fixed it 'good enough' for now


-- 
Allan Jude