Re: [UPDATE] sysutils/bsdconfig snapshot

(replying to my own request with some additional details on what to try in this update)

On May 7, 2013, at 1:36 PM, Teske, Devin wrote:

> Hello,
> 
> I've taken a new snapshot of HEAD usr.sbin/bsdconfig and made it available through the ports tree (as sysutils/bsdconfig). The last snapshot was quite some time months ago, so a lot has changed since then.
> 

Files have moved, there's now a /usr/share/bsdconfig directory for shared include files, and the fake language files have been removed. That's what you'll notice from what the new port installs.


> Most notably, we now have the beginnings of the package management module, edging ever-closer to a 1.0 release status.
> 

Of which it's important to note that this initial work is based on the old pkg_tools and not pkgng. We'll convert it to pkgng later, but first we need to make sure the framework for interacting with huge package data is usable.

One of the challenges in testing this feature is that on 9.1, there aren't any packages available (yet; that should change eventually -- being restored retroactively).

However, if you don't use the "FTP" media types, you'll get a lot further. What I've been using as a test is to download the 9.0-RELEASE packages/INDEX file and throw that into a local "packages" directory and then point the "Directory" media type at the parent directory where the "packages" directory was created.

For example, …

cd /tmp
mkdir packages
fetch ftp://ftp-archive.freebsd.org/pub/FreeBSD-Archive/old-releases/i386/9.0-RELEASE/packages/INDEX

Then use the "Directory" media type and enter a value of "/tmp". The "packages/INDEX" file will be found in the media ("/tmp" directory) and the the packages interface will then be presented.

If you're already running 9.0-RELEASE, then you can choose the "FTP Passive" media type, then choose the second menu item title "URL" and enter the URL "ftp://ftp-archive.freebsd.org".

ASIDE: /me notes that he really ought to change the "FTP" routine to use a default $FTPMODE of "auto" instead of "active" (currently, "FTP" means explicitly active, and "FTP Passive" means explicitly passive).

Once you're in the package interface, you will see:

1. List of package categories. NOTE: For a package category to be listed, it has to have at least one package. This list also includes a dummy category named "All".

2. For each category (including the "All" dummy category), the number of:

2.a. packages available in that category

2.b. packages that are installed from that category

2.c. "selected" packages (a "selected" package is one that has been been marked for install, re-install, un-install, or marked as a dependency of a package to install/re-install)

are displayed.

3. Above the category list is a "> Review" menu item. Use this to review the selections you've made.

Choosing a category intuitively displays the packages available from that category.

In the list of packages, packages that are already installed are marked with an "X"; those marked for install have an "I"; re-install have an "R"; un-install, "U"; and dependencies are marked with a "D".

Selecting a package that is not marked will turn the mark into an "I".

Selecting a package that is marked with an "X" will bring up a menu asking if you would like to mark it for Uninstall, Re-install, or return it back to just being installed.

Selecting a package that is marked as a dependency turns it into a package requested for installation (D mark becomes I mark). If without leaving the package selection screen you then re-select the same package (which was a dependency turned into an "I"), the mark goes away, but if you exit and return to the package selection menu, the mark is restored as a "D". This is because package dependencies are not calculated until you return to the category list (when counts must be updated).

When you're navigating the menus, you may notice an asterisk (*) on some menu items. This indicates that a page has been "cached". Unlike sysinstall which would spend ~10 minutes (on my hardware) processing the INDEX file, building the in-memory structures for all category submenus, bsdconfig only generates the menus you request, and does them on-the-fly.

The first time you are presented with the list of package categories, no category has an asterisk next to it. The first time you enter a package category, this is a small 1-5 second delay (depending on hardware) to generate the menu(s) for that category. When you return to the category list, that category will now have an asterisk displayed next to it to let you know:

a. this category has been visited and

b. the menus have been cached in-memory so re-entering this category will be fast (less than one second).

Another notable feature once you get to the package menus is that not all packages are displayed on one page. There is a global PACKAGES_MENU_SIZE which defaults to 2000 and sets the maximum number of packages that can appear on one page (in contrast, sysinstall would display all 21,000+ packages on one menu for the All category; sysinstall did not display "pages"). Part of this is due to a limitation in that we construct the menus using dialog(1) and therefore are limited by ARGMAX, but it also makes navigating the menus more efficient as the screens can redraw faster with less content.

===

In the above, we explored entering the packages module in a straight-forward manner. However, because bsdconfig is scriptable, there's also the scripted fashion.

Here are three I am scripting the package menu for my testing:

Method 1: Make a sysinstall-style script…

$ cat install.cfg
_directoryPath=/tmp
mediaSetDirectory
configPackages
$ bsdconfig -f install.cfg

NOTE: Again, /tmp has my "packages" directory with nothing but an "INDEX" file within

Method 2: Make a runnable script…

$ cat localpkg.sh
#!/bin/sh
. /usr/local/share/bsdconfig/script.subr
nonInteractive=YES
_directoryPath=/tmp
mediaSetDirectory
configPackages
$ ./localpkg.sh

Method 3: So-called "HERE" document or standard pipe

$ bsdconfig -f- <<EOF
_directoryPath=/tmp
mediaSetDirectory
configPackages
EOF

or

$ echo "_directoryPath=/tmp;mediaSetDirectory;configPackages" | bsdconfig -f-

===

And then there's how to debug this beast…

Like sysinstall, you can add "debug=1" to your script (in any of the above 3 forms of scripting).

For example:

#!/bin/sh
debug=1
. /usr/local/share/bsdconfig/script.subr
nonInteractive=YES
_directoryPath=/tmp
mediaSetDirectory
configPackages

However, you should be aware… unlike sysinstall, the default reaction to "debug=1" is to print lots of debugging information to the console.

If you want bsdconfig to instead put its debug statements into a file (like sysinstall, which created "sysinstall.debug"), set the debugFile environment variable. For example:

#!/bin/sh
debugFile=bsdconfig.debug
. /usr/local/share/bsdconfig/script.subr
nonInteractive=YES
_directoryPath=/tmp
mediaSetDirectory
configPackages

But you can optionally add a "+" to the beginning of the filename and then debug output will appear both on the console and in the debug file (minus the leading "+" of course). For example:

#!/bin/sh
debugFile=+bsdconfig.debug
. /usr/local/share/bsdconfig/script.subr
nonInteractive=YES
_directoryPath=/tmp
mediaSetDirectory
configPackages

But wait, there's more…

You can access these features from the command-line. Any one of the following is considered valid (see "bsdconfig -h" for more info):

bsdconfig -d # enables debug to console

bsdconfig -D bsdconfig.debug # enables debug to file

bsdconfig -D +bsdconfig.debug # enables debug to file and console

bsdconfig -d packages

bsdconfig packages -d

bsdconfig -D +foo packages

bsdconfig packages -D +foo




> I'd like to see if there are any interested folks out there that could give my updated version a go and provide some feedback. Any/all feedback would be greatly appreciated.
> 

Go through all the menus. One of the things you'll notice is that the program as a whole should feel more fluid. Everytime you enter a "submenu", the last item you selected is remembered and if/when you return that menu, your selection is restored.

I also went through and killed a lot of "--clear" arguments to dialog(1) and employed the rarely used "--keep-tite" arguments in an effort to make transitions between screens much more fluid.

I'm interested to know how the program "feels".

Also… I spent considerable effort working out the "startup" module and "usermgmt" modules. For the "startup" module, I implemented a third (fourth?) layer of cache management (lol) which speeds up the menus even further. For the "usermgmt" module I spent a lot of time making sure we never penalize choices that are merely the result of "exploring".

I spent a lot of time just sitting down in the program and exploring trying to make the program smooth through-and-through. Please let me know if you see any kinks or hiccups that detract from a smooth experience. A lot of this stuff is as much "User eXperience" programming than it is "User Interface" programming (or back-end programming).



> Just an FYI however… this code is only expected to work on 9.0-R or higher.

As mentioned earlier… in some cases it works better on 9.0 than 9.1 (e.g., lack of 9.1-RELEASE packages on the main FTP repositories). But package INDEX access aside with FTP media type, the experience between 9.x/10.x/HEAD should be consistent.
-- 
Cheers,
Devin

_____________
The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.