Summary: experiences with NanoBSD, successes and nits on a Soekris 4801

While in San Francisco a week or two ago, Sam Leffler kindly handed me a 
Soekris net4801 system.  After borrowing the necessary power parts from 
Mark Murray here in Cambridge, and picking up a slightly overpriced flash 
disk from a local electronics store, I decided I'd experiment some with 
NanoBSD and see where it got me.  In particular, I wanted to configure a 
DHCP/NAT/firewall box, and figured that this should be relatively straight 
forward, as it's likely one of the most common uses of embedded hardware 
of this type.

I general, I was quite pleased with the experience.  NanoBSD is fairly 
straight forward to configre and adapt.  I ran into a number of nits, 
several related to the recent dhclient changes that have not fit well into 
the NanoBSD environment.  I thought I'd provide a brief summary of what 
worked, and what didn't, on the way to having it up and running.  I should 
point out that I'm now reaching the rest of the world through the box, and 
was pleased with how easy the NanoBSD work made it to trivially assemble 
flash images -- really a boon for this sort of thing.  Kudos also to the 
Soekris guys -- it's a pleasure to work with a box that's so well tuned 
for running an open source OS in an embedded network environment.

Nits are identified with an XXX in the left of the text.  These are 
opportunities for people with interests in this space to investigate what 
went wrong :-).

- First, I configured src/tools/tools/nanobsd/make.conf.

 	- Primarily, this was to configure a kernel that did not have
 	  debugging enabled, disable SMP, and so on.  I also enabled
 	  WITH_GEOM_VOL and GEOM_VOL in the kernel.  See below for more on
 	  this.

 	- I deleted NO_KLDLOAD and the NANOBSD_PRUNE of usr/share/misc.
 	  I'm too much of a vi guy to go without termcap entries.

 	- I added NET4801 to the CUSTOMIZE= list.

 	- I enabled CUSTOMIZE_ARGS=${.CURDIR}/localfiles.

- I created a src/tools/tools/nanobsd/etc directory in which to place my
   local configuration files.  In particular, rc.conf, ipfw.conf, and
   dhcpd.conf.

- I created an rc.conf file.  There are three if_sis interfaces in the
   net4801 system -- I used the first interface for an external network
   port, and the second and third interfaces for two different internal
   networks (one wired, one wireless).  Due to the nature of the network
   environment I run in, it was necessary for me to configure a specific
   ethernet address as part of the ifconfig_sis0 line.  I was pleased that
   it's now possible to combine directives with DHCP for ifconfig -- I was
   displeased that this only sort of worked.  First nit:

XXX As I posted last night, the ifconfig_sis0="ether XXX DHCP" line worked
   fine for launching dhclient and running ifconfig first, but it caused
   natd to fail to match the interface as requiring -dynamic, requiring me
   me to manually configure -dynamic for natd in rc.conf.

- I downloaded the ISC dhcpd package to use for a DHCP server, and added a
   packages directive to localfiles to cause it to be installed on the
   image.  Next nit.

XXX It appears package installs involving user/group additions don't work
   with NanoBSD.  Specifically, when the package is added, it looks like it
   tries to add the user to the build system's password/group files, and
   never end up in the DESTDIR.  As a result, DHCPd couldn't switch to the
   dhcpd user.  I added rc.conf lines to force DHCPd to use the existing
   _dhcp user, rather than modifying the src/etc files.  Obviously, not
   pleased that my build system /etc files were modified either.

- I configured dhcpd.conf and configured localfiles to install it in the
   right place.

On boot, I run into the following as yet unresolved nits:

XXX As I posted last night, dhclient now exits if there is no link,
   relying on devd to restart it.  Unfortunately, devd relies on C++, and
   NanoBSD disables CXX, meaning that once dhclient detects a down link on
   a NanoBSD box, that link will never have IP configured on it again.
   This is a big problem.

XXX Every boot, I get savecore warnings:

   Loading configuration files.
   kenv: unable to get dumpdev
   No suitable dump device was found.
   ...
   /etc/rc: WARNING: Dump device does not exist.  Savecore not run.

   I've not yet tracked down why that is, but suspect a missing directive,
   no swap, or the like.  This isn't worth an upper-case WARNING.

XXX I get a sendmail warning every boot:

   sendmail: execing /usr/libexec/sendmail/sendmail: No such file or
   directory

   This is likely vi.recover running and trying to do something with
   sendmail.  Unhelpful, given no sendmail is present.  Maybe a better
   failure mode is desirable here?

XXX I liked that it seemed easy to do incremental rebuilds with NanoBSD,
   but found I had to do a fair amount of manually deleting intermediate
   targets to get it to work.  One thing I found that really didn't work
   was that if you install any third party packages, you have to blow away
   the DESTDIR install tree or when you re-build, the packages conflict and
   abort the image creation.

XXX The following FAILURE message from ATA seems a bit extreme, perhaps 
the word warning would be more appropriate:

   ad1: FAILURE - SETFEATURES ENABLE RCACHE status=51<READY,DSC,ERROR> error=4<AB
   ad1: FAILURE - SETFEATURES ENABLE WCACHE status=51<READY,DSC,ERROR> error=4<AB

However, the box is now up and running -- at least, until the link goes 
down and dhclient exits.  The box is quite neat, and although there are a 
fair number of details above, they're largely integration problems, 
several from recent changes that probably haven't filtered into the 
NanoBSD world yet.  Most are relatively minor -- the big issue is devd, 
which is something that probably is needed in the embedded world.  I think 
what NanoBSD wants isn't NO_CXX, it's DONT_INSTALL_BUILDTOOLS -- don't 
need a compiler, but using C++ applications should be fine.

Robert N M Watson