Re: LUA boot loader coming very soon

On Monday, February 12, 2018 02:31:46 PM Warner Losh wrote:
> On Mon, Feb 12, 2018 at 10:12 AM, John Baldwin <jhb_at_freebsd.org> wrote:
> 
> > On Monday, February 12, 2018 08:27:27 AM Warner Losh wrote:
> > > Greetings,
> > >
> > > As you may know, the Lua (http://www.lua.org) boot loader has been in
> > the
> > > works for some time. It started out life as a GSoC in 2014 by Pedro Souza
> > > mentored by Wojciech A. Koszek. Rui Paulo created a svn project branch to
> > > try to integrate it. I rebased that effort into a github branch which
> > Pedro
> > > Arthur fixed up. Over the past year, I've been cleaning up the boot
> > loader
> > > for other reasons, and found the time was ripe to start integrating this
> > > into the tree. However, those integration efforts have taken a while as
> > my
> > > day-job work on the boot loader took priority. In the mean time, Ed Maste
> > > and the FreeBSD Foundation funded Zakary Nafziger to enhance the original
> > > GSoC Lua scripts to bring it closer to parity with the evolution of the
> > > FORTH menu system since the GSoC project started.
> > >
> > > I'm pleased to announce that all these threads of development have
> > > converged and I'll be pushing the FreeBSD Lua Loader later today. This
> > > loader uses Lua as its scripting language instead of FORTH. While
> > > co-existance is planned, the timeline for it is looking to be a few weeks
> > > and I didn't want to delay pushing this into the tree for that.
> > >
> > > To try the loader, you'll need to build WITHOUT_FORTH=yes and
> > > WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do this,
> > > you can do it in src/stand and install the result (be sure to have the
> > > options for both the build and the install). This will replace your
> > current
> > > /boot/loader that is scripted with FORTH to one that's scripted with Lua.
> > > It will install the lua scripts in /boot/lua. The boot is scripted with
> > > /boot/lua/loader.lua instead of /boot/loader.rc. You are strongly advised
> > > to create a backup copy of /boot/loader before testing (eg cp
> > /boot/loader
> > > /boot/loader_forth), since you'll need to boot that from boot2 if
> > something
> > > goes wrong. I've tested it extensively, though, with userboot.so and it's
> > > test program, so all the initial kinks of finding the lua scripts, etc
> > have
> > > been worked out.
> > >
> > > While it's possible to build all the /boot/loader variants with Lua, I've
> > > just tested a BIOS booting /boot/loader both with and without menus
> > > enabled. I've not tested any of the other variants and the instructions
> > for
> > > testing some of them may be rather tedious (especially UEFI, if you want
> > a
> > > simple path to back out). Since there's not been full convergence
> > testing,
> > > you'll almost certainly find bumps in this system. Also, all the
> > > build-system APIs are likely not yet final.
> > >
> > > I put  MFC after a month on the commit. Due to the heroic (dare I say
> > > almost crazy) work of Kyle Evans on merging all the revs from -current to
> > > 11, I'm planning a MFC to 11 after the co-existence issues are hammered
> > > out. In 11, FORTH will be the default, and Lua will  be built by default,
> > > but users will have to do something to use it. 12, both FORTH and Lua
> > will
> > > be built and installed, with Lua as default (barring unforeseen
> > > complications). Once the co-existence stuff goes in, I imagine we'll make
> > > the switch to Lua by default shortly after that. In 13, FORTH will be
> > > removed unless there's a really really compelling case made to keep it.
> > >
> > > So please give it a spin and give me any feedback, documentation updates
> > > and/or bug fixes. I'm especially interested in reviews from people that
> > > have embedded Lua in other projects or experts in Lua that can improve
> > the
> > > robustness of the menu code.
> >
> > Do you have some memory usage numbers for LUA vs forth for the different
> > BIOS loaders (text/data/bss sizes)?  For the EFI case we probably have lots
> > of room, but for the non-EFI case we are constrained to 0xa0000 - 0xa000
> > for the text/data/bss and stack (in some cases like PXE booting the top
> > can be lower than 0xa0000).  I'm not sure if we have any other platforms
> > with similar memory constraints.
> >
> 
> Lua is about 60-70k larger than FORTH for this application:
> 
> Forth based:
> -r-xr-xr-x  1 root  wheel  385024 Feb 12 14:14
> /home/imp/roots/amd64/boot/loader
> -r-xr-xr-x  1 root  wheel  450560 Feb 12 14:14
> /home/imp/roots/amd64/boot/zfsloader
> Lua based:
> -r-xr-xr-x  1 root  wheel  450560 Feb 12 14:20
> /home/imp/roots/amd64/boot/loader
> -r-xr-xr-x  1 root  wheel  520192 Feb 12 14:20
> /home/imp/roots/amd64/boot/zfsloader
> 
> So with Lua:
> %  size loader.bin
>     text    data     bss      dec       hex   filename
>   411840   22264   48304   482408   0x75c68   loader.bin
> % size zfsloader.bin
>     text    data     bss      dec       hex   filename
>   478712   22968   52368   554048   0x87440   zfsloader.bin
> 
> so we're at 555k out of 640k before we look at heap usage. Of course, these
> numbers are only going to get a lot worse as we pile-in new crypto
> features... I know that the normal loader works, but I've not tried the
> zfsloader.

Woof, on stable/11 the forth ones are quite a bit smaller:

-r-xr-xr-x  1 root  wheel  315392 Dec 16 13:39 /boot/loader
-r-xr-xr-x  1 root  wheel  356352 Dec 16 13:39 /boot/zfsloader

% size /usr/obj/usr/src/sys/boot/i386/loader/loader.bin 
    text    data     bss      dec       hex   filename
  274913   22624   46560   344097   0x54021   /usr/obj/usr/src/sys/boot/i386/loader/loader.bin
% size /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin 
    text    data     bss      dec       hex   filename
  313865   23184   50624   387673   0x5ea59   /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin

Note that it's not 640k, it's 0xa0000 (640k) - the start address of the 
loader: 0xa000 (40k), so text/data/bss + stack has 600k of room.  The heap
isn't in low memory anymore, so we don't have to worry about that.  555k for
ZFS loader leaves around 45k for stack.  If the ZFS loader code is anything
like the kernel code then ZFS probably uses deeper stacks than UFS. :(

One of the larger uses of stack in the loader is probably gzipfs, so loading
a mfsroot.gz from the loader might be a good test.  Unfortunately, there isn't
a good way to really detect stack overflow other than silent corruption in
bss.  If we could create a 'volatile int canary' in the bss and force it to be
the last thing in the bss we could check the value of 'canary' in the loader's
main loop perhaps and complain if it is ever non-zero.

-- 
John Baldwin