Re: LUA boot loader coming very soon

On Mon, Feb 12, 2018 at 2:53 PM, John Baldwin <jhb_at_freebsd.org> wrote:

> On Monday, February 12, 2018 02:31:46 PM Warner Losh wrote:
> > On Mon, Feb 12, 2018 at 10:12 AM, John Baldwin <jhb_at_freebsd.org> wrote:
> >
> > > On Monday, February 12, 2018 08:27:27 AM Warner Losh wrote:
> > > > Greetings,
> > > >
> > > > As you may know, the Lua (http://www.lua.org) boot loader has been
> in
> > > the
> > > > works for some time. It started out life as a GSoC in 2014 by Pedro
> Souza
> > > > mentored by Wojciech A. Koszek. Rui Paulo created a svn project
> branch to
> > > > try to integrate it. I rebased that effort into a github branch which
> > > Pedro
> > > > Arthur fixed up. Over the past year, I've been cleaning up the boot
> > > loader
> > > > for other reasons, and found the time was ripe to start integrating
> this
> > > > into the tree. However, those integration efforts have taken a while
> as
> > > my
> > > > day-job work on the boot loader took priority. In the mean time, Ed
> Maste
> > > > and the FreeBSD Foundation funded Zakary Nafziger to enhance the
> original
> > > > GSoC Lua scripts to bring it closer to parity with the evolution of
> the
> > > > FORTH menu system since the GSoC project started.
> > > >
> > > > I'm pleased to announce that all these threads of development have
> > > > converged and I'll be pushing the FreeBSD Lua Loader later today.
> This
> > > > loader uses Lua as its scripting language instead of FORTH. While
> > > > co-existance is planned, the timeline for it is looking to be a few
> weeks
> > > > and I didn't want to delay pushing this into the tree for that.
> > > >
> > > > To try the loader, you'll need to build WITHOUT_FORTH=yes and
> > > > WITH_LOADER_LUA=yes. Fortunately, you needn't do a full world to do
> this,
> > > > you can do it in src/stand and install the result (be sure to have
> the
> > > > options for both the build and the install). This will replace your
> > > current
> > > > /boot/loader that is scripted with FORTH to one that's scripted with
> Lua.
> > > > It will install the lua scripts in /boot/lua. The boot is scripted
> with
> > > > /boot/lua/loader.lua instead of /boot/loader.rc. You are strongly
> advised
> > > > to create a backup copy of /boot/loader before testing (eg cp
> > > /boot/loader
> > > > /boot/loader_forth), since you'll need to boot that from boot2 if
> > > something
> > > > goes wrong. I've tested it extensively, though, with userboot.so and
> it's
> > > > test program, so all the initial kinks of finding the lua scripts,
> etc
> > > have
> > > > been worked out.
> > > >
> > > > While it's possible to build all the /boot/loader variants with Lua,
> I've
> > > > just tested a BIOS booting /boot/loader both with and without menus
> > > > enabled. I've not tested any of the other variants and the
> instructions
> > > for
> > > > testing some of them may be rather tedious (especially UEFI, if you
> want
> > > a
> > > > simple path to back out). Since there's not been full convergence
> > > testing,
> > > > you'll almost certainly find bumps in this system. Also, all the
> > > > build-system APIs are likely not yet final.
> > > >
> > > > I put  MFC after a month on the commit. Due to the heroic (dare I say
> > > > almost crazy) work of Kyle Evans on merging all the revs from
> -current to
> > > > 11, I'm planning a MFC to 11 after the co-existence issues are
> hammered
> > > > out. In 11, FORTH will be the default, and Lua will  be built by
> default,
> > > > but users will have to do something to use it. 12, both FORTH and Lua
> > > will
> > > > be built and installed, with Lua as default (barring unforeseen
> > > > complications). Once the co-existence stuff goes in, I imagine we'll
> make
> > > > the switch to Lua by default shortly after that. In 13, FORTH will be
> > > > removed unless there's a really really compelling case made to keep
> it.
> > > >
> > > > So please give it a spin and give me any feedback, documentation
> updates
> > > > and/or bug fixes. I'm especially interested in reviews from people
> that
> > > > have embedded Lua in other projects or experts in Lua that can
> improve
> > > the
> > > > robustness of the menu code.
> > >
> > > Do you have some memory usage numbers for LUA vs forth for the
> different
> > > BIOS loaders (text/data/bss sizes)?  For the EFI case we probably have
> lots
> > > of room, but for the non-EFI case we are constrained to 0xa0000 -
> 0xa000
> > > for the text/data/bss and stack (in some cases like PXE booting the top
> > > can be lower than 0xa0000).  I'm not sure if we have any other
> platforms
> > > with similar memory constraints.
> > >
> >
> > Lua is about 60-70k larger than FORTH for this application:
> >
> > Forth based:
> > -r-xr-xr-x  1 root  wheel  385024 Feb 12 14:14
> > /home/imp/roots/amd64/boot/loader
> > -r-xr-xr-x  1 root  wheel  450560 Feb 12 14:14
> > /home/imp/roots/amd64/boot/zfsloader
> > Lua based:
> > -r-xr-xr-x  1 root  wheel  450560 Feb 12 14:20
> > /home/imp/roots/amd64/boot/loader
> > -r-xr-xr-x  1 root  wheel  520192 Feb 12 14:20
> > /home/imp/roots/amd64/boot/zfsloader
> >
> > So with Lua:
> > %  size loader.bin
> >     text    data     bss      dec       hex   filename
> >   411840   22264   48304   482408   0x75c68   loader.bin
> > % size zfsloader.bin
> >     text    data     bss      dec       hex   filename
> >   478712   22968   52368   554048   0x87440   zfsloader.bin
> >
> > so we're at 555k out of 640k before we look at heap usage. Of course,
> these
> > numbers are only going to get a lot worse as we pile-in new crypto
> > features... I know that the normal loader works, but I've not tried the
> > zfsloader.
>
> Woof, on stable/11 the forth ones are quite a bit smaller:
>
> -r-xr-xr-x  1 root  wheel  315392 Dec 16 13:39 /boot/loader
> -r-xr-xr-x  1 root  wheel  356352 Dec 16 13:39 /boot/zfsloader
>
> % size /usr/obj/usr/src/sys/boot/i386/loader/loader.bin
>     text    data     bss      dec       hex   filename
>   274913   22624   46560   344097   0x54021   /usr/obj/usr/src/sys/boot/
> i386/loader/loader.bin
> % size /usr/obj/usr/src/sys/boot/i386/zfsloader/zfsloader.bin
>     text    data     bss      dec       hex   filename
>   313865   23184   50624   387673   0x5ea59   /usr/obj/usr/src/sys/boot/
> i386/zfsloader/zfsloader.bin
>
> Note that it's not 640k, it's 0xa0000 (640k) - the start address of the
> loader: 0xa000 (40k), so text/data/bss + stack has 600k of room.  The heap
> isn't in low memory anymore, so we don't have to worry about that.  555k
> for
> ZFS loader leaves around 45k for stack.  If the ZFS loader code is anything
> like the kernel code then ZFS probably uses deeper stacks than UFS. :(
>

Yea. We support more FS types than in 11 now I think (at least more than in
10),
so we might want to look to trimming that. I don't think we need 3
different compressed
filesystem formats compiled in by default, for example...


> One of the larger uses of stack in the loader is probably gzipfs, so
> loading
> a mfsroot.gz from the loader might be a good test.  Unfortunately, there
> isn't
> a good way to really detect stack overflow other than silent corruption in
> bss.  If we could create a 'volatile int canary' in the bss and force it
> to be
> the last thing in the bss we could check the value of 'canary' in the
> loader's
> main loop perhaps and complain if it is ever non-zero.
>

That's not a terrible idea... Assuming that the corruption is such that
it's right at the end of bss, and not something a bit before it...

Warner