FreeBSD as read-only firmware

Hello all!
Some time ago I got somewhere idea, that base OS should be RO - readonly.
And should be updated easily (ACID) and with possibility of fast rollback.

So, basically ideas was to have some RO root, with unionfs with other dirs.
But then all machines was real (not virtualised), and predict some best
layout of partitioning was hard.
ZFS was nice, but it is not supported by unionfs (not fully IIRC).

When I'm started working with virtual machines this idea is came back, and
with help from Andriy Gapon, I made some progress.
So, here I got some proof-of-concept, with FreeBSD in RO mode, and I want
to share.
Comments and thoughts appreciated!

So, how to achieve RO FreeBSD? Firstly, you need create media with OS
itself, and have it in .vmdk format (you can import it into any VM machine,
and boot from it).
My scripts doing install in memory-based md disk, dump it to flat file,
creating from flat file .vmdk (ATA) by VBoxManage, and convert descriptor
file, so VMware Esxi can import disk as SCSI (much faster than ATA).

Here disks I have in VM:

r24243.vmdk 750Mb #OS -- I did not tried to  shrink it as possible, it
could be much less if you want to.
disk1.vmdk 2Gb #  128M = etcfs, 1.8Gb = localfs
disk2.vmdk XXXGb # data here
disk3.vmdk 8Gb #swap

here's how look mount:

/dev/gpt/r242434 on / (ufs, local, read-only) # r24243.vmdk
devfs on /dev (devfs, local, multilabel)
/dev/gpt/etcfs on /etc-rw (ufs, local, noatime) # disk1.vmdk
<above>:/etc-rw on /etc (unionfs, local)
/dev/gpt/localfs on /usr/local (ufs, local, noatime) # disk1.vmdk
/dev/md0 on /tmp (ufs, local)
<above>:/usr/local/usr-bin-rw on /usr/bin (unionfs, local)
<above>:/usr/local/root-rw on /root (unionfs, local)
devfs on /usr/local/var-root/named/dev (devfs, local, multilabel)
/dev/gpt/data on /usr/local/data (ufs, local) # disk2.vmdk

Also, /var is symlinked to /usr/local/var-root.

So, this setup shows how to upgrade OS = just replace  r24243.vmdk with
some  REL10.vmdk  or else rev.
Also, /usr/local update possible - just setup all in some other machine,
and bring new  disk1.vmdk with fresh programs (Assuming all data is moved
or symlinked to partition with data).

The only caveat I got - is Perl (and probably some other programs) required
to place some symlinks to /usr/bin <- so I had to create usr-bin-rw.
Maybe there will be some other problems, but now I have production server
set up in such way, working and not complainig :)

So, this all raised such questions in my head - is it OK to have configs,
scripts, data dirs, and default configs all messed in /etc ? :)
I understand that this is how it historically happen, but have all
partition in RW mode just to be able edit few files...
Not want produce flamewar here, but this dir I think should be revised in
some future.


You can view my scripts here:
https://github.com/yerenkow/freebsd-vm-image/tree/master/freebsd-firmware

Thanks!

-- 
Regards,
Alexander Yerenkow