Re: VirtualBox network connectivity broken on recent -CURRENT

On Sun, Jun 05, 2016 at 12:35:29PM -0700, Don Lewis wrote:
> On  4 Jun, To: mmacy_at_nextbsd.org wrote:
> > On  4 Jun, Matthew Macy wrote:
> >> 
> >> 
> >> 
> >>  ---- On Sat, 04 Jun 2016 19:04:42 -0700 Randy Westlund
> >>  <rwestlun_at_gmail.com> wrote ----
> >>  > On Fri, Jun 03, 2016 at 05:11:24PM -0700, Don Lewis wrote: 
> >>  > > It looks like something changed in -CURRENT to break network 
> >>  > > connectivity to VirtualBox guests.  This was last known to work
> >>  > > with r299139 (May 6th) and is definitely broken with r301229.
> >>  >  
> >>  > I've been having VirtualBox networking problems as well.  I can't
> >>  > get my VMs on the network recently, but I don't recall when it
> >>  > last worked. Everything looks right from the guest (the arp cache
> >>  > shows the VirtualBox NAT router), but tcpdump on the host shows no
> >>  > traffic.  I haven't had time to investigate further :/
> >>  > 
> >> 
> >> The odds of it being fixed will increase greatly if someone would do a
> >> bisect and test.
> > 
> > I started bisecting a bit earlier today.  So far all I know is r300184
> > is also broken.
> 
> The culprit turned out to be r300043.  There seems to be some
> sort of ABI compatiblity issue that was resolved when I rebuilt and
> reinstalled virtualbox-ose-kmod.  There should probably be note in
> src/UPDATING if so.  The .kmod files that I had been
> running were probably built with r299139.
Not ABI, bit KBI.  The revision changed layout of the struct thread,
which occurs semi-regularly on HEAD, but we maintain the layout for
struct proc and struct thread for existing fields on stable.

I also changes the layouts of struct proc and thread today, please see
r301456.

I am not sure about __FreeBSD_version bump for that changes, since they
do not add or remove any consumable interfaces.  The sole purpose of the
bump would be to delineate the modules build time.

> 
> I used to have PORTS_MODULES=emulators/virtualbox-ose-kmod in
> /etc/make.conf, but took it out because pkg would also try to
> intall/upgrade it from my local pkg repo.
> 
> What is strange is that the kmods would successfully load even on very
> recent kernels, but when I did a "pkg install -f virtualbox-ose-kmod"
> to fetch a recently rebuilt version (just a few days old), it would not
> load with a r300043 kernel.  Is there a check to prevent a module with a
> newer version from loading on an older kernel, but not the reverse?
Yes, the __FreeBSD_version from the headers pack used for module build
is embedded into module metadata, and module loader checks it against
the version of the kernel.  It makes sense on stable, but not much on
HEAD, where only exact match would ever reasonable.

Even on stable, we only guarantee partial KBI stability, mostly
to interfaces relevant for device drivers, and not for e.g. ABI emulators.
For such modules which are too tied to the kernel, I added the
DECLARE_MODULE_TIED() declarator.  Might be vbox kmods need to start
using that (if not using _TIED already).