Re: FreeBSD 11 - BETA-1 Xen DOMU loses network when jail (VIMAGE) starts

On Tue, Aug 09, 2016 at 03:09:37PM +0100, Miguel C wrote:
> On Tuesday, August 9, 2016, Roger Pau Monné <roger.pau_at_citrix.com> wrote:
> 
> > On Tue, Aug 09, 2016 at 12:12:34PM +0100, Miguel C wrote:
> > > Melhores Cumprimentos // Best Regards
> > > -----------------------------------------------
> > > *Miguel Clara*
> > > *IT - Sys Admin & Developer*
> > >
> > > On Tue, Aug 9, 2016 at 9:55 AM, Roger Pau Monné <roger.pau_at_citrix.com
> > <javascript:;>>
> > > wrote:
> > >
> > > > On Sat, Jul 23, 2016 at 08:46:49PM +0100, Miguel C wrote:
> > > > > Just as a note using netgraph (with jng script as a workaround)
> > works....
> > > > >
> > > > > Also manually creating a bridge in the domu and adding xn0 as a
> > member
> > > > > makes this fail.... so the issue is indeed related to the bridge.
> > > > >
> > > > > I'll open a PR later in case someone want to look into it, but I'm
> > happy
> > > > it
> > > > > works with netgraph.
> > > >
> > > > I seem to be able to use xn* interfaces with bridges without problems:
> > > >
> > > > xn0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST>
> > metric 0
> > > > mtu 1500
> > > >         options=3<RXCSUM,TXCSUM>
> > > >         ether 00:16:3e:74:3d:76
> > > >         nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > > >         media: Ethernet manual
> > > >         status: active
> > > > bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0
> > mtu
> > > > 1500
> > > >         ether 02:77:3d:4a:18:00
> > > >         inet 172.16.1.140 netmask 0xffffff00 broadcast 172.16.1.255
> > > >         nd6 options=9<PERFORMNUD,IFDISABLED>
> > > >         groups: bridge
> > > >         id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
> > > >         maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
> > > >         root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
> > > >         member: xn0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
> > > >                 ifmaxaddr 0 port 2 priority 128 path cost 2000000
> > > >
> > > > Is this a GENERIC kernel or are you using some custom
> > > > configuration/patches?
> > > > Can you provide some more information about how to reproduce this?
> > > >
> > > > GENERIC + VIMAGE, but that's just it, no other custom changes or
> > patches.
> > >
> > > Note however that this is under a NetbBSD Dom0, and I see the "vifXX"
> > > interface disappear in the Dom0 side when the bridge is create on FreeBSD
> > > DomU.
> > >
> > > I'm actually happy with netgraph, although I've never played with it, and
> > > seems more complex, the script provide in /share/examples is perfect to
> > use
> > > with "jail.conf" and pf seems happy in FreeBSD-11 (which is not CURRENT,
> > > should we move this to a different mailing list!?) too, no panics so far.
> > >
> > > I suspect the main issue, since it works fine for you is the fact that
> > this
> > > is in a NetBSD Dom0.
> >
> > Oh, from your previous email I thought that it was the interface inside of
> > the DomU that disappeared. Does then same happen on a NetBSD Dom0 with a
> > NetBSD DomU?
> >
> > Sorry I should have explained better, and no it does not happen with other
> guests not even FreeBSD 9 or 10, but VIMAGE has major issues there and some
> have been fixed in 11 (panics while using of for example), and I also
> needed a patch for xn to even work (also related to NetBSD dom0) but bridge
> did not give any issues.
> 
> It seems with 11 when I add xn0 to the bridge the dom0 thinks the interface
> was disconnected, and when that happens I guess the vif bridge script ( on
> dom0 ) destroys the interface.

Can you paste the output of `xenstore-ls -fp` on the Dom0 when that happens? 
Also are there any messages in the Dom0 dmesg?

You might want to modify src/sys/arch/xen/xen/xennetback_xenbus.c (on the 
NetBSD sources) to define XENDEBUG_NET, so that you get verbose output.

Roger.