Re: VIMAGE: vnet, epair and lots of jails on bridgeX - routing

On Sat, 10 Feb 2018 09:54:49 +0100
Marko Zec <zec_at_fer.hr> wrote:

> On Sat, 10 Feb 2018 08:52:21 +0100
> "O. Hartmann" <ohartmann_at_walstatt.org> wrote:
> 
> > Am Fri, 09 Feb 2018 16:43:17 +0000
> > "Bjoern A. Zeeb" <bzeeb-lists_at_lists.zabbadoz.net> schrieb:
> >   
> > > On 9 Feb 2018, at 16:22, O. Hartmann wrote:
> > >     
> > > > Am Thu, 8 Feb 2018 09:31:15 +0100
> > > > "O. Hartmann" <ohartmann_at_walstatt.org> schrieb:
> > > >
> > > > Is this problem to trivial?      
> > > 
> > > I read through it yesterday and found myself in the position that I
> > > need a whiteboard or paper and pencil or an ASCII art of your
> > > situation.  But by the time I made it to the question I was
> > > basically lost.  Could you massively simplify this and maybe
> > > produce the ASCII art?
> > > 
> > > /bz
> > > _______________________________________________
> > > freebsd-current_at_freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to
> > > "freebsd-current-unsubscribe_at_freebsd.org"    
> > 
> > All right.
> > 
> > I'm not much of an artist and at this very moment, I haven't much
> > experience with neat ASCII art tools. But I'll provide a sketch
> > later, but I also will simplify  the situation.
> > 
> > Consider three "vswitches", basically based on the creation of
> > bridges, bridge0, bridge1, bridge2. Create at least three individual
> > vnet-jails attached to each vbridge. Those jails have epair pseudo
> > devices. The jail itself owns the "a-part" of the epair and the
> > b-part is "member of the bridge". Each jail's epairXXXa has an IP
> > assigned of the network the vswitch is part of. I mention a- and
> > b-part of the epair here, because I thought it could matter, but I
> > think for symmetry reasons it doesn't.
> > 
> > Now consider a further, special jail. This jail is supposed to have
> > three epair devices, each one is reaching into one of the vbridges.
> > This jail is the router/routing jail. Later, this jail should filter
> > via IPFW the traffic between the three vbridges according to rules,
> > but this doesn't matter here, beacuase the basics are not working as
> > expected.
> > 
> > Now the problems. It doesn't matter on which jail of the three
> > vswitches I login, the moment a vbridge has more than two member
> > epairs (one  is alway member of the routing jail, now consider a
> > database jail and a webserver jail), pinging each jail or the routing
> > jail fails. It works sometimes for a couple of ICMP packets and then
> > stops.
> > 
> > If each vbridge has only one member jail, I have NO PROBLEMS
> > traversing accordingly to the static routing rules from one vbridge
> > to any other, say from vbridge1 to vbridge0 or vbridge2 and any
> > permutation of that.
> > 
> > The moment any of the bridges gets an additional member epair
> > interface (so the bridge has at least three members including the on
> > reaching into the virtual router jail) the vbridge seems to operate
> > unpredictable (to me). Pinging jails memeber of that vbridge are
> > unreachable.
> > 
> > Technical information:
> > 
> > The kernel has options IPFIREWALL, VIMAGE. The host's ipfw (kernel)
> > declines packets by default. Each jail is configured to have ipfw
> > "open".
> > 
> > Thanks for the patience.  
> 
> If you could provide a script which sets up the topology you described
> in two lengthy posts then others could reproduce this, and your chances
> of getting useful feedback would certainly increase.
> 
> We also have a graphical tool (https://github.com/imunes/imunes) which
> can set up a topology like you described in a few clicks of a mouse,
> albeit using netgraph and ng_eiface instead of epairs, but I assume this
> is irellevant as long as you are not aiming for maximum packet
> throughputs.  If you attempt to use this tool, note that selecting
> "stpswitch" will create if_bridge instances, whereas "lanswitch"
> creates ng_bridge instances.
> 
> Good luck,
> 
> Marko
> 
> 
> > 
> > Kind regards,
> > 
> > O. Hartmann  
> 
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"

Hello Marko,

thanks for your response. First of all: I looked at "imunes". From the first
glimpse it looks great! Something really usefull; I regret not having a port
for this tool or the chance to package it via poudriere.

The problems I faced seem to be related to a bug Olivier Cochard-Labbe pointed
me at:

 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=176671

Checking the MAC of the epairs created revealed, that either doubles or even
more occur on the host side of the epair (in my case, all the b-parts of an
epair), or, if there is nothing irregular, then the a-parts (owned by the
VIMAGE jail) have same MAC. The more jails I create, the more ambiguous MACs
are present.

It is the first time I ran into a more complex network topology and thanks for
the hint using netgraph. 

Kind regards,

Oliver