Re: ezjails, systat -ifstat, and multiple network cards

On 2014-02-13 13:59, Preston Hagar wrote:
> I have a server setup with FreeBSD-10.0-RELEASE.  It has 3 Intel gigabit
> network cards in it, em0, em1, and em2.  I have multiple ezjails setup that
> run various things.
> 
> One jail, called db, runs a postgresql database.  It was my intention to
> give it em0 all to itself.   The other jails and host machine should be
> going through em2.  em1 currently isn't being used.
> 
> If I do an ifconfig, I see that em0 has the alias IP for my db jail and em2
> has the alias IP for all other jails.  All the jails respond to network
> traffic as expected and seemingly work fine.
> 
> The weird thing is when I do a systat -ifstat from the host, it should
> essentially all traffic going through em0.  Some of the jails that run off
> of em2 (as defined in their jail config files and seen in ifconfig) have
> large data transfers and/or are web servers with lots of photos.  I have
> even tried to manually scp a large file out of a jail setup through em2 and
> the numbers don't seem to budge.
> 
> If I do netstat -i -b -n -I  and check em0 and em2, it seems to support the
> numbers shown by systat -ifstat.  However, if I use trafshow or iftop (both
> of which require choosing one interface at a time), they both seem to
> indicate the traffic flowing through the interfaces as I would expect.
> 
> So I was curious if anyone had seen something like this before or had any
> ideas of what is going on.  I have net.fibs=2 set in /boot/loader.conf, but
> in all the jails I current have jail_name_fib="" as I haven't got around to
> fullying setting up fibs.  Is that perhaps the issue?  Is there any way to
> determine with certainty which jail is using which interface short of
> physically pulling a network cable and seeing what stops working?
> 
> Here are the relevant lines from my db (the one that should be on em0)
> config:
> 
> export jail_db_hostname="db"
> export jail_db_ip="em0|10.1.10.2"
> 
> From another jail on em2 called www:
> 
> export jail_www_hostname="www"
> export jail_www_ip="em2|10.1.10.7"
> 
> from ifconfig
> 
> em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
> ether 08:60:6e:13:94:06
> inet 10.1.1.4 netmask 0xffff0000 broadcast 10.1.255.255
> inet6 fe80::a60:6eff:fe13:9406%em0 prefixlen 64 scopeid 0x1
> inet 10.1.10.2 netmask 0xffffffff broadcast 10.1.10.2
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> 
> em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
> ether 68:05:ca:13:74:2a
> inet 10.1.1.2 netmask 0xffff0000 broadcast 10.1.255.255
> inet6 fe80::6a05:caff:fe13:742a%em2 prefixlen 64 scopeid 0x3
> inet 10.1.10.3 netmask 0xffffffff broadcast 10.1.10.3
> inet 10.1.10.1 netmask 0xffffffff broadcast 10.1.10.1
> inet 10.1.10.8 netmask 0xffffffff broadcast 10.1.10.8
> inet 10.1.10.10 netmask 0xffffffff broadcast 10.1.10.10
> inet 10.1.10.4 netmask 0xffffffff broadcast 10.1.10.4
> inet 10.1.10.9 netmask 0xffffffff broadcast 10.1.10.9
> inet 10.1.10.7 netmask 0xffffffff broadcast 10.1.10.7
> nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> media: Ethernet autoselect (1000baseT <full-duplex>)
> status: active
> 
> 
> Let me know if any more detail would be helpful or if you have any ideas of
> things to check.
> 
> Thanks,
> 
> Preston
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
> 

All traffic going out from the jails will using the routing table from
the host system. The routing table will use the network card that is in
the same subnet as your default gateway to route the traffic to the
internet.

In your case, I would imagine this is 10.1.1.4/16 (and 10.1.1.2/16).

'netstat -rn' will tell the tale, but I imagine it is whichever was
added first.

If you want to have separate routing tables per jail, you'd have to
either use FIBs, and set the jails to use the different FIBs, or use
VNET jails and have a routing table in each jail.

-- 
Allan Jude