Re: ezjails, systat -ifstat, and multiple network cards

On Thu, Feb 13, 2014 at 2:22 PM, Allan Jude <freebsd_at_allanjude.com> wrote:

> On 2014-02-13 13:59, Preston Hagar wrote:
> > I have a server setup with FreeBSD-10.0-RELEASE.  It has 3 Intel gigabit
> > network cards in it, em0, em1, and em2.  I have multiple ezjails setup
> that
> > run various things.
> >
> > One jail, called db, runs a postgresql database.  It was my intention to
> > give it em0 all to itself.   The other jails and host machine should be
> > going through em2.  em1 currently isn't being used.
> >
> > If I do an ifconfig, I see that em0 has the alias IP for my db jail and
> em2
> > has the alias IP for all other jails.  All the jails respond to network
> > traffic as expected and seemingly work fine.
> >
> > The weird thing is when I do a systat -ifstat from the host, it should
> > essentially all traffic going through em0.  Some of the jails that run
> off
> > of em2 (as defined in their jail config files and seen in ifconfig) have
> > large data transfers and/or are web servers with lots of photos.  I have
> > even tried to manually scp a large file out of a jail setup through em2
> and
> > the numbers don't seem to budge.
> >
> > If I do netstat -i -b -n -I  and check em0 and em2, it seems to support
> the
> > numbers shown by systat -ifstat.  However, if I use trafshow or iftop
> (both
> > of which require choosing one interface at a time), they both seem to
> > indicate the traffic flowing through the interfaces as I would expect.
> >
> > So I was curious if anyone had seen something like this before or had any
> > ideas of what is going on.  I have net.fibs=2 set in /boot/loader.conf,
> but
> > in all the jails I current have jail_name_fib="" as I haven't got around
> to
> > fullying setting up fibs.  Is that perhaps the issue?  Is there any way
> to
> > determine with certainty which jail is using which interface short of
> > physically pulling a network cable and seeing what stops working?
> >
> > Here are the relevant lines from my db (the one that should be on em0)
> > config:
> >
> > export jail_db_hostname="db"
> > export jail_db_ip="em0|10.1.10.2"
> >
> > From another jail on em2 called www:
> >
> > export jail_www_hostname="www"
> > export jail_www_ip="em2|10.1.10.7"
> >
> > from ifconfig
> >
> > em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
> > ether 08:60:6e:13:94:06
> > inet 10.1.1.4 netmask 0xffff0000 broadcast 10.1.255.255
> > inet6 fe80::a60:6eff:fe13:9406%em0 prefixlen 64 scopeid 0x1
> > inet 10.1.10.2 netmask 0xffffffff broadcast 10.1.10.2
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet autoselect (1000baseT <full-duplex>)
> > status: active
> >
> > em2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
> >
> options=4219b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,WOL_MAGIC,VLAN_HWTSO>
> > ether 68:05:ca:13:74:2a
> > inet 10.1.1.2 netmask 0xffff0000 broadcast 10.1.255.255
> > inet6 fe80::6a05:caff:fe13:742a%em2 prefixlen 64 scopeid 0x3
> > inet 10.1.10.3 netmask 0xffffffff broadcast 10.1.10.3
> > inet 10.1.10.1 netmask 0xffffffff broadcast 10.1.10.1
> > inet 10.1.10.8 netmask 0xffffffff broadcast 10.1.10.8
> > inet 10.1.10.10 netmask 0xffffffff broadcast 10.1.10.10
> > inet 10.1.10.4 netmask 0xffffffff broadcast 10.1.10.4
> > inet 10.1.10.9 netmask 0xffffffff broadcast 10.1.10.9
> > inet 10.1.10.7 netmask 0xffffffff broadcast 10.1.10.7
> > nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
> > media: Ethernet autoselect (1000baseT <full-duplex>)
> > status: active
> >
> >
> > Let me know if any more detail would be helpful or if you have any ideas
> of
> > things to check.
> >
> > Thanks,
> >
> > Preston
> > _______________________________________________
> > freebsd-current_at_freebsd.org mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "
> freebsd-current-unsubscribe_at_freebsd.org"
> >
>
> All traffic going out from the jails will using the routing table from
> the host system. The routing table will use the network card that is in
> the same subnet as your default gateway to route the traffic to the
> internet.
>
> In your case, I would imagine this is 10.1.1.4/16 (and 10.1.1.2/16).
>
> 'netstat -rn' will tell the tale, but I imagine it is whichever was
> added first.
>
> If you want to have separate routing tables per jail, you'd have to
> either use FIBs, and set the jails to use the different FIBs, or use
> VNET jails and have a routing table in each jail.
>
> --
> Allan Jude
>
>
Makes sense, thank you.  I'll setup the FIBs.

Preston