Re: Issues with Jails/Routes/FIBs

On Nov 25, 2010, at 1:38 PM, Kevin Mai <kma_at_mrecic.gov.ar> wrote:

> Hi folks! 
> 
> I'm facing an issue here while trying to define separate routing tables for each jail and host. 
> 
> Let me show you briefly how it's done: 
> 
> The server has 3 physical NICs, each one connected to a different network (say, public network A, public network B, and LAN). 
> 
> Currently, the default gateway is set to be the LAN gateway, even though the two jails can see their own public network subnet. 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 172.16.2.1 UGS 1 3935 bce2 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 172.16.2.127 link#3 UHS 0 0 lo0 
> 100.16.97.0/24 link#1 U 0 0 bce0 
> 100.16.97.5 link#1 UHS 0 0 lo0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 100.16.98.5 link#2 UHS 0 0 lo0 
> 
> 100.16.97.0/24 and 100.16.98.0/24 are the two public networks and 172.16.2.0/24 is the LAN. 
> 
> I have already tried removing devfs rules from the jails, setting securelevel to -1 but I'm still out of luck.. 
> 
> I know setfib can define alternate routing tables, and I even created a default gateway for two fibs, 1 & 2: 
> 
> [root_at_mrefns09 ~]# setfib 2 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.98.100 UGS 14 906 bce1 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 39 bce0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 
> [root_at_mrefns09 ~]# setfib 1 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.97.100 UGS 0 1758 bce0 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 44 bce0 
> 100.16.98.0/24 link#2 U 0 4 bce1 
> 
> And i've added the proper settings in rc.conf.. 
> 
> jail_athea97_ip="100.16.97.5 netmask 255.255.255.0" 
> jail_athea97_fib=1 
> 
> 
> jail_athea98_ip="100.16.98.5 netmask 255.255.255.0" 
> jail_athea98_fib=2 
> 
> Am I missing something? because once I get into the jail the routing table is the same: 
> 
> [root_at_athea97 /]# netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 172.16.2.1 UGS 13 6175 bce2 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 172.16.2.127 link#3 UHS 0 0 lo0 
> 100.16.97.0/24 link#1 U 0 0 bce0 
> 100.16.97.5 link#1 UHS 0 0 lo0 
> 100.16.98.0/24 link#2 U 0 0 bce1 
> 100.16.98.5 link#2 UHS 0 0 lo0 
> 
> [root_at_athea97 /]# setfib 1 netstat -rn 
> Routing tables 
> 
> Internet: 
> Destination Gateway Flags Refs Use Netif Expire 
> default 100.16.97.100 UGS 15 1814 bce0 
> 127.0.0.1 link#5 UH 0 0 lo0 
> 172.16.2.0/24 link#3 U 0 0 bce2 
> 100.16.97.0/24 link#1 U 0 44 bce0 
> 100.16.98.0/24 link#2 U 0 4 bce1 
> 
> The other jail is acting the same way. I know that since I'm doing a jexec, the shell will have the host's route because, but, how can I know if it's getting the alternate routing table? 
> 
> Thanks, 
> 
> Kevin

Try ssh'ing into one of the jails from the public side. The jail should honor the FIB configuration from that perspective. Are things behaving as you expect in the jail at that point?

As you've figured out, when jexec'ing into the jail from the host machine, you inherit the FIB of your current shell.

I think this due to the design of FreeBSD's multiple routing tables -- and not a bug :)

-Brandon