Re: Issues with Jails/Routes/FIBs

On 11/25/10 11:38 AM, Kevin Mai wrote:
> Hi folks!
>
> I'm facing an issue here while trying to define separate routing tables for each jail and host.

You don't say what version of FreeBSD you are using..
> Let me show you briefly how it's done:
>
> The server has 3 physical NICs, each one connected to a different network (say, public network A, public network B, and LAN).
>
> Currently, the default gateway is set to be the LAN gateway, even though the two jails can see their own public network subnet.

can you explain what you mean by that?
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 172.16.2.1 UGS 1 3935 bce2
> 127.0.0.1 link#5 UH 0 0 lo0
> 172.16.2.0/24 link#3 U 0 0 bce2
> 172.16.2.127 link#3 UHS 0 0 lo0
> 100.16.97.0/24 link#1 U 0 0 bce0
> 100.16.97.5 link#1 UHS 0 0 lo0
> 100.16.98.0/24 link#2 U 0 0 bce1
> 100.16.98.5 link#2 UHS 0 0 lo0
>
> 100.16.97.0/24 and 100.16.98.0/24 are the two public networks and 172.16.2.0/24 is the LAN.
>
> I have already tried removing devfs rules from the jails, setting securelevel to -1 but I'm still out of luck..
>
> I know setfib can define alternate routing tables, and I even created a default gateway for two fibs, 1&  2:
>
> [root_at_mrefns09 ~]# setfib 2 netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 100.16.98.100 UGS 14 906 bce1
> 127.0.0.1 link#5 UH 0 0 lo0
> 172.16.2.0/24 link#3 U 0 0 bce2
> 100.16.97.0/24 link#1 U 0 39 bce0
> 100.16.98.0/24 link#2 U 0 0 bce1
>
> [root_at_mrefns09 ~]# setfib 1 netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 100.16.97.100 UGS 0 1758 bce0
> 127.0.0.1 link#5 UH 0 0 lo0
> 172.16.2.0/24 link#3 U 0 0 bce2
> 100.16.97.0/24 link#1 U 0 44 bce0
> 100.16.98.0/24 link#2 U 0 4 bce1
>
> And i've added the proper settings in rc.conf..
>
> jail_athea97_ip="100.16.97.5 netmask 255.255.255.0"
> jail_athea97_fib=1

ooh I hadn't seen that..
(goes to a recent machine to look at new jail confif stuff.)

cool. Hadn't seen that.. but it should work.
>
> jail_athea98_ip="100.16.98.5 netmask 255.255.255.0"
> jail_athea98_fib=2
>
> Am I missing something? because once I get into the jail the routing table is the same:

well HOW do you get into the jail?
the fib is inherited from your parent process so if your process is 
not descended from
whatever you started the jail with, you will not inherrit the right 
fib unless
you specifically include setfib() in the command.

> [root_at_athea97 /]# netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 172.16.2.1 UGS 13 6175 bce2
> 127.0.0.1 link#5 UH 0 0 lo0
> 172.16.2.0/24 link#3 U 0 0 bce2
> 172.16.2.127 link#3 UHS 0 0 lo0
> 100.16.97.0/24 link#1 U 0 0 bce0
> 100.16.97.5 link#1 UHS 0 0 lo0
> 100.16.98.0/24 link#2 U 0 0 bce1
> 100.16.98.5 link#2 UHS 0 0 lo0
>
> [root_at_athea97 /]# setfib 1 netstat -rn
> Routing tables
>
> Internet:
> Destination Gateway Flags Refs Use Netif Expire
> default 100.16.97.100 UGS 15 1814 bce0
> 127.0.0.1 link#5 UH 0 0 lo0
> 172.16.2.0/24 link#3 U 0 0 bce2
> 100.16.97.0/24 link#1 U 0 44 bce0
> 100.16.98.0/24 link#2 U 0 4 bce1
>
> The other jail is acting the same way. I know that since I'm doing a jexec, the shell will have the host's route because, but, how can I know if it's getting the alternate routing table?

you need to do setfib 1 jexec {your command}
OR
you should make your original jail command start up (however 
indirectly) sshd or telnetd or whatever and
connect in via that. In that case your process will be descended from 
the original
jailed process and you will get the effect you want.

However I have to point out to you that if you are using freebsd 8, 
you might want to investigate the
"VIMAGE" extensions to jail.  this does even more of what you want by 
associating the network
changes with the jail and not just the process that was jailed.
(check the 'vnet' option to the jail command.)



> Thanks,
>
> Kevin
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>