Re: vnet/jail crashdump

Ronald Klop wrote:
> Hi,
> 
> After stopping a jail I get a crashdump.
> core.txt: 
> https://www.klop.ws/core_2eef39c581f90f2f0c4921e43f1998c1/core.txt.0
> 
> Jail.conf:
> ----------
> exec.stop = "/bin/sh /etc/rc.shutdown";
> exec.clean;
> 
> exec.prestart = "ifconfig bridge0 > /dev/null 2> /dev/null || ( ifconfig 
> bridge0 create && ifconfig bridge0 addm vtnet0 && ifconfig bridge0 up)";
> 
> exec.consolelog = "/var/log/jail_${name}_console.log";
> 
> mount.devfs;
> path = "/data/jails/$name";
> host.hostname = "$name";
> mount.fstab = "/data/jails/fstab.$name";
> vnet;
> allow.mlock;
> devfs_ruleset="110";
> 
> freebsd12 {
>     osrelease = 12.1-RELEASE-p4;
>     osreldate = 1201000;
>     vnet.interface = "epair0b";
>     # make sure the exec.prestart has a "+=" as we de it in the global 
> definition
>     # when checking for the bridge
>     exec.prestart += "ifconfig epair0 create up";
>     exec.prestart += "ifconfig bridge0 addm epair0a";
>     exec.prestart += "ifconfig epair0b link 02:xxxxxx:0c";
>     exec.start = "dhclient epair0b";
>     exec.start += "/bin/sh /etc/rc";
>     exec.poststop  = "ifconfig bridge0 deletem epair0a";
>     exec.poststop += "ifconfig epair0a destroy";
> 
> }
> freebsd13 {
>     vnet.interface = "epair1b";
>     # make sure the exec.prestart has a "+=" as we de it in the global 
> definition
>     # when checking for the bridge
>     exec.prestart += "ifconfig epair1 create up";
>     exec.prestart += "ifconfig bridge0 addm epair1a";
>     exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d";
>     exec.start = "dhclient epair1b";
>     exec.start += "/bin/sh /etc/rc";
>     exec.poststop  = "ifconfig bridge0 deletem epair1a";
>     exec.poststop += "ifconfig epair1a destroy";
> }
> ----------
> 
> What can I do to help debug?
> 


Don't understand why you have these 2 statements

      exec.prestart += "ifconfig epair1b link 02:xxxxxx:0d";
      exec.start = "dhclient epair1b";

There is a well known bug with bridge vnet tear down since release 9.0. 
Their is a rewrite of if_bridge going on right now to fix the problem 
and increase the performance of if_bridge. As of today this fix is not 
in 12.2 stable or 13.0 current.

There also looks like a bug in jail(8) when you have both vnet jails and 
non-vnet jails being started on the same host at the same time. In most 
cases the host just loses internet access until all the jails are 
stopped. Some times you will get a system crash.


This jail.conf def seems to work around the bridge tear down problem

#  vnet jail using the bridge/epair method on 12.1
v0jail1 {
host.hostname   = "v0jail1";
path            = "/usr/jails/v0jail1";
mount.fstab     = "/usr/local/etc/fstab/v0jail1";
exec.consolelog = "/var/log/v0jail1.console.log";
mount.devfs;
devfs_ruleset   = "4";
vnet            = "new";
vnet.interface  = "epair55b";
exec.prestart   = "ifconfig epair55  create up";
exec.prestart  += "ifconfig bridge0 addm epair55a";
exec.prestart  += "ifconfig epair55a descr vnet-v0jail1";
exec.prestart  += "ifconfig bridge0 inet 10.0.48.2 netmask 255.255.255.0 
alias";
exec.start      = "/bin/sh /etc/rc";
exec.start     += "ifconfig epair55b inet 10.0.48.1 netmask 255.255.255.0";
exec.start     += "route add default 10.0.48.2";
exec.prestop    = "ifconfig epair55b -vnet v0jail1";
exec.stop       = "/bin/sh /etc/rc.shutdown";
exec.poststop   = "ifconfig bridge0 deletem epair55a";
exec.poststop  += "sleep 2";
exec.poststop  += "ifconfig epair55a destroy";
exec.poststop  += "ifconfig bridge0 inet 10.0.48.2 -alias";
}

Remember that your host firewall processes all traffic in & out of the 
host including any vnet jail traffic. Yes a vnet jail has its own stack 
and can have its own firewall, but the host firewall still has the last 
say. The host must NAT any private ip addresses used by the vnet jails.

jail.conf jail definitions are based on hard codded ip addresses. You 
can not use the host dhcp to assign local lan private ip addresses to a 
jail.

You may find this helpful

https://forums.freebsd.org/threads/vnet-jail-with-public-internet-access-using-the-bridge-epair-method.76071/