Re: ipfw and ipsec processing order for outgoing packets wrong

From: Vincent Poy <vincepoy_at_gmail.com>
Date: Sun, 03 Jul 2005 00:45:55 -0000
Hi,

I don't know how to explain my problem but it goes something like this...

root_at_bigbang [2:05am][/home/vince] >> ipfw show
00049     1557131    244839199 skipto 100 ip from 208.201.244.224/29 to any
00050 12072800468 917651580916 divert 8668 ip from any to any via xl0
00100       69518      8548222 allow ip from any to any via lo0
00200           0            0 deny ip from any to 127.0.0.0/8
00300           0            0 deny ip from 127.0.0.0/8 to any
63000           0            0 allow ip from any to 10.0.0.0/8 out
63001           0            0 allow ip from any to 172.16.0.0/12 out
63002         312        16048 allow ip from any to 192.168.0.0/16 out
63003       24237      2952214 allow ip from any to 208.201.244.224/29 out
63004      667879    129410867 queue 1 tcp from any to any tcpflags ack out
63005           1           40 queue 2 tcp from any to any dst-port 22,23 out
63006       38782      3364689 queue 2 udp from any to any not
dst-port 80,443 out
63007       43021      2194871 queue 3 ip from any to any dst-port 80,443 out
63008        5467       405319 queue 4 ip from any to any out
65000     1795325    424479044 allow ip from any to any
65535           0            0 deny ip from any to any

The counters for queue 1 keeps increasing when I do a ftp out even for
non-ACK packets but the other counters for queue 2-4 doesn't move at
all so it seems like everything is going out one queue instead of what
the rules actually say.  I have one pipe configured as 480Kbit/sec
which is what rules 63005-63008 does.

ipfw pipe show and ipfw queue show would seem normal except the Source
IP and Destination IP is stuck with the first processed queues
information while only the counters for queue 1 updates.

root_at_bigbang [2:12am][/home/vince] >> ipfw pipe show
00001: 480.000 Kbit/s    0 ms   50 sl. 0 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
q00001: weight 100 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3748  205.188.179.233/5190  673549 137223155 
0    0 2303
q00002: weight 66 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 udp  208.201.244.225/1026   208.201.224.11/53    40022  3470523  0    0   0
q00003: weight 33 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3750  199.181.132.105/80    43058  2196795  0    0   0
q00004: weight 1 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3748  205.188.179.233/5190  5492   407173  0    0   0
root_at_bigbang [2:12am][/home/vince] >> ipfw queue show
q00001: weight 100 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3748  205.188.179.233/5190  673550 137223195 
0    0 2303
q00002: weight 66 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 udp  208.201.244.225/1026   208.201.224.11/53    40025  3470881  0    0   0
q00003: weight 33 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3750  199.181.132.105/80    43058  2196795  0    0   0
q00004: weight 1 pipe 1   50 sl. 1 queues (1 buckets) droptail
    mask: 0x00 0x00000000/0x0000 -> 0x00000000/0x0000
BKT Prot ___Source IP/port____ ____Dest. IP/port____ Tot_pkt/bytes Pkt/Byte Drp
  0 tcp  208.201.244.226/3748  205.188.179.233/5190  5493   407225  0    0   0

I don't know how else I would test this.  

Cheers,
Vince
 
On Mon, 1 Nov 2004 09:35:58 +0200, Ari Suutari <ari_at_suutari.iki.fi> wrote:
 
> >I am experiencing the same problem as well when I updated from a March
> > 6, 2004 -CURRENT to the October 19, 2004 -CURRENT.  The problem still
> > exists with the October 27, 2004 -CURRENT.  I'm using ipfw/dummynet
> > for outgoing queues with the ACK packets having the highest priority
> > in it's own queue.  However, it seems like while the queues are there,
> > the information on ipfw queue show doesn't update at all as the Source
> > and Destination IP is still the same as the first packet after bootup
> > while the counters change but the ACK packets are not sent on it's own
> > queue but rather with all other packets.  I know it is related with
> > pfil_hook when ipfw was converted.
> 
>    This is not related to pfil_hook conversion. The problem is also present
> in
>    FreeBSD 4.x-STABLE (just tested it). I think that history of ipfw and
> ipsec
>    interaction goes like this:
> 
>    - in the very beginning, a packet that was processed by ipsec didn't
>      hit ipfw at all in unencrypted form, ie. one was able to able to
> filter esp
>      and ah protocols only.
> 
>    - someone fixed this, apparently for incoming packets only, but this
>       some folks were upset by the fact that they would have to add a rule
>       for unencrypted protocols into ipfw. At that time (in ipfw1), there
> was
>       possibility to check that unencrypted packet actually came from ipsec
>       (ie. ipfw ipsec flag wasn't implemented)
> 
>    - IPSEC_FILTERGIF option was added. If set, incoming packets go
>       through ipfw twice (encrypted and unencrypted). If not set, packets
> go
>       to ipfw only once (encrypted).
> 
>    Currently outgoing packets are always processed like IPSEC_FILTERGIF was
>    not set (I like to have it set, because I need quite fine-grained
> firewalling
>    even inside my ipsec tunnels, which are between different companies).
> What
>    I was suggesting (ie. moving pfil_hook processing in ip_output before
>    ipsec stuff) wasn't really correct: This change should be conditional
> based on
>    IPSEC_FILTERGIF setting: The change I described should be done only
>    when IPSEC_FILTERGIF is set.
> 
>    Now, ip_output is quite central part in ip stack. I would be happy if
> someone
>    who knows that part better than me could implement this (I can sure test
> it easily).
> 
>        Ari S.
> 
> 
> 
> >
> > Cheers,
> > Vince
> >
> > On Sat, 30 Oct 2004 09:27:50 +0300, Ari Suutari <ari_at_suutari.iki.fi>
> > wrote:
> >> Hi,
> >>
> >> I noticed that processing order of ipsec and ipfw (pfil_hook) is not
> >> correct for outgoing packets. Currently, ipsec processing is done first,
> >> which makes packets to go through without firewall inspection.
> >> This might be a security problem for someone, but at least it
> >> breaks stateful rule handling.
> >>
> >> My test setup is (all freebsd 5.3-rc1 machines):
> >>
> >> freebsd laptop <-> ipsec tunnel <->freebsd server
> >>
> >> When server sends packet to laptop, it now goes like this:
> >>
> >> ip_output -> ipsec -> ip_output -> ipfw -> network
> >>
> >> It should go like this:
> >>
> >> ip_output -> ipfw -> ipsec -> ip_output -> ipfw -> network
> >>
> >> I think that this could be fixed by just moving pfil_hook
> >> processing in ip_output before ipsec processing.
> >>
> >>     Ari S.
> >>
> >> _______________________________________________
> >> freebsd-current_at_freebsd.org mailing list
> >> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to
> >> "freebsd-current-unsubscribe_at_freebsd.org"
> >>
> >>
> >
> 
>
_______________________________________________
freebsd-net_at_freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe_at_freebsd.org"
Received on Sat Jul 02 2005 - 22:45:55 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:38:38 UTC