CURRENT as gateway on not-so-fast hardware: where is a bottlneck?

Hello, Current.

 I'm  using  Soekris  net5501  (AMD  Geode  500Mhz,  512MiB of RAM) as
all-in-one  home  router:  it  connects  my  network  to ISP via PPPoE
(40Mbit shaped channel, ng0, over 100Mbit ethernet, interface vr1) and acts as AP for my
WiFi (Atheros card). It connected to my wired home network with gigabit Intel card
(em0, old PCI one).

  One of servers in my network acts as storage and torrent box in same
time. It is connected to wire network (gigabit one).

  And I found strange thing: when torrents are active (40Mbit up and
down, from em0 to ng0 over vr1 with firewall and NAT) router could
pass only about 0.1MiB/s of trafic from internal (gigabit) network
into WiFi. Yes, 0.1MiB/s. When torrents are not active it could
provide from 1.5 to 2.2MiB/s, depending on time of day (and state of
environment), so in this case radio channel is limiting factor.

 Answer looks trivial: router CPU is bottleneck. But here is one additional
detail: `top' never shows less than 50% of idle when torrents are
active. And `idle' time with torrents traffic is ALWAYS is higher than
without them, but with WiFi traffic.

  Here is typical top-5 when torrents AND WiFi is active (WiFi speed
is terrible):

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     155 ki31     0K     8K RUN      3:29 75.20% idle
   11 root     -72    -     0K    80K WAIT     2:33 12.50% intr{swi1: netisr 0}
   11 root     -92    -     0K    80K WAIT     0:39  6.40% intr{irq15: ath0 ata1}
   14 root     -16    -     0K     8K -        0:12  5.76% yarrow
    0 root     -92    0     0K    72K -        0:10  1.17% kernel{ath0 taskq}

 Here is typical top-5 when ONLY WiFi is active (please, note: idle is
less than in previous case!):

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     155 ki31     0K     8K RUN      3:12 38.57% idle
   11 root     -92    -     0K    80K WAIT     0:31 34.67% intr{irq15: ath0 ata1}
   11 root     -72    -     0K    80K WAIT     2:12 13.67% intr{swi1: netisr 0}
    0 root     -92    0     0K    72K -        0:08  7.47% kernel{ath0 taskq}
   14 root     -16    -     0K     8K -        0:10  4.69% yarrow

 Here is typical top-5 only with torrents:

    PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     155 ki31     0K     8K RUN      3:47 82.67% idle
   11 root     -72    -     0K    80K WAIT     3:00 10.60% intr{swi1: netisr 0}
   14 root     -16    -     0K     8K -        0:13  5.18% yarrow
   11 root     -92    -     0K    80K WAIT     0:40  0.78% intr{irq15: ath0 ata1}
 1870 root      20    0 10684K  2268K RUN      0:02  0.59% top

 And here is typical top-5 with no traffic

  PID USERNAME PRI NICE   SIZE    RES STATE    TIME   WCPU COMMAND
   10 root     155 ki31     0K     8K RUN      4:36 98.58% idle
   11 root     -72    -     0K    80K WAIT     3:22  5.08% intr{swi1: netisr 0}
   14 root     -16    -     0K     8K -        0:14  0.78% yarrow
   11 root     -92    -     0K    80K WAIT     0:41  0.68% intr{irq15: ath0 ata1}
 1870 root      20    0 10684K  2268K RUN      0:02  0.29% top

  I'm wonder, why adding torrent traffic slows down WiFi transfer BUT
gives much more idle time than WiFi transfer alone?

I'm using CURRENT r239228 with SCHED_ULE.

-- 
// Black Lion AKA Lev Serebryakov <lev_at_FreeBSD.org>