Re: GET BUF: dmamap load failure - 12 + kernel panic [was: NETIO UDP benchmark problem]

From: Viktor CISTICZ <viktor_at_cisti.cz>
Date: Fri, 21 Aug 2009 14:26:51 +0200
Hello! Thanks for changing subject, this is better.

Dne 20/8/2009, napsal "Miroslav Lachman" <000.fbsd_at_quip.cz>:

>I changed subject to better describing one, because it is not problem in
>netio itself.
>
>Viktor CISTICZ wrote:
>> Hello,
>> recently i have been testing two servers  via crosslink. Both have
>> Freebsd installed >
>>
>> twin1$ uname -a
>> FreeBSD twin1 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Sun Aug 16
>> 22:57:29 CEST 2009
>> viktor_at_twin1:/usr/obj/usr/src/sys/GEN_NO_DBG  amd64
>>
>> twin2$ uname -a
>> FreeBSD kitt.twin2 8.0-BETA2 FreeBSD 8.0-BETA2 #0: Wed Jul 22 15:05:19
>> CEST 2009     viktor_at_kitt.twin2:/usr/obj/usr/src/sys/GEN_NO_DBG  amd64
>>
>> GEN_NO_DBG is GENERIC kernel without debugging options
>>
>> Twin2 is virtual machine on the same hardware as twin1, vmware used.
>>
>> I have this set installed on both machines >
>> twin1$ pkg_info
>> NetPIPE-3.7.1       A self-scaling network benchmark
>> gettext-0.17_1      GNU gettext package
>> gmake-3.81_3        GNU version of 'make' utility
>> iperf-2.0.4         A tool to measure maximum TCP and UDP bandwidth
>> libiconv-1.13.1     A character set conversion library
>> libtool-2.2.6a      Generic shared library support script
>> netio-1.26          Network benchmark
>> netperf-2.4.5       Network performance benchmarking package
>> portaudit-0.5.13    Checks installed ports against a list of security
>> vulnerabi
>> portmaster-2.9      Manage your ports without external databases or
>> languages
>> screen-4.0.3_6      A multi-screen window manager
>> ttcp-1.12           Benchmarking tool for analysing TCP and UDP
>> performance
>> unzip-5.52_5        List, test and extract compressed files in a ZIP
>> archive
>>
>> Both machines are connected via crosslink >
>> twin1# ifconfig
>> igb0 : public interface
>> igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=13b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,TSO4>
>>         ether 00:30:48:c8:f3:91
>>         inet 10.10.10.10 netmask 0xffffff00 broadcast 10.10.10.255
>>         media: Ethernet autoselect (1000baseT <full-duplex>)
>>         status: active
>>
>> twin2# ifconfig
>> em0: public interface
>> em1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
>>         options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
>>         ether 00:0c:29:4b:f1:76
>>         inet 10.10.10.20 netmask 0xffffff00 broadcast 10.10.10.255
>>         media: Ethernet autoselect (1000baseT <full-duplex>)
>>         status: active
>>
>> I have set up twin2 as a server for netio >
>> kitt.twin2# netio -s -p 1122
>>
>> And then tested from twin1 tcp test >
>> twin1# netio -p 1122 -t 10.10.10.20
>>
>> It was allright, i've got some results.
>>
>> But when I tried UDP, it failed.
>>
>> The server is still the same >
>> kitt.twin2# netio -s -p 1122
>>
>> Client
>> twin1# netio -p 1122 -u 10.10.10.20
>>
>> After around 1 minute, twin1 server stopped responding on public
>> interface, i've been disconnected. Via remote console i could access
>> the machine, it was acting normally. I could ping both interfaces. I
>> could even ping from other side of crosslink, from twin2 the privat
>> interface, but no reply on public interface.
>> The interface was shown UP in ifconfig, no messages in /var/log/messages.
>>
>> Then i executed ifconfig igb1 down && ifconfig igb1 up and it worked
>> again.
>
>Did you ifconfig down + up igb1 or igb0? As you said "server stopped
>responding on public interface", but you have igb0 marked as public
>interface and igb1 as private crosslink interface in ifconfig above.
Server stopped responding on igb0(public interface) from outside of the
box. Via igb1 there was ping response.
>Are you really saying that heavy UDP load on private interface igb1
>caused freez of public interface igb0 and then kernel panic?
Yeah, test was run via igb1, after it crashed, the server was not
responding on public interface igb0 and crashed.

>
>> I have then executed netio udp test again with 2k udp packets
>>
>> The server is still the same >
>> kitt.twin2# netio -s -p 1122
>>
>> twin1# netio -p 1122 -u -b 2k 10.10.10.20
>>
>> The same problem on twin1, but now it has crashed the computer >
>> via remote console i could see this >
>> twin1# GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>> GET BUF: dmamap load failure - 12
>>
>> then it executed core dump and restarted.
>>
>> I am lost, thanks for any advise.
>
>Did you run it as root or regular user?
>
>Can you add output of netstat -m right before interface freez / kernel
>panic?
>
>And last - can you reproduce it with kernel with debugging options enabled?
>
>Miroslav Lachman
>
I have done another testing, this time i've booted twin1 server with
GENERIC kernel from 8.0-BETA2-amd64 and it has happened again.
this is what i've done on twin1 (client), i have done it as a normal
user >

twin1$ netio -u -p 1122  10.10.10.20



NETIO - Network Throughput Benchmark, Version 1.26

(C) 1997-2005 Kai Uwe Rommel



UDP connection established.

Packet size  1k bytes:  6494 KByte/s (32%) Tx,  114228 KByte/s (73%) Rx.

Packet size  2k bytes:  115122 KByte/s (0%) Tx,


on kitt.twin2 (server)

kitt.twin2# netio -s -p 1122

this appeared on kitt.twin2 >

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

sendto(): No buffer space available

Sending to client, packet size  1k ...

Receiving from client, packet size  2k ...

Then twin1 stopped working and kernel panic.

On another terminal on kitt.twin2 I ran (while 1, while netstat -m, while
sleep 1, while end) >
67272K/12510K/79782K bytes allocated to network (current/cache/total)

0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

0/0/0 requests for jumbo clusters denied (4k/9k/16k)

0/0/0 sfbufs in use (current/peak/max)

0 requests for sfbufs denied

0 requests for sfbufs delayed

0 requests for I/O initiated by sendfile

0 calls to protocol drain routines

35585/8200/43785 mbufs in use (current/cache/total)

3892/4926/8818/25600 mbuf clusters in use (current/cache/total/max)

3892/4812 mbuf+clusters out of packet secondary zone in use
(current/cache)

12648/152/12800/12800 4k (page size) jumbo clusters in use
(current/cache/total/max)

0/0/0/6400 9k jumbo clusters in use (current/cache/total/max)

0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)

67272K/12510K/79782K bytes allocated to network (current/cache/total)

0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)

0/0/0 requests for jumbo clusters denied (4k/9k/16k)

0/0/0 sfbufs in use (current/peak/max)

0 requests for sfbufs denied

0 requests for sfbufs delayed

0 requests for I/O initiated by sendfile

0 calls to protocol drain routines

I have captured via remote console what happened on twin1 (rewritten from
the picture)>
twin1#
the same netstat -m statistic procedure as at kitt.twin2>

0/0/0/3200 16k jumbo clusters in use (current/cache/total/max)
60243K/5841K/66085K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
34079/2791/36870 mbufs in use
514/2320/2834/25600 mbuf clusters in use
0/512 mbuf +clusters out of packet secondary
12674/126/12800/12800 4k (page size) jumbo clusters in use
0/0/0/6400 9k jumbo clusters in use
0/0/0/3200 16k jumbo clusters in use
60243K/5841K/66085K bytes alocated to network
..

and during the crash >
twin1#

Memory modified after free 0xffffff0083eb800(256) val=a8 _at_
0xffffff00083eb818
Memory modified after free 0xffffff0083ec600(256) val=a8 _at_
0xffffff00083ec618
there was many lines of this similar text.

VC
Received on Fri Aug 21 2009 - 10:26:50 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:54 UTC