Re: sk ethernet driver: watchdog timeout

On Thu, Apr 08, 2004 at 12:17:06AM +1000, Bruce Evans wrote:
[...]
> The following patch reduces the problem on A7V8X-E a little.  It limits
> the tx queue to 1 packet and fixes handling of the timeout on txeof.
> The first part probably makes the second part a no-op.  Without this,
> my A7V8X-E hangs on even light nfs activity (e.g., copying a 1MB file
> to nfs).  With it, it takes heavier nfs activity to hang (makeworld
> never completes, and a flood ping always hangs).
> 
> I first suspected an interrupt-related bug, but the bug seems to be
> more hardware-specific.  Examination of the output queues shows that
> the tx sometimes just stops before processing all packets.  Resetting
> in sk_watchdog() doesn't always fix the problem, and the timeout usually
> stops firing after a couple of unsuccessful resets, giving a completely
> hung device.  But the problem may be related to interrupt timing, since
> it is much smaller under RELENG_4.  RELENG_4 hangs about as often
> without this hack as -current does with it.
> 
> nv0 hangs similarly.  fxp0 just works.
> 
> %%%
> Index: if_sk.c
> ===================================================================
> RCS file: /home/ncvs/src/sys/pci/if_sk.c,v
> retrieving revision 1.78
> diff -u -2 -r1.78 if_sk.c
> --- if_sk.c	31 Mar 2004 12:35:51 -0000	1.78
> +++ if_sk.c	1 Apr 2004 07:33:58 -0000
> _at__at_ -1830,4 +1830,9 _at__at_
>  	SK_IF_LOCK(sc_if);
> 
> +	if (sc_if->sk_cdata.sk_tx_cnt > 0) {
> +		SK_IF_UNLOCK(sc_if);
> +		return;
> +	}
> +
>  	idx = sc_if->sk_cdata.sk_tx_prod;
> 
> _at__at_ -1853,4 +1858,5 _at__at_
>  		 */
>  		BPF_MTAP(ifp, m_head);
> +		break;
>  	}
> 
> _at__at_ -2000,5 +2031,4 _at__at_
>  		sc_if->sk_cdata.sk_tx_cnt--;
>  		SK_INC(idx, SK_TX_RING_CNT);
> -		ifp->if_timer = 0;
>  	}
> 
> _at__at_ -2007,4 +2037,6 _at__at_
>  	if (cur_tx != NULL)
>  		ifp->if_flags &= ~IFF_OACTIVE;
> +
> +	ifp->if_timer = (sc_if->sk_cdata.sk_tx_cnt == 0) ? 0 : 5;
> 
>  	return;
> %%%
> 
Always recharging the timer to 5 when there's some TX work still
left is a bug.  With DEVICE_POLLING (yes, I have plans to add
polling(4) support for sk(4) too), sk_txeof() will be called
periodically, and if the card gets stuck, the if_timer will
never downgrade to zero, and sk_watchdog() will never be called.
Without DEVICE_POLLING, recharging it back to 5 even when
if_timer reaches 0 is still pointless, because when if_timer is
0 while in the sk_txeof(), it means it's called by sk_watchdog()
which will reinit the card and both RX and TX lists, making them
empty, so having the if_timer with the value of 5 _after_
executing the watchdog cleaning and having _no_ TX activity at
all may cause a second (false) watchdog.  My version of the
TX fixes (which also fixes resetting of IFF_OACTIVE):

%%%
Index: if_sk.c
===================================================================
RCS file: /home/ncvs/src/sys/pci/if_sk.c,v
retrieving revision 1.78
diff -u -p -r1.78 if_sk.c
--- if_sk.c	31 Mar 2004 12:35:51 -0000	1.78
+++ if_sk.c	8 Apr 2004 19:10:50 -0000
_at__at_ -1998,14 +1998,14 _at__at_ sk_txeof(sc_if)
 			sc_if->sk_cdata.sk_tx_chain[idx].sk_mbuf = NULL;
 		}
 		sc_if->sk_cdata.sk_tx_cnt--;
+		ifp->if_flags &= ~IFF_OACTIVE;
 		SK_INC(idx, SK_TX_RING_CNT);
-		ifp->if_timer = 0;
 	}
 
 	sc_if->sk_cdata.sk_tx_cons = idx;
 
-	if (cur_tx != NULL)
-		ifp->if_flags &= ~IFF_OACTIVE;
+	if (sc_if->sk_cdata.sk_tx_cnt == 0)
+		ifp->if_timer = 0;
 
 	return;
 }
%%%

We have been running the 3COM 3C940 card on 4.9 (and from today
on 4.10-BETA) without any problems and under a heavy TX load.


Cheers,
-- 
Ruslan Ermilov
ru_at_FreeBSD.org
FreeBSD committer