Re: NFS issues since upgrading to 13-RELEASE

From: Olav Gjerde <olav_at_backupbay.com>
Date: Mon, 19 Apr 2021 10:30:05 +0200
I have tried D29690 patch and reverting back to r367492 this weekend.
Neither made any difference for my system.
There is also a reddit thread about this
https://www.reddit.com/r/freebsd/comments/mqol4o/nfs_issues_since_upgrading_to_13release/

On Sat, Apr 17, 2021 at 1:10 AM Rick Macklem <rmacklem_at_uoguelph.ca> wrote:

> Just fyi, I just got a "recursed on non-recursed mutex" panic in
> socantrcvmore() with the D29690 patch, so you might not
> want to test with that one yet.
>
> rick
>
> ________________________________________
> From: owner-freebsd-current_at_freebsd.org <owner-freebsd-current_at_freebsd.org>
> on behalf of Olav Gjerde <olav_at_backupbay.com>
> Sent: Thursday, April 15, 2021 3:21 PM
> To: Allan Jude
> Cc: freebsd-current_at_freebsd.org
> Subject: Re: NFS issues since upgrading to 13-RELEASE
>
> CAUTION: This email originated from outside of the University of Guelph.
> Do not click links or open attachments unless you recognize the sender and
> know the content is safe. If in doubt, forward suspicious emails to
> IThelp_at_uoguelph.ca
>
>
> Well something do happen if I restart NFS Service on FreeBSD , it works for
> like 10 seconds then it gets unresponsive again.
>
> This is my output from `nfsstat -d 1`
>
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 8.00  1025    8.00  8.02 17170  134.54  2.01 72716  142.54  0.07  51  34
> 8.00  2273   17.76  7.99 31273  244.07  2.01 133267  261.83  0.14  20  82
> 8.03  4889   38.33  7.99 25885  202.07  2.06 119340  240.40  0.13  21  81
> [===== Read =====]  [===== Write ====]  [=========== Total ============]
> KB/t   tps    MB/s  KB/t   tps    MB/s  KB/t   tps    MB/s    ms  ql  %b
> 7.98  8811   68.64  8.00 12997  101.54  2.22 78396  170.18  0.15   1  80
> 7.99   922    7.20  8.00  3798   29.68  2.10 17965   36.87  0.09   0  11
> 8.07  2959   23.31  0.00     0    0.00  2.67  8938   23.31  0.86  32  72
> 7.97  7088   55.18  0.00     0    0.00  2.66 21233   55.18  1.05  16  98
> 7.98  4666   36.38  0.00     0    0.00  2.66 13986   36.38  0.36   9  29
> 8.00  4513   35.24  8.00  7662   59.86  2.20 44188   95.10  0.27  10  49
> 7.98  4799   37.40  8.00 11422   89.23  2.16 60076  126.63  0.19   0  51
> 8.00  4322   33.76  0.00     0    0.00  2.67 12967   33.76  0.89   0  42
> 8.02  4839   37.91  0.00     0    0.00  2.67 14550   37.91  0.54  17  41
> 8.01  4516   35.32  0.00     0    0.00  2.67 13569   35.32  0.57  27  38
> 7.95  4459   34.62  8.00  1195    9.34  2.49 18109   43.96  0.55   0  45
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
> 0.00     0    0.00  0.00     0    0.00  0.00     0    0.00  0.00   0   0
>
>
>
> On Thu, Apr 15, 2021 at 9:07 PM Olav Gjerde <olav_at_backupbay.com> wrote:
>
> > I have the same issue, using Ubuntu 20.10 with Linux 5.8 kernel. The
> Linux
> > NFS client will get unresponsive and it does not recover in my case, even
> > if I restart NFS on FreeBSD. I upgraded from FreeBSD 12.1-RELEASE though.
> >
> > On Thu, Apr 15, 2021 at 8:36 PM Allan Jude <allanjude_at_freebsd.org>
> wrote:
> >
> >> On 4/15/2021 9:22 AM, Chris Roose wrote:
> >> > I posted this in -questions and someone suggested I post here as well.
> >> >
> >> > I'm having NFS availability issues between my Proxmox client and
> >> FreeBSD server (10G link) since upgrading to 13-RELEASE. And
> unfortunately
> >> I upgraded my ZFS pool to v2.0.0 before I noticed the issue, so I'm
> kind of
> >> stuck.
> >> >
> >> > Periodically, the NFS server (I've tried both v3 and v4.2 clients)
> will
> >> go unresponsive for several minutes. I never had this problem on 12.2,
> and
> >> as far as I can tell it's not a disk or network I/O issue. I'll get
> several
> >> "nfs: server not responding, still trying" messages on the client and a
> few
> >> minutes later it usually recovers. It's not clear to me yet what's
> causing
> >> the block. Restarting nfsd on the server will resolve the issue if it
> >> doesn't clear itself.
> >> >
> >> > Any pointers for troubleshooting this? I've been looking through
> >> vmstat, gstat, top, etc. when the problem occurs, but I haven't been
> able
> >> to pinpoint the issue. I can get pcap, but it would be from the hosts,
> >> because I don't have a 10G tap or managed switch.
> >> >
> >>
> >> run `nfsstat -d 1` and try to capture a few lines from before, during,
> >> and after the stall, and that may provide some insight.
> >>
> >> Specifically, does the queue length grow, suggesting it is waiting on
> >> the I/O subsystem, or does it just stop getting traffic all together.
> >>
> >>
> >> --
> >> Allan Jude
> >> _______________________________________________
> >> freebsd-current_at_freebsd.org mailing list
> >> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> >> To unsubscribe, send any mail to "
> freebsd-current-unsubscribe_at_freebsd.org
> >> "
> >>
> >
> >
> > --
> > Kind Regards / Med Vennlig Hilsen
> >
> > Olav Grønås Gjerde
> >
> > BackupBay Gjerde
> > Madlaforen 35
> > 4042 HAFRSFJORD
> > Norway
> > Phone: +47 918 000 59
> >
>
>
> --
> Kind Regards / Med Vennlig Hilsen
>
> Olav Grønås Gjerde
>
> BackupBay Gjerde
> Madlaforen 35
> 4042 HAFRSFJORD
> Norway
> Phone: +47 918 000 59
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>


-- 
Kind Regards / Med Vennlig Hilsen

Olav Grønås Gjerde

BackupBay Gjerde
Madlaforen 35
4042 HAFRSFJORD
Norway
Phone: +47 918 000 59
Received on Mon Apr 19 2021 - 06:30:20 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:28 UTC