On 30 Nov 2009, at 08:40, Eirik Øverby wrote: >>> Short follow-up: Making OpenBSD use TCP mounts (it defaults to UDP) seems to solve the issue. >>> >>> So this is a UDP-NFS-related problem, it would seem? >> >> Could well be. Let's try another debugging tactic -- there are two possible things going on here: resource leak, and resource exhaustion leading to deadlock. If you shut down to single user mode from multi-user, and let the system quiesce for a few minutes, then run netstat -m, what does it look like? Do vast numbers of mbufs+clusters get freed, or do they remain accounted for as allocated? > > It's been sitting in single-user mode for about 15 minutes now, no change in allocation. > I'll reboot in about 15 minutes, then try to mount from a FreeBSD box using UDP - if that causes the same issues, I guess it's not an OpenBSD specific issue but a UDP issue "in general". Next step would be to try to reproduce the same between two VMs on my own box, as this box needs to return to production soonish - if we manage to reproduce elsewhere.. This sounds like a good plan -- especially reproducing it on a non-production box :-). I agree it's most likely that the OpenBSD NFS client simply does something a little differently than the other NFS clients you are dealing with, triggering an edge case in our NFS server code. But, to be clear, I think it's much more likely that the bug is in the NFS over UDP code than UDP itself, given the complexity of the NFS code (although a UDP bug can't be ruled out). Robert > > Other ideas/suggestions? > > /Eirik > >> (If they remain allocated, they were likely leaked, since most/all sockets will have been closed, releasing their resources on shutdown to single user when all processes are killed) >> >> The theory of an mbuf leak in NFS isn't an unlikely theory -- the socket code there continues to change, and rare edge cases frequently lead to leaks (per my earlier e-mail). Perhaps there's a case the OpenBSD client is triggering that other NFS clients normally don't. If we think that's the case, the next step is usually to narrow down what causes the leak to trigger a lot (i.e., the backup starting), and then grab a packet trace that we can analyze with wireshark. We'll want to look at the types of errors being returned for RPCs and, in particular, if there's one that happens about the same number of times as the resource has leaked over the same window, look at the code and see if that error case is handled properly. >> >> If this is definitely an NFS leak bug, we should get the NFS folks attention by sticking "NFS mbuf leak" in the subject line and CC'ing rmacklem/dfr. :-) >> >> Robert >> >> >> >> >>> /Eirik >>> >>> On 30. nov. 2009, at 11.22, Eirik Øverby wrote: >>> >>>> Hi, >>>> >>>> I have something that might be more interesting than any counter ... >>>> It seems to me as if the problem *only* manifests itself when an OpenBSD box is backing up to this FreeBSD 8.0-NFS-ZFS server. All other boxes are FreeBSD, and I have so far today been unable to reproduce the problem from any of those. As soon as I interrupted the backup running from OpenBSD, the mbuf cluster usage stabilized. >>>> >>>> How's that for a mystery in the morning? >>>> >>>> /Eirik >>>> >>>> On 29. nov. 2009, at 15.29, Robert Watson wrote: >>>> >>>>> On Sun, 29 Nov 2009, Eirik Øverby wrote: >>>>> >>>>>> I just did that (-rxcsum -txcsum -tso), but the numbers still keep rising. I'll wait and see if it goes down again, then reboot with those values to see how it behaves. But right away it doesn't look too good .. >>>>> >>>>> It would be interesting to know if any of the counters in the output of netstat -s grow linearly with the allocation count in netstat -m. Often times leaks are associated with edge cases in the stack (typically because if they are in common cases the bug is detected really quickly!) -- usually error handling, where in some error case the unwinding fails to free an mbuf that it should free. These are notoriously hard to track down, unfortunately, but the stats output (especially where delta alloc is linear to delta stat) may inform the situation some more. >>>>> >>>>> Robert N M Watson >>>>> Computer Laboratory >>>>> University of Cambridge >>>> >>>> _______________________________________________ >>>> freebsd-current_at_freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-current >>>> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" >>>> >>> >> >> _______________________________________________ >> freebsd-current_at_freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-current >> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" >> >Received on Mon Nov 30 2009 - 14:36:27 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:58 UTC