thread on sleepqueue does not wake up after timeout

From: Andriy Gapon <avg_at_FreeBSD.org>
Date: Tue, 22 Oct 2019 13:08:59 +0300
We observe a problem that happens very rarely (about once a month across many
test machines).  The problem is that a thread remain in sleepq_timedwait() even
after its timeout expires.  The thread's td_slpcallout looks like the callout
has fired.  But the thread's state looks like it was never notified.
E.g.:
(kgdb) p td->td_slpcallout
$1 = {c_links = {le = {le_next = 0xfffff800108e6470, le_prev =
0xfffffe0000be6ea8}, sle = {sle_next = 0xfffff800108e6470}, tqe = {tqe_next =
0xfffff800108e6470, tqe_prev = 0xfffffe0000be6ea8}}, c_time = 160957479343159,
  c_precision = 268435450, c_arg = 0xfffff80184602000, c_func =
0xffffffff807481d0 <sleepq_timeout>, c_lock = 0x0, c_flags = 2, c_iflags = 272,
c_cpu = 6, c_exec_time = 160957506517070} [*]
(kgdb) p/x td->td_flags
$5 = 0x80000004
(kgdb) p td->td_sqqueue
$8 = 0
(kgdb) p td->td_sleepqueue
$9 = (struct sleepqueue *) 0x0
(kgdb) p td->td_wchan
$10 = (void *) 0xfffff802b990df38


Has anyone seen anything like this problem?
Any advice on how to diagnose it?

Thanks!

P.S.
c_exec_time is our addition, we set this field right before firing a callback
and we reset it to zero when a callout is (re-)scheduled.

-- 
Andriy Gapon
Received on Tue Oct 22 2019 - 08:09:05 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:22 UTC