Re: panic: Assertion pgrp->pg_jobc > 0 failed at kern_proc.c:816

On Mon, Dec 28, 2020 at 12:44:18PM -0800, John Baldwin wrote:
> On 12/28/20 12:24 PM, John Baldwin wrote:
> > I got this panic again today in a VM when quitting a gdb
> > session after killing a child process via 'kill'.
> > 
> > panic: Assertion pgrp->pg_jobc > 0 failed at /git/bhyve/sys/kern/kern_proc.c:816
> > cpuid = 1
> > time = 1609185862
> > KDB: stack backtrace:
> > db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe00946547f0
> > vpanic() at vpanic+0x181/frame 0xfffffe0094654840
> > panic() at panic+0x43/frame 0xfffffe00946548a0
> > _refcount_update_saturated() at _refcount_update_saturated/frame 0xfffffe00946548e0
> > killjobc() at killjobc+0x6a6/frame 0xfffffe0094654940
> > exit1() at exit1+0x6af/frame 0xfffffe00946549b0
> > sys_sys_exit() at sys_sys_exit+0xd/frame 0xfffffe00946549c0
> > amd64_syscall() at amd64_syscall+0x12e/frame 0xfffffe0094654af0
> > fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0094654af0
> > --- syscall (1, FreeBSD ELF64, sys_sys_exit), rip = 0x8024c8f0a, rsp = 0x7fffffffe358, rbp = 0x7fffffffe370 ---
> > KDB: enter: panic
> > [ thread pid 44034 tid 102484 ]
> > Stopped at      kdb_enter+0x37: movq    $0,0x10ab066(%rip)
> > 
> > From what I can tell, the child process that was killed via 'kill' has
> > not yet exited and is stuck in ptracestop() from fork():
> > 
> > (kgdb) where
> > #0  sched_switch (td=0xfffffe0094001a00, flags=<optimized out>) at /git/bhyve/sys/kern/sched_ule.c:2147
> > #1  0xffffffff80bf4015 in mi_switch (flags=266) at /git/bhyve/sys/kern/kern_synch.c:542
> > #2  0xffffffff80bfeba5 in thread_suspend_switch (td=<optimized out>, p=<optimized out>)
> >     at /git/bhyve/sys/kern/kern_thread.c:1477
> > #3  0xffffffff80bef04b in ptracestop (td=0xfffffe0094001a00, sig=17, si=0x0)
> >     at /git/bhyve/sys/kern/kern_sig.c:2642
> > #4  0xffffffff80ba1a54 in fork_return (td=0xfffffe0094001a00, frame=0xfffffe0094671b00)
> >     at /git/bhyve/sys/kern/kern_fork.c:1106
> > #5  0xffffffff80ba18b0 in fork_exit (callout=0xffffffff80ba1950 <fork_return>, arg=0xfffffe0094001a00, 
> >     frame=0xfffffe0094671b00) at /git/bhyve/sys/kern/kern_fork.c:1069
> > #6  <signal handler called>
> > #7  0x00000008007b71aa in ?? ()
> > 
> > kgdb can't find the panicking process due to the zombproc removal, so I will
> > have to go work on kgdb to recover from that change. :(
> 
> I've come up with a shorter reproducer (original was trying to debug a perl script
> in OpenSSL's test suite).
> 
> Compile this program:
> 
> #include <sys/types.h>
> #include <sys/wait.h>
> #include <err.h>
> #include <stdio.h>
> #include <stdlib.h>
> #include <unistd.h>
> 
> int
> main(void)
> {
> 	pid_t pid, wpid;
> 
> 	pid = fork();
> 	if (pid == -1)
> 		err(1, "fork");
> 	if (pid == 0) {
> 		printf("I'm in the child\n");
> 		exit(1);
> 	}
> 	printf("I'm in the parent\n");
> 	wpid = waitpid(pid, NULL, 0);
> 	if (wpid < 0)
> 		err(1, "waitpid");
> 
> 	return (0);
> }
> 
> Then in gdb do the following:
> 
> # gdb101 ./forktest
> ...
> (gdb) catch fork
> Catchpoint 1 (fork)
> (gdb) r
> Starting program: /mnt/forktest/forktest 
> 
> Catchpoint 1 (forked process 830), _fork () at _fork.S:4
> 4       _fork.S: No such file or directory.
> (gdb) kill
> Kill the program being debugged? (y or n) y
> [Inferior 1 (process 828) killed]
> (gdb) quit
> <panic>
> 
https://reviews.freebsd.org/D27816