Re: kernel panic and fun debugging

On Sat, Oct 17, 2020 at 1:17 PM Steve Kargl <
sgk_at_troutmask.apl.washington.edu> wrote:

> On Sat, Oct 17, 2020 at 12:58:48PM -0600, Warner Losh wrote:
> > On Sat, Oct 17, 2020, 12:00 PM Andrey V. Elsukov <bu7cher_at_yandex.ru>
> wrote:
> >
> > > On 15.10.2020 09:56, Steve Kargl wrote:
> > > > Just had a kernel panic.  Best info I give you is
> > > >
> > > > % uname -a
> > > > FreeBSD mobile 13.0-CURRENT FreeBSD 13.0-CURRENT #1 r366176M: Sat
> Sep 26
> > > 10:35:23 PDT 2020     kargl_at_mobile
> :/usr/obj/usr/src/i386.i386/sys/MOBILE
> > > i386
> > > >
> > > > % kgdb gdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1
> > > > ...
> > > > Reading symbols from /usr/lib/debug/boot/kernel/kernel.debug...
> > > > /usr/ports/devel/gdb/work-py37/gdb-9.2/gdb/inferior.c:283:
> > > internal-error: struct inferior *find_inferior_pid(int): Assertion
> `pid !=
> > > 0' failed.
> > > > A problem internal to GDB has been detected,
> > > > further debugging may prove unreliable.
> > >
> > > Hi,
> > >
> > > do you have /var/crash/core.txt.1 file?
> > > It may have some useful info. Also did you try an old version
> > > /usr/libexec/kgdb ?
> > >
> >
> > I got this same error, btw, when I pointed kgdb at the wrong kernel for
> the
> > core file.
> >
> > But you can 'more' the uncompressed vmcore to get at least a traceback if
> > the proper kernel is gone...
> >
> > Warner
> >
>
> There are only 2 kernel.debug on the system.
>
> % locate kernel.debug
> /usr/lib/debug/boot/kernel/kernel.debug
> /usr/lib/debug/boot/kernel.old/kernel.debug
>
> Using
>
> % kgdb /usr/lib/debug/boot/kernel/kernel.debug vmcore.1
> % kgdb /usr/lib/debug/boot/kernel.old/kernel.debug vmcore.1
>
> Both have the same result.


In a pinch, you can use /boot/kernel/kernel if you suspect a difference...


> I didn't realize the panic text
> was recorded int vmcore.1.  If I extracted everything correctly,
> here's what happen
>
> WARNING !drm_modeset_is_locked(&crtc->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:621
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x23219ec2 at drm_atomic_helper_check_modeset+0x92
> #2 0x23091d90 at intel_atomic_check+0x70
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&crtc->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:621
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x23219ec2 at drm_atomic_helper_check_modeset+0x92
> #2 0x23091d90 at intel_atomic_check+0x70
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&dev->mode_config.connection_mutex) failed
> at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:666
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x2321a005 at drm_atomic_helper_check_modeset+0x1d5
> #2 0x23091d90 at intel_atomic_check+0x70
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&plane->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d
> #2 0x23092da7 at intel_atomic_check+0x1087
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&plane->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d
> #2 0x23092da7 at intel_atomic_check+0x1087
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&plane->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d
> #2 0x23092da7 at intel_atomic_check+0x1087
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> WARNING !drm_modeset_is_locked(&plane->mutex) failed at
> /usr/ports/graphics/drm-current-kmod/work/drm-kmod-drm_v5.4.62_1/drivers/gpu/drm/drm_atomic_helper.c:871
> #0 0x2318d4b9 at linux_dump_stack+0x19
> #1 0x2321adbd at drm_atomic_helper_check_planes+0x8d
> #2 0x23092da7 at intel_atomic_check+0x1087
> #3 0x2321913e at drm_atomic_check_only+0x38e
> #4 0x23219461 at drm_atomic_commit+0x11
> #5 0x23224793 at drm_client_modeset_commit_atomic+0xb3
> #6 0x2322458e at drm_client_modeset_commit_force+0x5e
> #7 0x23260851 at drm_fb_helper_restore_fbdev_mode_unlocked+0x71
> #8 0x2325b092 at vt_kms_postswitch+0x132
> #9 0xa8bec5 at vt_fb_postswitch+0x15
> #10 0xa9153d at vt_window_switch+0xfd
> #11 0xa8f59c at vtterm_cngrab+0x1c
> #12 0xbec5bf at termcn_cngrab+0xf
> #13 0xb4b6f6 at cngrab+0x16
> #14 0xb9fce2 at vpanic+0xd2
> #15 0xb9fc04 at panic+0x14
> #16 0xe2c06a at vm_fault_lookup+0x13a
> #17 0xe2b8e7 at vm_fault+0x77
> <4>WARN_ON(!mutex_is_locked(&dev->struct_mutex))
>
>
> <4>WARN_ON(!mutex_is_locked(&fbc->lock))WARN_ON(!mutex_is_locked(&fbc->lock))
> panic: vm_fault_lookup: fault on nofault entry, addr: 0
> cpuid = 1
> time = 1602744240
> KDB: stack backtrace:
> db_trace_self_wrapper(1,22b9e8b4,9c8f69,fdabf8,12742000,...) at
> db_trace_self_wrapper+0x28/frame 0x22b9e898
> kdb_backtrace(22c28700,1,121ad98,22b9e960,22b9e9a0,...) at
> kdb_backtrace+0x2b/frame 0x22b9e8f0
> vpanic(f2d22a,22b9e92c,22b9e92c,22b9e950,e2c06a,...) at vpanic+0x130/frame
> 0x22b9e90c
> panic(f2d22a,f1fec9,0,22b9e974,87a2328,...) at panic+0x14/frame 0x22b9e920
> vm_fault_lookup(ffffffff,0,0,0,701,...) at vm_fault_lookup+0x13a/frame
> 0x22b9e950
> vm_fault(121ad98,0,1,0,0) at vm_fault+0x77/frame 0x22b9e9d0
> vm_fault_trap(121ad98,0,1,0,0,0) at vm_fault_trap+0x44/frame 0x22b9e9f8
> trap_pfault(0,0,0) at trap_pfault+0x119/frame 0x22b9ea40
> trap(22b9eb08,8,28,28,40,...) at trap+0x2fb/frame 0x22b9eafc
> calltrap() at 0xffc0319f/frame 0x22b9eafc
> --- trap 0xc, eip = 0x231a72c4, esp = 0x22b9eb48, ebp = 0x22b9eb48 ---
> __linux_clflushopt(0,90000,0,3bea6000,2,...) at
> __linux_clflushopt+0x24/frame 0x22b9eb48
> drm_clflush_sg(34cca3c0,34992dc0,22b9ebb0,22b9eb8c,ba82b8,...) at
> drm_clflush_sg+0x129/frame 0x22b9eb70
> shmem_put_pages(34dab8c0,34cca3c0,0,0,22fe6038,...) at
> shmem_put_pages+0x3f/frame 0x22b9ebd0
> __i915_gem_object_put_pages(34dab8c0,0,0,22c28700,22fe6038,...) at
> __i915_gem_object_put_pages+0x9d/frame 0x22b9ebf4
> i915_gem_flush_free_objects(22fe6000,22b9ec5c,231981bd,22fe87bc,885aec0,...)
> at i915_gem_flush_free_objects+0x11e/frame 0x22b9ec20
> __i915_gem_free_work(22fe87bc,885aec0,0,885aed8,0,...) at
> __i915_gem_free_work+0x11/frame 0x22b9ec2c
> linux_work_fn(22fe87bc,1) at linux_work_fn+0xed/frame 0x22b9ec5c
> taskqueue_run_locked(0,22c28700,22b9ecd4,b67a57,885aec0,...) at
> taskqueue_run_locked+0x1c4/frame 0x22b9eca8
> taskqueue_thread_loop(885aec0,22b9ece8) at
> taskqueue_thread_loop+0x9e/frame 0x22b9ecb8
> fork_exit(beb5d0,885aec0,22b9ece8,0,0,...) at fork_exit+0x67/frame
> 0x22b9ecd4
> fork_trampoline() at 0xffc0340e/frame 0x22b9ecd4
> --- kthread start
> KDB: enter: panic
>


I'm guessing that there's a locking issue with the 32-bit drm emulation
code. This leads to unsynchronized access which leads to corrupt and this
error. But that's a wild guess.

At a guess, I'd say this is likely a bug in the 32-bit linux emulation, but
that's just a crazy WAG. I'd expect it would take a fair amount to track
this down...  And it would be useful to know what the first of these
warnings that eventually lead to this was. But my familiarity with the drm
code and how it plumbs into the VM is quite low.

Warner