Re: Panic on shutdown _at_r328436: "Unholding 6 with cnt = -559038242"

From: Warner Losh <imp_at_bsdimp.com>
Date: Sun, 28 Jan 2018 20:18:59 -0700
On Sun, Jan 28, 2018 at 7:38 PM, Cy Schubert <Cy.Schubert_at_cschubert.com>
wrote:

> In message <2effa324-c428-6135-371b-acb00c803d29_at_freebsd.org>, Allan
> Jude write
> s:
> > On 2018-01-28 16:28, Warner Losh wrote:
> > > On Sun, Jan 28, 2018 at 2:22 PM, thomas masper <
> thomas.masper_at_gmail.com>
> > > wrote:
> > >
> > >> Hi,
> > >> similar panic happen to me when extracting a pendrive from laptop USB
> port
> > >> (I tried 3 different pendrive).
> > >> No issue if I reboot or shutdown. I don't know if those two issues are
> > >> related.
> > >>
> > >
> > > Do you have a reproducible test case? Ideally, it would be 'insert and
> > > remove usb thumb drive' but maybe there's more steps between insert and
> > > removal.
> > >
> > > Warner
> > >
> > >
> > >
> > >> panic: Releasing 6 with cnt = -559038242
>
> Converting this to hex we get DEADC0DE.


vm/uma_dbg.c:static const uint32_t uma_junk = 0xdeadc0de;

Use after free it is then...

Warner


> > >>
> > >> GNU gdb (GDB) 8.0.1 [GDB v8.0.1 for FreeBSD]
> > >> Copyright (C) 2017 Free Software Foundation, Inc.
> > >> License GPLv3+: GNU GPL version 3 or later <
> http://gnu.org/licenses/gpl.
> > >> html
> > >>>
> > >> This is free software: you are free to change and redistribute it.
> > >> There is NO WARRANTY, to the extent permitted by law.  Type "show
> copying"
> > >> and "show warranty" for details.
> > >> This GDB was configured as "x86_64-portbld-freebsd12.0".
> > >> Type "show configuration" for configuration details.
> > >> For bug reporting instructions, please see:
> > >> <http://www.gnu.org/software/gdb/bugs/>.
> > >> Find the GDB manual and other documentation resources online at:
> > >> <http://www.gnu.org/software/gdb/documentation/>.
> > >> For help, type "help".
> > >> Type "apropos word" to search for commands related to "word"...
> > >> Reading symbols from /boot/kernel/kernel...Reading symbols from
> > >> /usr/lib/debug//boot/kernel/kernel.debug...done.
> > >> done.
> > >>
> > >> Unread portion of the kernel message buffer:
> > >> da0 at umass-sim0 bus 0 scbus4 target 0 lun 0
> > >> da0: <Generic Flash Disk 8.07>  s/n 30E47C20 detached
> > >> (da0:umass-sim0:0:0:0): Periph destroyed
> > >> panic: Releasing 6 with cnt = -559038242
> > >> cpuid = 0
> > >> time = 1517158352
> > >> KDB: stack backtrace:
> > >> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > >> 0xfffffe00593838c0
> > >> vpanic() at vpanic+0x18d/frame 0xfffffe0059383920
> > >> panic() at panic+0x43/frame 0xfffffe0059383980
> > >> dadiskgonecb() at dadiskgonecb+0x42/frame 0xfffffe00593839a0
> > >> g_disk_providergone() at g_disk_providergone+0x25/frame
> 0xfffffe00593839d0
> > >> g_destroy_provider() at g_destroy_provider+0xae/frame
> 0xfffffe00593839f0
> > >> g_wither_washer() at g_wither_washer+0x87/frame 0xfffffe0059383a30
> > >> g_run_events() at g_run_events+0x3ca/frame 0xfffffe0059383a70
> > >> fork_exit() at fork_exit+0x84/frame 0xfffffe0059383ab0
> > >> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0059383ab0
> > >> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> > >> KDB: enter: panic
> > >>
> > >> __curthread () at ./machine/pcpu.h:229
> > >> 229 __asm("movq %%gs:%1,%0" : "=r" (td)
> > >> (kgdb) #0  __curthread () at ./machine/pcpu.h:229
> > >> #1  doadump (textdump=0) at /usr/src/sys/kern/kern_shutdown.c:346
> > >> #2  0xffffffff8040a08b in db_dump (dummy=<optimized out>,
> > >>     dummy2=<unavailable>, dummy3=<unavailable>, dummy4=<unavailable>)
> > >>     at /usr/src/sys/ddb/db_command.c:574
> > >> #3  0xffffffff80409e59 in db_command (last_cmdp=<optimized out>,
> > >>     cmd_table=<optimized out>, dopager=<optimized out>)
> > >>     at /usr/src/sys/ddb/db_command.c:481
> > >> #4  0xffffffff80409bd4 in db_command_loop ()
> > >>     at /usr/src/sys/ddb/db_command.c:534
> > >> #5  0xffffffff8040cdff in db_trap (type=<optimized out>,
> code=<optimized
> > >> out>)
> > >>     at /usr/src/sys/ddb/db_main.c:250
> > >> #6  0xffffffff80b0d923 in kdb_trap (type=3, code=-61456, tf=<optimized
> > >> out>)
> > >>     at /usr/src/sys/kern/subr_kdb.c:697
> > >> #7  0xffffffff80f7b498 in trap (frame=0xfffffe00593837f0)
> > >>     at /usr/src/sys/amd64/amd64/trap.c:547
> > >> #8  <signal handler called>
> > >> #9  kdb_enter (why=0xffffffff811f101e "panic", msg=<optimized out>)
> > >>     at /usr/src/sys/kern/subr_kdb.c:479
> > >> #10 0xffffffff80ac8d3a in vpanic (fmt=<optimized out>,
> > >> ap=0xfffffe0059383960)
> > >>     at /usr/src/sys/kern/kern_shutdown.c:800
> > >> #11 0xffffffff80ac8dc3 in panic (
> > >>     fmt=0xffffffff81b1bbd8 <cnputs_mtx> "\257\257\033\201\377\377\377\
> > >> 377")
> > >>     at /usr/src/sys/kern/kern_shutdown.c:738
> > >> #12 0xffffffff80368bb2 in da_periph_release (periph=<optimized out>,
> > >>     token=DA_REF_GEOM) at /usr/src/sys/cam/scsi/scsi_da.c:1591
> > >> #13 dadiskgonecb (dp=<optimized out>) at
> > >> /usr/src/sys/cam/scsi/scsi_da.c:1904
> > >> #14 0xffffffff80a0fdd5 in g_disk_providergone (pp=0xfffff80003e8b700)
> > >>     at /usr/src/sys/geom/geom_disk.c:783
> > >> #15 0xffffffff80a15f9e in g_destroy_provider (pp=0xfffff80003e8b700)
> > >>     at /usr/src/sys/geom/geom_subr.c:746
> > >> #16 0xffffffff80a15e17 in g_wither_washer ()
> > >>     at /usr/src/sys/geom/geom_subr.c:461
> > >> #17 0xffffffff80a112da in g_run_events ()
> > >>     at /usr/src/sys/geom/geom_event.c:297
> > >> #18 0xffffffff80a89444 in fork_exit (
> > >>     callout=0xffffffff80a138c0 <g_event_procbody>, arg=0x0,
> > >>     frame=0xfffffe0059383ac0) at /usr/src/sys/kern/kern_fork.c:1039
> > >> #19 <signal handler called>
> > >> (kgdb)
> > >>
> > >>
> > >> uname -a
> > >> FreeBSD laptopW530.tommyBSD.org 12.0-CURRENT FreeBSD 12.0-CURRENT #13
> > >> r328509M: Sun Jan 28 15:38:35 CET 2018
> > >> tommy_at_laptopW530.tommyBSD.org:/usr/obj/usr/src/amd64.amd64/
> sys/GENERIC
> > >> amd64
> > >>
> > >> Regards,
> > >> thomas
> > >>
> > >>
> > >> On Fri, Jan 26, 2018 at 4:07 PM, David Wolfskill <
> david_at_catwhisker.org>
> > >> wrote:
> > >>
> > >>> On Fri, Jan 26, 2018 at 07:47:48AM -0700, Warner Losh wrote:
> > >>>> On Fri, Jan 26, 2018 at 5:29 AM, David Wolfskill <
> david_at_catwhisker.org
> > >>>
> > >>>> wrote:
> > >>>>
> > >>>>> This is on my "build machine" (laptop is still building updated
> ports
> > >>>>> for today, so I don't know yet whether or not it encounters this.)
> > >>>>>
> > >>>>
> > >>>> Running a kernel with INVARIANTS, right?
> > >>>
> > >>> Yes -- GENERIC.
> > >>>
> > >>>>> I had performed a source-based update from r328393 to r328436,
> > >>>>> rebooted, performed "make delete-old-libs", and all seemed well.
> > >>>>>
> > >>>>
> > >>>> This has my change 328415 in it.
> > >>>
> > >>> :-)
> > >>>
> > >>>>> I then issued "sudo shutdown -p now", and serial console shows:
> > >>>>> panic: Unholding 6 with cnt = -559038242
> > >>>>> cpuid = 3
> > >>>>> time = 1516968697
> > >>>>> KDB: stack backtrace:
> > >>>>> db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame
> > >>>>> 0xfffffe00004288c0
> > >>>>> vpanic() at vpanic+0x18d/frame 0xfffffe0000428920
> > >>>>> panic() at panic+0x43/frame 0xfffffe0000428980
> > >>>>> dadiskgonecb() at dadiskgonecb+0x42/frame 0xfffffe00004289a0
> > >>>>> g_disk_providergone() at g_disk_providergone+0x25/frame
> > >>> 0xfffffe00004289d0
> > >>>>> g_destroy_provider() at g_destroy_provider+0xae/frame
> > >>> 0xfffffe00004289f0
> > >>>>> g_wither_washer() at g_wither_washer+0x87/frame 0xfffffe0000428a30
> > >>>>> g_run_events() at g_run_events+0x3ca/frame 0xfffffe0000428a70
> > >>>>> fork_exit() at fork_exit+0x84/frame 0xfffffe0000428ab0
> > >>>>> fork_trampoline() at fork_trampoline+0xe/frame 0xfffffe0000428ab0
> > >>>>> --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
> > >>>>> KDB: enter: panic
> > >>>>> [ thread pid 13 tid 100044 ]
> > >>>>> Stopped at      kdb_enter+0x3b: movq    $0,kdb_why
> > >>>>> db>
> > >>>>>
> > >>>>
> > >>>> That's no good. We're releasing a reference to the da peripheral
> > >> because
> > >>>> geom has finished with the disk and is giving us a final callback
> so we
> > >>> can
> > >>>> drop the reference we took when we created the geom. Trouble is, cnt
> > >>> should
> > >>>> be like 1 always for this code, but it's not. It looks like it may
> be
> > >>> bytes
> > >>>> to a pointer :(
> > >>>>
> > >>>>
> > >>>>> As noted, this is a build machine, and it was to be powered off for
> > >>>>> the rest of the day anyway, so I don't need to get it up & running
> > >>>>> immediately: I can poke at the ddb prompt, given some clues.
> > >>>>>
> > >>>>
> > >>>> I don't suppose you can attach kgdb to this machine? I'd be
> interested
> > >> to
> > >>>> see what the contents of the softc are...a
> > >>>
> > >>> Pointer to how to do that?
> > >>>
> > >>> I do have ddb right now....
> > >>>
> > >>>> ....
> > >>>> Thanks for the report. This is quite troubling.
> > >>>
> > >>> Well, let's get it fixed, then! :-)
> > >>>
> > >>>> Warner
> > >>>> ....
> > >>>
> > >>> I should still have access to the serial console after I get in to
> the
> > >>> office (heading out shortly).
> > >>>
> > >>> Peace,
> > >>> david
> > >>> --
> > >>> David H. Wolfskill                              david_at_catwhisker.org
> > >>> "unfortunately, no trust!” -- well, of course!  You reap what you
> sow.
> > >>>
> > >>> See http://www.catwhisker.org/~david/publickey.gpg for my public
> key.
> > >>>
> > >> _______________________________________________
> > >> freebsd-current_at_freebsd.org mailing list
> > >> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > >> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_
> freebsd.org"
> > >>
> > > _______________________________________________
> > > freebsd-current_at_freebsd.org mailing list
> > > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_
> freebsd.org"
> > >
> >
> > I've been seeing this today while working on my laptop.
> >
> > 1) insert USB stick.
> > 2) mount UFS partition to /mnt
> > 3) copy a file off
> > 4) umount /mnt
> > 5) remove usb stick
> > 6) instant panic
> >
> > Oddly, it is the same negative number every time (-559038242), so it
> > isn't random/memory corruption.
> >
> >
> > --
> > Allan Jude
> > _______________________________________________
> > freebsd-current_at_freebsd.org mailing list
> > https://lists.freebsd.org/mailman/listinfo/freebsd-current
> > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_
> freebsd.org"
> >
>
>
> --
> Cheers,
> Cy Schubert <Cy.Schubert_at_cschubert.com>
> FreeBSD UNIX:  <cy_at_FreeBSD.org>   Web:  http://www.FreeBSD.org
>
>         The need of the many outweighs the greed of the few.
>
>
>
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org"
>
>
Received on Mon Jan 29 2018 - 02:19:01 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:14 UTC