Re: [PANIC]: rw_lock panic in in_pcballoc() in r185864

On Wed, Dec 10, 2008 at 07:23:49PM +0000, Robert Watson wrote:
> On Wed, 10 Dec 2008, Roman Divacky wrote:
> 
> >#9  0xc06bd28b in calltrap () at ../../../i386/i386/exception.s:165
> >#10 0xc0528cc9 in _rw_wlock_hard (rw=0xc45a00a4, tid=3293569600, file=0x0,
> >   line=0) at ../../../kern/kern_rwlock.c:616
> >#11 0xc05eae42 in in_pcballoc (so=0xc459e000, pcbinfo=0xc0794b40)
> >   at ../../../netinet/in_pcb.c:238
> >#12 0xc060b403 in udp_attach (so=0xc459e000, proto=0, td=0xc44fe240)
> >   at ../../../netinet/udp_usrreq.c:1131
> >#13 0xc0586df5 in socreate (dom=2, aso=0xc3e77c6c, type=2, proto=0,
> >#14 0xc058d974 in socket (td=0xc44fe240, uap=0xc3e77cf8)
> >---Type <return> to continue, or q <return> to quit---Dec 10 17:29:23 
> >witten log
> >in: ROOT LOGIN (root) ON ttyv1
> >
> >   at ../../../kern/uipc_syscalls.c:178
> >#15 0xc06d8010 in syscall (frame=0xc3e77d38) at 
> >../../../i386/i386/trap.c:1076
> >#16 0xc06bd320 in Xint0x80_syscall () at ../../../i386/i386/exception.s:261
> >#17 0x00000033 in ?? ()
> >Previous frame inner to this frame (corrupt stack?)
> >
> >(kgdb) p *pcbinfo
> >$2 = {ipi_listhead = 0xc0794b24, ipi_count = 1, ipi_hashbase = 0xc42fe000,
> > ipi_hashmask = 127, ipi_porthashbase = 0xc42fce00, ipi_porthashmask = 127,
> > ipi_lastport = 0, ipi_lastlow = 0, ipi_lasthi = 0, ipi_zone = 0xc1471360,
> > ipi_gencnt = 0, ipi_lock = {lock_object = {lo_name = 0xc0713b87 "udp",
> >     lo_flags = 69926928, lo_data = 0, lo_witness = 0x0},
> >   rw_lock = 3293569600}, ipi_pspare = {
> >(kgdb) p *pcbinfo->ipi_zone
> >$4 = {uz_name = 0xc0716712 "udpcb", uz_lock = 0xc147ed88,
> > uz_keg = 0xc147ed80, uz_link = {le_next = 0x0, le_prev = 0xc147eda8},
> > uz_full_bucket = {lh_first = 0x0}, uz_free_bucket = {lh_first = 0x0},
> > uz_ctor = 0, uz_dtor = 0, uz_init = 0, uz_fini = 0, uz_allocs = 0,
> > uz_frees = 0, uz_fails = 0, uz_fills = 0, uz_count = 23, uz_cpu = {{
> >     uc_freebucket = 0x0, uc_allocbucket = 0x0, uc_allocs = 0,
> >     uc_frees = 0}}}
> >
> >the code tries to rw_rwlock() the inp->inp_lock, the inp is allocated from 
> >an UMA zone which has no constructor and in the in_pcballoc() the rwlock 
> >is never initialized. I believe that's why it crashes
> >
> >can someone confirm/fix that?
> 
> Hmm.  I disagree with the diagnosis, although clearly there's a problem 
> here of some sort since panicking is, well, bad.  Each protocol using the 
> inpcb framework provides its own UMA zone and is responsible for 
> initializing the lock on the inpcb when memory is first pulled into the 
> cache.  For UDP, that occurs in udp_inpcb_init(), the init function passed 
> into uma_zcreate() in udp_init().  Notice that when in_pcballoc() zeroes 
> the inpcb, it stops at inp_zero_size, which is before inp_lock, and since 
> the inpcb zone for UDP is set to be a no-free zone, there's no INP lock 
> destroy.
 
hm... I wondered how it could work before that.... anyway, I got this crash

> So, given that this code has worked for quite a long time for many people, 
> this really raises two questions: (1) how reproduceable is this and at what 
> point does it kick in during the boot/runtime, and (2) when did this start 
> happening in terms of updating your source?

ad (1): I get this on every boot when dhcp kicks in (it uses udp I believe)
ie. 100% reproducible

ad (2): Mon Dec  8 23:02:27 CET 2008 kernel (svn up-dated at most 10 minutes
before that) works 100% ok

I have the crash dump and the kernel at hand so I can do basically anything
you ask me to do :) anything I can provide?

thnx

roman