Re: panic: solaris assert: sa.sa_magic == 0x2F505A (0x4d5ea364 == 0x2f505a), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 625

Andriy Gapon <avg_at_FreeBSD.org> wrote:

> on 01/04/2013 17:18 Fabian Keil said the following:
> > #10 0xffffffff8130323f in assfail3 (a=<value optimized out>, lv=<value optimized out>, op=<value optimized out>, rv=<value optimized out>, f=<value optimized out>, l=<value optimized out>)
> >     at /usr/src/sys/modules/opensolaris/../../cddl/compat/opensolaris/kern/opensolaris_cmn_err.c:89
> > #11 0xffffffff8117924e in zfs_space_delta_cb (bonustype=<value optimized out>, data=0xffffff8015eeb8c0, userp=0xfffffe004261c640, groupp=0xfffffe004261c648)
> >     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:625
> > #12 0xffffffff8110003b in dmu_objset_userquota_get_ids (dn=0xfffffe004261c358, before=0, tx=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1249
> > #13 0xffffffff811071b6 in dnode_sync (dn=0xfffffe004261c358, tx=0xfffffe00186e1300) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dnode_sync.c:554
> > #14 0xffffffff810ff98b in dmu_objset_sync_dnodes (list=0xfffffe00691a5250, newlist=<value optimized out>, tx=<value optimized out>)
> >     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:910
> > #15 0xffffffff810ff825 in dmu_objset_sync (os=0xfffffe00691a5000, pio=<value optimized out>, tx=0xfffffe00186e1300)
> >     at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1027
> > #16 0xffffffff8110cb0d in dsl_dataset_sync (ds=0xfffffe001f3d0c00, zio=0x780, tx=0xfffffe00186e1300) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_dataset.c:1411
> > #17 0xffffffff8111399a in dsl_pool_sync (dp=0xfffffe0069ec4000, txg=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dsl_pool.c:409
> > #18 0xffffffff8112f0ee in spa_sync (spa=0xfffffe0050f00000, txg=3292) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/spa.c:6328
> > #19 0xffffffff81137c45 in txg_sync_thread (arg=0xfffffe0069ec4000) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/txg.c:493
> > #20 0xffffffff80569c1a in fork_exit (callout=0xffffffff811378d0 <txg_sync_thread>, arg=0xfffffe0069ec4000, frame=0xffffff80dc9d6c00) at /usr/src/sys/kern/kern_fork.c:991
> > #21 0xffffffff8086a1ee in fork_trampoline () at exception.S:602
> > #22 0x0000000000000000 in ?? ()
> > Current language:  auto; currently minimal
> > (kgdb) f 12
> > #12 0xffffffff8110003b in dmu_objset_userquota_get_ids (dn=0xfffffe004261c358, before=0, tx=<value optimized out>) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/dmu_objset.c:1249
> > 1249		error = used_cbs[os->os_phys->os_type](dn->dn_bonustype, data,
> > (kgdb) p *dn
> 
> Could you please also provide *dn->dn_phys?

vmcore.12:

(kgdb)  p *dn->dn_phys
Cannot access memory at address 0xffffff8019492800
(kgdb)  p *dn->dn_dbuf
$1 = {db = {db_object = 0, db_offset = 28491776, db_size = 16384, db_data = 0xffffff8019492000}, db_objset = 0xfffffe002bc62400, db_dnode_handle = 0xfffffe002bc62420, db_parent = 0xfffffe005f1071c0, 
  db_hash_next = 0x0, db_blkid = 1739, db_blkptr = 0xffffff801936c580, db_level = 0 '\0', db_mtx = {lock_object = {lo_name = 0xffffffff811d8696 "db->db_mtx", lo_flags = 40960000, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, db_holds = {rc_count = 2}, db_buf = 0xfffffe005f34c798, db_changed = {cv_description = 0xffffffff811d86a2 "db->db_changed", cv_waiters = 0}, 
  db_data_pending = 0xfffffe004bcc0c00, db_last_dirty = 0xfffffe004bcc0c00, db_link = {list_next = 0xfffffe005f3506d0, list_prev = 0xfffffe0030c392a0}, db_user_ptr = 0xfffffe005f269000, 
  db_user_data_ptr_ptr = 0x0, db_evict_func = 0xffffffff81105770 <dnode_buf_pageout>, db_immediate_evict = 0 '\0', db_freed_in_flight = 0 '\0', db_dirtycnt = 1 '\001'}

vmcore.13:

(kgdb)  p *dn->dn_phys
Cannot access memory at address 0xffffff8015eeb800
(kgdb) p *dn->dn_dbuf
$1 = {db = {db_object = 0, db_offset = 28491776, db_size = 16384, db_data = 0xffffff8015eeb000}, db_objset = 0xfffffe00691a5000, db_dnode_handle = 0xfffffe00691a5020, db_parent = 0xfffffe004254ab60, 
  db_hash_next = 0x0, db_blkid = 1739, db_blkptr = 0xffffff8015d65580, db_level = 0 '\0', db_mtx = {lock_object = {lo_name = 0xffffffff811d8696 "db->db_mtx", lo_flags = 40960000, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, db_holds = {rc_count = 2}, db_buf = 0xfffffe0042bedcf0, db_changed = {cv_description = 0xffffffff811d86a2 "db->db_changed", cv_waiters = 0}, 
  db_data_pending = 0xfffffe00406d6500, db_last_dirty = 0xfffffe00406d6500, db_link = {list_next = 0xfffffe0042758c10, list_prev = 0xfffffe0069cab5f8}, db_user_ptr = 0xfffffe0069f70000, 
  db_user_data_ptr_ptr = 0x0, db_evict_func = 0xffffffff81105770 <dnode_buf_pageout>, db_immediate_evict = 0 '\0', db_freed_in_flight = 0 '\0', db_dirtycnt = 1 '\001'}

vmcore.14:
(kgdb)  p *dn->dn_phys
Cannot access memory at address 0xffffff8014d21800
(kgdb)  p *dn->dn_dbuf
$1 = {db = {db_object = 0, db_offset = 28508160, db_size = 16384, db_data = 0xffffff8014d21000}, db_objset = 0xfffffe0073372000, db_dnode_handle = 0xfffffe0073372020, db_parent = 0xfffffe00369eec40, 
  db_hash_next = 0x0, db_blkid = 1740, db_blkptr = 0xffffff8014ac3600, db_level = 0 '\0', db_mtx = {lock_object = {lo_name = 0xffffffff811d8696 "db->db_mtx", lo_flags = 40960000, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, db_holds = {rc_count = 2}, db_buf = 0xfffffe0036b759d8, db_changed = {cv_description = 0xffffffff811d86a2 "db->db_changed", cv_waiters = 0}, 
  db_data_pending = 0xfffffe0034467700, db_last_dirty = 0xfffffe0034467700, db_link = {list_next = 0xfffffe0036c17970, list_prev = 0xfffffe00737bc950}, db_user_ptr = 0xfffffe00369fe000, 
  db_user_data_ptr_ptr = 0x0, db_evict_func = 0xffffffff81105770 <dnode_buf_pageout>, db_immediate_evict = 0 '\0', db_freed_in_flight = 0 '\0', db_dirtycnt = 1 '\001'}

vmcore.15:
(kgdb)  p *dn->dn_phys
Cannot access memory at address 0xffffff8030e73800
(kgdb) p *dn->dn_dbuf
$1 = {db = {db_object = 0, db_offset = 28508160, db_size = 16384, db_data = 0xffffff8030e73000}, db_objset = 0xfffffe0028fe2800, db_dnode_handle = 0xfffffe0028fe2820, db_parent = 0xfffffe005ac4dd20, 
  db_hash_next = 0x0, db_blkid = 1740, db_blkptr = 0xffffff8030d49600, db_level = 0 '\0', db_mtx = {lock_object = {lo_name = 0xffffffff811d8696 "db->db_mtx", lo_flags = 40960000, lo_data = 0, 
      lo_witness = 0x0}, sx_lock = 1}, db_state = DB_CACHED, db_holds = {rc_count = 2}, db_buf = 0xfffffe0010130a68, db_changed = {cv_description = 0xffffffff811d86a2 "db->db_changed", cv_waiters = 0}, 
  db_data_pending = 0xfffffe0057b85200, db_last_dirty = 0xfffffe0057b85200, db_link = {list_next = 0xfffffe005ad57890, list_prev = 0xfffffe00568765f8}, db_user_ptr = 0xfffffe005a7a1000, 
  db_user_data_ptr_ptr = 0x0, db_evict_func = 0xffffffff81105770 <dnode_buf_pageout>, db_immediate_evict = 0 '\0', db_freed_in_flight = 0 '\0', db_dirtycnt = 1 '\001'}

> > The offending sa_magic in the panic message is always the same.
> 
> Which part, left side or right side?

I meant the left side, but it looks like I only compared the first two panics:

fk_at_r500 /usr/crash $grep ^panic: core.txt.1[2-5]
core.txt.12:panic: solaris assert: sa.sa_magic == 0x2F505A (0x4d5ea364 == 0x2f505a), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 625
core.txt.13:panic: solaris assert: sa.sa_magic == 0x2F505A (0x4d5ea364 == 0x2f505a), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 625
core.txt.14:panic: solaris assert: sa.sa_magic == 0x2F505A (0x4be9810a == 0x2f505a), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 625
core.txt.15:panic: solaris assert: sa.sa_magic == 0x2F505A (0x4be9810a == 0x2f505a), file: /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c, line: 625

The matches seem to correlate with the matching dn->dn_dbuf->db->db_offset
values above.

Fabian