zdb -m : Assertion failed: (tq->tq_freelist != NULL)

From: Dan Mack <mack_at_macktronics.com>
Date: Sat, 17 May 2014 08:43:55 -0500 (CDT)
zdb -m crashed for me today, here's some details in case its a new
problem.  The errors on disk4 were fixed during the last scrub but I guess
that could be related and I'll retry after that drive is replaced.

% uname -a
FreeBSD darkstor 11.0-CURRENT FreeBSD 11.0-CURRENT #4 r265941: Mon May 12 22:57:13 CDT 2014     root_at_darkstor:/usr/obj/usr/src/sys/GENERIC  amd64

config:

 	NAME           STATE     READ WRITE CKSUM
 	tank           ONLINE       0     0     0
 	  raidz1-0     ONLINE       0     0     0
             gpt/disk0  ONLINE       0     0     0
 	    gpt/disk1  ONLINE       0     0     0
             gpt/disk2  ONLINE       0     0     0
             gpt/disk3  ONLINE       0     0     0
             gpt/disk4  ONLINE       0     0     7
         cache
           gpt/larc5    ONLINE       0     0     0

root_at_darkstor:~ # zdb -m tank

Metaslabs:
 	vdev          0
 	metaslabs   187   offset                spacemap          free
 	---------------   -------------------   ---------------   -------------
 	metaslab      0   offset            0   spacemap     36   free    4.78G
 	metaslab      1   offset    800000000   spacemap    136   free    12.3G
 	metaslab      2   offset   1000000000   spacemap    138   free    1.59G
 	metaslab      3   offset   1800000000   spacemap    145   free    1.25G
 	metaslab      4   offset   2000000000   spacemap    147   free    7.54G
 	metaslab      5   offset   2800000000   spacemap    149   free    5.59G
 	metaslab      6   offset   3000000000   spacemap    157   free    1.33G
 	metaslab      7   offset   3800000000   spacemap    158   free    1.31G
 	metaslab      8   offset   4000000000   spacemap    161   free    1.19G
 	metaslab      9   offset   4800000000   spacemap    200   free    1.40G
 	metaslab     10   offset   5000000000   spacemap    203   free    7.89G
 	metaslab     11   offset   5800000000   spacemap    214   free    11.0G
 	metaslab     12   offset   6000000000   spacemap  93606   free    7.18G
 	metaslab     13   offset   6800000000   spacemap   4128   free    3.38G
 	metaslab     14   offset   7000000000   spacemap   4337   free    1.43G
 	metaslab     15   offset   7800000000   spacemap   4992   free    7.58G
 	metaslab     16   offset   8000000000   spacemap   5102   free    7.38G
 	metaslab     17   offset   8800000000   spacemap   8520   free    1.20G
 	metaslab     18   offset   9000000000   spacemap 102401   free    5.75G
 	metaslab     19   offset   9800000000   spacemap  90113   free    1.29G
 	metaslab     20   offset   a000000000   spacemap  20483   free    1.53G
 	metaslab     21   offset   a800000000   spacemap  20591   free    1.32G
 	metaslab     22   offset   b000000000   spacemap  28996   free    1.29G
 	metaslab     23   offset   b800000000   spacemap  37284   free    1.57G
 	metaslab     24   offset   c000000000   spacemap  37286   free     9.6G
 	metaslab     25   offset   c800000000   spacemap  37361   free    6.96G
 	metaslab     26   offset   d000000000   spacemap  41470   free    1.33G
 	metaslab     27   offset   d800000000   spacemap  41473   free    1.14G
 	metaslab     28   offset   e000000000   spacemap  41475   free    1.26G
 	metaslab     29   offset   e800000000   spacemap  41477   free     953M
 	metaslab     30   offset   f000000000   spacemap  41491   free    1.16G
 	metaslab     31   offset   f800000000   spacemap  41494   free    1.22G
 	metaslab     32   offset  10000000000   spacemap  41497   free     955M
 	metaslab     33   offset  10800000000   spacemap  41499   free    1.06G
 	metaslab     34   offset  11000000000   spacemap  41502   free    1.27G
 	metaslab     35   offset  11800000000   spacemap  41505   free    1.27G
 	metaslab     36   offset  12000000000   spacemap     35   free    13.1G
 	metaslab     37   offset  12800000000   spacemap    135   free    1.57G
 	metaslab     38   offset  13000000000   spacemap    137   free    13.1G
 	metaslab     39   offset  13800000000   spacemap    139   free    1.34G
 	metaslab     40   offset  14000000000   spacemap    146   free    1.13G
 	metaslab     41   offset  14800000000   spacemap    159   free    7.70G
 	metaslab     42   offset  15000000000   spacemap    160   free    8.37G
 	metaslab     43   offset  15800000000   spacemap    181   free    12.3G
 	metaslab     44   offset  16000000000   spacemap    179   free    6.54G
 	metaslab     45   offset  16800000000   spacemap   4098   free     987M
 	metaslab     46   offset  17000000000   spacemap   4126   free    1.10G
 	metaslab     47   offset  17800000000   spacemap   4336   free    1.27G
 	metaslab     48   offset  18000000000   spacemap   4127   free    12.6G
 	metaslab     49   offset  18800000000   spacemap  20484   free    1.13G
 	metaslab     50   offset  19000000000   spacemap  37162   free    11.9G
 	metaslab     51   offset  19800000000   spacemap  41465   free    4.34G
 	metaslab     52   offset  1a000000000   spacemap  41507   free    1.19G
 	metaslab     53   offset  1a800000000   spacemap   8519   free    2.77G
 	metaslab     54   offset  1b000000000   spacemap  15838   free    1.21G
 	metaslab     55   offset  1b800000000   spacemap  15917   free    1.61G
 	metaslab     56   offset  1c000000000   spacemap  20482   free    1.50G
 	metaslab     57   offset  1c800000000   spacemap  20590   free    2.72G
 	metaslab     58   offset  1d000000000   spacemap  28995   free    1.27G
 	metaslab     59   offset  1d800000000   spacemap  90253   free    1.41G
 	metaslab     60   offset  1e000000000   spacemap  37285   free    6.80G
 	metaslab     61   offset  1e800000000   spacemap  41469   free    1.23G
 	metaslab     62   offset  1f000000000   spacemap  41511   free    1.22G
 	metaslab     63   offset  1f800000000   spacemap  41472   free    14.7G
 	metaslab     64   offset  20000000000   spacemap  41513   free    1.07G
 	metaslab     65   offset  20800000000   spacemap  41476   free    1.32G
 	metaslab     66   offset  21000000000   spacemap  41490   free    1.30G
 	metaslab     67   offset  21800000000   spacemap  41493   free    12.5G
 	metaslab     68   offset  22000000000   spacemap  41496   free    15.2G
 	metaslab     69   offset  22800000000   spacemap  41498   free    1.75G
 	metaslab     70   offset  23000000000   spacemap  41501   free    14.4G
 	metaslab     71   offset  23800000000   spacemap  41504   free    15.5G
 	metaslab     72   offset  24000000000   spacemap     34   free    1.43G
 	metaslab     73   offset  24800000000   spacemap    141   free    1.25G
 	metaslab     74   offset  25000000000   spacemap    199   free    8.03G
 	metaslab     75   offset  25800000000   spacemap   8193   free    13.8G
 	metaslab     76   offset  26000000000   spacemap  41508   free    6.52G
 	metaslab     77   offset  26800000000   spacemap   4991   free    3.11G
 	metaslab     78   offset  27000000000   spacemap    196   free     9.8G
 	metaslab     79   offset  27800000000   spacemap    180   free    1.30G
 	metaslab     80   offset  28000000000   spacemap   5101   free    9.24G
 	metaslab     81   offset  28800000000   spacemap   4097   free     9.8G
 	metaslab     82   offset  29000000000   spacemap  41509   free    8.03G
 	metaslab     83   offset  29800000000   spacemap  94209   free    5.21G
 	metaslab     84   offset  2a000000000   spacemap  41521   free    1.25G
 	metaslab     85   offset  2a800000000   spacemap  90190   free    7.09G
 	metaslab     86   offset  2b000000000   spacemap  41523   free    1.31G
 	metaslab     87   offset  2b800000000   spacemap  41464   free    8.43G
 	metaslab     88   offset  2c000000000   spacemap  99170   free    14.4G
 	metaslab     89   offset  2c800000000   spacemap  41525   free    1.27G
 	metaslab     90   offset  2d000000000   spacemap  41515   free    16.0G
 	metaslab     91   offset  2d800000000   spacemap  41528   free    1.31G
 	metaslab     92   offset  2e000000000   spacemap  20481   free    3.08G
 	metaslab     93   offset  2e800000000   spacemap  29180   free    1.47G
 	metaslab     94   offset  2f000000000   spacemap  28994   free    2.70G
 	metaslab     95   offset  2f800000000   spacemap  41474   free    15.8G
 	metaslab     96   offset  30000000000   spacemap  41553   free    1.22G
 	metaslab     97   offset  30800000000   spacemap  41568   free    1.62G
 	metaslab     98   offset  31000000000   spacemap  41510   free    16.0G
 	metaslab     99   offset  31800000000   spacemap  41471   free    1.50G
 	metaslab    100   offset  32000000000   spacemap  41512   free    15.9G
 	metaslab    101   offset  32800000000   spacemap  41570   free    1.20G
 	metaslab    102   offset  33000000000   spacemap  41489   free    1.47G
 	metaslab    103   offset  33800000000   spacemap  90254   free    1.64G
 	metaslab    104   offset  34000000000   spacemap  90194   free    17.4G
 	metaslab    105   offset  34800000000   spacemap  41571   free    1.12G
 	metaslab    106   offset  35000000000   spacemap  94221   free    1.69G
 	metaslab    107   offset  35800000000   spacemap  94210   free    5.75G
 	metaslab    108   offset  36000000000   spacemap  41573   free    1.25G
 	metaslab    109   offset  36800000000   spacemap  41575   free    1.25G
 	metaslab    110   offset  37000000000   spacemap  41577   free    1.20G
 	metaslab    111   offset  37800000000   spacemap  41579   free    1.17G
 	metaslab    112   offset  38000000000   spacemap  41581   free    1.12G
 	metaslab    113   offset  38800000000   spacemap  41583   free    1.33G
 	metaslab    114   offset  39000000000   spacemap  41585   free    1.17G
 	metaslab    115   offset  39800000000   spacemap  41544   free    11.5G
 	metaslab    116   offset  3a000000000   spacemap  41587   free    1.14G
 	metaslab    117   offset  3a800000000   spacemap  41589   free    1.27G
 	metaslab    118   offset  3b000000000   spacemap  41594   free    1.19G
 	metaslab    119   offset  3b800000000   spacemap  41596   free    1.13G
 	metaslab    120   offset  3c000000000   spacemap  41520   free    2.15G
 	metaslab    121   offset  3c800000000   spacemap  90191   free    16.3G
 	metaslab    122   offset  3d000000000   spacemap  41522   free    2.14G
 	metaslab    123   offset  3d800000000   spacemap  41598   free    1.14G
 	metaslab    124   offset  3e000000000   spacemap  41600   free    1.21G
 	metaslab    125   offset  3e800000000   spacemap  41524   free    16.5G
 	metaslab    126   offset  3f000000000   spacemap  41602   free    1.28G
 	metaslab    127   offset  3f800000000   spacemap  41527   free    16.3G
 	metaslab    128   offset  40000000000   spacemap  41604   free    1.31G
 	metaslab    129   offset  40800000000   spacemap  41606   free    1.24G
 	metaslab    130   offset  41000000000   spacemap  41608   free    1.04G
 	metaslab    131   offset  41800000000   spacemap  41611   free    1.09G
 	metaslab    132   offset  42000000000   spacemap  41552   free    1.97G
 	metaslab    133   offset  42800000000   spacemap  41567   free    14.5G
 	metaslab    134   offset  43000000000   spacemap  41613   free    1.22G
 	metaslab    135   offset  43800000000   spacemap  41978   free    1.31G
 	metaslab    136   offset  44000000000   spacemap  41980   free    1.34G
 	metaslab    137   offset  44800000000   spacemap  41569   free    16.8G
 	metaslab    138   offset  45000000000   spacemap  46091   free    12.6G
 	metaslab    139   offset  45800000000   spacemap  53250   free    1.29G
 	metaslab    140   offset  46000000000   spacemap  53400   free    1.36G
 	metaslab    141   offset  46800000000   spacemap  61303   free    1.30G
 	metaslab    142   offset  47000000000   spacemap  61594   free    1.23G
 	metaslab    143   offset  47800000000   spacemap  68835   free    1.26G
 	metaslab    144   offset  48000000000   spacemap  41572   free    17.3G
 	metaslab    145   offset  48800000000   spacemap  41574   free    10.6G
 	metaslab    146   offset  49000000000   spacemap  41576   free    12.3G
 	metaslab    147   offset  49800000000   spacemap  41578   free    11.5G
 	metaslab    148   offset  4a000000000   spacemap  41580   free    11.6G
 	metaslab    149   offset  4a800000000   spacemap  41582   free    12.9G
 	metaslab    150   offset  4b000000000   spacemap  41584   free    11.7G
 	metaslab    151   offset  4b800000000   spacemap  46078   free    12.5G
 	metaslab    152   offset  4c000000000   spacemap  41586   free    12.3G
 	metaslab    153   offset  4c800000000   spacemap  41588   free    10.7G
 	metaslab    154   offset  4d000000000   spacemap  41593   free    15.1G
 	metaslab    155   offset  4d800000000   spacemap  41595   free    6.25G
 	metaslab    156   offset  4e000000000   spacemap  69103   free    1.29G
 	metaslab    157   offset  4e800000000   spacemap  77110   free    1.28G
 	metaslab    158   offset  4f000000000   spacemap  80851   free    1.24G
 	metaslab    159   offset  4f800000000   spacemap  41597   free    11.2G
 	metaslab    160   offset  50000000000   spacemap  41599   free     9.6G
 	metaslab    161   offset  50800000000   spacemap  85363   free    13.0G
 	metaslab    162   offset  51000000000   spacemap  41601   free    11.2G
 	metaslab    163   offset  51800000000   spacemap  90192   free    14.1G
 	metaslab    164   offset  52000000000   spacemap  41603   free     9.8G
 	metaslab    165   offset  52800000000   spacemap  41605   free    10.3G
 	metaslab    166   offset  53000000000   spacemap  41607   free    10.5G
 	metaslab    167   offset  53800000000   spacemap  41610   free    25.0G
 	metaslab    168   offset  54000000000   spacemap  90252   free    13.9G
 	metaslab    169   offset  54800000000   spacemap  94211   free    1.51G
 	metaslab    170   offset  55000000000   spacemap  41612   free    1.99G
 	metaslab    171   offset  55800000000   spacemap  41977   free    25.9G
 	metaslab    172   offset  56000000000   spacemap  41979   free    1.95G
 	metaslab    173   offset  56800000000   spacemap  97745   free    9.04G
 	metaslab    174   offset  57000000000   spacemap  46090   free    6.09G
 	metaslab    175   offset  57800000000   spacemap  53249   free    25.9G
 	metaslab    176   offset  58000000000   spacemap  53399   free    1.79G
 	metaslab    177   offset  58800000000   spacemap  61302   free    29.4G
 	metaslab    178   offset  59000000000   spacemap  61593   free    1.90G
 	metaslab    179   offset  59800000000   spacemap  69102   free    2.85G
 	metaslab    180   offset  5a000000000   spacemap  99169   free    27.7G
 	metaslab    181   offset  5a800000000   spacemap 102402   free    15.5G
 	metaslab    182   offset  5b000000000   spacemap      0   free      32G
 	metaslab    183   offset  5b800000000   spacemap      0   free      32G
 	metaslab    184   offset  5c000000000   spacemap      0   free      32G
 	metaslab    185   offset  5c800000000   spacemap      0   free      32G
 	metaslab    186   offset  5d000000000   spacemap      0   free      32G

Assertion failed: (tq->tq_freelist != NULL), file /usr/src/cddl/lib/libzpool/../../../cddl/contrib/opensolaris/lib/libzpool/common/taskq.c, line 289.
Abort (core dumped)

(gdb) where
#0  0x0000000801ca6b7a in thr_kill () from /lib/libc.so.7
#1  0x0000000801d68a59 in abort () from /lib/libc.so.7
#2  0x0000000801912321 in taskq_dispatch () from /lib/libzpool.so.2
#3  0x0000000801912b02 in taskq_destroy () from /lib/libzpool.so.2
#4  0x0000000801912b95 in system_taskq_fini () from /lib/libzpool.so.2
#5  0x000000080191495e in kernel_fini () from /lib/libzpool.so.2
#6  0x000000000040874f in ?? ()
#7  0x00000000004050bf in ?? ()
#8  0x0000000800631000 in ?? ()
#9  0x0000000000000000 in ?? ()
(gdb)

Let me know if I can help by providing any more information,

dan
--
Dan Mack
Received on Sat May 17 2014 - 11:51:21 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:49 UTC