Re: Unkillable and runaway processes

From: Kenneth Vestergaard Schmidt <kvs_at_pil.dk>
Date: Fri, 07 Sep 2007 08:31:09 +0200
Pawel Jakub Dawidek <pjd_at_FreeBSD.org> writes:

> On Wed, Sep 05, 2007 at 05:25:09PM +0200, Kenneth Vestergaard Schmidt wrote:
>> Pawel Jakub Dawidek <pjd_at_FreeBSD.org> writes:
>> >> I could use some pointers on what to do - is there some way I can debug
>> >> this better, maybe give some better info?
>> >
>> > Try disabling ZIL. This looks like a bug was already reported by Kris.
>> > This was already reported to OpenSolaris.
>> 
>> We disabled ZIL, but the bug crept up again. I've just now rebooted the
>> machine with DDB-support, so I can get some more info, per Kris's mail.

That would be Kip's mail.

>> The full state of the process hanging is 'zfs:(&tx->tx_quiesce_done_cv)'
>>  - it cycles between RUN, CPUx and this one.
>
> Hmm, this means it didn't deadlock...

Here's some debugging output. PIDs 14870, 14933 and 12458 are just
spinning and being useless. Funnily, all three have rename() in their
backtrace.


db> ps
  pid  ppid  pgrp   uid   state   wmesg     wchan    cmd
20724 20720 20724     0  R+      CPU 0               sysctl
20720  1012 20720     0  S+      pause    0xd1e1a864 csh
20421   948   948   125  S       kqread   0xc9ad1600 pickup
14933 14931  4351     0  R                           rsync
14932 14931  4351     0  S       select   0xc07d68b8 ssh
14931 14930  4351     0  S       select   0xc07d68b8 rsync
14930 14661  4351     0  S       wait     0xd1c2b804 sh
14870 14866  4351     0  S       zfs:(&tx 0xc73e6f24 rsync
14867 14866  4351     0  S       select   0xc07d68b8 ssh
14866 14865  4351     0  S       select   0xc07d68b8 rsync
14865 14856  4351     0  S       wait     0xd26c62ac sh
14856  4355  4351     0  S       piperd   0xc8f94630 perl5.8.8
14661  4355  4351     0  S       piperd   0xd1fcb630 perl5.8.8
12458 12457  4351     0  R                           rsync
12457 12397  4351     0  S       select   0xc07d68b8 rsync
12397  4355  4351     0  S       piperd   0xd2090dec perl5.8.8
 4360  4357  4357     0  S       piperd   0xd1d55630 postdrop
 4357  4348  4357     0  Ss      piperd   0xd1fcbdec sendmail
 4355  4354  4351     0  S       wait     0xd20b0000 perl5.8.8
 4354  4351  4351     0  S       wait     0xd2050804 perl5.8.8
 4351  4348  4351     0  Ss      wait     0xd20482ac sh
 4348   970   970     0  S       piperd   0xc741d318 cron
 1013     1     1     0  S       nanslp   0xc07d07e4 getty
 1012     1  1012     0  Ss+     wait     0xd1c2c000 login
 1011     1  1011     0  Ss+     ttyin    0xc71d6410 getty
 1010     1  1010     0  Ss+     ttyin    0xc71c5410 getty
  995     1   995     0  Ss      select   0xc07d68b8 inetd
  970     1   970     0  Ss      nanslp   0xc07d07e4 cron
  963     1   963     0  Ss      select   0xc07d68b8 sshd
  957   948   948   125  S       kqread   0xd1eff480 qmgr
  948     1   948     0  Ss      kqread   0xd1e14e80 master
  866     1   866     0  Ss      select   0xc07d68b8 ntpd
  776     1   776     0  Ss      select   0xc07d68b8 syslogd
  721     1   721     0  Ss      select   0xc07d68b8 devd
  360     0     0     0  SL      zfs:(&tq 0xd151877c [zil_clean]
  359     0     0     0  SL      zfs:(&tq 0xd1518848 [zil_clean]
  358     0     0     0  SL      zfs:(&tq 0xd1518914 [zil_clean]
  357     0     0     0  SL      zfs:(&tq 0xd15189e0 [zil_clean]
  356     0     0     0  SL      zfs:(&tq 0xd1518aac [zil_clean]
  355     0     0     0  SL      zfs:(&tq 0xd1518b78 [zil_clean]
  354     0     0     0  SL      zfs:(&tq 0xd10ba1e8 [zil_clean]
  353     0     0     0  SL      zfs:(&tq 0xd10ba2b4 [zil_clean]
  352     0     0     0  SL      zfs:(&tq 0xd10ba380 [zil_clean]
  351     0     0     0  SL      zfs:(&tq 0xd1518c44 [zil_clean]
  350     0     0     0  SL      zfs:(&tq 0xd1518d10 [zil_clean]
  349     0     0     0  SL      zfs:(&tq 0xd1518ddc [zil_clean]
  348     0     0     0  SL      zfs:(&tq 0xd1518ea8 [zil_clean]
  347     0     0     0  SL      zfs:(&tq 0xd1849050 [zil_clean]
  346     0     0     0  SL      zfs:(&tq 0xd184911c [zil_clean]
  345     0     0     0  SL      zfs:(&tq 0xd18491e8 [zil_clean]
  344     0     0     0  SL      zfs:(&tq 0xd10ba44c [zil_clean]
  343     0     0     0  SL      zfs:(&tq 0xd18492b4 [zil_clean]
  342     0     0     0  SL      zfs:(&tq 0xd10ba518 [zil_clean]
  341     0     0     0  SL      zfs:(&tq 0xd10ba5e4 [zil_clean]
  340     0     0     0  SL      zfs:(&tq 0xd1849380 [zil_clean]
  339     0     0     0  SL      zfs:(&tq 0xd184944c [zil_clean]
  338     0     0     0  SL      zfs:(&tq 0xd10ba6b0 [zil_clean]
  337     0     0     0  SL      zfs:(&tq 0xd10ba77c [zil_clean]
  336     0     0     0  SL      zfs:(&tq 0xd1849518 [zil_clean]
  335     0     0     0  SL      zfs:(&tq 0xd18495e4 [zil_clean]
  334     0     0     0  SL      zfs:(&tq 0xd10ba848 [zil_clean]
  333     0     0     0  SL      zfs:(&tq 0xd18496b0 [zil_clean]
  332     0     0     0  SL      zfs:(&tq 0xd184977c [zil_clean]
  331     0     0     0  SL      zfs:(&tq 0xd1849848 [zil_clean]
  330     0     0     0  SL      zfs:(&tq 0xd1849914 [zil_clean]
  329     0     0     0  SL      zfs:(&tq 0xd10ba914 [zil_clean]
  328     0     0     0  SL      zfs:(&tq 0xd18499e0 [zil_clean]
  327     0     0     0  SL      zfs:(&tq 0xd1849aac [zil_clean]
  326     0     0     0  SL      zfs:(&tq 0xd1849b78 [zil_clean]
  325     0     0     0  SL      zfs:(&tq 0xd10ba9e0 [zil_clean]
  324     0     0     0  SL      zfs:(&tq 0xd1849c44 [zil_clean]
  323     0     0     0  SL      zfs:(&tq 0xd1849d10 [zil_clean]
  322     0     0     0  SL      zfs:(&tq 0xd10baaac [zil_clean]
  321     0     0     0  SL      zfs:(&tq 0xd0c77848 [zil_clean]
  320     0     0     0  SL      zfs:(&tq 0xd10bab78 [zil_clean]
  319     0     0     0  SL      zfs:(&tq 0xd0c77914 [zil_clean]
  318     0     0     0  SL      zfs:(&tq 0xd0c779e0 [zil_clean]
  317     0     0     0  SL      zfs:(&tq 0xd0c77aac [zil_clean]
  316     0     0     0  SL      zfs:(&tq 0xd10bac44 [zil_clean]
  315     0     0     0  SL      zfs:(&tq 0xd0c77b78 [zil_clean]
  314     0     0     0  SL      zfs:(&tq 0xd10bad10 [zil_clean]
  313     0     0     0  SL      zfs:(&tq 0xd0c77c44 [zil_clean]
  312     0     0     0  SL      zfs:(&tq 0xd0c77d10 [zil_clean]
  311     0     0     0  SL      zfs:(&tq 0xd0c77ddc [zil_clean]
  310     0     0     0  SL      zfs:(&tq 0xd10baddc [zil_clean]
  309     0     0     0  SL      zfs:(&tq 0xd10baea8 [zil_clean]
  308     0     0     0  SL      zfs:(&tq 0xd1518050 [zil_clean]
  307     0     0     0  SL      zfs:(&tq 0xd151811c [zil_clean]
  306     0     0     0  SL      zfs:(&tq 0xd15181e8 [zil_clean]
  305     0     0     0  SL      zfs:(&tq 0xd0c77ea8 [zil_clean]
  304     0     0     0  SL      zfs:(&tq 0xd0ea7050 [zil_clean]
  303     0     0     0  SL      zfs:(&tq 0xd15182b4 [zil_clean]
  302     0     0     0  SL      zfs:(&tq 0xd1518380 [zil_clean]
  301     0     0     0  SL      zfs:(&tq 0xd151844c [zil_clean]
  300     0     0     0  SL      zfs:(&tq 0xd1518518 [zil_clean]
  299     0     0     0  SL      zfs:(&tq 0xd15185e4 [zil_clean]
  298     0     0     0  SL      zfs:(&tq 0xd15186b0 [zil_clean]
  297     0     0     0  SL      zfs:(&tq 0xd0ea7c44 [zil_clean]
  296     0     0     0  SL      zfs:(&tq 0xd0ea7d10 [zil_clean]
  295     0     0     0  SL      zfs:(&tq 0xd0ea7ddc [zil_clean]
  294     0     0     0  SL      zfs:(&tq 0xd0ea7ea8 [zil_clean]
  293     0     0     0  SL      zfs:(&tq 0xd0ea711c [zil_clean]
  292     0     0     0  SL      zfs:(&tq 0xd10b9050 [zil_clean]
  291     0     0     0  SL      zfs:(&tq 0xd10b911c [zil_clean]
  290     0     0     0  SL      zfs:(&tq 0xd0ea71e8 [zil_clean]
  289     0     0     0  SL      zfs:(&tq 0xd0ea72b4 [zil_clean]
  288     0     0     0  SL      zfs:(&tq 0xd10b91e8 [zil_clean]
  287     0     0     0  SL      zfs:(&tq 0xd0ea7380 [zil_clean]
  286     0     0     0  SL      zfs:(&tq 0xd10b92b4 [zil_clean]
  285     0     0     0  SL      zfs:(&tq 0xd0ea744c [zil_clean]
  284     0     0     0  SL      zfs:(&tq 0xd10b9380 [zil_clean]
  283     0     0     0  SL      zfs:(&tq 0xd0ea7518 [zil_clean]
  282     0     0     0  SL      zfs:(&tq 0xd0ea75e4 [zil_clean]
  281     0     0     0  SL      zfs:(&tq 0xd10b944c [zil_clean]
  280     0     0     0  SL      zfs:(&tq 0xd10b9518 [zil_clean]
  279     0     0     0  SL      zfs:(&tq 0xd10b95e4 [zil_clean]
  278     0     0     0  SL      zfs:(&tq 0xd10b96b0 [zil_clean]
  277     0     0     0  SL      zfs:(&tq 0xd10b977c [zil_clean]
  276     0     0     0  SL      zfs:(&tq 0xd10b9848 [zil_clean]
  275     0     0     0  SL      zfs:(&tq 0xd0ea76b0 [zil_clean]
  274     0     0     0  SL      zfs:(&tq 0xd10b9914 [zil_clean]
  273     0     0     0  SL      zfs:(&tq 0xd10b99e0 [zil_clean]
  272     0     0     0  SL      zfs:(&tq 0xd10b9aac [zil_clean]
  271     0     0     0  SL      zfs:(&tq 0xd10b9b78 [zil_clean]
  270     0     0     0  SL      zfs:(&tq 0xd10b9c44 [zil_clean]
  269     0     0     0  SL      zfs:(&tq 0xd10b9d10 [zil_clean]
  268     0     0     0  SL      zfs:(&tq 0xd10b9ddc [zil_clean]
  267     0     0     0  SL      zfs:(&tq 0xd10b9ea8 [zil_clean]
  266     0     0     0  SL      zfs:(&tq 0xd0ea777c [zil_clean]
  265     0     0     0  SL      zfs:(&tq 0xd10ba050 [zil_clean]
  264     0     0     0  SL      zfs:(&tq 0xd10ba11c [zil_clean]
  263     0     0     0  SL      zfs:(&tq 0xd0ea7848 [zil_clean]
  262     0     0     0  SL      zfs:(&tq 0xd08c7518 [zil_clean]
  261     0     0     0  SL      zfs:(&tq 0xd08c75e4 [zil_clean]
  260     0     0     0  SL      zfs:(&tq 0xd08c76b0 [zil_clean]
  259     0     0     0  SL      zfs:(&tq 0xd08c777c [zil_clean]
  258     0     0     0  SL      zfs:(&tq 0xd08c7848 [zil_clean]
  257     0     0     0  SL      zfs:(&tq 0xd08c7914 [zil_clean]
  256     0     0     0  SL      zfs:(&tq 0xd08c79e0 [zil_clean]
  255     0     0     0  SL      zfs:(&tq 0xd08c7aac [zil_clean]
  254     0     0     0  SL      zfs:(&tq 0xd0ea7914 [zil_clean]
  253     0     0     0  SL      zfs:(&tq 0xd0ea79e0 [zil_clean]
  252     0     0     0  SL      zfs:(&tq 0xd08c7b78 [zil_clean]
  251     0     0     0  SL      zfs:(&tq 0xd08c7c44 [zil_clean]
  250     0     0     0  SL      zfs:(&tq 0xd08c7d10 [zil_clean]
  249     0     0     0  SL      zfs:(&tq 0xd0ea7aac [zil_clean]
  248     0     0     0  SL      zfs:(&tq 0xd08c7ddc [zil_clean]
  247     0     0     0  SL      zfs:(&tq 0xd0ea7b78 [zil_clean]
  246     0     0     0  SL      zfs:(&tq 0xd08c7ea8 [zil_clean]
  245     0     0     0  SL      zfs:(&tq 0xc75d3ddc [zil_clean]
  244     0     0     0  SL      zfs:(&tq 0xd0c77050 [zil_clean]
  243     0     0     0  SL      zfs:(&tq 0xd0c7711c [zil_clean]
  242     0     0     0  SL      zfs:(&tq 0xc75d3d10 [zil_clean]
  241     0     0     0  SL      zfs:(&tq 0xd0c771e8 [zil_clean]
  240     0     0     0  SL      zfs:(&tq 0xc75d3b78 [zil_clean]
  239     0     0     0  SL      zfs:(&tq 0xc75d3aac [zil_clean]
  238     0     0     0  SL      zfs:(&tq 0xd0c772b4 [zil_clean]
  237     0     0     0  SL      zfs:(&tq 0xd0c77380 [zil_clean]
  236     0     0     0  SL      zfs:(&tq 0xd0c7744c [zil_clean]
  235     0     0     0  SL      zfs:(&tq 0xc75d39e0 [zil_clean]
  234     0     0     0  SL      zfs:(&tq 0xd0c77518 [zil_clean]
  233     0     0     0  SL      zfs:(&tq 0xd0c775e4 [zil_clean]
  232     0     0     0  SL      zfs:(&tq 0xc75d3914 [zil_clean]
  231     0     0     0  SL      zfs:(&tq 0xd0c776b0 [zil_clean]
  230     0     0     0  SL      zfs:(&tq 0xc75d3848 [zil_clean]
  229     0     0     0  SL      zfs:(&tq 0xd0c7777c [zil_clean]
  228     0     0     0  SL      zfs:(&tq 0xc7a6c2b4 [zil_clean]
  227     0     0     0  SL      zfs:(&tq 0xc75d36b0 [zil_clean]
  226     0     0     0  SL      zfs:(&tq 0xc75d35e4 [zil_clean]
  225     0     0     0  SL      zfs:(&tq 0xc7a6c380 [zil_clean]
  224     0     0     0  SL      zfs:(&tq 0xc75d411c [zil_clean]
  223     0     0     0  SL      zfs:(&tq 0xc7a6c44c [zil_clean]
  222     0     0     0  SL      zfs:(&tq 0xc7a6c518 [zil_clean]
  221     0     0     0  SL      zfs:(&tq 0xc7a6c5e4 [zil_clean]
  220     0     0     0  SL      zfs:(&tq 0xc7a6c6b0 [zil_clean]
  219     0     0     0  SL      zfs:(&tq 0xc75d41e8 [zil_clean]
  218     0     0     0  SL      zfs:(&tq 0xc7a6c77c [zil_clean]
  217     0     0     0  SL      zfs:(&tq 0xc7a6c848 [zil_clean]
  216     0     0     0  SL      zfs:(&tq 0xc75d42b4 [zil_clean]
  215     0     0     0  SL      zfs:(&tq 0xc7a6c914 [zil_clean]
  214     0     0     0  SL      zfs:(&tq 0xc7a6c9e0 [zil_clean]
  213     0     0     0  SL      zfs:(&tq 0xc75d4380 [zil_clean]
  212     0     0     0  SL      zfs:(&tq 0xc7a6caac [zil_clean]
  211     0     0     0  SL      zfs:(&tq 0xc75d444c [zil_clean]
  210     0     0     0  SL      zfs:(&tq 0xc7a6cb78 [zil_clean]
  209     0     0     0  SL      zfs:(&tq 0xc7a6cc44 [zil_clean]
  208     0     0     0  SL      zfs:(&tq 0xc7a6cd10 [zil_clean]
  207     0     0     0  SL      zfs:(&tq 0xc75d4518 [zil_clean]
  206     0     0     0  SL      zfs:(&tq 0xc7a6cddc [zil_clean]
  205     0     0     0  SL      zfs:(&tq 0xc7a6cea8 [zil_clean]
  204     0     0     0  SL      zfs:(&tq 0xd08c7050 [zil_clean]
  203     0     0     0  SL      zfs:(&tq 0xd08c711c [zil_clean]
  202     0     0     0  SL      zfs:(&tq 0xd08c71e8 [zil_clean]
  201     0     0     0  SL      zfs:(&tq 0xd08c72b4 [zil_clean]
  200     0     0     0  SL      zfs:(&tq 0xd08c7380 [zil_clean]
  199     0     0     0  SL      zfs:(&tq 0xd08c744c [zil_clean]
  198     0     0     0  SL      zfs:(&tq 0xc75d3ea8 [zil_clean]
  197     0     0     0  SL      zfs:(&tq 0xc75d3c44 [zil_clean]
  196     0     0     0  SL      zfs:(&tq 0xc75d377c [zil_clean]
  195     0     0     0  SL      zfs:(&tq 0xc75d3050 [zil_clean]
  194     0     0     0  SL      zfs:(&tq 0xc75d311c [zil_clean]
  193     0     0     0  SL      zfs:(&tq 0xc75d31e8 [zil_clean]
  192     0     0     0  SL      zfs:(&tq 0xc75d45e4 [zil_clean]
  191     0     0     0  SL      zfs:(&tq 0xc75d46b0 [zil_clean]
  189     0     0     0  SL      zfs:(&tq 0xc75d477c [zil_clean]
  188     0     0     0  SL      zfs:(&tq 0xc75d32b4 [zil_clean]
  187     0     0     0  SL      zfs:(&tq 0xc75d3380 [zil_clean]
  186     0     0     0  SL      zfs:(&tq 0xc75d344c [zil_clean]
  185     0     0     0  SL      zfs:(&tq 0xc75d3518 [zil_clean]
  183     0     0     0  SL      zfs:(&tx 0xc73e6f2c [txg_thread_enter]
  182     0     0     0  SL      zfs:(&tx 0xc73e6f0c [txg_thread_enter]
  181     0     0     0  RL                          [txg_thread_enter]
  180     0     0     0  SL      vgeom:io 0xc73d8a08 [vdev:worker da11]
  179     0     0     0  SL      vgeom:io 0xc73d89c8 [vdev:worker da10]
  178     0     0     0  SL      vgeom:io 0xc73d8948 [vdev:worker da9]
  177     0     0     0  SL      vgeom:io 0xc7a1c748 [vdev:worker da8]
  176     0     0     0  SL      vgeom:io 0xc73d8888 [vdev:worker da7]
  175     0     0     0  SL      vgeom:io 0xc73d8848 [vdev:worker da6]
  174     0     0     0  SL      vgeom:io 0xc73d7848 [vdev:worker da5]
  175     0     0     0  SL      vgeom:io 0xc73d8848 [vdev:worker da6]
  174     0     0     0  SL      vgeom:io 0xc73d7848 [vdev:worker da5]
  173     0     0     0  SL      vgeom:io 0xc73d8788 [vdev:worker da4]
  172     0     0     0  SL      vgeom:io 0xc73d8708 [vdev:worker da3]
  171     0     0     0  SL      vgeom:io 0xc7a1c9c8 [vdev:worker da2]
  170     0     0     0  SL      vgeom:io 0xc7a1cdc8 [vdev:worker da1]
  169     0     0     0  SL      vgeom:io 0xc7a1cbc8 [vdev:worker da0]
  168     0     0     0  SL      zfs:(&tq 0xc75d4848 [spa_zio_intr_5]
  167     0     0     0  SL      zfs:(&tq 0xc75d4848 [spa_zio_intr_5]
  166     0     0     0  SL      zfs:(&tq 0xc75d4914 [spa_zio_issue_5]
  165     0     0     0  SL      zfs:(&tq 0xc75d4914 [spa_zio_issue_5]
  164     0     0     0  SL      zfs:(&tq 0xc75d49e0 [spa_zio_intr_4]
  163     0     0     0  SL      zfs:(&tq 0xc75d49e0 [spa_zio_intr_4]
  162     0     0     0  SL      zfs:(&tq 0xc75d4aac [spa_zio_issue_4]
  161     0     0     0  SL      zfs:(&tq 0xc75d4aac [spa_zio_issue_4]
  160     0     0     0  SL      zfs:(&tq 0xc75d4b78 [spa_zio_intr_3]
  159     0     0     0  SL      zfs:(&tq 0xc75d4b78 [spa_zio_intr_3]
  158     0     0     0  SL      zfs:(&tq 0xc75d4c44 [spa_zio_issue_3]
  157     0     0     0  SL      zfs:(&tq 0xc75d4c44 [spa_zio_issue_3]
  156     0     0     0  SL      zfs:(&tq 0xc75d4d10 [spa_zio_intr_2]
  155     0     0     0  SL      zfs:(&tq 0xc75d4d10 [spa_zio_intr_2]
  154     0     0     0  SL      zfs:(&tq 0xc75d4ddc [spa_zio_issue_2]
 153     0     0     0  SL      zfs:(&tq 0xc75d4ddc [spa_zio_issue_2]
  152     0     0     0  SL      zfs:(&tq 0xc75d4ea8 [spa_zio_intr_1]
  151     0     0     0  SL      zfs:(&tq 0xc75d4ea8 [spa_zio_intr_1]
  150     0     0     0  SL      zfs:(&tq 0xc7a6c050 [spa_zio_issue_1]
  149     0     0     0  SL      zfs:(&tq 0xc7a6c050 [spa_zio_issue_1]
  148     0     0     0  SL      zfs:(&tq 0xc7a6c11c [spa_zio_intr_0]
  147     0     0     0  SL      zfs:(&tq 0xc7a6c11c [spa_zio_intr_0]
  146     0     0     0  SL      zfs:(&tq 0xc7a6c1e8 [spa_zio_issue_0]
  145     0     0     0  SL      zfs:(&tq 0xc7a6c1e8 [spa_zio_issue_0]
  108     0     0     0  SL      zfs:(&ar 0xc759440c [arc_reclaim_thread]
  105     0     0     0  SL      zfs:(&tq 0xc75d4050 [system_taskq]
  104     0     0     0  SL      zfs:(&tq 0xc75d4050 [system_taskq]
   43     0     0     0  SL      -        0xc07d0614 [schedcpu]
   42     0     0     0  SL      sdflush  0xc07dbb24 [softdepflush]
   41     0     0     0  SL      syncer   0xc07d060c [syncer]
   40     0     0     0  SL      vlruwt   0xc71c7000 [vnlru]
   39     0     0     0  SL      psleep   0xc07d6d44 [bufdaemon]
   38     0     0     0  SL      pgzero   0xc07dc5d8 [pagezero]
   37     0     0     0  SL      psleep   0xc07dc2f4 [vmdaemon]
   36     0     0     0  SL      psleep   0xc07dc2bc [pagedaemon]
   35     0     0     0  SL      pftm     0xc047ab60 [pfpurge]
   34     0     0     0  WL                          [irq12: psm0]
   33     0     0     0  WL                          [irq1: atkbd0]
   32     0     0     0  LL     *Giant    0xc7088550 [swi0: sio]
   31     0     0     0  SL      -        0xc715043c [fdc0]
   30     0     0     0  WL                          [irq25: bge1]
   29     0     0     0  WL                          [irq24: bge0]
   28     0     0     0  WL                          [irq18: ips0]
   27     0     0     0  SL      idle     0xc71962e4 [mpt_raid0]
   26     0     0     0  SL      idle     0xc7196000 [mpt_recovery0]
   25     0     0     0  WL                          [irq22: mpt0]
   24     0     0     0  WL                          [irq15: ata1]
   23     0     0     0  WL                          [irq14: ata0]
   22     0     0     0  WL                          [irq7: acpi0]
   21     0     0     0  WL                          [swi2: cambio]
   20     0     0     0  SL      ccb_scan 0xc07b82f4 [xpt_thrd]
   19     0     0     0  WL                          [swi6: task queue]
   18     0     0     0  WL                          [swi6: Giant taskq]
    9     0     0     0  SL      -        0xc70e4e80 [thread taskq]
   17     0     0     0  WL                          [swi5: +]
    8     0     0     0  SL      -        0xc70fb100 [acpi_task_2]
    7     0     0     0  SL      -        0xc70fb100 [acpi_task_1]
    6     0     0     0  SL      -        0xc70fb100 [acpi_task_0]
    5     0     0     0  SL      -        0xc70fb200 [kqueue taskq]
   16     0     0     0  SL      -        0xc07d0614 [yarrow]
    4     0     0     0  SL      -        0xc07ce6ec [g_down]
    3     0     0     0  SL      -        0xc07ce6e8 [g_up]
    2     0     0     0  SL      -        0xc07ce6e0 [g_event]
   15     0     0     0  WL                          [swi1: net]
   14     0     0     0  WL                          [swi3: vm]
   13     0     0     0  RL      CPU 1               [swi4: clock sio]
   12     0     0     0  RL                          [idle: cpu0]
   11     0     0     0  RL                          [idle: cpu1]
    1     0     1     0  SLs     wait     0xc709dab0 [init]
   10     0     0     0  SL      audit_wo 0xc07db588 [audit]
    0     0     0     0  WLs                         [swapper]

db> show proc 14870
Process 14870 (rsync) at 0xd26c3ab0:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 14866 at 0xd1c2bab0
 ABI: FreeBSD ELF32
 arguments: /usr/local/bin/rsync
 threads: 1
100379                   D       zfs:(&tx 0xc73e6f24 rsync
db> thread 100379
[thread pid 14870 tid 100379 ]
sched_switch+0x19d:     testb   $0x20,0x68(%esi)
db> bt
Tracing pid 14870 tid 100379 td 0xd26c4e00
sched_switch(d26c4e00,0,1,181d9d4,8567e45,...) at sched_switch+0x19d
mi_switch(1,0,d26c4e00,fc81da0c,c05b08c6,...) at mi_switch+0x146
sleepq_switch(d26c4e00,0,c07640c2,21b,0,...) at sleepq_switch+0x7f
sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36
_cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c
txg_wait_open(c73e6e00,4b65728e,0,d5,c88e3bfc,...) at txg_wait_open+0xd5
dmu_tx_wait(d0fe6800,2,0,0,0,...) at dmu_tx_wait+0xe0
zfs_freebsd_rename(fc81dba4,fc81db7c,fc81dbc0,fc81dc0c,fc81dc68,...) at zfs_freebsd_rename+0x41e
VOP_RENAME_APV(c7592e00,fc81dba4,101,c05ee4d5,5009c10,...) at VOP_RENAME_APV+0x46
kern_rename(d26c4e00,bfbfd0c8,bfbfd8c8,0,fc81dd2c,...) at kern_rename+0x2e1
rename(d26c4e00,fc81dcfc,8,0,5a0a382,...) at rename+0x29
syscall(fc81dd38) at syscall+0x305
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcbbc, ebp = 0xbfbfcbc8 ---

db> show proc 14933
Process 14933 (rsync) at 0xd1f752ac:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 14931 at 0xd1f6bab0
 ABI: FreeBSD ELF32
 arguments: /usr/local/bin/rsync
 threads: 1
100321                   RunQ                        rsync
db> thread 100321
[thread pid 14933 tid 100321 ]
sched_switch+0x19d:     testb   $0x20,0x68(%esi)
db> bt
Tracing pid 14933 tid 100321 td 0xd1f72200
sched_switch(d1f72200,0,1,173a9d4,8567e2f,...) at sched_switch+0x19d
mi_switch(1,0,d1f72200,fc73aa0c,c05b08c6,...) at mi_switch+0x146
sleepq_switch(d1f72200,0,c07640c2,21b,0,...) at sleepq_switch+0x7f
sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36
_cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c
txg_wait_open(c73e6e00,4b657007,0,d5,d91633b0,...) at txg_wait_open+0xd5
dmu_tx_wait(d160bd00,2,0,0,0,...) at dmu_tx_wait+0xe0
zfs_freebsd_rename(fc73aba4,fc73ab7c,fc73abc0,fc73ac0c,fc73ac68,...) at zfs_freebsd_rename+0x41e
VOP_RENAME_APV(c7592e00,fc73aba4,101,119,5009c10,...) at VOP_RENAME_APV+0x46
kern_rename(d1f72200,bfbfd0c8,bfbfd8c8,0,fc73ad2c,...) at kern_rename+0x2e1
rename(d1f72200,fc73acfc,8,d1f72200,fc73ad2c,...) at rename+0x29
syscall(fc73ad38) at syscall+0x305
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcbbc, ebp = 0xbfbfcbc8 ---

db> show proc 12458
Process 12458 (rsync) at 0xd1e1c804:
 state: NORMAL
 uid: 0  gids: 0, 0, 5
 parent: pid 12457 at 0xd1c2dab0
 ABI: FreeBSD ELF32
 arguments: /usr/local/bin/rsync
 threads: 1
100293                   RunQ                        rsync
db> thread 100293
[thread pid 12458 tid 100293 ]
sched_switch+0x19d:     testb   $0x20,0x68(%esi)
db> bt
Tracing pid 12458 tid 100293 td 0xd18c5a00
sched_switch(d18c5a00,0,1,15d29d4,858acc6,...) at sched_switch+0x19d
mi_switch(1,0,d18c5a00,fc5d2a0c,c05b08c6,...) at mi_switch+0x146
sleepq_switch(d18c5a00,0,c07640c2,21b,0,...) at sleepq_switch+0x7f
sleepq_wait(c73e6f24,c73e6eac,c758ddf9,1,0,...) at sleepq_wait+0x36
_cv_wait(c73e6f24,c73e6eac,c758dce6,19e,c73e6f1c,...) at _cv_wait+0x13c
txg_wait_open(c73e6e00,4b7d2f7f,0,d5,d26010ec,...) at txg_wait_open+0xd5
dmu_tx_wait(c987f980,2,0,0,0,...) at dmu_tx_wait+0xe0
zfs_freebsd_rename(fc5d2ba4,fc5d2b7c,fc5d2bc0,fc5d2c0c,fc5d2c68,...) at zfs_freebsd_rename+0x41e
VOP_RENAME_APV(c7592e00,fc5d2ba4,101,c8c63c08,5009c10,...) at VOP_RENAME_APV+0x46
kern_rename(d18c5a00,bfbfd0a8,bfbfd8a8,0,fc5d2d2c,...) at kern_rename+0x2e1
rename(d18c5a00,fc5d2cfc,8,c714b400,7df1b67,...) at rename+0x29
syscall(fc5d2d38) at syscall+0x305
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (128, FreeBSD ELF32, rename), eip = 0x28103ccb, esp = 0xbfbfcb9c, ebp = 0xbfbfcba8 ---

/Kenneth
Received on Fri Sep 07 2007 - 04:31:15 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:39:17 UTC