On Mon, 2004-05-31 at 14:22 +1000, Tim Robbins wrote: > I'm not sure where to go from here. A deadlock doesn't seem likely, but > it's possible that background fsck could lock up the system for quite > some time by using this sysctl. How long did you wait before dropping to > ddb (approximately)? Not sure, probably my problem is related or same (I have posted the message to current_at_ before): -- About 3-5 time per day recent current lockup on my notebook. Symptoms: system responsive, not locked processes works, I can switch X to sc0 and back. processes locked on 'sysctl lock' (see below) allways there is one process sleeping on allproc usually it is related with high activity (this case evolution gets bunch of night mail and pass it through spam-assassin) I do not know why proc 1037 have such strange backtrace but suspect that evolution was forked before exec spamc (or like). proc 658 was created after lock (I switched on free console and type login) - login was executed but even does not ask password. May be it is same issue as reported there about lockup of top and ps. Unfortunately there was no comconsole, so I have retype most important info: db> ps 1037 ... 0000000 new [INACTIVE] evolution-1.5 ... 790 ... 008c001 stop(threaded) evolution-1.5 thread ... [SUSP] thread ... [SLPQ spltwt ...][SLP] thread ... [SUSP] thread ... [SLPQ select ...][SLP][SUSP] thread ... [SUSP] 781 ... 0004001 [SLPQ sysctl lock ...][SLP] wireless-applet ... 731 ... 0004000 [SLPQ sysctl lock ...][SLP] metacity ... 477 ... 0000000 [SLPQ allproc ...][SLP] cron ... 658 ... 0004002 [SLPQ sysctl lock ...][SLP] login ... db> tr 1037 sched_switch(0,...) (null)(1,..) at 0 _end(1,...) _end(1,...) ... a lot of _end(), different args on stack _end(1,...) _end() at ... db> tr 781 sched_switch() mi_switch() sleepq_switch() sleepq_wait() cv_wait() _sx_xlock() userland_sysctl() __syscall() ... db> tr 731 sched_switch() mi_switch() sleepq_switch() sleepq_wait() cv_wait() _sx_xlock() userland_sysctl() __syscall() ... db> tr 477 sched_switch() mi_switch() sleepq_switch() sleepq_wait() cv_wait() _sx_xlock() fork1() fork() syscall() ... db> tr 658 sched_switch() mi_switch() sleepq_switch() sleepq_wait() cv_wait() _sx_xlock() userland_sysctl() __syscall() ... Once again, with serial console this time. Symptoms are the same. Processes locket on 'sysctl lock' and 'allproc'. DDB output: db> c Debugger("manual escape to debugger") Stopped at Debugger+0x54: xchgl %ebx,in_Debugger.0 db> ps pid proc uarea uid ppid pgrp flag stat wmesg wchan cmd 2420 c296cdc0 cc929000 207 829 757 0000000 new [INACTIVE] evolution-1.5 2394 c29fb370 cc94d000 207 820 820 0006002 [SLPQ allproc 0xc0693cac][SLP] sleep 2372 c296c6e0 cc925000 207 2371 2357 0004000 [SLPQ select 0xc0698144][SLP] ssh 2371 c2962c08 cc8d7000 207 2359 2357 0004000 [SLPQ sbwait 0xc29bc51c][SLP] vzcpcon 2370 c296c1b8 cc8da000 207 2369 2357 0004000 [SLPQ select 0xc0698144][SLP] ssh 2369 c2962a50 cc8d6000 207 2360 2357 0004000 [SLPQ sbwait 0xc29bc7ec][SLP] vzcpcon 2368 c2962370 cc8d2000 207 2357 2357 0000100 [SLPQ lockf 0xc29ddcc0] [SLP] httpd 2367 c285a6e0 cc81f000 207 2357 2357 0000100 [SLPQ lockf 0xc29dd840] [SLP] httpd 2366 c2939a50 cc890000 207 2357 2357 0000100 [SLPQ lockf 0xc29e1c00] [SLP] httpd 2365 c2939370 cc88c000 207 2363 2357 0004000 [SLPQ select 0xc0698144][SLP] ssh 2364 c2939898 cc88f000 207 2357 2357 0000100 [SLPQ select 0xc0698144][SLP] httpd 2363 c285a898 cc820000 207 2361 2357 0004000 [SLPQ sbwait 0xc29bbc9c][SLP] vzcpcon 2362 c2939dc0 cc892000 207 2357 2357 0000100 [SLPQ lockf 0xc29e4dc0] [SLP] httpd 2361 c29391b8 cc887000 207 2357 2357 0000100 [SLPQ lockf 0xc29e0580] [SLP] httpd 2360 c29621b8 cc8d1000 207 2357 2357 0000100 [SLPQ lockf 0xc29e12c0] [SLP] httpd 2359 c2939c08 cc891000 207 2357 2357 0000100 [SLPQ lockf 0xc29e47c0] [SLP] httpd 2358 c296c528 cc924000 207 2357 2357 0000100 [SLPQ lockf 0xc29dd9c0] [SLP] httpd 2357 c2357898 cc7a6000 0 1 2357 0000000 [SLPQ select 0xc0698144][SLP] httpd 2351 c2962000 cc8d0000 0 2343 2351 0004002 [SLPQ ttyin 0xc2882a40] [SLP] tcsh 2349 c23596e0 cc7d2000 207 2340 2349 0004002 [SLPQ select 0xc0698144][SLP] gnuserv 2343 c285ac08 cc822000 207 2342 2343 0004002 [SLPQ pause 0xc281f000] [SLP] tcsh 2342 c296c370 cc923000 207 1 757 0004100 [SLPQ select 0xc0698144][SLP] xterm 2340 c2962dc0 cc8d8000 207 2338 2338 0004002 [SLPQ select 0xc0698144][SLP] xemacs-21.5-b15 2338 c285a1b8 cc81c000 207 2335 2338 0004002 [SLPQ wait 0xc285a1b8] [SLP] sh 2335 c2858dc0 cc7f6000 207 2334 2335 0004002 [SLPQ ttyin 0xc296a240] [SLP] tcsh 2334 c296c000 cc8d9000 207 1 757 0004100 [SLPQ select 0xc0698144][SLP] xterm 1826 c285adc0 cc823000 207 1772 757 0000000 [SLPQ allproc 0xc0693cac][SLP] opera 1777 c29626e0 cc8d4000 207 1776 757 0004001 [SLPQ select 0xc0698144][SLP] sim 1776 c2962528 cc8d3000 207 1 757 0004000 [SLPQ pause 0xc29a5000] [SLP] tcsh 1772 c296c898 cc926000 207 1 757 0004001 [SLPQ pause 0xc27c9000] [SLP] opera 854 c285a370 cc81d000 207 1 854 0000000 [SLPQ select 0xc0698144][SLP] perl 839 c2939528 cc88d000 207 829 757 0004100 [SLPQ select 0xc0698144][SLP] camel-lock-helper 838 c29396e0 cc88e000 207 1 723 000c001 (threaded) evolution- alarm-not thread 0xc2985690 ksegrp 0xc2358c00 [SLPQ kserel 0xc2358c5c][SLP] thread 0xc235a930 ksegrp 0xc2358c00 [SLPQ select 0xc0698144][SLP] thread 0xc28e6bd0 ksegrp 0xc2358c00 [SLPQ select 0xc0698144][SLP] thread 0xc224e7e0 ksegrp 0xc2358900 [SLPQ ksesigwait 0xc29397e0][SLP] 833 c28586e0 cc7f2000 207 1 723 000c001 (threaded) evolution- data-serv thread 0xc295a7e0 ksegrp 0xc212c600 [SLPQ kserel 0xc212c65c][SLP] thread 0xc295a150 ksegrp 0xc212c600 [SLPQ select 0xc0698144][SLP] thread 0xc28e6000 ksegrp 0xc212c600 [SLPQ select 0xc0698144][SLP] thread 0xc28e6150 ksegrp 0xc2358980 [SLPQ ksesigwait 0xc28587e0][SLP] 829 c285aa50 cc821000 207 1 757 008c001 stop(threaded) evolution-1.5 thread 0xc295ae70 ksegrp 0xc285f180 [SUSP] thread 0xc28e6d20 ksegrp 0xc285f180 [SLPQ spltwt 0xc0e87678][SLP] thread 0xc29fd540 ksegrp 0xc285f180 [SUSP] thread 0xc28e67e0 ksegrp 0xc285f180 [SLPQ select 0xc0698144][SLP] [SUSP] thread 0xc295aa80 ksegrp 0xc2358880 [SUSP] 820 c2939000 cc82b000 207 802 820 0004002 [SLPQ wait 0xc2939000] [SLP] sh 802 c2357a50 cc7a7000 207 801 802 0004002 [SLPQ pause 0xc2326000] [SLP] tcsh 801 c2359528 cc7d1000 207 1 757 0004100 [SLPQ select 0xc0698144][SLP] xterm 799 c218fdc0 cbba2000 207 1 723 0004001 [SLPQ select 0xc0698144][SLP] notification-area-a 797 c2858370 cc7f0000 207 1 723 0004001 [SLPQ select 0xc0698144][SLP] clock-applet 795 c2858528 cc7f1000 207 1 723 0004001 [SLPQ select 0xc0698144][SLP] mixer_applet2 793 c28581b8 cc7ef000 207 1 723 0004001 [SLPQ select 0xc0698144][SLP] wnck-applet 789 c2357c08 cc7a8000 207 1 759 0004000 [SLPQ select 0xc0698144][SLP] mapping-daemon 771 c2359dc0 cc7d6000 207 1 723 000c001 (threaded) gnome-vfs- daemon thread 0xc2985930 ksegrp 0xc212c880 [SLPQ kserel 0xc212c8dc][SLP] thread 0xc29fd150 ksegrp 0xc212c880 [SLPQ select 0xc0698144][SLP] thread 0xc224ee70 ksegrp 0xc212c880 [SLPQ select 0xc0698144][SLP] thread 0xc235ad20 ksegrp 0xc2358f00 [SLPQ ksesigwait 0xc2359ec0][SLP] 767 c2359a50 cc7d4000 207 1 767 000c101 (threaded) gkrellm thread 0xc29fd690 ksegrp 0xc212c980 [SLPQ kserel 0xc212c9dc][SLP] thread 0xc29fdd20 ksegrp 0xc212c980 [SLPQ thrd_sleep 0xc2510b4c][SLP] thread 0xc235ae70 ksegrp 0xc2358e80 [SLPQ ksesigwait 0xc2359b50][SLP] 759 c2858000 cc7ea000 207 1 759 000c001 (threaded) nautilus thread 0xc29fd000 ksegrp 0xc212c800 [SLPQ kserel 0xc212c85c][SLP] thread 0xc28e53f0 ksegrp 0xc212c800 [SLPQ select 0xc0698144][SLP] thread 0xc28e5d20 ksegrp 0xc212c800 [SLPQ select 0xc0698144][SLP] thread 0xc285b000 ksegrp 0xc285f000 [SLPQ ksesigwait 0xc2858100][SLP] 757 c2858a50 cc7f4000 207 1 757 0004001 [SLPQ select 0xc0698144][SLP] gnome-panel 755 c224ca50 cc784000 207 1 755 0004000 [SLPQ sysctl lock 0xc069466c][SLP] metacity 740 c2359c08 cc7d5000 207 1 723 0004100 [SLPQ select 0xc0698144][SLP] xscreensaver 727 c2858898 cc7f3000 207 1 723 0004001 [SLPQ select 0xc0698144][SLP] gnome-settings-daem 725 c2858c08 cc7f5000 207 1 725 0004000 [SLPQ select 0xc0698144][SLP] gnome-smproxy 723 c285a000 cc7f7000 207 1 723 0004000 [SLPQ select 0xc0698144][SLP] bonobo-activation-s 721 c224c1b8 cc737000 207 1 696 0000000 [SLPQ select 0xc0698144][SLP] gnome-keyring-daemo 714 c224cc08 cc785000 207 1 696 0004001 [SLPQ select 0xc0698144][SLP] gconfd-2 712 c2357dc0 cc7a9000 207 696 712 0000000 [SLPQ select 0xc0698144][SLP] ssh-agent 696 c2357528 cc7a4000 207 608 696 0004001 [SLPQ select 0xc0698144][SLP] gnome-session 693 c212aa50 cbb64000 0 1 693 0004002 [SLPQ sysctl lock 0xc069466c][SLP] login 669 c23576e0 cc7a5000 0 1 669 0004002 [SLPQ ttyin 0xc25fba10] [SLP] getty 668 c218f1b8 cbb9b000 0 1 668 0004002 [SLPQ ttyin 0xc25fc010] [SLP] getty 667 c2359370 cc7ac000 0 1 667 0004002 [SLPQ ttyin 0xc2171010] [SLP] getty 666 c224c528 cc739000 0 1 666 0004002 [SLPQ ttyin 0xc21a7e10] [SLP] getty 665 c23571b8 cc7a2000 0 1 665 0004002 [SLPQ ttyin 0xc2171210] [SLP] getty 664 c2359000 cc7aa000 0 1 664 0004002 [SLPQ ttyin 0xc21a7810] [SLP] getty 663 c224cdc0 cc786000 0 1 663 0004002 [SLPQ ttyin 0xc21a7410] [SLP] getty 651 c218f370 cbb9c000 26 1 651 0000100 [SLPQ select 0xc0698144][SLP] exim-4.30-1 627 c2357370 cc7a3000 0 1 627 0000000 [SLPQ select 0xc0698144][SLP] moused 611 c224c898 cc783000 0 608 611 0004000 [SLPQ select 0xc0698144][SLP] XFree86 608 c218fa50 cbba0000 0 605 608 0000100 [SLPQ select 0xc0698144][SLP] gdm-binary 605 c218fc08 cbba1000 0 1 605 0000100 [SLPQ select 0xc0698144][SLP] gdm-binary 563 c2357000 cc7a1000 0 539 539 0004100 [SLPQ nanslp 0xc069481c][SLP] ipp 539 c23591b8 cc7ab000 0 1 539 0000000 [SLPQ select 0xc0698144][SLP] cupsd 457 c218f6e0 cbb9e000 0 1 457 0000100 [SLPQ select 0xc0698144][SLP] sshd 404 c224c6e0 cc73a000 0 1 404 0000000 [SLPQ select 0xc0698144][SLP] usbd 401 c218f528 cbb9d000 0 1 401 0000000 [SLPQ select 0xc0698144][SLP] moused 303 c218f898 cbb9f000 0 1 303 0000000 [SLPQ select 0xc0698144][SLP] syslogd 281 c224c370 cc738000 0 1 281 0000000 [SLPQ dev cv 0xc069522c][SLP] devd 186 c224c000 cc736000 0 1 186 0000000 [SLPQ pause 0xc226d000] [SLP] adjkerntz 45 c212ac08 cbb65000 0 0 0 0000204 [SLPQ vlruwt 0xc212ac08][SLP] vnlru 44 c212adc0 cbb66000 0 0 0 0000204 [SLPQ syncer 0xc06941e0][SLP] syncer 43 c218c000 cbb6d000 0 0 0 0000204 [SLPQ psleep 0xc06984cc][SLP] bufdaemon 42 c218c1b8 cbb6e000 0 0 0 000020c [SLPQ pgzero 0xc069ca28][SLP] pagezero 41 c218c370 cbb6f000 0 0 0 0000204 [SLPQ psleep 0xc069ca80][SLP] vmdaemon 40 c218c528 cbb70000 0 0 0 0000204 [SLPQ psleep 0xc069ca6c][SLP] pagedaemon 39 c218c6e0 cbb71000 0 0 0 0000204 [IWAIT] swi0: tty:sio 9 c218c898 cbb72000 0 0 0 0000204 [SLPQ cbb cv 0xc2172354][SLP] cbb0 38 c218ca50 cbb73000 0 0 0 0000204 [SLPQ usbtsk 0xc07b1c80][SLP] usbtask 37 c218cc08 cbb98000 0 0 0 0000204 [SLPQ usbevt 0xc21a3210][SLP] usb0 36 c218cdc0 cbb99000 0 0 0 0000204 [SLPQ tzpoll 0xc068bea4][SLP] acpi_thermal 8 c218f000 cbb9a000 0 0 0 0000204 [SLPQ actask 0xc068becc][SLP] acpi_task2 7 c2121528 cbb34000 0 0 0 0000204 [SLPQ actask 0xc068becc][SLP] acpi_task1 6 c21216e0 cbb35000 0 0 0 0000204 [SLPQ actask 0xc068becc][SLP] acpi_task0 5 c2121898 cbb36000 0 0 0 0000204 [SLPQ - 0xc214eb80] [SLP] taskqueue 35 c2121a50 cbb37000 0 0 0 0000204 [IWAIT] swi6:+ 34 c2121c08 cbb38000 0 0 0 0000204 [IWAIT] swi7: task queue 33 c2121dc0 cbb39000 0 0 0 0000204 [IWAIT] swi3: cambio 32 c212a000 cbb3a000 0 0 0 0000204 [IWAIT] swi2: camnet 31 c212a1b8 cbb5f000 0 0 0 0000204 [IWAIT] swi7: acpitaskq 30 c212a370 cbb60000 0 0 0 0000204 [IWAIT] swi5:+ 29 c212a528 cbb61000 0 0 0 0000204 [SLPQ - 0xc068c0e0] [SLP] random 4 c212a6e0 cbb62000 0 0 0 0000204 [SLPQ - 0xc068ec48] [SLP] g_down 3 c212a898 cbb63000 0 0 0 0000204 [SLPQ - 0xc068ec44] [SLP] g_up 2 c0fef1b8 cb334000 0 0 0 0000204 [SLPQ - 0xc068ec3c] [SLP] g_event 28 c0fef370 cb335000 0 0 0 0000204 [IWAIT] swi1: net 27 c0fef528 cb336000 0 0 0 0000204 [IWAIT] swi4: vm 26 c0fef6e0 cb337000 0 0 0 000020c [RUNQ] swi8: tty:sio clock 25 c0fef898 cb35c000 0 0 0 0000204 [IWAIT] irq15: ata1 24 c0fefa50 cb35d000 0 0 0 0000204 [IWAIT] irq14: ata0 23 c0fefc08 cb35e000 0 0 0 0000204 [IWAIT] irq13: 22 c0fefdc0 cb35f000 0 0 0 0000204 [IWAIT] irq12: psm0 21 c2121000 cbb31000 0 0 0 0000204 [IWAIT] irq11: 20 c21211b8 cbb32000 0 0 0 0000204 [IWAIT] irq10: sio1 19 c2121370 cbb33000 0 0 0 0000204 [CPU 0] irq9: cbb0 fxp0 ++++ 18 c0fe8000 cb2e2000 0 0 0 0000204 [IWAIT] irq8: rtc 17 c0fe81b8 cb32b000 0 0 0 0000204 [IWAIT] irq7: 16 c0fe8370 cb32c000 0 0 0 0000204 [IWAIT] irq6: 15 c0fe8528 cb32d000 0 0 0 0000204 [IWAIT] irq5: 14 c0fe86e0 cb32e000 0 0 0 0000204 [IWAIT] irq4: sio0 13 c0fe8898 cb32f000 0 0 0 0000204 [IWAIT] irq3: 12 c0fe8a50 cb330000 0 0 0 0000204 [IWAIT] irq1: atkbd0 11 c0fe8c08 cb331000 0 0 0 0000204 [IWAIT] irq0: clk 10 c0fe8dc0 cb332000 0 0 0 000020c [Can run] idle 1 c0fef000 cb333000 0 0 1 0004200 [SLPQ wait 0xc0fef000] [SLP] init 0 c068ed40 c081f000 0 0 0 0000200 [SLPQ sched 0xc068ed40] [SLP] swapper db> tr 755 sched_switch(c224de70,d99f5503,929c897c,c06b1354,c224de70) at sched_switch+0xd5 mi_switch(1,c0515ca0,c0515a42,bfbfd000,1) at mi_switch+0x1ab sleepq_switch(c069466c,cc75eba4,c04c7d0f,c069466c,c069466c) at sleepq_switch+0x16f sleepq_wait(c069466c,c069466c,c069313c,c064b519,1) at sleepq_wait+0x11 cv_wait(c069466c,c069313c,2ffffff,bfbfdcd4,bfbfdcd4) at cv_wait+0x12f _sx_xlock(c0694640,0,0,2000,ffffffff) at _sx_xlock+0x72 userland_sysctl(c224de70,cc75ec7c,2,bfbfdd20,bfbfdcd4) at userland_sysctl+0xed __sysctl(c224de70,cc75ed14,18,c0686da0,6) at __sysctl+0xd4 syscall(20002f,80b002f,bfbf002f,2,bfbfdd20) at syscall+0x2f0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x288a7457, esp = 0xbfbfdc4c, ebp = 0xbfbfdc88 --- db> tr 693 sched_switch(c0ff0e70,3d17d003,29f41a1b,c06b1354,c0ff0e70) at sched_switch+0xd5 mi_switch(1,c0515ca0,c0515a42,28069000,1) at mi_switch+0x1ab sleepq_switch(c069466c,cb35bba4,c04c7d0f,c069466c,c069466c) at sleepq_switch+0x16f sleepq_wait(c069466c,c069466c,c069313c,c064b519,1) at sleepq_wait+0x11 cv_wait(c069466c,c069313c,2d00610,bfbfecb0,bfbfecb0) at cv_wait+0x12f _sx_xlock(c0694640,0,0,80,c2a0421c) at _sx_xlock+0x72 userland_sysctl(c0ff0e70,cb35bc7c,2,28069a0c,bfbfecb0) at userland_sysctl+0xed __sysctl(c0ff0e70,cb35bd14,18,4,6) at __sysctl+0xd4 syscall(2f,2f,2f,2,28069a0c) at syscall+0x2f0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (202, FreeBSD ELF32, __sysctl), eip = 0x2805accf, esp = 0xbfbfec48, ebp = 0xbfbfec84 --- db> tr 1826 sched_switch(c285b150,1dcd1203,e8f479c,c06b1354,c285b150) at sched_switch+0xd5 mi_switch(1,c0515ca0,c0515a42,7,1) at mi_switch+0x1ab sleepq_switch(c0693cac,cc7fdc54,c04c7d0f,c0693cac,c0693cac) at sleepq_switch+0x16f sleepq_wait(c0693cac,c0693cac,c0693820,c064aa0b,1) at sleepq_wait+0x11 cv_wait(c0693cac,c0693820,c29ff528,21,21) at cv_wait+0x12f _sx_xlock(c0693c80,0,0,6,0) at _sx_xlock+0x72 fork1(c285b150,24030,0,cc7fdcd4,c285b150) at fork1+0x3a6 linux_clone(c285b150,cc7fdd14,cc7fdd0c,cc7fdd10,2) at linux_clone+0x69 syscall(bfbf002f,bfbf002f,cc90002f,bf3ffd24,0) at syscall+0x2f0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (120, Linux ELF, linux_clone), eip = 0x28dc2c4e, esp = 0x8dc232c, ebp = 0x8dc237c --- db> tr 2394 sched_switch(c2985bd0,ebae7c03,f173380a,c06b1354,c2985bd0) at sched_switch+0xd5 mi_switch(1,c0515ca0,c0515a42,c055905f,1) at mi_switch+0x1ab sleepq_switch(c0693cac,cc8f8c5c,c04c7d0f,c0693cac,c0693cac) at sleepq_switch+0x16f sleepq_wait(c0693cac,c0693cac,c0693820,c064aa0b,1) at sleepq_wait+0x11 cv_wait(c0693cac,c0693820,c2985bd0,c29fb370,c29fb370) at cv_wait+0x12f _sx_xlock(c0693c80,0,0,cc8f8cc8,200f) at _sx_xlock+0x72 exit1(c2985bd0,0,c0517b46,c2985bd0,cc8f8d40) at exit1+0xce5 sys_exit(c2985bd0,cc8f8d14,4,4,1) at sys_exit+0x67 syscall(2f,2f,2f,bfbfe680,bfbfe68c) at syscall+0x2f0 Xint0x80_syscall() at Xint0x80_syscall+0x1d --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x280be0d3, esp = 0xbfbfe62c, ebp = 0xbfbfe648 --- db> tr 2420 sched_switch(0,1,0,c2a02084,4) at sched_switch+0xd5 db> This happens only under heavy load (with swapping) > Tim > _______________________________________________ > freebsd-current_at_freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-current > To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org" -- Vladimir B. Grebenchikov vova_at_fbsd.ruReceived on Fri Jun 25 2004 - 09:58:32 UTC
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:37:58 UTC