Re: Complete hang on 9.0-RELEASE

Hi,

On Mon, Apr 16, 2012 at 5:50 PM, Arnaud Lacombe <lacombar_at_gmail.com> wrote:
> Hi,
>
> [for the record...]
>
> On Tue, Feb 14, 2012 at 11:41 AM, Arnaud Lacombe <lacombar_at_gmail.com> wrote:
>> Hi folks,
>>
>> For the records, I was running some tests yesterday on top of a
>> 9.0-RELEASE, amd64, kernel when the box hanged. At the time of the
>> hang, the box was running a process with about 2800 threads with heavy
>> IPC between 1400 writers and 1400 readers. The box was in single user
>> mode (/bin/sh coming from FreeBSD 7.4-STABLE). Here is the beginning
>> of the dmesg:
>>
>> Copyright (c) 1992-2012 The FreeBSD Project.
>> Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
>>        The Regents of the University of California. All rights reserved.
>> FreeBSD is a registered trademark of The FreeBSD Foundation.
>> FreeBSD 9.0-RELEASE #0: Tue Jan  3 07:46:30 UTC 2012
>>    root_at_farrell.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
>> CPU: Intel(R) Atom(TM) CPU D510   _at_ 1.66GHz (1666.70-MHz K8-class CPU)
>>  Origin = "GenuineIntel"  Id = 0x106ca  Family = 6  Model = 1c  Stepping = 10
>>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>>  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
>>  AMD Features=0x20000800<SYSCALL,LM>
>>  AMD Features2=0x1<LAHF>
>>  TSC: P-state invariant, performance statistics
>> real memory  = 2137587712 (2038 MB)
>> avail memory = 2037841920 (1943 MB)
>> Event timer "LAPIC" quality 400
>> ACPI APIC Table: <070611 APIC1125>
>> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
>> FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
>>  cpu0 (BSP): APIC ID:  0
>>  cpu1 (AP/HT): APIC ID:  1
>>  cpu2 (AP): APIC ID:  2
>>  cpu3 (AP/HT): APIC ID:  3
>>
>> I will restart the test and see if this happens again.
>>
> I reproduced the previous problem on 10-CURRENT from r233917, on the
> following platform (here running 8.2-RELEASE):
>
> FreeBSD is a registered trademark of The FreeBSD Foundation.
> FreeBSD 8.2-RELEASE #0: Thu Feb 17 02:41:51 UTC 2011
>    root_at_mason.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC amd64
> Timecounter "i8254" frequency 1193182 Hz quality 0
> CPU: Intel(R) Atom(TM) CPU D525   _at_ 1.80GHz (1800.01-MHz K8-class CPU)
>  Origin = "GenuineIntel"  Id = 0x106ca  Family = 6  Model = 1c  Stepping = 10
>  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
>  Features2=0x40e31d<SSE3,DTES64,MON,DS_CPL,TM2,SSSE3,CX16,xTPR,PDCM,MOVBE>
>  AMD Features=0x20100800<SYSCALL,NX,LM>
>  AMD Features2=0x1<LAHF>
>  TSC: P-state invariant
> real memory  = 2136539136 (2037 MB)
> avail memory = 2043772928 (1949 MB)
> ACPI APIC Table: <010312 APIC0947>
> FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
> FreeBSD/SMP: 1 package(s) x 2 core(s) x 2 HTT threads
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP/HT): APIC ID:  1
>  cpu2 (AP): APIC ID:  2
>  cpu3 (AP/HT): APIC ID:  3
>
> Complete system freeze while running about 2400 threads. I had to
> power cycle the system to get it back alive. I discussed a way to
> debug this with attilio_at_ on freebsd-stable_at_, but still did not had
> time to implement it.
>
10-CURRENT from r233917 hanged again today while running 3600 threads.
I enabled WITNESS and INVARIANTS on that specific kernel, secretly
hoping that they would trigger some meaningful information, but they
did not. I would guess my last attempt is to enable SW_WATCHDOG, and
gather some state information out of DDB when the watchdog trigger, if
it does...

Btw, this issue seems to be specifically happening on Atom/ICH8M
platform running amd64 kernel, as I've never seen it on other
platforms, and yet ran extensive tests. I am not entirely sure it
happens on i386. I would need to check.

 - Arnaud