Re: CURRENT: make -jX buildworld doesn't work

From: Ian Lepore <freebsd_at_damnhippie.dyndns.org>
Date: Wed, 14 Mar 2012 09:08:56 -0600
On Wed, 2012-03-14 at 14:47 +0100, O. Hartmann wrote:
> This is no compalin, since make buildworld works with one thread.
> 
> But I'd like to report a funny thing I witnessed.
> 
> On two boxes equipted with Core2Dou CPUS (E8500,  2 cores/threads,
> Q6600, 4 cores/threads) a parallel make buildworld works fine with the
> most recent sources of FreeBSD 10.0-CURRENT/amd64 (both boxes have 8 GB
> RAM, both boxes use a very close/similar setup and configuration).
> 
> I moved in my lab towards a brand new Sandy-Bridge-E box with 32GB RAM
> and the CPU is a Core i7-3930X with six cores/12 threads. On this box,
> even "make buildworld" with -j2 fails to build. It builds fine with a
> vanilla "make buildworld".
> 
> Also funny is, that even with only one thread, the 3 GHz Core2Duo
> methusalem systems "outperform" in compile time the 3,2 GHZ driven Intel
> youngster.
> 
> Maybe there is an issue in FreeBSD 10 with the TURBO BOOST? I'm a bit
> time constraint now, but I'm willing to do some tests with advices from
> the experts.
> 
> Lets give you some informations I think it could be valuable. Please
> request more, if this isn't sufficient.
> 
> I realize/I'm aware that both hardware and OS are brandnew! But
> hopefully this is something important and could be "fixed" - if my
> observation is a real observation ...
> 
> Regards,
> Oliver
> 
> KERNEL:
> 
> # CPU frequency control
> device          cpufreq
> device          cpuctl
> device          nvram
> device          coretemp
> 
> /etc/rc.conf:
> performance_cx_lowest="HIGH"    # Online CPU idle state
> performance_cpu_freq="NONE"     # Online CPU frequency
> economy_cx_lowest="HIGH"        # Offline CPU idle state
> economy_cpu_freq="NONE"         # Offline CPU frequency
> 
> #> sysctl kern.timecounter
> kern.timecounter.tc.i8254.mask: 65535
> kern.timecounter.tc.i8254.counter: 52702
> kern.timecounter.tc.i8254.frequency: 1193182
> kern.timecounter.tc.i8254.quality: 0
> kern.timecounter.tc.HPET.mask: 4294967295
> kern.timecounter.tc.HPET.counter: 2109995661
> kern.timecounter.tc.HPET.frequency: 14318180
> kern.timecounter.tc.HPET.quality: 950
> kern.timecounter.tc.ACPI-fast.mask: 16777215
> kern.timecounter.tc.ACPI-fast.counter: 3590445
> kern.timecounter.tc.ACPI-fast.frequency: 3579545
> kern.timecounter.tc.ACPI-fast.quality: 900
> kern.timecounter.tc.TSC-low.mask: 4294967295
> kern.timecounter.tc.TSC-low.counter: 1510440780
> kern.timecounter.tc.TSC-low.frequency: 12537463
> kern.timecounter.tc.TSC-low.quality: -100
> kern.timecounter.stepwarnings: 0
> kern.timecounter.hardware: HPET
> kern.timecounter.choice: TSC-low(-100) ACPI-fast(900) HPET(950) i8254(0)
> dummy(-1000000)
> kern.timecounter.tick: 5
> kern.timecounter.invariant_tsc: 1
> kern.timecounter.smp_tsc: 0
> 
> #
> #> dmesg | grep cpu
>  cpu0 (BSP): APIC ID:  0
>  cpu1 (AP): APIC ID:  1
>  cpu2 (AP): APIC ID:  2
>  cpu3 (AP): APIC ID:  3
>  cpu4 (AP): APIC ID:  4
>  cpu5 (AP): APIC ID:  5
>  cpu6 (AP): APIC ID:  6
>  cpu7 (AP): APIC ID:  7
>  cpu8 (AP): APIC ID:  8
>  cpu9 (AP): APIC ID:  9
>  cpu10 (AP): APIC ID: 10
>  cpu11 (AP): APIC ID: 11
> cpu0: <ACPI CPU> on acpi0
> cpu1: <ACPI CPU> on acpi0
> cpu2: <ACPI CPU> on acpi0
> cpu3: <ACPI CPU> on acpi0
> cpu4: <ACPI CPU> on acpi0
> cpu5: <ACPI CPU> on acpi0
> cpu6: <ACPI CPU> on acpi0
> cpu7: <ACPI CPU> on acpi0
> cpu8: <ACPI CPU> on acpi0
> cpu9: <ACPI CPU> on acpi0
> cpu10: <ACPI CPU> on acpi0
> cpu11: <ACPI CPU> on acpi0
> coretemp0: <CPU On-Die Thermal Sensors> on cpu0
> est0: <Enhanced SpeedStep Frequency Control> on cpu0
> p4tcc0: <CPU Frequency Thermal Control> on cpu0
> coretemp1: <CPU On-Die Thermal Sensors> on cpu1
> est1: <Enhanced SpeedStep Frequency Control> on cpu1
> p4tcc1: <CPU Frequency Thermal Control> on cpu1
> coretemp2: <CPU On-Die Thermal Sensors> on cpu2
> est2: <Enhanced SpeedStep Frequency Control> on cpu2
> p4tcc2: <CPU Frequency Thermal Control> on cpu2
> coretemp3: <CPU On-Die Thermal Sensors> on cpu3
> est3: <Enhanced SpeedStep Frequency Control> on cpu3
> p4tcc3: <CPU Frequency Thermal Control> on cpu3
> coretemp4: <CPU On-Die Thermal Sensors> on cpu4
> est4: <Enhanced SpeedStep Frequency Control> on cpu4
> p4tcc4: <CPU Frequency Thermal Control> on cpu4
> coretemp5: <CPU On-Die Thermal Sensors> on cpu5
> est5: <Enhanced SpeedStep Frequency Control> on cpu5
> p4tcc5: <CPU Frequency Thermal Control> on cpu5
> coretemp6: <CPU On-Die Thermal Sensors> on cpu6
> est6: <Enhanced SpeedStep Frequency Control> on cpu6
> p4tcc6: <CPU Frequency Thermal Control> on cpu6
> coretemp7: <CPU On-Die Thermal Sensors> on cpu7
> est7: <Enhanced SpeedStep Frequency Control> on cpu7
> p4tcc7: <CPU Frequency Thermal Control> on cpu7
> coretemp8: <CPU On-Die Thermal Sensors> on cpu8
> est8: <Enhanced SpeedStep Frequency Control> on cpu8
> p4tcc8: <CPU Frequency Thermal Control> on cpu8
> coretemp9: <CPU On-Die Thermal Sensors> on cpu9
> est9: <Enhanced SpeedStep Frequency Control> on cpu9
> p4tcc9: <CPU Frequency Thermal Control> on cpu9
> coretemp10: <CPU On-Die Thermal Sensors> on cpu10
> est10: <Enhanced SpeedStep Frequency Control> on cpu10
> p4tcc10: <CPU Frequency Thermal Control> on cpu10
> coretemp11: <CPU On-Die Thermal Sensors> on cpu11
> est11: <Enhanced SpeedStep Frequency Control> on cpu11
> p4tcc11: <CPU Frequency Thermal Control> on cpu11
> 
> 
> 
> #>  sysctl dev.cpu
> dev.cpu.0.%desc: ACPI CPU
> dev.cpu.0.%driver: cpu
> dev.cpu.0.%location: handle=\_SB_.P000
> dev.cpu.0.%pnpinfo: _HID=none _UID=0
> dev.cpu.0.%parent: acpi0
> dev.cpu.0.coretemp.delta: 54
> dev.cpu.0.coretemp.resolution: 1
> dev.cpu.0.coretemp.tjmax: 91.0C
> dev.cpu.0.coretemp.throttle_log: 0
> dev.cpu.0.temperature: 37.0C
> dev.cpu.0.freq: 3200
> dev.cpu.0.freq_levels: 3201/130000 3200/130000 3000/119000 2800/109000
> 2600/99000 2400/89000 2200/80000 2000/71000 1800/62000 1600/54000
> 1400/46000 1225/40250 1200/39000 1050/34125 900/29250 750/24375
> 600/19500 450/14625 300/9750 150/4875
> dev.cpu.0.cx_supported: C1/1
> dev.cpu.0.cx_lowest: C1
> dev.cpu.0.cx_usage: 100.00% last 759us
> dev.cpu.1.%desc: ACPI CPU
> dev.cpu.1.%driver: cpu
> dev.cpu.1.%location: handle=\_SB_.P001
> dev.cpu.1.%pnpinfo: _HID=none _UID=0
> dev.cpu.1.%parent: acpi0
> dev.cpu.1.coretemp.delta: 54
> dev.cpu.1.coretemp.resolution: 1
> dev.cpu.1.coretemp.tjmax: 91.0C
> dev.cpu.1.coretemp.throttle_log: 0
> dev.cpu.1.temperature: 37.0C
> dev.cpu.1.cx_supported: C1/1
> dev.cpu.1.cx_lowest: C1
> dev.cpu.1.cx_usage: 100.00% last 24782us
> dev.cpu.2.%desc: ACPI CPU
> dev.cpu.2.%driver: cpu
> dev.cpu.2.%location: handle=\_SB_.P002
> dev.cpu.2.%pnpinfo: _HID=none _UID=0
> dev.cpu.2.%parent: acpi0
> dev.cpu.2.coretemp.delta: 53
> dev.cpu.2.coretemp.resolution: 1
> dev.cpu.2.coretemp.tjmax: 91.0C
> dev.cpu.2.coretemp.throttle_log: 0
> dev.cpu.2.temperature: 38.0C
> dev.cpu.2.cx_supported: C1/1
> dev.cpu.2.cx_lowest: C1
> dev.cpu.2.cx_usage: 100.00% last 23314us
> dev.cpu.3.%desc: ACPI CPU
> dev.cpu.3.%driver: cpu
> dev.cpu.3.%location: handle=\_SB_.P003
> dev.cpu.3.%pnpinfo: _HID=none _UID=0
> dev.cpu.3.%parent: acpi0
> dev.cpu.3.coretemp.delta: 53
> dev.cpu.3.coretemp.resolution: 1
> dev.cpu.3.coretemp.tjmax: 91.0C
> dev.cpu.3.coretemp.throttle_log: 0
> dev.cpu.3.temperature: 38.0C
> dev.cpu.3.cx_supported: C1/1
> dev.cpu.3.cx_lowest: C1
> dev.cpu.3.cx_usage: 100.00% last 8973us
> dev.cpu.4.%desc: ACPI CPU
> dev.cpu.4.%driver: cpu
> dev.cpu.4.%location: handle=\_SB_.P004
> dev.cpu.4.%pnpinfo: _HID=none _UID=0
> dev.cpu.4.%parent: acpi0
> dev.cpu.4.coretemp.delta: 50
> dev.cpu.4.coretemp.resolution: 1
> dev.cpu.4.coretemp.tjmax: 91.0C
> dev.cpu.4.coretemp.throttle_log: 0
> dev.cpu.4.temperature: 41.0C
> dev.cpu.4.cx_supported: C1/1
> dev.cpu.4.cx_lowest: C1
> dev.cpu.4.cx_usage: 100.00% last 9446us
> dev.cpu.5.%desc: ACPI CPU
> dev.cpu.5.%driver: cpu
> dev.cpu.5.%location: handle=\_SB_.P005
> dev.cpu.5.%pnpinfo: _HID=none _UID=0
> dev.cpu.5.%parent: acpi0
> dev.cpu.5.coretemp.delta: 50
> dev.cpu.5.coretemp.resolution: 1
> dev.cpu.5.coretemp.tjmax: 91.0C
> dev.cpu.5.coretemp.throttle_log: 0
> dev.cpu.5.temperature: 41.0C
> dev.cpu.5.cx_supported: C1/1
> dev.cpu.5.cx_lowest: C1
> dev.cpu.5.cx_usage: 100.00% last 50156us
> dev.cpu.6.%desc: ACPI CPU
> dev.cpu.6.%driver: cpu
> dev.cpu.6.%location: handle=\_SB_.P006
> dev.cpu.6.%pnpinfo: _HID=none _UID=0
> dev.cpu.6.%parent: acpi0
> dev.cpu.6.coretemp.delta: 48
> dev.cpu.6.coretemp.resolution: 1
> dev.cpu.6.coretemp.tjmax: 91.0C
> dev.cpu.6.coretemp.throttle_log: 0
> dev.cpu.6.temperature: 43.0C
> dev.cpu.6.cx_supported: C1/1
> dev.cpu.6.cx_lowest: C1
> dev.cpu.6.cx_usage: 100.00% last 19304us
> dev.cpu.7.%desc: ACPI CPU
> dev.cpu.7.%driver: cpu
> dev.cpu.7.%location: handle=\_SB_.P007
> dev.cpu.7.%pnpinfo: _HID=none _UID=0
> dev.cpu.7.%parent: acpi0
> dev.cpu.7.coretemp.delta: 48
> dev.cpu.7.coretemp.resolution: 1
> dev.cpu.7.coretemp.tjmax: 91.0C
> dev.cpu.7.coretemp.throttle_log: 0
> dev.cpu.7.temperature: 43.0C
> dev.cpu.7.cx_supported: C1/1
> dev.cpu.7.cx_lowest: C1
> dev.cpu.7.cx_usage: 100.00% last 17630us
> dev.cpu.8.%desc: ACPI CPU
> dev.cpu.8.%driver: cpu
> dev.cpu.8.%location: handle=\_SB_.P008
> dev.cpu.8.%pnpinfo: _HID=none _UID=0
> dev.cpu.8.%parent: acpi0
> dev.cpu.8.coretemp.delta: 56
> dev.cpu.8.coretemp.resolution: 1
> dev.cpu.8.coretemp.tjmax: 91.0C
> dev.cpu.8.coretemp.throttle_log: 0
> dev.cpu.8.temperature: 35.0C
> dev.cpu.8.cx_supported: C1/1
> dev.cpu.8.cx_lowest: C1
> dev.cpu.8.cx_usage: 100.00% last 30858us
> dev.cpu.9.%desc: ACPI CPU
> dev.cpu.9.%driver: cpu
> dev.cpu.9.%location: handle=\_SB_.P009
> dev.cpu.9.%pnpinfo: _HID=none _UID=0
> dev.cpu.9.%parent: acpi0
> dev.cpu.9.coretemp.delta: 56
> dev.cpu.9.coretemp.resolution: 1
> dev.cpu.9.coretemp.tjmax: 91.0C
> dev.cpu.9.coretemp.throttle_log: 0
> dev.cpu.9.temperature: 35.0C
> dev.cpu.9.cx_supported: C1/1
> dev.cpu.9.cx_lowest: C1
> dev.cpu.9.cx_usage: 100.00% last 14794us
> dev.cpu.10.%desc: ACPI CPU
> dev.cpu.10.%driver: cpu
> dev.cpu.10.%location: handle=\_SB_.P00A
> dev.cpu.10.%pnpinfo: _HID=none _UID=0
> dev.cpu.10.%parent: acpi0
> dev.cpu.10.coretemp.delta: 50
> dev.cpu.10.coretemp.resolution: 1
> dev.cpu.10.coretemp.tjmax: 91.0C
> dev.cpu.10.coretemp.throttle_log: 0
> dev.cpu.10.temperature: 41.0C
> dev.cpu.10.cx_supported: C1/1
> dev.cpu.10.cx_lowest: C1
> dev.cpu.10.cx_usage: 100.00% last 12779us
> dev.cpu.11.%desc: ACPI CPU
> dev.cpu.11.%driver: cpu
> dev.cpu.11.%location: handle=\_SB_.P00B
> dev.cpu.11.%pnpinfo: _HID=none _UID=0
> dev.cpu.11.%parent: acpi0
> dev.cpu.11.coretemp.delta: 49
> dev.cpu.11.coretemp.resolution: 1
> dev.cpu.11.coretemp.tjmax: 91.0C
> dev.cpu.11.coretemp.throttle_log: 0
> dev.cpu.11.temperature: 42.0C
> dev.cpu.11.cx_supported: C1/1
> dev.cpu.11.cx_lowest: C1
> dev.cpu.11.cx_usage: 100.00% last 1628us
> 

There was a change (r232793) a few days ago to make turbo boost work,
more info in this thread:

http://lists.freebsd.org/pipermail/freebsd-current/2012-March/032434.html

I wasn't able to get it to work just by tweaking the rc.conf knobs, but
I suspect the reason may be because I have non-standard devd.conf. I
worked around it and haven't had time to look for the cause yet.

In trying to explain the compile time differences between two systems,
one of the first things I'd look at would be differences in the disks.
In my experience, IO performance has as big an influence on build time
as processor speed and number of cores.

I just updated my -current sources and did a fresh buildworld and
buildkernel using both -j2 and -j12 and had no problems on a 6-core
Xeon.  I know "make xdev" fails with -jN but I haven't seen failures on
any other targets.  There was a checkin recently that added some .ORDER
stuff for -jN but it only affected building usr.sbin/acpi.

-- Ian
Received on Wed Mar 14 2012 - 14:09:06 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:40:25 UTC