buildworld on CPU-A, installworld on CPU-B ends up with SIGILL

From: Ruslan Garipov <>
Date: Mon, 25 Nov 2019 19:06:11 +0500

I want to build kernel and world (of FreeBSD 13.0-CURRENT) on a fast
virtual machine for other ones (all the virtual machines are hosted on
VMware EXSi hypervisors, which have different physical CPUs).

After `make -j16 buildworld` has finished successfully on the build
machine, I get there, for example,
/usr/obj/usr/src/amd64.amd64/tmp/legacy/bin/install program having the
shlxq instruction (one from the BMI2 instruction set extensions). This
eventually causes make installkernel and make installworld to fail with
SIGILL on a virtual machine which must consume built world and kernel,
and which is hosted on another ESXi instance, with older physical CPU
(read: with CPU not implementing shlxq).

The build machine (FreeBSD 13.0-CURRENT r354802) builds (x)install using
the following commands (a part of buildworld):

$ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree
-I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD
-MF.depend.xinstall.o -MTxinstall.o -std=gnu99 -Wno-format-zero-length
-Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include
-c /usr/src/usr.bin/xinstall/xinstall.c -o xinstall.o
$ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree
-I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -MD
-MF.depend.getid.o -MTgetid.o -std=gnu99 -Wno-format-zero-length
-Qunused-arguments -I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include
-c /usr/src/contrib/mtree/getid.c -o getid.o
$ cc -O2 -pipe -O2 -march=x86-64 -pipe -I/usr/src/contrib/mtree
-I/usr/src/lib/libnetbsd -DHAVE_STRUCT_STAT_ST_FLAGS=1 -std=gnu99
-Wno-format-zero-length -Qunused-arguments
-I/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/include -static
-L/usr/obj/usr/src/amd64.amd64/tmp/legacy/usr/lib -o xinstall xinstall.o
getid.o -L/usr/obj/usr/src/amd64.amd64/tmp/obj-tools/lib/libmd -lmd -legacy

This produces xintstall with `shlxq`s:

$ llvm-objdump --disassemble xinstall | grep -c shlxq

I believe statically linked /usr/lib/libmd.a is a stuff which brings
`shlxq`s into the xinstall.  I didn't examine it further, sorry...

My question is: is it possible to buildworld without issuing
instructions which are native for the host CPU?  I have neither
/etc/make.conf, nor /etc/src.conf on the build machine.  Is it possible
at all for FreeBSD CURRENT to be built outside a host and installed on
the host later?

Just as a reference:

My build machine has Intel(R) Xeon(R) Gold 6150 CPU that supports BMI2:

# cpucontrol -i 7 /dev/cpuctl0
cpuid level 0x7: 0x00000000 0xd19f6ffb 0x00000018 0xbc000000

(Bit 08 in EBX is set)

, and a consuming machine has Intel(R) Xeon(R) CPU E5-4617 CPU that
doesn't support BMI2:

# cpucontrol -i 7 /dev/cpuctl0
cpuid level 0x7: 0x00000000 0x00000002 0x00000000 0xbc000000

(Bit 08 in EBX is unset).

Both machines install kernel and world without any problem, if they were
built locally.


Yours truly,
Ruslan Garipov <ruslanngaripov at gmail dot com>.
MinGW-w64 developer, with write after approval
GEDKeeper collaborator
Received on Mon Nov 25 2019 - 13:06:22 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:22 UTC