Re: HEADS UP: sparc64 backend for llvm/clang imported

On Wed, Aug 19, 2015 at 04:19:03PM -0400, Kurt Lidl wrote:
> > Dimitry Andric wrote this message on Fri, Feb 28, 2014 at 20:22 +0100:
> >> In r262613 I have merged the clang-sparc64 branch back to head.  This
> >> imports an updated sparc64 backend for llvm and clang, allowing clang to
> >> bootstrap itself on sparc64, and to completely build world.  To be able
> >> to build the GENERIC kernel, there is still one patch to be finalized,
> >> see below.
> >>
> >> If you have any sparc64 hardware, and are not afraid to encounter rough
> >> edges, please try out building and running your system with clang.  To
> >> do so, update to at least r262613, and enable the following options in
> >> e.g. src.conf, or in your build environment:
> >>
> >> WITH_CLANG=y
> >> WITH_CLANG_IS_CC=y
> >> WITH_LIBCPLUSPLUS=y  (optional)
> >>
> >> Alternatively, if you would rather keep gcc as /usr/bin/cc for the
> >> moment, build world using just WITH_CLANG, enabling clang to be built
> >> (by gcc) and installed.  After installworld, you can then set CC=clang,
> >> CXX=clang++ and CPP=clang-cpp for building another world.
> >>
> >> For building the sparc64 kernel, there is one open issue left, which is
> >> that sys/sparc64/include/pcpu.h uses global register variables, and this
> >> is not supported by clang.  A preliminary patch for this is attached,
> >> but it may or may not blow up your system, please beware!
> >>
> >> The patch changes the pcpu and curpcb global register variables into
> >> inline functions, similar to what is done on other architectures.
> >> However, the current approach is not optimal, and the emitted code is
> >> slightly different from what gcc outputs.  Any improvements to this
> >> patch are greatly appreciated!
> >>
> >> Last but not least, thanks go out to Roman Divacky for his work with
> >> llvm/clang upstream in getting the sparc64 backend into shape.
> >
> > Ok, I have a new pcpu patch to try.  I have only compile tested it.
> >
> > It is available here:
> > https://www.funkthat.com/~jmg/sparc64.pcpu.patch
> >
> > I've also attached it.
> >
> > Craig, do you mind testing it?
> >
> > This patch also removes curpcb as it appears to not be used by any
> > sparc64 C code.  A GENERIC kernel compiles fine, and fxr only turns up
> > curpcb used in machdep code, and no references to it under sparc64.
> >
> > This is not a proper solution in that
> > it can mean counters/stats can be copied/moved to other cpus overwriting
> > the previous values if a race happens...  We use
> > PCPU_SET(mem, PCPU_GET(mem) + val) for PCPU_ADD, not great, but it's
> > no worse than what we were previously using..
> >
> > Until we get a proper fix which involves mapping all the cpu's PCPU
> > data on all CPUs, this will have to sufice..
> >
> > This patch is based upon, I believe, a patch from Marius and possibly
> > modified by rdivacky.
> >
> > Thanks for testing..
> 
> The above message was posted a while ago, and I decided that I would
> give the patch a test run on a spare sparc that I have, now that the
> instability problem with multiprocessor sparc64 machines has been
> resolved.
> 
> So, I have an up-to-date stable/10 V240 (2x1.5Ghz cpus, 8GB of memory),
> running a completely stock r286861.  That all seems to work just fine.
> 
> I applied the patch referenced in the email:
> 
> https://www.funkthat.com/~jmg/sparc64.pcpu.patch
> 
> (it applied cleanly), and then rebuilt the kernel on the machine,
> using the stock gcc 4.2.1 compiler.
> 
> When rebooting with that kernel, the machine panics pretty early
> in the boot:
> 
> FreeBSD 10.2-STABLE #3 r286861M: Wed Aug 19 14:28:45 EDT 2015
>      lidl_at_spork.pix.net:/usr/obj/usr/src/sys/GENERIC sparc64
> gcc version 4.2.1 20070831 patched [FreeBSD]
> real memory  = 8589934592 (8192 MB)
> avail memory = 8379719680 (7991 MB)
> cpu0: Sun Microsystems UltraSparc-IIIi Processor (1503.00 MHz CPU)
> cpu1: Sun Microsystems UltraSparc-IIIi Processor (1503.00 MHz CPU)
> FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs
> random device not loaded; using insecure entropy
> panic: trap: illegal instruction (kernel)
> cpuid = 0
> KDB: stack backtrace:
> #0 0xc05750e0 at panic+0x20
> #1 0xc08db9f8 at trap+0x558
> Uptime: 1s
> Automatic reboot in 15 seconds - press a key on the console to abort
> Rebooting...
> timeout shutting down CPUs.
> 
> So, the patch to get rid of the pcpu usage (as a prereq to poking
> at the clang compiler issues) does not work properly.
> 

As I pointed out when that patch was posted, the approach taken by
it assumes a CPU can access foreign PCPU data, which currently isn't
true on sparc64. So the patch is at least incomplete but also may
have further issues.

Such a patch is no longer a prerequisite for compiling a sparc64
kernel with clang, though, as clang meanwhile has been told to
grok at least the global registers used by the PCPU code.

Besides some default options like the choice of code model not
being appropriate for FreeBSD, clang-compiled loader and kernel
don't work due to two major problems present in clang up to at
least version 3.6.0: a) it uses a different stack layout than
GCC so any unwinding code fails and b) it produces broken code
for recursive function calls. YMMV.

Marius