I Found What The Real Problem Is... (you're gonna hate me) (re: I *really* need help PLEASE - buildworld failing on mkdep libstdc++ can't find unwind.h but it *is* there)

I thank everyone who tried to help me on this problem.

I'm 99% certain the problem stems from known compiler
bugs when using the lang/gcc* ports.

I can actually point to a few PRs in GCC's bug database
that may be related.  The timings of the commits that
fix these bugs, together with the lang/gcc* snapshots,
and the fact that the -Current system gcc has not been
updated since 3.3.3-prerelease, all conspired to cause
these glitches in the build process. 
That is my theory, anyway.

The GCC-3.3.4 status page at
<http://gcc.gnu.org/ml/gcc/2004-03/msg01654.html>
begins a thread on some of these optimization bugs that
have regressed back into the current lang/gcc33 snapshot. 
The consensus there seems to be that these bugs must be
fixed before 3.3.4 is released as final.

If you follow the previous-msg thread up from this post,
I requested to update lang/gcc33 (aka 3.3.4) with a new
snapshot.  But I'm not certain we'd get all the
regressed bugs fixed just yet.

My previous msg was actually related to my trying to
switch back to using system gcc.

However, the 3.3.3 was released in mid-February, and these
bugs were supposed to be fixed in it.  At this time I'd
like to find some way of using _this_ release as our
system gcc, esp. if we cannot get a current snapshot of
3.3.4 (update lang/gcc33).


How I got led to this discovery, for documentations's sake:

I tried everything, esp. DES's cleandir procedure, several
times.  (That is a good thing to know how to do.)  Besides
that, I always blow away /usr/obj/* when building a fresh
world or kernel.

In my earlier msg, I'd stated I had run 'make installworld'
using a /usr/obj bzip2ball from the jpsnap site, dated
20040410, which would've matched rather closely to my CTM
status at that time.  I felt my /usr/include was already
'fixed' by this action.  To prove this, I needed a current
'live CD' to compare against.  (The jpsnap server is
terribly slow; I've e-mailed the sysop there.  None of the
ftp*.tw mirrors have any -Currents in April at all.)  I was
finally able to get a live-5.2-current ISO dated 20040410,
close enough to the CTM deltas I'd applied when I posted my
original plea for help.

After installing world from the bzip2ball, and recreating
the stuff in /usr/include with DES's proc, I ran
'diff -r' between my /usr/include and the one on the
live-5.2-20040410 CD.  The diffs therein could all be
explained: some diffs were commits done since 20040410,
other diffs (esp. files "only in" the CD) were from stuff
not built due to NO_foo=yes set in /etc/make.conf. 
Absolutely no real differences at all.

Also, the installworld from the bzip2ball *should* have
fixed the libs so that I could run system gcc and skirt
around the known lang/gcc* bugs.  I previously mentioned
that I'd cleaned out /lib and /libexec, and fixed ldconfig
in rc.conf (some ports were built with lang/gcc33 and must
continue to use libgcc from lang/gcc33, so that had to
remain as the last entry in ldconfig). 
None of that fixed the problem.

Okay.  We need to treat this as tho we're using a buggy gcc. 
I still can't rule out bad C src as the real problem, but we
need to find another way to skirt around these known bugs.

One of the optimizing PRs on GCC mentioned -O[1] was
affected as well as -Os, but -O2 was okay.  So I changed
make.conf to add -O2 to all CFLAGS and permutations thereof
I could think-up.

BINGO!  We got thru a buildworld without a problem!

The only thing I changed was to use -O2.

Just for good measure, I installed that version with -O2
and a fresh kernel, rebooted, and started a whole new build
process with _those_ libs active.  I have installed that
doubled-up whole fresh version and am running it now.

I am testing builds of world & kernel as of this morning's
CTM, setting -Os again, and so far we haven't hit a glitch;
this slow Puny Pentium2 has already passed the point where
building libstdc++ was broken, so we got a good -Os build
of it, at least.

I really need to urge to y'all the importance of using a
good working compiler to do all testing & debugging.  The
final release of 3.3.3 was in February, and we have no
ports or system gcc using it AFAIK.  These bugs are
supposed to be fixed in it.  I'm tracking -Current via
CTM, and 'gcc --version' now shows:
gcc (GCC) 3.3.3 [FreeBSD] 20031106
Copyright (C) 2003 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

We desparately need the 200402xx final version, please.


And: Unless the snapshots get updated, too, we cannot use
them at all.


Thank you for putting up with my long writeup.  I'm hoping
this will show the importance of getting gcc updated ASAP.


  --  Paul Seniura
      System Specialist
      State of Okla. D.O.T.