Re: CURRENT: buildworld fails

On Wed, May 23, 2012 at 1:52 PM, O. Hartmann
<ohartman_at_zedat.fu-berlin.de> wrote:
> On 05/19/12 22:33, Garrett Cooper wrote:
>> On Sat, May 19, 2012 at 6:55 AM, O. Hartmann
>> <ohartman_at_zedat.fu-berlin.de> wrote:
>>> Since approx. a week for now, I can not build FreeBSD 10.0-CURRENT/amd64
>>> anymore! This happens to be on ALL(!) FreeBSD 10 boxes around here I
>>> maintain.
>>
>> ...
>>
>>> b) build and install /usr/src/lib via "make clean cleandepend depend obj
>>> all install" doesn't work anymore, it fails with
>>> ."/usr/src/lib/Makefile", line 179: Malformed conditional (${MK_NAND} !=
>>> "no")
>>> "/usr/src/lib/Makefile", line 181: if-less endif
>>
>> Your mk files in /usr/share/mk are out of synch with your build tree.
>> If you opt out of using buildworld, then you need to do 'make -C
>> share/mk install' beforehand.

...

> My problems occured around the May, 15th and hit ALL of my FreeBSD
> 10-CURRENT/amd64 boxes on which I do most time daily buildworld. Two of
> them could be fixed, I did this two days ago by

The problem has been on and off "mild" breakage for the past couple
days (yacc, format string qualifier issues, etc).

> cd /usr/src
> make installincludes
> make -C {lib|libexec|sbin ...} clean cleandepend depend obj all install
>
> On one specific machine it didn't work that way. I found out that
> several binaries in the system's tree remained dated on 15th May, so
> like "ld" or other essential pieces. Therefore I suspect a crushed system.

I ran into a similar issue when my copy of mtree went MIA after a
freshly installed world corrupted my VM during an nlm panic
(thankfully it was my VM and not something else).

> Does anyhow do have any idea how to repair the system?

Trial and error for the base system is the only method I'm aware of
right now, but if portions of the base system are corrupted and not
others, I would suspect all of the contents on disk (esp after recent
reports about UFS filesystem corruption with SUJ).

> I can compile a kernel, but I can not compile ports (or even update them, I receive
> multiple weird errors of missing header files or "changed" libstdc++.so.

You can resolve the ports issue via another port that's shell based or
C based. Memory serves me correctly it was
ports-mgmt/port-maintenance-tools ..

> The idea was to take a very recent binary installation set and install
> this selective peice by piece over the existing installation, but I need
> to preserve /etc and other directories. Any ideas ?

You can save files that would be in the dist tarballs on the install
media to another directory (/etc/*, customizations in /root, /usr,
etc). Then you can extract the tarballs (I would use a test directory
first though). Passing certain flags to tar to just copy out some
binaries might help.

> Help is appreciated and thanks in advance,

Lesson to maybe learn:
1. Always have a backup machine running a stable version of FreeBSD
that is fast enough that you can use to generate images and push out
to machines on the fly. If not possible, always have a static
toolchain on hand (I would assume that it's safe to say that anything
listed in bootstrap-tools would be game for the toolchain).
2. Expect warts on CURRENT right after a major release cycle has run
its course. If this is too much to worry about, don't run 10-CURRENT
and instead run 9-STABLE, etc.

Cheers,
-Garrett