Re: Ordering of files in zoneinfo [Was Re: sort.core error doing installworld on Current.]

From: Warner Losh <imp_at_bsdimp.com>
Date: Thu, 23 Apr 2020 10:49:09 -0600
On Thu, Apr 23, 2020 at 10:45 AM Xin LI <delphij_at_gmail.com> wrote:

> Hi,
>
> Thanks for raising this.
>
> I have took a look at the change history, it seems that the find operation
> was introduced in r245265
> <https://svnweb.freebsd.org/base?view=revision&revision=245265> (brooks_at_,
> to support packaged base) and sort was initially implemented as find -s in
> r289451 <https://svnweb.freebsd.org/base?view=revision&revision=289451>
> (ngie_at_, to make METALOG reproducible) then as sort in r328958
> <https://svnweb.freebsd.org/base?view=revision&revision=328958> (imp_at_,
> for portability).
>
> I wonder if we could drop the sort and replace ${TZS} in line 100 with
> ${TZS:O} instead?
>

I haven't thought carefully about that, but a quick look suggests that it's
OK. It happens in a bootstrapped make, so that will work everywhere which
addresses the issues around r328958. Using TZS:O should give us the same
built-to-build stability we need. I don't recall the issue with a lot of
clarity, so there's small chance I'm missing something.

Warner


> By the way, looking at
> https://github.com/freebsd/pkg/blob/master/libpkg/metalog.c , I wonder if
> the sort should really happen in pkg(8) instead?
>
> On Fri, Apr 17, 2020 at 7:28 AM Johan Hendriks <joh.hendriks_at_gmail.com>
> wrote:
>
>> Op 17-04-2020 om 13:30 schreef Johan Hendriks:
>> >
>> > Op 17-04-2020 om 12:47 schreef Rodney W. Grimes:
>> >>>> Op 17-04-2020 om 03:31 schreef Rodney W. Grimes:
>> >>>>>> On Thu, Apr 16, 2020 at 12:39 PM Kevin Oberman
>> >>>>>> <rkoberman_at_gmail.com> wrote:
>> >>>>>>
>> >>>>>>> So you some how had a sort core dump sitting in
>> >>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir. The
>> >>>>>>> questions, how
>> >>>>>>> did get there? I'd take a look at the date on the file and, it
>> >>>>>>> it is older
>> >>>>>>> than the buildworld, just assume that it was left-over garbage.
>> >>>>>>> In either
>> >>>>>>> case, you can delete it and do another installworld.
>> >>>>>>>
>> >>>>>>> That should most likely fix things, but, if the buildworld or
>> >>>>>>> installworld
>> >>>>>>> had a crash of sort(1) that left the file, further investigation
>> >>>>>>> might be
>> >>>>>>> needed. Re-making the zoneinfo would help track it down should
>> >>>>>>> this be a re
>> >>>>>>> al bug, but it's my uneducated guess that it's not.
>> >>>>>>> --
>> >>>>>>> Kevin Oberman, Part time kid herder and retired Network Engineer
>> >>>>>>> E-mail: rkoberman_at_gmail.com
>> >>>>>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
>> >>>>>>>
>> >>>>>> Please forgive that awful post! I missed a part of your message
>> >>>>>> by laziness.
>> >>>>>>
>> >>>>>> It's odd that the error of sort(1) crashing was not caught by the
>> >>>>>> script.
>> >>>>> Yes, that is a Makefile flaw someplace.
>> >>>>> Further there must be a wildcard being used to decide which files to
>> >>>>> install, that is a further Makefile flaw.  Wildcards should NOT be
>> >>>>> used
>> >>>>> in the source of an install list, exactly because of this type of
>> >>>>> cruft
>> >>>>> that can be dropped in an obj dir.
>> >>  From src/share/zoneinfo/Makefile at about line 93:
>> >> 92  if make(*install*)
>> >> 93  TZS!= cd ${TZBUILDDIR} && find * -type f | LC_ALL=C sort
>> >>                                   ^^^^ this is a very bad thing to do
>> >> in a Makefile.
>> >>
>> >> 94  .endif
>> >>
>> >> Now I still don't know why sort cored, but I am sure this is the line
>> >> that did it.
>> >>
>> >>>>>> Clearly, sort should NOT crash! Again, a re-build of zoneinfo
>> >>>>>> might catch
>> >>>>>> something. Looking at the core might tell you which "sort" was
>> >>>>>> involved...
>> >>>>>> the one you just built or the one in the base system. This could
>> >>>>>> be just a
>> >>>>>> FOTU, but I would not bet on it.
>> >>>>> I suspect a recent zoneinfo commit as the root cause.
>> >>>>>
>> >>>> I have no idea how to bypass this issue.
>> >>>> I have used sort from the latest snapshot and placed that file on the
>> >>>> system and in the build dir, but i keep getting the core
>> >>>>
>> >>>> How can i test an build and install part for zoneinfo
>> >>>>
>> >>>> If i go into the dir /usr/src/share/zoneinfo and do make install it
>> >>>> does
>> >>>> not work, do i need to add something?
>> >>> Can you show us the output from
>> >>> cd /usr/src/share/zoneinfo
>> >>> make clean && make depend && make all && make install
>> >>> Someplace in that we should get to see sort crashing...
>> >>>
>> > On both machines my src.conf file is the same.
>> >
>> > I will start over from a clean world by doing a make cleanworld and
>> > see if it then still gives the errors
>> > Maybe some old artifacts are hanging around.
>> >
>> >
>> >
>> >>>
>> >>>> Thank you both for your time
>> >>>>
>> >>>>>> --
>> >>>>>> Kevin Oberman, Part time kid herder and retired Network Engineer
>> >>>>>> E-mail: rkoberman_at_gmail.com
>> >>>>>> PGP Fingerprint: D03FB98AFA78E3B78C1694B318AB39EF1B055683
>> >>>>>>
>> >>>>>>
>> >>>>>>> On Thu, Apr 16, 2020 at 11:49 AM Johan Hendriks
>> >>>>>>> <joh.hendriks_at_gmail.com>
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>> I have a machine running FreeBSD head.
>> >>>>>>>> rev 13.0-CURRENT #11 r360008
>> >>>>>>>>
>> >>>>>>>> This is a quite powerful machine, so i thought it was a good
>> >>>>>>>> idea to let
>> >>>>>>>> that server do the build and for my virtualbox machine i can
>> >>>>>>>> use the
>> >>>>>>>> powerful machine to do a installword over NFS.
>> >>>>>>>>
>> >>>>>>>> But when i did the make installworld step the client so to say
>> >>>>>>>> gives an
>> >>>>>>>> error.
>> >>>>>>>>
>> >>>>>>>> install   -o root -g wheel -m 444
>> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/Zulu
>> >>>>>>>> /usr/share/zoneinfo/Zulu
>> >>>>>>>> install   -o root -g wheel -m 444
>> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/posixrules
>> >>>>>>>> /usr/share/zoneinfo/posixrules
>> >>>>>>>> install   -o root -g wheel -m 444
>> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/sort.core
>> >>>>>>>> /usr/share/zoneinfo/sort.core
>> >>>>>>>> install:
>> >>>>>>>> /usr/obj/usr/src/amd64.amd64/share/zoneinfo/builddir/sort.core:
>> >>>>>>>> Permission denied
>> >>>>>>>> *** Error code 71
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> bmake[5]: stopped in /usr/src/share/zoneinfo
>> >>>>>>>> *** Error code 1
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> bmake[4]: stopped in /usr/src/share
>> >>>>>>>> *** Error code 1
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> bmake[3]: stopped in /usr/src
>> >>>>>>>> *** Error code 1
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> bmake[2]: stopped in /usr/src
>> >>>>>>>> *** Error code 1
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> bmake[1]: stopped in /usr/src
>> >>>>>>>> *** Error code 1
>> >>>>>>>>
>> >>>>>>>> Stop.
>> >>>>>>>> make: stopped in /usr/src
>> >>>>>>>> .ERROR_TARGET='installworld'
>> >>>>>>>> .ERROR_META_FILE=''
>> >>>>>>>> .MAKE.LEVEL='0'
>> >>>>>>>> MAKEFILE=''
>> >>>>>>>> .MAKE.MODE='normal'
>> >>>>>>>> _ERROR_CMD='.PHONY'
>> >>>>>>>> .CURDIR='/usr/src'
>> >>>>>>>> .MAKE='make'
>> >>>>>>>> .OBJDIR='/usr/obj/usr/src/amd64.amd64'
>> >>>>>>>> .TARGETS='installworld'
>> >>>>>>>> DESTDIR=''
>> >>>>>>>> LD_LIBRARY_PATH=''
>> >>>>>>>> MACHINE='amd64'
>> >>>>>>>> MACHINE_ARCH='amd64'
>> >>>>>>>> MAKEOBJDIRPREFIX='/usr/obj'
>> >>>>>>>> MAKESYSPATH='/usr/src/share/mk'
>> >>>>>>>> MAKE_VERSION='20181221'
>> >>>>>>>> PATH='/sbin:/bin:/usr/sbin:/usr/bin'
>> >>>>>>>> SRCTOP='/usr/src'
>> >>>>>>>> OBJTOP='/usr/obj/usr/src/amd64.amd64'
>> >>>>>>>>
>> >>>>>>>> It looks likes sort coredumps in the usr/share/zoneinfo part of
>> >>>>>>>> the base.
>> >>>>>>>> As it has no permission on the NFS share it errors out.
>> >>>>>>>> On the server itself, the installworld goes well, but it leaves a
>> >>>>>>>> sort.core file behind in /usr/share/zoneinfo
>> >>>>>>>>
>> >>>>>>>> cd /usr/share/zoneinfo
>> >>>>>>>> ls -al
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>> _______________________________________________
>> >>>>>> freebsd-current_at_freebsd.org mailing list
>> >>>>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> >>>>>> To unsubscribe, send any mail to
>> >>>>>> "freebsd-current-unsubscribe_at_freebsd.org"
>> >>>>>>
>> >>>> _______________________________________________
>> >>>> freebsd-current_at_freebsd.org mailing list
>> >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> >>>> To unsubscribe, send any mail to
>> >>>> "freebsd-current-unsubscribe_at_freebsd.org"
>> >>>>
>> >>> --
>> >>> Rod Grimes rgrimes_at_freebsd.org
>> >>> _______________________________________________
>> >>> freebsd-current_at_freebsd.org mailing list
>> >>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> >>> To unsubscribe, send any mail to
>> >>> "freebsd-current-unsubscribe_at_freebsd.org"
>> >>>
>> I have rebuild everything on the host and did a make cleanworld.
>> Al is fine now.
>> I should have done that before i asked here.
>> Sorry to have wasted your time.
>> But we did find a Makefile that should be doing things differently.
>>
>> regards
>> Johan
>>
>>
>> _______________________________________________
>> freebsd-current_at_freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>> To unsubscribe, send any mail to "freebsd-current-unsubscribe_at_freebsd.org
>> "
>>
>
Received on Thu Apr 23 2020 - 14:49:22 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:23 UTC