Re: src: continued use of Subversion for getting updates

From: Warner Losh <imp_at_bsdimp.com>
Date: Fri, 25 Dec 2020 15:11:23 -0700
[[ stupid gmail sent this too fast :( ]]

On Fri, Dec 25, 2020 at 3:07 PM Warner Losh <imp_at_bsdimp.com> wrote:

>
>
> On Fri, Dec 25, 2020 at 2:41 PM Steffen Nurpmeso <steffen_at_sdaoden.eu>
> wrote:
>
>> Ulrich Spörlein wrote in
>>  <X+ZUa7NyatH2ktYI_at_acme.spoerlein.net>:
>>  |On Wed, 2020-12-23 at 15:35:45 +0100, Steffen Nurpmeso wrote:
>>  |>Jeffrey Bouquet wrote in
>>  |> <E1ks2I6-0005nh-BQ_at_rmmprod05.runbox>:
>>  |>|On Wed, 23 Dec 2020 11:13:07 +0100, Johan Hendriks
>> <joh.hendriks_at_gmail.c\
>>  |>|om> wrote:
>>  |>|> On 23/12/2020 09:49, Warner Losh wrote:
>>  |>|>> On Wed, Dec 23, 2020, 1:48 AM Graham Perrin <
>> grahamperrin_at_gmail.com> \
>>  |>|>> wrote:
>>  |> ...
>>  |>|> First of all a big thank you for all your time and effort you and
>> all
>>  |>|> the other people put in this tremendous task.
>>  |>
>>  |>Yes, it is great to have FreeBSD as a stable git repository now,
>>  ...
>>  |>I really dislike that vendor imports have been tagged.  Because
>>  |>there is only one tag namespace you cannot avoid getting all this
>>  |>cruft.  I mean, it is too late now, but one could have used
>>  |>per-vendor import branches and step them via "git rm -rf '*' &&
>>  |>tar -xf newball && git add . && git commit bla" or whatever, and
>>  |>then join them in.  It does not matter for those who have all the
>>  |
>>  |That's basically what was done? I don't understand what you're saying
>>  |here ...
>>
>> Well, cgit-beta did not have had all these tags if i recall
>> correctly, did it?  I mean it has been two months or so since
>> i last had it because "git fetch" bailed here due to the errors
>> that i have reported, and fetching more than a gigabyte for
>> brand-new fetches devastates here.
>>
>
> It had them, but not under the refs/head/vendor space but under the
> refs/vendor space.
>
> The multiple gigabyte fetch is because we changed the hashes two or three
> times in the last few months.
>
> But i _think_ all the tags below refs/tags/vendor/ like
>>
>>   vendor/wpa/2.9
>>   vendor/wpa_supplicant/0.3.8
>>   vendor/wpa_supplicant/0.5.10
>>   vendor/wpa_supplicant/0.5.11
>>   vendor/x86emu/4.6
>>   vendor/xe/1.13
>>
>> etc. did not exist in cgit-beta?  I surely would have said
>> something once comments have been requested, wouldn't i?
>>
>
> They did exist. They were under refs/vendor rather than refs/head/vendor
> though.
>
>
>> The thing is if i do
>>
>>   #?0|kent:free-src.git$ git ls-remote|wc -l
>>   From https://git.freebsd.org/src.git
>>   6814
>>
>> This is a tremendous amount of head references that need to be
>> compared.
>>
>>   #?0|kent:free-src.git$ cd ../net-src.git/
>>   #?0|kent:net-src.git$ git ls-remote|wc -l
>>   From https://github.com/NetBSD/src.git
>>   609
>>   #?0|kent:net-src.git$ cd ../open-src.git/
>>   #?0|kent:open-src.git$ git ls-remote|wc -l
>>   From https://github.com/openbsd/src.git
>>
>
>   34
>
>
These are both artificial repos, that are export-only views.


>   #?0|kent:open-src.git$ cd ../dfly-src.git/
>>   #?0|kent:dfly-src.git$ git ls-remote|wc -l
>>   From git://git.dragonflybsd.org/dragonfly.git
>>   349
>>
>
> You can twek
>

You can tweak the default refs that you pull to pull less.


> Linux is a little bit more
>>
>>   #?0|kent:linux.git$ git ls-remote|wc -l
>>   From https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
>>   7019
>>
>> but this specific repository tracks all release candidates and all
>> the update releases, which alone go into the hundreds.
>>
>> Anyhow, the difference is the number of local references that
>> effectively have to be compared against remote heads.  For example
>> my local Linux repo is this:
>>
>>   [remote "origin"]
>>   url = https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/
>>   fetch = +refs/heads/linux-4.19.y:refs/remotes/origin/linux-4.19.y
>>   fetch = +refs/heads/linux-5.10.y:refs/remotes/origin/linux-5.10.y
>>   fetch = +refs/heads/master:refs/remotes/origin/master
>>
>> And if i "sr" (show-ref) i get:
>>
>>   #?0|kent:linux.git$ git sr|wc -l
>>   925
>>
>> which is a lot due to Linux update policy, but it only relates to
>> the project itself, there is not one reference of what is
>> effectively vendor data like for example
>>
>>     Merge tag 'wireless-drivers-next-2020-12-12' of
>>     git://
>> git.kernel.org/pub/scm/linux/kernel/git/kvalo/wireless-drivers-next
>>
>> may it be in refs/tags or anywhere else, whereas with FreeBSD and
>>
>>   [remote "origin"]
>>   url = https://git.freebsd.org/src.git
>>   fetch = +refs/heads/releng/5.5:refs/remotes/origin/releng/5.5
>>   fetch = +refs/heads/releng/6.4:refs/remotes/origin/releng/6.4
>>   fetch = +refs/heads/releng/7.4:refs/remotes/origin/releng/7.4
>>   fetch = +refs/heads/releng/8.4:refs/remotes/origin/releng/8.4
>>   fetch = +refs/heads/releng/9.3:refs/remotes/origin/releng/9.3
>>   fetch = +refs/heads/releng/10.3:refs/remotes/origin/releng/10.4
>>   fetch = +refs/heads/releng/11.4:refs/remotes/origin/releng/11.4
>>   fetch = +refs/heads/releng/12.2:refs/remotes/origin/releng/12.2
>>   fetch = +refs/heads/stable/12:refs/remotes/origin/stable/12
>>   fetch = +refs/heads/main:refs/remotes/origin/main
>>
>> there is
>>
>>   #?0|kent:free-src.git$ git sr|wc -l
>>   2137
>>
>> but if i go for "the real" FreeBSD itself it is just
>>
>>   #?0|kent:free-src.git$ git sr | grep -v vendor | wc -l
>>   19
>>
>
You might be happier tracking on github, once we start pushing there as the
vendor branches won't be published there.


> namely
>>
>>   a62107ed19a1095158f454132a3b6ec536a4de7c refs/remotes/origin/main
>>   8411c9ac24aabdbbde468778b35da58dc3c15178 refs/remotes/origin/releng/10.4
>>   4adbf1f6686ffdc4c0555a018e14ec63b5981534 refs/remotes/origin/releng/11.4
>>   2120d07af09cb830873554ba5405c5d3e51b41cc refs/remotes/origin/releng/12.2
>>   d4c364b3dcf6dfe4e20c70d31912aad7e6219c14 refs/remotes/origin/releng/5.5
>>   0df48150179039ce25e09b48c85aa39bffdbb4c4 refs/remotes/origin/releng/6.4
>>   1c7fe7463d0204f08806b9da6d5d50fb7f75c5ad refs/remotes/origin/releng/7.4
>>   554a1309dbd71bb59ed4e97ea5e0bc829dad0f04 refs/remotes/origin/releng/8.4
>>   b06b7e647fe7b4eefbd29369cf0c24e1902bf72a refs/remotes/origin/releng/9.3
>>   f4d0bc6aa6b90cbb0ea6cb993d9a10e36f5f4a4c refs/remotes/origin/stable/12
>>   aed278eb8da79777d64e06bbc94ee0a018885477 refs/tags/release/10.3.0
>>   20dbcb9a99d5c4b8ceb4897f27c53ab9fb853167 refs/tags/release/11.4.0
>>   906434f95e83f03e0ac62d2544dad1656b5df1e9 refs/tags/release/12.2.0
>>   1185954e6d623ada4ef32f25687ecd5e5c003545 refs/tags/release/5.5.0
>>   44b4768239e96f6d6ba8a43cb2542e1bc83dec94 refs/tags/release/6.4.0
>>   c31c07d1a9283bd28bb00ce519dfca69e529f452 refs/tags/release/7.4.0
>>   174c77559c8cc317c743876f55f02bec233735f4 refs/tags/release/8.1.0
>>   45cf4d6a59488ea4113bde749bb61de33882cc46 refs/tags/release/8.4.0
>>   1fb916120b12fdc14c7f93aa5f14849db0efa166 refs/tags/release/9.3.0
>>
>> and thus
>>
>>   #?0|kent:free-src.git$ git sr | grep vendor | wc -l
>>   2118
>>
>> Which is a pity since all these references will be checked during
>> "git fetch" unless i am mistaken.
>
>
Yes.  So far it's been doing it quite quickly for me, but I'm decently
connected...

 |>repository, but you decided to loose one of the strengts of git,
>>  |>selective tracking.  Also i think it causes updates to require
>>  |>more network traffic, i see this with the repos i have at
>>  |>repo.or.cz, the one with few heads/tags is minimal, the other
>>  |>requires hundreds of kilobytes just for the check that happens
>>  |>many times a day.  All these references have to be compared each
>>  |>and every time.  I think.  On the other hand, a few years back
>>  |>i accidentally "heard" a discussion about improving the network
>>  |>protocol and that "head" reference matching, iirc, so it may
>>  |>change in the future.
>>  |
>>  |That's a valid point, we debated whether to keep vendor tags and
>> decided
>>  |for now to replicate what we have in SVN. We can still delete all the
>>  |vendor tags on the main repo anytime we want ...
>>
>> I personally would track that in the commit message of the import
>> on the vendor branch that anyway exists(!), and then when merging
>> this into the mainline, but not create a real tag in the tag
>> namespace.  Also the backups/ and such, because why?
>>
>
We need tags to keep track of what's been done, and to revert and do other
management things with vendor imports.

Warner
Received on Fri Dec 25 2020 - 21:11:36 UTC

This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:26 UTC