Re: Plans for git

I've taken the time to clean up this email (which I wrote on my phone last
thing yesterday) and expand it a little.

https://bsdimp.blogspot.com/2020/09/freebsd-subversion-to-git-migration.html

is the result. I'm planning on doing a number of videos about this, as well
as blog posts, FreeBSD Journal articles and handbook updates.

Hopefully that will help explain why we're headed that way, as well as
provide resources for what to do after the cut-over, what users can expect,
etc.

Warner

On Sat, Sep 19, 2020 at 12:21 AM Warner Losh <imp_at_bsdimp.com> wrote:

>
>
> On Fri, Sep 18, 2020, 11:31 PM Zaphod Beeblebrox <zbeeble_at_gmail.com>
> wrote:
>
>> Hrm.  Maybe what I hear others saying, tho, and not entirely being
>> replied to is just a nice concise document of the why.  What I hear you
>> saying is that GIT has momentum and that it's popular... (and I accept that
>> --- it is evidently true), but then I hear handwaving about features, but
>> no list of features that are a clear win/loose.
>>
>
> Apache has switched to git from subversion. They are the current
> caretakers of subversion so it's future is very much in doubt.
>
> Git have more support for newer CI tools than subversion. This will allow
> us, once things are fully phased in, to increase the quality of the code
> going into the tree, as well as greatly reduce build breakages and
> accidental regressions.
>
> Gits merging facilities are much better than subversion. You can more
> easily curate patches as well since git supports a rebase workflow. This
> allows cleaner patches that are logical bits for larger submissions.
> Subversion can't do this.
>
> Git can easily and robustly be mirrored. Subversion can be mirrored, but
> that mirroring is far from robust. One of the snags in the git migration is
> that different svn mirrors have different data than the main repo or each
> other. Git can cryptographically sign tags and commits. Subversion cannot.
>
> Mirroring also opens up more 3rd party plug ins. Gitlab can do some
> things, Github others, in terms of automated testing and continuous
> integration. Tests can be run when branches are pushed. Both these
> platforms have significant collaboration tools as well, which will support
> groups going off and creating new features for FreeBSD.  While one can use
> these things to a limited degree, with subversion mirrored to github, the
> full power of these tools isn't fully realized without a conversion to git.
>
> All of this is before the skillset arguments are made about kids today
> just knowing git, but needing to learn subversion and how that has had
> increasing been a source of friction.  This argument is the closest one to
> being handwavy since I've not voted the data showing the growing proportion
> of projects using git (iirc, it's 85% or 90%).
>
> These are the main ones. The three down sides are lack of $FreeBSD$
> support and tags in general. Git has a weaker count of commits than
> subversion. And the BSDL git clients are less mature than the GPL'd ones.
>
> Certainly the only clear things a quick search turns up that seem relevant
>> is that GIT is GPL2.0 and SVN is Apache2.0.  This was enough for LLVM vs
>> GCC and the repository is a core function, but I suppose not a necessary
>> function for forked projects that can't abide, so...
>>
>
> OPENBSD has got, which is being ported in earnest. It has an agreeable
> license.
>
> Anyways... some concise record of thoughts might calm the gnawing
>> sensation in my gut...
>>
>
> Yes. I've started doing a series of short videos explaining the change,
> why we are doing it and what to do in the new world order. I'll be doing
> blog entries as well as turning that material into handbook entries. I have
> some of those written.
>
> Does this help?
>
> Warner
>
> On Thu, Sep 3, 2020 at 4:19 PM Warner Losh <imp_at_bsdimp.com> wrote:
>>
>>> On Thu, Sep 3, 2020 at 1:32 PM Chris <bsd-lists_at_bsdforge.com> wrote:
>>>
>>> > On 2020-09-03 11:33, Kristof Provost wrote:
>>> > > On 3 Sep 2020, at 19:56, Chris wrote:
>>> > >> Why was the intention to switch NOT announced as such MUCH sooner?
>>> > >>
>>> > > There was discussion about a possible switch to git on the
>>> freebsd-git
>>> > > mailing
>>> > > list as early as February 2017:
>>> > >
>>> >
>>> https://lists.freebsd.org/pipermail/freebsd-git/2017-February/000092.html
>>> > >
>>> > > Ed gave a talk about FreeBSD and git back in 2018:
>>> > > https://www.youtube.com/watch?v=G8wQ88d85s4
>>> > >
>>> > > The Git Transition Working group was mentioned in the quarterly
>>> status
>>> > > reports a
>>> > > year ago:
>>> > https://www.freebsd.org/news/status/report-2019-07-2019-09.html
>>> > > and
>>> > > https://www.freebsd.org/news/status/report-2019-04-2019-06.html
>>> > It appears to me that the references you make here all allude to
>>> > _investigation_
>>> > of a _possibility_ as opposed to a _likelihood_, or an _inevitable_.
>>> > IMHO a change as significant as this, which will require tossing years
>>> of
>>> > tooling
>>> > out the window, and an untold amount of _re_tooling. Should have
>>> created an
>>> > announcement at _least_ as significant as a new release gets on the
>>> mailing
>>> > lists.
>>> >
>>>
>>> Yes. We've been working on this for years. We've been working on the
>>> retooling in earnest for the last 6 months. We didn't make an
>>> announcement
>>> because we wanted to have most of the pieces in place before we did that.
>>> We've been doing the multi-year process for multiple years now. I'll also
>>> point out that only the 'git' people showed. A number of other ideas were
>>> suggested, but nothing else was stood up in a serious way (hg came the
>>> closest to setting up an exporter). Since there was really no
>>> alternatives
>>> being proposed to git, the process was less visible than if we'd had to
>>> have had a hg vs git bake off. One step has lead to the next, with much
>>> status reporting done for years.
>>>
>>> Subversion, btw, is not viable in the long run. The Apache foundation has
>>> moved all its projects from svn to git. The velocity of features with svn
>>> has diminished, and the number of CI/CT/etc tool sets that supported svn
>>> well has been dropping over time. It's also been identified as a barrier
>>> to
>>> entry for the project. Sure, some people climb the learning curve to
>>> learn
>>> and understand it, but our survey data has shown that for every one that
>>> did take the time, many others did not and simply didn't contribute.
>>>
>>> All of these issues have been discussed at length at conferences for the
>>> last 5 years at least, with increasing levels of sophistication. Had
>>> there
>>> been a BSDcan this year, I'd hoped to do a talk / BOF on this to help
>>> socialize the ideas and to get people's feedback in real time (rather
>>> than
>>> the slow and difficult process of getting it just in email).
>>>
>>> We've also talked about this in the BSD office hours and core election
>>> forums over the past year.
>>>
>>> We've been rolling out the beta git repo now for 3 months. People have
>>> found issues and we've been correcting them. We've heard from people that
>>> their automated tooling needs a bit of transition time, so we'll be
>>> exporting the stable/11 and stable/12 branches back into subversion (and
>>> the release branches) after the conversion for the life of those
>>> branches, though we don't plan on doing it for the current nor stable/13
>>> branches (due to the inefficiencies of git-svn, we need separate trees
>>> for
>>> each, and each tree takes over a day to setup). We know we need some
>>> better
>>> docs (we have some decent preliminary ones, but there's some missing bits
>>> in them, and they are too long for more casual users). We need to spell
>>> out
>>> different commit policies, how we're going to have to shift our "MFC"
>>> terminology because that's very CVS/SVN centric (in git a merge means a
>>> more specific thing than it does in svn. A cherry-pick in git is also
>>> called a merge in svn, for example). We need to document to the
>>> developers
>>> how to make the shift (this doc is mostly complete and was posted
>>> earlier,
>>> though it could use some TLC). We'd hoped to have the documents done, the
>>> policies written and vetted and have a good test system before making an
>>> announcement. The last thing I wanted was to make a big announcement,
>>> only
>>> to fail to deliver. And since each step of the process followed
>>> naturally,
>>> there wasn't a clear A/B decision point where an announcement could have
>>> been generated. We were getting close to publishing the final schedule
>>> when
>>> this thread popped up, though, since it is finally feeling like it will
>>> be
>>> real soon. I also had hoped to refine things so that existing developers
>>> and users would have only minor disruption at the cut over because things
>>> were well documented.
>>>
>>> And once we have all the basics of phase 1 (which is just going to be 'do
>>> FreeBSD's current workflow in git, making minimal changes initially),
>>> we'll
>>> start on the refinements to the workflow that will help improve the
>>> quality
>>> of code, make it easier to integrate changes, etc. There's quite the
>>> diversity of views and opinions in the larger open source world on what
>>> best practices are here, with different projects doing different things.
>>> It
>>> will take some time for the project to adopt these new tools, new work
>>> flows and realize that in some cases a diversity of tools can be an
>>> advantage rather than a liability. This may be especially true as the
>>> needs
>>> of the ports tree differ from the needs of the src tree and what's
>>> optimal
>>> for one may not work too well for the other.
>>>
>>> So a lot of my reaction in the past few days has been frustration that
>>> this
>>> came up about a week before we had all our ducks in a row to talk about
>>> it
>>> effectively and talk about the transition plans. I apologize for being so
>>> snarky about it.
>>>
>>> Warner
>>> _______________________________________________
>>> freebsd-current_at_freebsd.org mailing list
>>> https://lists.freebsd.org/mailman/listinfo/freebsd-current
>>> To unsubscribe, send any mail to "
>>> freebsd-current-unsubscribe_at_freebsd.org"
>>>
>>