Re: contrib/jemalloc

On Apr 13, 2012, at 3:43 PM, David O'Brien wrote:
> On Thu, Apr 12, 2012 at 01:19:56PM -0700, Jason Evans wrote:
>> On Apr 12, 2012, at 11:41 AM, David O'Brien wrote:
> It looks like you could run './FREEBSD-upgrade extract' from a check out
> of ssh://svn.freebsd.org/base/vendor/jemalloc.
> 
> I'm unclear what the './FREEBSD-upgrade rediff' step is for.  I'm having
> trouble following what "then regenerate diffs to update line offsets" is
> for.

The 'rediff' step regenerates the diff so that it precisely matches the differences as they apply to the code about to be imported.  If this step were omitted, patch fuzz would accumulate in FREEBSD-diffs.

> How do you handle the difference between FreeBSD having
> include/malloc_np.h and your GIT repo having
> jemalloc/include/jemalloc/jemalloc.h[.in]?
> 
> Does './FREEBSD-upgrade extract' or './FREEBSD-upgrade rediff' patch
> 'include/malloc_np.h' and 'include/stdlib.h'?
> Especially with the '#define JEMALLOC_P(s) s' wrapping.
> Is that part of the './FREEBSD-upgrade rediff' step?

stdlib.h+malloc_np.h and jemalloc.h are different enough that they require separate maintenance.  Alas, not all programming can be automated; if interfaces change, manual intervention is required.

> contrib/jemalloc/FREEBSD-upgrade doesn't describe the "commit into
> FreeBSD subversion repo" step, that I can see.  So I am not sure what
> gets committed vs. what's in git://canonware.com/jemalloc.git.

Every difference that remains in the tree afterwards is to be committed.  That might require some 'svn add' and/or 'svn rm' commands prior to 'svn commit'.

>> The only upside to vendor branch merges I can think of is
>> that if any jemalloc sources were manually modified between imports,
>> merging would fail rather than silently overwriting the changes.
> 
> I feel that is an important check-and-balance.
> 
> Not following our established procedure also means that the average
> developer(committer) and commercial consumer will have their expectations
> fails.  One expects to be able to find the stock vendor sources in
> ssh://svn.freebsd.org/base/vendor/ and to be able to find FreeBSD local
> changes to the sources thru 'svn diff' against
> ssh://obrien_at_svn.freebsd.org/base.

Regarding established procedure, I read through several of the contrib/*/FREEBSD-upgrade files when trying to understand established procedure, and the conclusion I came to is that many of the vendor imports would be painful for a new person to take over.  In contrast, contrib/jemalloc/FREEBSD-upgrade provides nearly complete automation, and documentation on how to deal with the manual parts of the process, should the need arise.

>> However, this presumes that changes aren't making it upstream.
> 
> That is not an unreasonable presumption.  Unless contrib/jemalloc/
> is locked so that you are aware of every commit FreeBSD and
> git://canonware.com/jemalloc.git can diverge at times.

Today I updated the FREEBSD-upgrade script to take care of this possibility.

	http://people.freebsd.org/~jasone/patches/jemalloc_20120413a.patch

One problem that doesn't fit neatly into the standard vendor import procedure is that jemalloc.3 is a generated file.  It isn't feasible to keep FreeBSD-specific changes out of the vendor branch unless jemalloc's build system is imported, or manual changes are made directly to the generated file (yuck).  In contrast, the FREEBSD-upgrade script leaves all non-essential files out of the imports, resulting in a cleaner import.

Imagine we're looking at the svn history three years from now, and I've croaked, leaving the FreeBSD copy of jemalloc to drift where it will.  svn might contain something like the following sequence of relevant commits:

- (jasone) Import jemalloc 7ca0fdfb85b2a9fc7a112e158892c098e004385b.
- (jasone) Import jemalloc [some other git rev].
- (jasone) Import jemalloc [some other git rev].
- (jasone) Import jemalloc 3.0.0.
- (jasone) Import jemalloc 3.0.1.
- (obrien) Fix aligned_alloc() to stop snatching helpless children.
- (jasone) Import jemalloc 3.2.0.
- (obrien) Stop protecting children; they're not as helpless as they appear.
- (obrien) Import jemalloc 3.2.1.  May jasone RIP.
- (obrien) This allocator sucks.  Gut it.

What gets lost?  As near as I can tell, the svn history tells the whole story, and 'svn blame' even reflects FreeBSD-specific changes that weren't made as part of an import commit (and FREEBSD-diffs provides a full audit trail for those).

It's worth mentioning that the less work imports are, the more often I will be able to do them.  I've put several days of effort into streamlining this process so that FreeBSD's jemalloc won't languish.  I've been working more or less full time on jemalloc for the past six weeks (thanks Facebook!), but I really have to wrap this up and get back to other projects.

Jason