Re: libXO-ification - Why - and is it a symptom of deeper issues?

On 16 Nov 2015, at 17:09, Elizabeth Myers <elizabeth_at_interlinked.me> wrote:
> 
> It seems to boil down to the golden rule: he who has the gold, makes the
> rules. Juniper wanted it, they're a non-trivial donor to the FreeBSD
> foundation and employ many devs, so they got their way.
> 
> That's all there is to it.

I think that’s a mischaracterisation.  

(Core hat on:) Juniper’s status as a donor to the FreeBSD Foundation has absolutely no baring on their ability to get code committed. The libxo code was accepted because it solves a problem that a number of FreeBSD users (and downstream consumers of FreeBSD) have.

Libucl is primarily developed by a PhD student.  He is not backed by a large corporation or an organisation that donates to the FreeBSD Foundation.  His code is accepted for precisely the same reason as libxo: it solves a problem that many people have identified is real.

Development is, however, driven by people willing to actually do the work, and being willing to listen to feedback from other developers.  If someone started committing a load of code that is only of use to them and makes life harder for everyone else, then Core would be quick to request that it be reverted.  This rarely happens, because we try hard to avoid giving commit bits to people who don’t play well with others.

Phil has put a lot of effort into libxo and, most importantly, listened to community feedback.  For example, his recent changes to libxo from feedback at BSDCam (where he led a session discussing it and related topics) means that libxo can now be used to trivially add localisation to the a load of base system utilities.  This is something that was not in the Juniper system that inspired libxo, because it is not something that they need (Juniper’s interface provides a choice between US English and US English).

This is part of the reason why Phil was recently awarded his commit bit: he isn’t just writing code that Juniper wants, he’s writing code that benefits both Juniper and the wider community and is willing to adapt it to provide wider benefits.  This is *precisely* how open source is supposed to work: Juniper benefits by (eventually) being able to reduce their diffs to upstream, everyone else benefits from having the new features, and development is led by consensus of what is useful.

(Core hat off:) I slightly disagree with Alan’s comment that librarification of base system utilities addresses the same problems.  There are three related problems:

- Being able to expose the same functionality as the base system utilities to C code.
- Being able to expose this functionality via bindings to high-level languages.
- Being able to drive complex scripting from the command line and shell scripts.

Libxo directly addresses the last of these points and inefficiently addresses the first and second.  Librarification would address the first and (possibly) the second.  They are overlapping requirements.  For some the second, the combination would likely be the best solution for a lot of requirements (i.e. have library calls that produce the JSON that Lua/Python/Ruby/JavaScript/Intercal can turn into native objects).

I would very much like to see all of the base system functionality exposed in terms of libraries, but this is a huge challenge.  Good API design is *hard*.  Tools like libucl and libxo allow people to build high-level wrappers and experiment with API design easily outside of the base system, in such a way that does not give us difficult C API compatibility requirements that we have to respect for the next few decades, and will allow us to be more informed when it comes to designing these APIs.

David