Re: libXO-ification - Why - and is it a symptom of deeper issues?

Thanks for the details Allan,

To be frank , Im not expecting any warm response. It wouldn't bother me if  you would all have answered me like rabid dogs, as long i would get answers to some questions I have. 

Libxo is not horrible per se, but adoption made me painfully aware of a deep problem in BSDs, the aforementioned issue of binary code reuse. The problem is a very difficult one, indeed, Microsoft for example  worked for decades on it. Libraries, framework, ATL , COM technologies where all attempts to solve this issue on the Windows platform.  And i dare to say they didn't managed to solve it completely :P

What I wanted to see if there are developers who aknowledge that there is an issue with binary code reuse, and if there are any plans to solve this proper. (see the end of David Chisnall message )

Also, you , the developers, are in much better position than outsiders to make the FreeBSD aware of those issue, and , start some projects sponsored by the foundation. You are the insiders, you are the ones which are closer to the decisional factors in the foundation of where the money will flow.





> On 16 Nov 2015, at 19:08, Allan Jude <allanjude_at_freebsd.org> wrote:
> 
> On 2015-11-15 14:44, Dan Partelly wrote:
>>> I would welcome competing ideas/solutions, but someone would have to actually build them, not just
>> 
>>> rattle off some ideas on the mailing list.
>> 
>> Am I missing the point of a mailing list ?  it is a place to present and exchange ideas, ask why some things are the way they are , and get criticism. Rattling ideas is generally where it all starts. You cant realistically expect one to start a pervasive project like transactioanl databases for an OS configuration without not one mailing list discussion, but a lot of consultation. 
>> 
> 
> Sorry, I didn't mean to say we shouldn't talk about this. The reason for
> the less-than-warm response you got from multiple people, is that this
> is the third or fourth such discussion on the mailing list (this is what
> mailing list archives are for). Once every so many months, someone rants
> about how libxo is horrible and is ruining the base system, and wants to
> lib-ize everything. And no one every does it. Meanwhile, libxo actually
> gets worked on.
> 
>> 
>>> There are tools available to filter json/ucl output
>> yes they are. vim is one . sed is another. awk is the third …  But a pipe needs both ends to talk the same language. I can generate json as output, i can filter it  , but what tool in FreeBSD will accept it as input ? None.  A
>> 
>>> and my forthcoming uclcmd
>> 
>> What uclcmd does / will do ?  
> 
> It can read, filter, and write UCL (as well as JSON, YAML, and msgpack.
> Hopefully it will be able to do NVLists in a few months).
> 
> It is designed to be able to view, extract information from, and edit
> config files.
> 
> AsiaBSDCon Paper:
> http://allanjude.com/talks/AsiaBSDCon2015_paper_-_UCL_for_FreeBSD.pdf <http://allanjude.com/talks/AsiaBSDCon2015_paper_-_UCL_for_FreeBSD.pdf>
> 
> BSDCan Dev Summit working group:
> 	Wiki Page: https://wiki.freebsd.org/201506DevSummit/UCL <https://wiki.freebsd.org/201506DevSummit/UCL>
> 	Slides: http://www.allanjude.com/bsd/BSDCan2015_-_UCL_Working_Group.pdf <http://www.allanjude.com/bsd/BSDCan2015_-_UCL_Working_Group.pdf>
> 	Additional Slides about libucl:
> http://www.allanjude.com/bsd/cebka_BSDCan2015-UCL.pdf <http://www.allanjude.com/bsd/cebka_BSDCan2015-UCL.pdf>
> 	Video: <apparently not online yet>
> 
> BSDCan Presentation:
> 	Slides: http://allanjude.com/talks/BSDCan2015_-_UCL_for_FreeBSD.pdf <http://allanjude.com/talks/BSDCan2015_-_UCL_for_FreeBSD.pdf>
> 	Video: https://www.youtube.com/watch?v=8l6bhKIDecg <https://www.youtube.com/watch?v=8l6bhKIDecg>
> 
>> 
>>> UCL is a good solution to having a universal config file, and is better
>>> than the bespoke config files each utility has
>> 
>> 
>> I agree with you, if you manage somehow to port every ad-hoc database FreeBSD has in base to ucl, (including all the bestiary we have in /etc ) and offer tools to parse them in the rc init system to fetch the settings. I dont expect this to be done in one release , but do you tend towards this in your projects ? One config format to rule them all is good. Yet another config format in selected places in base is evil.
>> 
> 
> The goal is to change everything, and I am making progress. The working
> group at BSDCan had quite a wish list of things to convert, including a
> bunch that I figured people would prefer not to touch, like login.conf
> (I need to polish that patch and get it up for review)
> 
>>> A transactional database
>>> might be better (for some uses, likely less so for some people), but I
>>> don't hear anyone volunteering to do that work.
>> 
>> IIRC Solaris uses sqlite. Might be a decent solution. ( a lot of the ad hoc unix databases are relational databases in disguise, with unix tools acting as operators)  And if you abstract a config interface API over UCL, someone could benefit  from it by plugging in a transactional backend one day. All you would have to do is not directly plug in UCL, but work on a orthogonal abstract API. You could even  model it after UCL API.
>> Please consider it. 
>> 
> 
> For bhyve, the basic design they have in mind is UCL config files that
> are parsed and presented to the application as an nvlist, which it then
> passed between modules of bhyve (like networking, out-of-band access,
> graphics, disk (vmdk, qcow, etc)), without the base bhyve application
> having to know what config options modules actually support. It seems
> like a reasonable design to me.
> 
> I don't know that further abuse of sqlite is actually a good idea, but I
> am still a bit new at this.
> 
>> 
>>> I would welcome competing ideas/solutions, but someone would have to actually build them, not just
>> 
>>> 
>>> lib-izing all of the utilities instead of using libxo is a better
>>> solution. It would likely be gratefully accepted if someone were to do
>>> it, but most likely, no one will.
>>> 
>>> libxo exists now, and most applications can be converted very quickly
>>> (although care does need to be taken, and it sometimes has not been).
>>> 
>>> There are tools available to filter json/ucl output, namely textproc/jq
>>> and my forthcoming uclcmd
>>> 
>>> One of the major other consumers of the json/xml output of libxo, is web
>>> control panels. This is why Juniper is doing the work, and I can think
>>> of a list of other appliance vendors who would love to replace fragile
>>> per-application parsers with a json parser to extract data from various
>>> command line tools.
>>> 
>>> UCL is a good solution to having a universal config file, and is better
>>> than the bespoke config files each utility has. A transactional database
>>> might be better (for some uses, likely less so for some people), but I
>>> don't hear anyone volunteering to do that work.
>>> 
>>> So, libxo and libucl may not be the very best solutions, but they are
>>> the ones that are moving forward. I would welcome competing
>>> ideas/solutions, but someone would have to actually build them, not just
>>> rattle off some ideas on the mailing list.
>>> 
>>> -- 
>>> Allan Jude
>>> 
>> 
> 
> 
> -- 
> Allan Jude