FreeBSD Project Quarterly Status Report - Third Quarter 2020 Introduction This report covers FreeBSD related projects for the period between July and September, and is the third of four planned reports for 2020. This quarter brings a good mix of additions and changes to the FreeBSD Project and community, from a diverse number of teams and people covering everything from architectures, continuous integration, wireless networking and drivers, over drm, desktop and third-party project work, as well as several team reports, along with many other interesting subjects too numerous to mention. As the world is still affected by the epidemic, we hope that this report can also serve as a good reminder that there is good work that can be done by people working together, even if we're apart. We hope you'll be as interested in reading it, as we've been in making it. Daniel Ebdrup Jensen, on behalf of the quarterly team. __________________________________________________________________ FreeBSD Team Reports * FreeBSD Foundation * FreeBSD Release Engineering Team * Cluster Administration Team * Continuous Integration * Ports Collection * FreeBSD Office team - 3rd quarter 2020 report * FreeBSD Graphics Team status report Projects * FreeBSD on Microsoft HyperV and Azure * Building FreeBSD on non-FreeBSD hosts * Git Migration Working Group * Linux compatibility layer update * LLDB Debugger Improvements * Lua usage in FreeBSD * NFS over TLS implementation * syzkaller on FreeBSD Kernel * DRM Drivers Update * DTS Update * DesignWare Ethernet adapter driver improvements * Google Summer of Code'20 Project - eBPF XDP Hooks * ENA FreeBSD Driver Update * IPSec Extended Sequence Number (ESN) support * NXP ARM64 SoC support * Addition of PowerPC64LE Architecture * ure - USB 3.0 Gigabit Ethernet Driver update * Stateless hardware offloads for VXLANs * Wireless updates * ZSTD Compression in ZFS Architectures * CheriBSD 2020 Q3 * FreeBSD/RISC-V Project Ports * Update to grub-bhyve * KDE on FreeBSD Documentation * DOCNG on FreeBSD Third-Party Projects * Potluck - Flavour & Image Repository for pot __________________________________________________________________ FreeBSD Team Reports Entries from the various official and semi-official teams, as found in the Administration Page. FreeBSD Foundation Contact: Deb Goodkin <deb_at_FreeBSDFoundation.org> The FreeBSD Foundation is a 501(c)(3) non-profit organization dedicated to supporting and promoting the FreeBSD Project and community worldwide. Funding comes from individual and corporate donations and is used to fund and manage software development projects, conferences and developer summits, and provide travel grants to FreeBSD contributors. The Foundation purchases and supports hardware to improve and maintain FreeBSD infrastructure and provides resources to improve security, quality assurance, and release engineering efforts; publishes marketing material to promote, educate, and advocate for the FreeBSD Project; facilitates collaboration between commercial vendors and FreeBSD developers; and finally, represents the FreeBSD Project in executing contracts, license agreements, and other legal arrangements that require a recognized legal entity. Here are some highlights of what we did to help FreeBSD last quarter: COVID-19 Impact to the Foundation Like other organizations, we put policies in place for all of our staff members to work from home. We also put a temporary ban on travel for staff members. We are continuing our work supporting the community and Project, but some of our work and responses may be delayed because of changes in some of our priorities and the impact of limited childcare for a few of our staff members. Partnerships and Commercial User Support We help facilitate collaboration between commercial users and FreeBSD developers. We also meet with companies to discuss their needs and bring that information back to the Project. Not surprisingly, the stay at home orders, combined with our company ban on travel during Q3 made in-person meetings non-existent. However, the team was able to continue meeting with our partners and commercial users virtually. These meetings help us understand some of the applications where FreeBSD is used. We are currently scheduling Zoom company meetings for Q4, please reach out if you would like to schedule a meeting with us. Fundraising Efforts Last quarter we raised $192,874.43! Thank you to the individuals and organizations that stepped in, to help fund our efforts. We'd like to thank Arm for their large contribution last quarter, which helped bring our 2020 fundraising effort to $521k. We hope other organizations will follow their lead and give back to help us continue supporting FreeBSD. These are trying times, and we deeply appreciate every donation that has come in from $5 to $150,000. We're still here giving 110% to supporting FreeBSD! We are 100% funded by donations, and those funds go towards software development work to improve FreeBSD, FreeBSD advocacy around the world, keeping FreeBSD secure, continuous integration improvements, sponsoring BSD-related and computing conferences (even the virtual events!), legal support for the Project, and many other areas. Please consider making a donation to help us continue and increase our support for FreeBSD. We also have the Partnership Program, to provide more benefits for our larger commercial donors. Find out more information about the partnership program and share with your companies! OS Improvements A number of FreeBSD Foundation grant recipients started, continued working on, or completed projects during the third quarter. These include: * Ongoing WiFi and Linux KPI layer improvements. * Linuxulator application compatibility. * DRM / Graphics driver updates. * Zstd compression for OpenZFS. * Online RAID-Z expansion. * Modernized LLDB target support for FreeBSD. You can find more details about most of these projects in other quarterly reports. Staff members also worked on a number of larger projects, including: * Run-Time Dynamic Linker (rtld) and kernel ELF loader improvements. * Rewritten UNIX domain socket locking. * Build infrastructure. * Open system call path handling support for O_BENEATH, O_RESOLVE_BENEATH. * arm64 support. * Migration to a Git repository. Many of these projects also have detailed entries in other quarterly report entries. Staff members also put in significant effort in many ways other than larger, individual projects. These include assisting with code reviews, bug report triage, security report triage and advisory handling, addressing syzkaller reports, and ongoing maintenance and bug fixes in functional areas such as the tool chain, developer tools, virtual memory kernel subsystem, low-level x86 infrastructure, sockets and protocols, and others. University of Waterloo Co-op With the transition to working from home, the Foundation decided to again take on three University of Waterloo Co-op students for the Fall 2020 term (September to December). Tiger returns for a second term, joined by new students Yang and Zac. Projects for the term include more work on ELF Tool Chain, application of Capsicum to additional utilities, testing and integration of FreePBX and Asterisk VOIP software, pkgbase, and exploring containerization tooling. Continuous Integration and Quality Assurance The Foundation provides a full-time staff member and funds projects on improving continuous integration, automated testing, and overall quality assurance efforts for the FreeBSD project. During the third quarter of 2020, Foundation staff continued improving and monitoring the Project's CI infrastructure, and working with experts to fix the failing builds and the regressions found by tests. The setting up of dedicated VM host for running tests is completed. New feature developments and the CI staging environment is in progress. We are also working with other teams in the Project for their testing needs. For example, tests of non-x86 architectures now run periodically, and improve the CI of the embedded systems. We are also working with many external projects and companies to improve the CI between their products and FreeBSD. See the FreeBSD CI section of this report for completed work items and detailed information. Supporting FreeBSD Infrastructure The Foundation provides hardware and support to improve the FreeBSD infrastructure. Last quarter, we continued supporting FreeBSD hardware located around the world. We coordinated efforts between the new NYI Chicago facility and clusteradm to start working on getting the facility prepared for some of the new FreeBSD hardware we are planning on purchasing. NYI generously provides this for free to the Project. We also worked on connecting with the new owners of the Bridgewater site, where most of the FreeBSD infrastructure is located. Some of the purchases we made for the Project last quarter to support infrastructure includes: * Spamhaus spam filtering software to limit the amount of spam on the mailing lists. * 5 application servers to run tasks like bugzilla, wiki, website, cgi, Phabricator, host git, etc. * 1 server to replace the old pkg server and provide a lot more IOPS to avoid the slowdowns seen during peak times of the day where the disks just cannot keep up with the request volume. * 1 server for exp-runs to make them faster. * 1 server to build packages more frequently. FreeBSD Advocacy and Education A large part of our efforts are dedicated to advocating for the Project. This includes promoting work being done by others with FreeBSD; producing advocacy literature to teach people about FreeBSD and help make the path to starting using FreeBSD or contributing to the Project easier; and attending and getting other FreeBSD contributors to volunteer to run FreeBSD events, staff FreeBSD tables, and give FreeBSD presentations. The FreeBSD Foundation sponsors many conferences, events, and summits around the globe. These events can be BSD-related, open source, or technology events geared towards underrepresented groups. We support the FreeBSD-focused events to help provide a venue for sharing knowledge, to work together on projects, and to facilitate collaboration between developers and commercial users. This all helps provide a healthy ecosystem. We support the non-FreeBSD events to promote and raise awareness of FreeBSD, to increase the use of FreeBSD in different applications, and to recruit more contributors to the Project. As is the case for most of us in this industry, COVID-19 has put our in-person events on hold. In addition to attending virtual events, we are continually working on new training initiatives and updating our selection of how-to guides to facilitate getting more folks to try out FreeBSD. Check out some of the advocacy and education work we did last quarter: * Launched our FreeBSD Fridays series of 101 classes. Topics included an Introduction to FreeBSD, FreeBSD Installfest, Introduction to Security, Introduction to ZFS and more. Videos of the past sessions and a schedule of upcoming events can be found here. * Attended and presented at OSI's State of the Source conference. The event was held virtually, September 9-11, 2020. * Launched the redesign of the FreeBSD Foundation Website. * Announced the 20th Anniversary of the FreeBSD Foundation. * Participated as an Admin for Google Summer of Code 2020 * Continued to promote the FreeBSD Office Hours series including holding our own Foundation led office hours. Videos from the one hour sessions can be found on the Project's YouTube Channel. You can watch ours here. * Interviewed members of the outgoing FreeBSD Core Team to get their thoughts on their term. * Began working with the FreeBSD Vendor Summit planning committee on the November 2020 Vendor Summit. * Promoted the Foundation's 20th Anniversary and our work to support the FreeBSD Project in the It's FOSS Article. FreeBSD Foundation Celebrates 20 Years of Promoting and Supporting FreeBSD Project. * Authored a Beginners Guide to FreeBSD for Fosslife. * Committed to sponsoring All Things Open as a media Sponsor. * Committed to sponsoring the OpenZFS Developers Summit at the Bronze level. * Became an International RISC-V Member. * Committed to giving a FreeBSD talk at the nerdear.la conference on October 20th. Keep up to date with our latest work in our monthly newsletters. Netflix provided an update on how and why they use FreeBSD in our latest Contributor Case Study. We help educate the world about FreeBSD by publishing the professionally produced FreeBSD Journal. As we mentioned previously, the FreeBSD Journal is now a free publication. Find out more and access the latest issues at https://www.FreeBSDfoundation.org/journal/. You can find out more about events we attended and upcoming events at https://www.FreeBSDfoundation.org/news-and-events/. Legal/FreeBSD IP The Foundation owns the FreeBSD trademarks, and it is our responsibility to protect them. We also provide legal support for the core team to investigate questions that arise. We updated our Trademark Usage Terms and Conditions on July 1, 2020. Go to the FreeBSD Foundation's web site to find out how we support FreeBSD and how we can help you! ### Other We welcomed Andrew Wafaa and Kevin Bowling to our board of directors, to help govern the Foundation and guide us with our strategic direction. We have more information about our new board members on our website. __________________________________________________________________ FreeBSD Release Engineering Team Links FreeBSD 12.2-RELEASE schedule URL: https://www.freebsd.org/releases/12.2R/schedule.html FreeBSD 12.2 test builds URL: https://www.freebsd.org/where.html#helptest FreeBSD development snapshots URL: https://download.freebsd.org/ftp/snapshots/ISO-IMAGES/ Contact: FreeBSD Release Engineering Team <re_at_FreeBSD.org> The FreeBSD Release Engineering Team is responsible for setting and publishing release schedules for official project releases of FreeBSD, announcing code freezes and maintaining the respective branches, among other things. During the third quarter of 2020, the Release Engineering Team started work on the 12.2-RELEASE cycle, the third release from the stable/12 branch. As of this writing, two BETA builds have been released, with the expectation there will be a third BETA build currently remaining on the schedule. The 12.2-RELEASE cycle will continue throughout October, with two RC builds currently planned, and RC3 scheduled on an as-needed basis. The 12.2-RELEASE is so far scheduled for final release on October 27. In addition to the 12.2-RELEASE, Glen Barber of the Release Engineering Team finished work to the release build tools and scripts to prepare for the conversion from Subversion to Git for the 13.0-RELEASE cycle. There are no plans to merge these changes to stable branches at this time; as discussed within the Git working group, we feel such a change on a stable branch would be too intrusive to our user base as well as downstream FreeBSD consumers. Development snapshot builds for 13.0-CURRENT have recently been built from the Git tree within the project, and further snapshot builds for 12.x and 11.x will continue to be built from Subversion. Additionally throughout the quarter, several development snapshots builds were released for the head, stable/12, and stable/11 branches. Finally, the Release Engineering Team would like to thank Marius Strobl for his time serving on the team; he had recently stepped down from the Deputy RE Lead role due to constraints on his time. The Team welcomes Colin Percival, who has accepted fulfilling this role. Much of this work was sponsored by Rubicon Communications, LLC (netgate.com) and the FreeBSD Foundation. __________________________________________________________________ Cluster Administration Team Links Cluster Administration Team members URL: https://www.freebsd.org/administration.html#t-clusteradm Contact: Cluster Administration Team <clusteradm_at_FreeBSD.org> The FreeBSD Cluster Administration Team consists of the people responsible for administering the machines that the Project relies on for its distributed work and communications to be synchronised. In this quarter, the team has worked on the following: * Work with the FreeBSD Foundation on hardware update for web services, mirror and package building servers. * Disable directory indexing on the package mirrors to resolve performance issues of the machine. + This was later relaxed to allow indexing of the parent directories but still disallow the large package directories. * Ongoing systems administration work: + Accounts management for committers. + Backups of critical infrastructure. + Keeping up with security updates in 3rd party software. Work in progress: * Setup Malaysia (KUL) mirror. * Setup Brazil (BRA) mirror. * Review the service jails and service administrators operation. * Infrastructure of building aarch64 and powerpc64 packages. + NVMe issues on PowerPC64 POWER9 blocking dual socket machine from being used as pkg builder. + Drive upgrade test for pkg builders (SSDs) courtesy of the FreeBSD Foundation. + Boot issues with Aarch64 reference machines. * New NYI.net sponsored colocation space in Chicago-land area. * Work with git working group for the git repository. * Searching for more providers that can fit the requirements for a generic mirrored layout or a tiny mirror. __________________________________________________________________ Continuous Integration Links FreeBSD Jenkins Instance URL: https://ci.FreeBSD.org FreeBSD Hardware Testing Lab URL: https://ci.FreeBSD.org/hwlab FreeBSD CI artifact archive URL: https://artifact.ci.FreeBSD.org FreeBSD CI weekly report URL: https://hackmd.io/_at_FreeBSD-CI FreeBSD Jenkins wiki URL: https://wiki.freebsd.org/Jenkins Hosted CI wiki URL: https://wiki.freebsd.org/HostedCI 3rd Party Software CI URL: https://wiki.freebsd.org/3rdPartySoftwareCI Tickets related to freebsd-testing_at_ URL: https://preview.tinyurl.com/y9maauwg FreeBSD CI Repository URL: https://github.com/freebsd/freebsd-ci Contact: Jenkins Admin <jenkins-admin_at_FreeBSD.org> Contact: Li-Wen Hsu <lwhsu_at_FreeBSD.org> Contact: freebsd-testing Mailing List Contact: IRC #freebsd-ci channel on EFNet The FreeBSD CI team maintains the continuous integration system of the FreeBSD project. The CI system firstly checks the committed changes can be successfully built, then performs various tests and analysis over the newly built results. The artifacts from those builds are archived in the artifact server for further testing and debugging needs. The CI team members examine the failing builds and unstable tests and work with the experts in that area to fix the codes or adjust test infrastructure. The details of these efforts are available in the weekly CI reports. During the third quarter of 2020, we continued working with the contributors and developers in the project to fulfill their testing needs and also keep collaborating with external projects and companies to improve their products and FreeBSD. Important changes: * All !x86 -test builds now trigger a new build on 22:00 UTC daily; this was not running very often because running all the tests in qemu takes lots of time. The work on improving the test execution speed and parallelism is in progress. The following is a list of the jobs affected: + Test build for FreeBSD HEAD on ARMv7. + Test build for FreeBSD HEAD on AArch64. + Test build for FreeBSD HEAD on MIPS64. + Test build for FreeBSD HEAD on PowerPC64. + Test build for FreeBSD HEAD on RISC-V64. * The build and test results will be sent to the dev-ci mailing list soon. Feedback and help with analysis is very appreciated! + A builder dedicated to run jobs using provisioned VMs is setup, this improves the stableness and reduces the execution time. + The result of FreeBSD-head-amd64-test_zfs is changed after OpenZFS importing; we encourage everyone to check and fix the failing and skipped test cases. New jobs added: * CI build for FreeBSD HEAD on PowerPC64LE. Work in progress: * Collecting and sorting CI tasks and ideas here. * Testing and merging pull requests in the the FreeBSD-ci repo. * Designing and implementing pre-commit CI building and testing, * Reduce the procedures of CI/test environment setting up for contributors and developers. * Setting up the CI stage environment and putting the experimental jobs on it. * Setting up public network access for the VM guest running tests. * Implementing automatic tests on bare metal hardware. * Adding drm ports building tests against -CURRENT. * Planning to run ztest and network stack tests. * Adding more external toolchain related jobs. * Improving the hardware lab to be more mature and adding more hardware. * Helping more 3rd software get CI on FreeBSD through a hosted CI solution. * Working with hosted CI providers to have better FreeBSD support. Please see freebsd-testing_at_ related tickets for more WIP information, and don't hesitate to join the effort! Sponsor: The FreeBSD Foundation __________________________________________________________________ Ports Collection Links About FreeBSD Ports URL: https://www.FreeBSD.org/ports/ Contributing to Ports URL: https://www.freebsd.org/doc/en_US.ISO8859-1/articles/contributing/ports -contributing.html FreeBSD Ports Monitoring URL: http://portsmon.freebsd.org/index.html Ports Management Team URL: https://www.freebsd.org/portmgr/index.html Contact: René Ladan <portmgr-secretary_at_FreeBSD.org> Contact: FreeBSD Ports Management Team <portmgr_at_FreeBSD.org> The Ports Management Team is responsible for overseeing the overall direction of the Ports Tree, building packages, and personnel matters. Below is what happened in the last quarter. We passed the landmark of 40,000 ports in the Ports Collection and are now around 40,400 ports. The last quarter saw 9335 commits to the HEAD branch and 481 commits to the 2020Q3 branch by respectively 167 and 63 committers. There are currently 2525 open problem reports of which 595 are unassigned. Compared to last quarter, this means a slight decrease in activity and also a slight increase in open PRs. During the last quarter we welcomed Rainer Hurling (rhurlin_at_) and said goodbye to Kevin Lo (kevlo_at_) and Grzegorz Blach (gblach_at_). The last three months saw new default versions for Perl (5.32), PostgreSQL (12) and PHP (7.4). Various packages also got updated: Firefox to 81.0.1, Chromium to 84.0.4147.135, Gnome to 3.36, Xorg to 1.20.9, Qt5 to 5.15.0, Emacs to 27.1, KDE Frameworks to 5.74.0 and pkg itself to 1.15.8. Never tired, antoine_at_ ran 30 exp-runs to test port version updates, on such diverse matters as: * Updating byacc in base to 20200330. * Check balancing of sed "y" command. * Use of brackets. * Removing the now redundant "port" argument from USES=readline. __________________________________________________________________ FreeBSD Office team - 3rd quarter 2020 report Links The FreeBSD Office project URL: https://wiki.freebsd.org/Office Contact: FreeBSD Office team ML <office_at_FreeBSD.org> Contact: Dima Panov <fluffy_at_FreeBSD.org> Contact: Li-Wen Hsu <lwhsu_at_FreeBSD.org> The FreeBSD Office team works on a number of office-related software suites and tools such as OpenOffice and LibreOffice. Work during this quarter focused on providing the latest stable release of LibreOffice suite and companion apps to all FreeBSD users. * Alongside with updating old stable branch to latest 6.4.x releases, current ports-tree now have a full-featured cutting-edge 7.0.1 bundle. * Conservative users can keep 6.4.x stable version by switching to use all-in-one editors/libreoffice6 port and even with i18n language pack (off by default). It will be kept updated at least till 7.1.0 version is released. We are looking for people to help the project. All unstable work with LibreOffice snapshots is staged in our WIP repository. The open bugs list contains all filed issues which need some attention. Patches, comments and objections are always welcome in the mailing list and bugzilla. __________________________________________________________________ FreeBSD Graphics Team status report Links Project GitHub page URL: https://github.com/FreeBSDDesktop Contact: FreeBSD Graphics Team <x11_at_freebsd.org> Contact: Niclas Zeising <zeising_at_freebsd.org> The FreeBSD X11/Graphics team maintains the lower levels of the FreeBSD graphics stack. This includes graphics drivers, graphics libraries such as the MESA OpenGL implementation, the X.org xserver with related libraries and applications, and Wayland with related libraries and applications. There have been several updates to the FreeBSD graphics stack and related libraries since the last report. Most notably, MESA related ports were changed to use the meson build system, instead of the autotools based one. This was needed since mesa upstream has deprecated and removed the autotools build system, and this paved the way for further mesa updates. While there was a need for a few minor corrections after the initial update, this update has been successful and made it possible to further update and improve the FreeBSD mesa port. There have also been several security fixes for xorg-server and libX11, so these ports have been updated to fix these issues. During the period, FreeBSD 12 was changed to improve the compatibility with input devices using udev/evdev and libinput. This change removes the need for local configuration and makes most mice, touchpads and keyboards work out of the box. This change will be in the upcoming FreeBSD 12.2 release. There have also been several updates to various libraries, both in the graphics and input stacks, and several userland drivers have been updated. Libraries such as libdrm and libevdev have been updated to include new FreeBSD support, developed by team members and added upstream. There has also been ongoing work to keep the various drm-kmod ports and packages up to date, mostly in response to changes in various FreeBSD versions. We have also continued our regularly scheduled bi-weekly meetings. People who are interested in helping out can find us on the x11_at_FreeBSD.org mailing list, or on our gitter chat. We are also available in #freebsd-xorg on EFNet. We also have a team area on GitHub where our work repositories can be found. __________________________________________________________________ Projects Projects that span multiple categories, from the kernel and userspace to the Ports Collection or external projects. FreeBSD on Microsoft HyperV and Azure Links Microsoft Azure article on FreeBSD wiki URL: https://wiki.freebsd.org/MicrosoftAzure Microsoft HyperV article on FreeBSD wiki URL: https://wiki.freebsd.org/HyperV Contact: FreeBSD Integration Services Team <bsdic_at_microsoft.com> Contact: Wei Hu <whu_at_FreeBSD.org> Contact: Li-Wen Hsu <lwhsu_at_FreeBSD.org> Li-Wen is working on the FreeBSD release code related to Azure for the -CURRENT, 12-STABLE and 11-STABLE branches. The work-in-progress is available here. The 11.4-RELEASE image on Azure Marketplace is published. We are testing the releng/12.2 branch and 12.2-RELEASE image will be published to Azure Marketplace soon after released. This project is sponsored by The FreeBSD Foundation, with resources provided by Microsoft. __________________________________________________________________ Building FreeBSD on non-FreeBSD hosts Links Wiki URL: https://wiki.freebsd.org/BuildingOnNonFreeBSD Contact: Alex Richardson <arichardson_at_freebsd.org> Until recently FreeBSD could only be built on a FreeBSD host. However, many popular free CI tools only allow building on Linux or macOS and therefore can not be used for building the FreeBSD base system. Furthermore, it is sometimes useful to cross-build FreeBSD for a remote machine or an emulator even if the build machine is not running FreeBSD. The goal of this project is to allow building the base system on Linux and macOS hosts. I started this project in 2017 to allow building CheriBSD on the Linux servers and desktops that many of us working on the CHERI project use. The first few patches were upstreamed in 2018 (see the 2018q3 report) and I merged the full set of patches to CheriBSD shortly after. Over the past two years I have slowly been upstreaming the remaining patches and finally committed the last required change in time for this report. As of September 2020 it should be possible to use the buildworld and buildkernel make targets to build a fully-functional FreeBSD installation on macOS and Linux hosts. We use this in our continuous integration system to build and test CheriBSD disk images for multiple architectures. I have also committed a GitHub Actions configuration upstream that takes approximately 10 minutes to build an amd64 kernel. This will ensure that changes that break crossbuilding from Linux/macOS can be detected easily. Upstreaming the crossbuilding changes has resulted in various build system cleanups. For example, we now no longer need to use lorder.sh when building libraries which speeds up the linking step a bit. The portability and bootstrapping changes should also make it easier to upgrade from older versions since we no longer rely on host headers in /usr/include matching those of the target system (e.g. when bootstrapping localedef, etc.). While this support for building on Linux and macOS should still be considered experimental, it should work in many cases. If you would like to give it a try, the following command line should successfully build an amd64 world on Linux and macOS systems that have packages for LLVM 10 (or newer) installed: MAKEOBJDIRPREFIX=/somewhere ./tools/build/make.py TARGET=amd64 TARGET_ARCH=amd64 buildworld Builds must be performed using the ./tools/build/make.py wrapper script since most Linux and macOS systems do not ship an appropriate version of bmake. Please let me know if you encounter any issues. Sponsor: DARPA __________________________________________________________________ Git Migration Working Group Links Git conversion tooling repo URL: https://github.com/freebsd/git_conv FreeBSD-git mailing list URL: https://lists.freebsd.org/mailman/listinfo/freebsd-git Beta doc git repo URL: https://cgit-beta.FreeBSD.org/doc Beta ports git repo URL: https://cgit-beta.FreeBSD.org/ports Beta src git repo URL: https://cgit-beta.FreeBSD.org/src Contact: Ed Maste <emaste_at_FreeBSD.org> Contact: Warner Losh <imp_at_FreeBSD.org> Contact: Ulrich Spörlein <uqs_at_FreeBSD.org> Work continues on FreeBSD's migration from Subversion to Git. Ulrich has addressed all known issues with svn2git and has been able to work around the inconsistent metadata and forced commit issues in the Subversion history. We still have additional documentation to write, and need to finish installing commit hooks (e.g. restricting branch creation, or ensuring appropriate data exists on cherry-pick commits). We expect to open the beta repository to test commits before the end of October. This is to allow testing of the commit hooks, and to allow developers to test access and become familiar with git operation. Commits in this repository will be deleted and the repository will be recreated at least once prior to the final migration. Those with an interest in the migration to Git are encouraged to subscribe to the FreeBSD-git mailing list and test out the beta src, ports, and/or doc repositories. You are also welcome check out the wiki, issues, README and other documentation at the Git conversion tooling repo. We currently expect to transition the src and doc repositories in mid-November. Additional investigation and experimentation with the ports repository is still underway. Sponsor: The FreeBSD Foundation (in part) __________________________________________________________________ Linux compatibility layer update Contact: Edward Tomasz Napierala <trasz_at_FreeBSD.org> Contact: Mark Johnston <markj_at_FreeBSD.org> Earlier Linuxulator work focused on code cleanups and improving diagnostic tools. Work has now shifted from cleanups to fixing actual applications. Current status is being tracked at Linux app status Wiki page. Initial focus was on applications that don't involve X11, mostly because they tend to be easier to test and debug, and the bug fixes are not application-specific. Foundation-sponsored work during this quarter included implementing a devfs(5) workaround to fix gettynam(3) inside jail/chroot, and workaround for the missing splice(2) syscall, which caused problems for grep and autotools. The Linux version reported to userspace was bumped to 3.10.0, which matches the kernel shipped with RHEL 7 and is neccessary for IBM's DB2 database installation to succeed. The BLKPBSZGET ioctl neccessary for Oracle database is supported now. There is now support for kcov(4), neccessary for syzcaller; as well as a number of fixes for issues reported by syzcaller, such as futex lock leaks. There were also more cleanups, including moving some Linuxulator-specific functionality related to error handling off from the syscall's fast code paths. The sysutils/debootstrap port, which provides an easy way to create Debian or Ubuntu jail, was updated to version 1.0.123. Finally there were some improvements to the documentation. Most of those changes have been merged to FreeBSD 12-STABLE, in order to ship with 12.2-RELEASE. There is increased involvement from other developers; this includes termios performance fixes, improved memfd support, implementing CLOCK_MONOTONIC_RAW required for Steam, madvise improvements, new compat.linux.use_emul_path sysctl. There is also ongoing work on tracking down the causes of failures related to Steam and WebKit, with fixes being first implemented in linuxulator-steam-utils. Sponsor: The FreeBSD Foundation __________________________________________________________________ LLDB Debugger Improvements Links Moritz Systems Project Description URL: https://www.moritz.systems/blog/lldb-debugger-improvements-for-freebsd/ Git Repository URL: https://github.com/moritz-systems/llvm-project Contact: Kamil Rytarowski <kamil_at_moritz.systems> Contact: Michal Górny <mgorny_at_moritz.systems> FreeBSD includes LLDB, the debugger in the LLVM family, in the base system. At present it has some limitations in comparison with the GNU GDB debugger, and does not yet provide a complete replacement. It relies on an obsolete plugin model in LLDB that causes growing technical debt. This project aims to bring LLDB closer to a fully featured replacement for GDB, and therefore for FreeBSD to feature a modern debugger for software developers. The legacy monolithic target supports the executed application being debugged in the same process space as the debugger. The modern LLDB plugin approach, used on other supported targets, executes the target process under a separate lldb-server process. This improves reliability and simplifies the process / thread model in LLDB itself. In addition, remote and local debugging will both be performed using the same approach. After the migration to the new process model is complete, the project will include reviewing the results of LLDB's test suite and fixing tests as time permits. The work is expected to be complete in 2020. The project schedule is divided into three milestones, each taking approximately one month: 1. Introduce new FreeBSD Remote Process Plugin for x86_64 with basic support and upstream to LLVM. 2. Ensure and add the mandated features in the project (process launch, process attach (pid), process attach (name), userland core files, breakpoints, watchpoints, threads, remote debugging) for FreeBSD/amd64 and FreeBSD/i386. 3. Iterate over the LLDB tests. Detect, and as time permits, fix bugs. Ensure bug reports for each non-fixed and known problem. Add missing man pages and update the FreeBSD Handbook. We are nearing the completion of the first milestone. The new plugin is getting into shape, and it can already run simple single-threaded programs. The supported features include single-stepping, breakpoints, memory and register I/O on amd64. Both plugins are supported simultaneously. The new plugin is used if FREEBSD_REMOTE_PLUGIN environment variable is set to any value, or if lldb-server is spawned directly. Otherwise, the old plugin is used for compatibility. Once the new plugin matures, we are planning to enable it unconditionally on the architectures that it is ported to. Sponsor: The FreeBSD Foundation __________________________________________________________________ Lua usage in FreeBSD Contact: Ed Maste <emaste_at_FreeBSD.org> Contact: Kyle Evans <kevans_at_FreeBSD.org> Contact: Ryan Moeller <freqlabs_at_FreeBSD.org> During this quarter, flua (FreeBSD Lua) was taught where to find base .lua modules in order to support require of .lua modules to be provided by the base system. flua also gained support for require of binary modules. A review for libjail bindings has also been submitted, pending review. libjail is an essential component if one wants to be able to write jail management utilities in flua. People interested in working with Lua in FreeBSD are welcome to get in contact to discuss other project ideas. To name a couple of potential projects, some interesting modules that have not been started but could prove useful (listed in no particular order): * libcrypt * libexpat * libnv * libxo There is also a small list of scripts that would do well with a port to flua: * certctl(8) __________________________________________________________________ NFS over TLS implementation Contact: Rick Macklem <rmacklem_at_freebsd.org> In an effort to improve NFS security, an internet draft which I expect will become an RFC soon specifies the use of TLS 1.3 to encrypt all data traffic on a Sun RPC connection used for NFS. Although NFS has been able to use sec=krb5p to encrypt data on the wire, this requires a Kerberos environment and, as such, has not been widely adopted. It also required that encryption/decryption be done in software, since only the RPC message NFS arguments are encrypted. Since Kernel TLS is capable of using hardware assist to improve performance and does not require Kerberos, NFS over TLS may be more widely adopted, once implementations are available. The coding for this project has now been completed. All required changes to the NFS and kernel RPC code have been committed to -CURRENT. The daemons are now believed to be complete, but will remain in base/projects/nfs-over-tls until -CURRENT has an OpenSSL library with the kernel TLS support incorporated in it. If this does not happen for FreeBSD-13, hopefully the patched OpenSSL and the daemons can become ports. To support clients such as laptops, the daemons that perform the TLS handshake may optionally handle client X.509 certificates from a site local CA. There are now exports(5) options to require client(s) to provide a valid X.509 certificate. While setting up system(s) for testing is still a little awkward, the documentation is now available for those who want to help with testing. The main limitation in the current implementation is that it uses TLS1.2 and not TLS1.3. This should change once the KERN_TLS rx patch includes TLS1.3 support. Third party testing would be appreciated. __________________________________________________________________ syzkaller on FreeBSD Contact: Mark Johnston <markj_at_FreeBSD.org> See the syzkaller entry in the 2019q1 quarterly report for an introduction to syzkaller. syzkaller, especially the public syzbot instance, continues to find bugs in the FreeBSD kernel. A number of these bugs have been fixed in subsystems such as the VFS name cache, the TCP and SCTP stacks, pf(4), the unix domain socket implementation, and the Linuxulator. The FreeBSD Foundation sponsored some work to enable cross-OS fuzzing. This makes it possible to fuzz the Linuxulator using syzkaller's Linux target. This effort quickly found several bugs; once the support is committed upstream we will hopefully be able to leverage syzbot to gain continuous testing of the Linux system call interface in addition to the native and 32-bit compatibility interfaces. Some work was also done to enable running syzkaller in a FreeBSD jail, with the eventual aim of making it easy to distribute binary images containing everything required to immediately start running syzkaller on a new host. Currently a number of setup steps are required, making deployment somewhat painful. Sponsor: The FreeBSD Foundation __________________________________________________________________ Kernel Updates to kernel subsystems/features, driver support, filesystems, and more. DRM Drivers Update Links drm-kmod URL: https://github.com/freebsd/drm-kmod/ Contact: Emmanuel Vadot <manu_at_FreeBSD.Org> The drm drivers for FreeBSD 13-CURRENT have been updated to match Linux 5.4.62 Then graphics/drm-current-kmod have been updated to follow this LTS release of Linux. For now graphics/drm-devel-kmod is also tracking this release but will be updated to a later revision of Linux drm drivers in the near future. A lot of linuxkpi code was removed from the ports or replaced with a BSD licenced implementation. Sponsor: The FreeBSD Foundation __________________________________________________________________ DTS Update Contact: Emmanuel Vadot <manu_at_FreeBSD.org> DTS files (Device Tree Sources) were updated to be on par with Linux 5.8 for HEAD and 5.6 for the 12-STABLE branch. __________________________________________________________________ DesignWare Ethernet adapter driver improvements Links WIP branch URL: https://github.com/gonzoua/freebsd/tree/rk_eth Contact: Oleksandr Tymoshenko <gonzo_at_FreeBSD.org> DesignWare Ethernet adapter IP is used in Rockchip and Allwinner SoCs. The driver was updated with following fixes: * Initialize clocks instead of relying on u-boot to do the right thing. * Sense media type and adjust controller configuration accordingly. * Add support for RMII PHY mode. Yet uncommitted changes include performance optimisation by adding support for multi-segment mbuf transmission. The next step is to try to get more performance boost by using interrupt coalescence. __________________________________________________________________ Google Summer of Code'20 Project - eBPF XDP Hooks Links Github diff link URL: https://github.com/Ankurk99/freebsd/tree/ebpf-import Project wiki URL: https://wiki.freebsd.org/SummerOfCode2020Projects/eBPFXDPHooksl Contact: Ankur Kothiwal <ankur_at_freebsd.org> The eBPF eXpress Data Path (XDP) allows eBPF programs to be run to filter received packets as early as possible, avoiding unnecessary processing overhead before the filter is run. The goal of this project is to extend an existing FreeBSD network driver (a virtual NIC like a VirtIO if_vtnet) to be able to call into an eBPF program when processing a newly received packet. In short, with XDP the driver must PASS (accept and process normally), DROP, TX or REDIRECT the packet as specified by the program. eBPF helper functions and maps for aiding in packet filtering will also be implemented. Implemented: * Register a eBPF probe when an interface is registered with pfil. * Activating eBPF probe. * Create hooks and link them to the pfil head when the eBPF XDP probe is activated and successfully list the XDP probes. * Create a xdp_rx function which will pass the received packets to the eBPF program where the packets can be further processed. This function will return XDP actions: DROP and PASS. * Register the xdp hook and link it to the pfil head. * Write an eBPF program to process (currently drop and pass) ICMP traffic - This is to test that the hook is working properly. * Write a loader function to load the ICMP filter program to the kernel. Future Work: * Currently we can only attach the XDP hook to PASS and DROP the packets - The work on detaching the hook is left. * The XDP action to "TX" and "REDIRECT" the packets. Final Deliverables: * Implemented XDP hook to pass and drop packets. * Created a loader program to attach the eBPF program to the kernel. * A test program to DROP ICMP filter. This code was done under the Google Summer of Code 2020 under the guidance of Ryan Stone (rstone_at_). The eBPF implementation for FreeBSD is still a work in progress and FreeBSD doesn't support eBPF yet. The basic implementation for eBPF was a GSoC'18 project, and is still under development. This project is based on that implementation so the XDP implementation for FreeBSD can only be merged into the FreeBSD source code once it supports eBPF. Currently this code is a work in progress and is merged to Ryan Stone's branch with support for the eBPF implementation. Sponsor: Google Summer of Code __________________________________________________________________ ENA FreeBSD Driver Update Links ENA README URL: https://github.com/amzn/amzn-drivers/blob/master/kernel/fbsd/ena/README Contact: Michal Krawczyk <mk_at_semihalf.com> Contact: Artur Rojek <ar_at_semihalf.com> Contact: Marcin Wojtas <mw_at_semihalf.com> ENA (Elastic Network Adapter) is the smart NIC available in the virtualized environment of Amazon Web Services (AWS). The ENA driver supports multiple transmit and receive queues and can handle up to 100 Gb/s of network traffic, depending on the instance type on which it is used. Completed since the last update: * Fix ENA compilation in case it is integrated into the kernel binary. * MFC of the ENA v2.2.0 driver to the FreeBSD 12.2. Work in progress: * Add feature that allows reading extra ENI (Elastic Network Interface) metrics about exceeding BW/pps limits. * Introduce full kernel RSS API support. * Allow reconfiguration of the RSS indirection table and hash key. * Evaluation and prototyping of the driver port to the iflib framework. Sponsor: Amazon.com Inc __________________________________________________________________ IPSec Extended Sequence Number (ESN) support Contact: Grzegorz Jaszczyk <jaz_at_semihalf.com> Contact: Patryk Duda <pdk_at_semihalf.com> Contact: Marcin Wojtas <mw_at_semihalf.com> Extended Sequence Number (ESN) is IPSec extension defined in RFC4303 Section 2.2.1. It makes possible to implement high-speed IPSec implementations where standard, 32-bit sequence number is not sufficient. A key feature of the ESN is that only low order 32 bits of sequence number are transmitted over the wire. High-order 32 bits are maintained by sender and receiver. Additionally high-order bits are included in the computation of Integrity Check Value (ICV) field. Extended Sequence Number support contains following: * Modification of existing anti-replay algorithm to fulfil ESN requirements. * Trigger soft lifetime expiration at 80% of UINT32_MAX when ESN is disabled. * Implement support for including ESN into ICV in cryptosoft engine in both encrypt and authenticate mode (eg. AES-CBC and SHA256 HMAC) and combined mode (eg. AES-GCM). * Implement support for including ESN into ICV in AES-NI engine in both encrypt and authenticate mode and combined mode. Completed since the last update: * Adjust implementation of crypto part to the reworked Open Crypto Framework. * Move the core ESN implementation from the crypto drivers to netipsec layer. * Make use of the newly introduced crp_aad mechanism for combined modes. * Introduce minor fixes and improvements. TODO: * Complete review process in Phabricator and merge patches in the tree. Sponsor: Stormshield __________________________________________________________________ NXP ARM64 SoC support Contact: Marcin Wojtas <mw_at_semihalf.com> Contact: Artur Rojek <ar_at_semihalf.com> Contact: Dawid Gorecki <dgr_at_semihalf.com> The Semihalf team initiated working on FreeBSD support for the NXP LS1046A SoC LS1046A are quad-core 64-bit ARMv8 Cortex-A72 processors with integrated packet processing acceleration and high speed peripherals including 10 Gb Ethernet, PCIe 3.0, SATA 3.0 and USB 3.0 for a wide range of networking, storage, security and industrial applications. Completed since the last update: * Upstreaming of the QorIQ SDHCI driver (r365054). With above the current Semihalf upstreaming activity is complete. The major out-of-tree supported components: * DPAA network controller support. * QSPI controller support. They work on 11.2-RELEASE, but still require significant effort to adopt to FreeBSD-CURRENT. Sponsor: Alstom Group __________________________________________________________________ Addition of PowerPC64LE Architecture Links Early notes URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-August/012043.html Announcement URL: https://lists.freebsd.org/pipermail/freebsd-ppc/2020-September/012098.h tml Contact: Brandon Bergren <bdragon_at_freebsd.org> As of r366063, experimental support for little-endian PowerPC64 (PowerPC64LE) is available in -CURRENT for POWER8 and POWER9 machines. In 2010, when FreeBSD was ported to PowerPC64, the average user would have been using a G5 PowerMac, a purely big-endian machine. While, at the time, a 32-bit PowerPC machine could run in little-endian, as well as POWER6 and POWER7, in practice, the complexities involved in managing it at the kernel level and lack of firmware support made it infeasible to support. When IBM designed POWER8, one main focus was to improve little-endian support, and bring it up to parity with big-endian. This improved support makes it practical to support a little-endian operating environment on what is traditionally a primarily big-endian platform. In 2020, with POWER9 being affordable for many users thanks to the Raptor Blackbird, semi-easy access to surplus POWER8 hardware, IBM having a major future focus on POWER little-endian, and the decay of big-endian support in modern video cards and graphical environments, there is demand for a little-endian version of FreeBSD on POWER. With FreeBSD/PowerPC64's transition in 2019 to the ELFv2 ABI as part of the 2019q4 PowerPC on Clang effort, the last major barrier to a little-endian port was eliminated. Since nobody else was working on it, and I had the skillset required to do the port, I decided to experiment one weekend with a little-endian kernel to see how difficult it would be to port. It turned out to be a lot more trivial than I was expecting. Three days later I had console support in qemu, and after another week of debugging, I had it fully up and running on hardware. FreeBSD PowerPC64LE is now an experimental MACHINE_ARCH in base, and is continuing to evolve at a rapid pace. Big-endian PowerPC64 is still the preferred platform for the foreseeable future, and will not be deprecated. Sponsor: Tag1 Consulting, Inc. __________________________________________________________________ ure - USB 3.0 Gigabit Ethernet Driver update Links svn commit: r365648 URL: https://svnweb.freebsd.org/changeset/base/365648 FreeBSD-SA-20:27.ure URL: https://www.freebsd.org/security/advisories/FreeBSD-SA-20:27.ure.asc D25809 major update to if_ure URL: https://reviews.freebsd.org/D25809 Contact: John-Mark Gurney <jmg_at_FreeBSD.org> The ure is a driver for handling the RealTek ethernet adapters, including the RTL8153 USB 3.0 Gigabit ethernet adapters. It is used in many ethernet dongles and docking stations. Previous to this update, the driver was limited in speed. In my testing, I was only able to get ~91Mbps. This limit was due to one packet per USB transfer. USB has a limit of 8000 transfers per second (1500 bytes/pkt * 8000 pkts/sec * 8bits/byte == 96 Mbps). This was acceptable for fast ethernet (RTL8152, 100Mbps), but with the additional support for Gigabit ethernet, it became a bottleneck. The updates add sending and receiving multiple packets in a single USB transfer, VLAN hardware tagging, and enable TCP and UDP checksum offloading. This increased the speed on gigabit ethernet to ~940 Mbps. In doing this work, a security vulnerability was discovered in the driver. Due to improper setting of a device register, on some devices, it caused packets to be fragmented when they shouldn't be and the driver was unable to handle them correctly. This allowed an attacker, who could generate large frames (say, ping packets, or large TCP transfers), to inject arbitrary packets into the network stack. This could allow the attacker to spoof traffic from other machines, and bypass VLAN protections. See the SA for more information. As part of this work, a script was created to run tests to validate that basic functionality of the driver (w/o options) work properly, and then iterate over each option to make sure that they function properly. This will be released at some point in the future. If you're interested in helping out, or testing it, let me know. __________________________________________________________________ Stateless hardware offloads for VXLANs Links r365867 URL: https://svnweb.freebsd.org/base?view=revision&revision=r365867 r365868 URL: https://svnweb.freebsd.org/base?view=revision&revision=r365868 r365869 URL: https://svnweb.freebsd.org/base?view=revision&revision=r365869 r365870 URL: https://svnweb.freebsd.org/base?view=revision&revision=r365870 r365871 URL: https://svnweb.freebsd.org/base?view=revision&revision=r365871 RFC6935 URL: https://tools.ietf.org/html/rfc6935 Contact: Navdeep Parhar <np_at_FreeBSD.org> VXLAN (Virtual eXtensible LAN) is a tunneling protocol in which Layer 2 traffic for a virtual LAN is encapsulated in UDP and transferred over Layer 3 networks between VTEPs (VXLAN Tunnel End Points). Traffic on the wire has two sets of networking headers: the headers for the encapsulation and the headers of the traffic being encapsulated. VXLANs are supported by if_vxlan(4) on FreeBSD. Modern NICs commonly support header checksum insertion and verification, TSO (TCP Segmentation Offload) on transmit, and RSS for load distribution on receive. But the default is to operate on the outermost headers. Some NICs can operate on the inner encapsulated frames as well. The commits listed above allow if_vxlan(4) to take advantage of such NICs. r365867 and r365868 add new mbuf checksum flags and ifnet capabilities. r365870 implements the kernel parts of the new capabilities and updates if_vxlan(4) to make use of them. r365871 implements driver support for the new capabilities in cxgbe(4). VXLAN and other tunneling protocols that use UDP explicitly allow zero checksum in the outer UDP header, even with IPv6. r365869 adds support for configuring one UDP/IPv6 port where zero checksums are allowed. This work was sponsored by Chelsio Communications and was implemented and tested using T6 (Terminator 6) NICs supported by cxgbe(4). It is available in 13.0-CURRENT (head) right now and will be available in 12-STABLE in the future. VXLANs can be created as usual and will automatically have checksum and TSO capabilities if the underlying physical interface supports VXLAN stateless offloads. Use ifconfig to list, disable, and enable checksum capabilities on the VXLAN interface. Use https://bugs.freebsd.org/bugzilla/ to report bugs. Future work: * Direct call into a vxlan input routine from the driver's receive routine. * LRO support in if_vxlan(4). * GENEVE support. Sponsor: Chelsio Communications __________________________________________________________________ Wireless updates Links The freebsd-wireless mailing list URL: https://lists.freebsd.org/mailman/listinfo/freebsd-wireless athp github repository URL: https://github.com/erikarn/athp Contact: Adrian Chadd <adrian_at_FreeBSD.org> Contact: Bjoern A. Zeeb <bz_at_FreeBSD.org> The following works happened in FreeBSD HEAD (some already in Q2) and were merged for 12.2-BETA2 and include net80211 and driver updates for better 11n and upcoming 11ac support. In more detail, this includes an ath(4) update, some run(4) 11n support, 11n for otus(4), A-MPDU, A-MSDU, A-MPDU+A-MSDU and Fast frames options, scanning fixes, enhanced PRIV checks for jails, restored parent device name printing, improvements for upcoming VHT support, lots of under-the-hood infrastructure improvements, new device IDs, and debug tools updates. If you have a chance please test before the release. Atheros 11ac driver athp In the last three months the athp(4) port of the ath10k driver has progressed well. Adrian reports the following important changes: * Per-node transmit buffering was implemented, required for correct hostap and QCA6174 behaviour. * Issues with ignoring sending some management frames got fixed; null-data frames were being filtered out and this caused undesirable hostap behaviour. * Transmit path refactoring reduced code duplication. * A fix on firmware start / VAP running tracking no longer stops the first VAP from coming active after VAP creation / ifconfig up. * Correcting hostap mode PHY configuration now allows non-VHT stations to associate and correctly exchange data with a VHT AP. * Addition of a crypto key configuration cache in the driver ensures the ieee80211_key details are available after the key is deleted; net80211 would reuse or free the state before the driver task would finish the firmware command. Newer Intel Wireless device support Initial work was done to integrate net80211 support in the LinuxKPI compat layer to get the wireless parts going. In addition, upstreaming code changes and working through problems and review started on two sides. One was trying to get mostly compile time changes upstreamed to the iwlwifi driver. The other is sorting out conflicting LinuxKPI changes to not break the DRM graphics drivers. Bjoern hopes that with some of that sorted out, he can soon go back to focus on the wireless parts and produce a new snapshot. rtw88 and brcmfmac As the Intel driver port and LinuxKPI advance, both the rtw88, and to a lower degree the brcmfmac, ports benefit from that. Bjoern lately also got a brcmfmac PCIe card and started to port support for that. This for the moment remains a free-time project. Work by Bjoern was sponsored by: Rubicon Communications, LLC (d/b/a "Netgate") and The FreeBSD Foundation __________________________________________________________________ ZSTD Compression in ZFS Contact: Allan Jude <allanjude_at_freebsd.org> Zstandard (ZSTD) is a modern high-performance compression algorithm designed to provide the compression ratios of gzip while offering much better performance. ZSTD has been adopted in FreeBSD for a number of other uses, including compressing kernel crash dumps, as a replacement for gzip or bzip for compressing log files, and for future versions of pkg(8). This effort to complete the integration of ZSTD into ZFS is funded by the FreeBSD Foundation. During the third quarter the integrating of ZSTD into OpenZFS was completed in the upstream OpenZFS repository, and the new OpenZFS 2.0 codebase was imported into 13-CURRENT. Completed milestones in this project: * Importing ZSTD 1.4.5 into OpenZFS, using the recent upstream zstd features that make it easier to embed zstd in other projects. * Changing the way compression levels are tracked and inherited. * Save and restore the compression level via an embedded block header. * Also store the version of zstd used in the embedded block header, for future-proofing. The checksum of a block may not match if zstd is upgraded, since it may compress the block more. * Add tests to ensure zstd compression and metadata survive ZFS replication. * Resolve possible negative interactions with L2ARC and ZFS Native Encryption. * Fix bug with L2ARC if the Compressed ARC feature is disabled. * Improve the ZFS feature activation code, so that zstd cannot create pools that will panic older versions of ZFS. With these changes, upgraded pools can compress data with zstd or zstd-fast, across a wide range of different compression levels. This will allow the storage administrator to select the performance-vs-compression tradeoff that best suits their needs. Tasks remaining to be completed: * Add a section to the FreeBSD Handbook ZFS chapter about zstd * Create more documentation around selecting a suitable compression level * Finish support for ZSTD in the FreeBSD boot loader (Warner Losh imp_at_freebsd.org) Sponsor: The FreeBSD Foundation __________________________________________________________________ Architectures Updating platform-specific features and bringing in support for new hardware platforms. CheriBSD 2020 Q3 Links Contact: Alex Richardson <arichardson_at_FreeBSD.org> Contact: Andrew Turner <andrew_at_FreeBSD.org> Contact: Brooks Davis <brooks_at_FreeBSD.org> Contact: Edward Tomasz Napierala <trasz_at_FreeBSD.org> Contact: George Neville-Neil <gnn_at_FreeBSD.org> Contact: Jessica Clarke <jrtc27_at_FreeBSD.org> Contact: John Baldwin <jhb_at_FreeBSD.org> Contact: Robert Watson <rwatson_at_FreeBSD.org> Contact: Ruslan Bukin <br_at_FreeBSD.org> CheriBSD extends FreeBSD to implement memory protection and software compartmentalization features supported by the CHERI instruction-set extensions. There are three architectural implementations of the CHERI protection model: CHERI-MIPS, CHERI-RISC-V, and Arm's forthcoming experimental Morello processor (due late 2021). CheriBSD is a research operating system with a stable baseline implementation into which various new research features have been, or are currently being, merged: * Arm Morello - We are preparing to open source our adaptation of CheriBSD to Arm's Morello architecture. The Morello branch is being updated to the most recent CheriBSD baseline, and patches are in review for upstreaming to our open-source repository. CheriBSD currently boots and runs statically linked CheriABI binaries on the Morello simulator, and dynamic linking support is in progress, with OS and toolchain bugs being worked on. We aim to make a first CheriBSD/Morello snapshot available alongside other open-source Morello software in mid-October 2020, however, our target for a more mature and usable implementation is December 2020. * Kernel spatial memory safety (pure-capability kernel) - The current CheriBSD kernel is a hybrid C program where only pointers to userspace are CHERI capabilities. This ensures that the kernel follows the intent of the application runtime and cannot be used to defeat bounds on application pointers. We have developed and will soon merge a pure-capability kernel where all pointers in the kernel are appropriately bounded capabilities. This vastly reduces the opportunity for buffer overflows. This spatial memory safety lays the groundwork for future work such as device driver compartmentalization and kernel temporal safety. * Userspace heap temporal memory safety (Cornucopia) - CHERI capabilities provide the necessary features to enable robust and efficient revocation of freed pointers. With Cornucopia we have implemented a light-weight revocation framework providing protection from use-after-reallocation bugs with an average cost below 2%. We aim to bring these overheads down further over the next year and merge this functionality into the mainline CheriBSD. * We have been working on updating the arm64 bhyve from Politehnica University of Bucharest to have it committed to FreeBSD. We have been upstreaming initial changes to help support this. * Baseline FreeBSD improvements - We are upstreaming (to FreeBSD) various bug fixes and tweaks for PCIe support, and support for the System MMU (SMMU) that will be present on the N1SDP and Morello SoCs. We have upstreamed support for cross-building FreeBSD from macOS and Linux (with some limitations; see separate entry on crossbuilding). We have also fixed implementation bugs in the RISC-V ABI. CHERI Documentation and Exercises * We have released [Capability Hardware Enhanced RISC Instructions: CHERI Instruction-Set Architecture (Version 8)](https://www.cl.cam.ac.uk/techreports/UCAM-CL-TR-951.pdf). Notable changes include promotion of CHERI-RISC-V to non-experimental and discussion of Arm's Morello prototype. * We have developed a set of [Adversarial CHERI Exercises and Missions](https://ctsrd-cheri.github.io/cheri-exercises) to introduce security researchers to CHERI protections. __________________________________________________________________ FreeBSD/RISC-V Project Links Wiki URL: https://wiki.freebsd.org/riscv Contact: Mitchell Horne <mhorne_at_FreeBSD.org> Contact: freebsd-riscv Mailing List Contact: IRC #freebsd-riscv on freenode The FreeBSD/RISC-V project is providing support for running FreeBSD on the RISC-V Instruction Set Architecture. This quarter saw several important bug fixes. A number of hangs in the system were identified and addressed, and a bug in QEMU's implementation of the Platform Level Interrupt Controller was fixed. This fix is included in the new devel/qemu50 and devel/qemu-devel ports. The end result of these fixes is that the test suite can now be reliably run to completion in QEMU. The entire run takes several hours, so CI has been configured to run the job once a day. There is active effort into reducing the time it takes to run the entire test suite. A new u-boot port was created: sysutils/u-boot-qemu-riscv64. This variant can be used as a secondary bootloader alongside OpenSBI to load and launch FreeBSD's loader(8) from an EFI System Partition. Next quarter will likely bring further fixes to address some of the failing test cases. __________________________________________________________________ Ports Changes affecting the Ports Collection, whether sweeping changes that touch most of the tree, or individual ports themselves. Update to grub-bhyve Links grub-bhyve Git Repository URL: https://gitlab.com/ctuffli/grub Contact: Chuck Tuffli <chuck_at_freebsd.org> bhyve is the hypervisor included in FreeBSD and other operating systems used to run virtual machines. When not using a boot ROM (i.e. UEFI), the user must load the guest operating system for bhyve. For non-FreeBSD guests, the loader is a version of GNU GRUB (a.k.a the GNU GRand Unified Bootloader) modified to interface with bhyve. This work is an effort to both update the base GRUB code to the latest version as well as improve the usability on FreeBSD. The current grub-bhyve is based on an older version of GRUB (circa 2015) and thus is missing more recent additions such as XFS file system and syslinux support. With the update, installing CentOS, for example, now does not require the extra step of changing the default file system to something other than XFS. Internally, the code has been restructured to be its own "platform" which should make it easier to keep in sync with upstream development. The major improvement is the ability to automatically find and load the GRUB configuration file from the guest disk image. With this change, it is not necessary to create a device map file or specify which Linux kernel or initrd image to use. More importantly, if the guest image updates its GRUB configuration, for example after updating the kernel, no changes are needed when invoking grub-bhyve. Note, this feature requires a new "disk" option: # grub-bhyve --disk=/zroot/vms/u18-mini/disk0.img --vm=u18-mini The automatic configuration file detection works with both GRUB configuration files (e.g. CentOS, Ubuntu) as well as syslinux configuration (e.g. Alpine). For the adventurous, there is experimental support for Fedora's BootLoaderSpec (a.k.a. blscfg) on the blscfg branch of the grub-bhyve Git repository. The code has been tested on a few Linux variants, but it would benefit from wider testing (and bug reports!). The new version does not have a Port but is easily built on FreeBSD. After cloning / downloading the source, run: $ PYTHON=python3.7 ./bootstrap $ MAKE=gmake ./configure --with-platform=bhyve $ gmake The resulting binary, grub-bhyve, will be in the grub-core/ directory. If you have success or troubles with it, please let me know. __________________________________________________________________ KDE on FreeBSD Links KDE FreeBSD URL: https://freebsd.kde.org/ KDE Community FreeBSD URL: https://community.kde.org/FreeBSD Contact: Adriaan de Groot <kde_at_FreeBSD.org> The KDE on FreeBSD project aims to package all of the software produced by the KDE Community for the FreeBSD ports tree. The software includes a full desktop environment called KDE Plasma, an IDE with the name KDevelop, a PIM suite known as Kontact and hundreds of other applications that can be used on any FreeBSD machine. With the continuation of the ever-so-peculiar era of almost-only-online, the KDE community has shifted gears and also gone for online events. The yearly conference, Akademy, was conducted online over video calls. Meanwhile, software continues to be released, so this quarter the kde_at_ team: * Put the beta of the next version of KDE Plasma, scheduled for official release in October 2020, into the Area51 development tree. Area51 is a fork of the FreeBSD ports tree where new development for KDE ports happens. * The monthly regular updates to the KDE Plasma desktop landed on-time and safely. * With three months in a quarter, there were also three releases of KDE Frameworks 5, including a new framework for handling DAV jobs. * The June applications update and its .1 release landed a bit late, but brings with it the usual raft of updates to KDE applications and libraries, * A new Digikam release, which arrived in the ports tree on the day of its release. * A new KDevelop release arrived a day after its release. This update contains a number of crash fixes for refactoring support. * Qt was updated to Qt 5.15, the last in the Qt5 series and an LTS version. Bugfix releases are expected, but the next major Qt will be Qt 6. On the infrastructure front, August saw some minor updates to CMake and ninja. As usual, kde_at_ continues to support the work of xorg_at_ and gnome_at_ in maintaining the Free Desktop stack on FreeBSD, including XOrg, poppler, and xdg-utils. A new MAINTAINER group, desktop_at_, has been created to provide shared ownership of that shared stack. With Python2 deprecation looming, the build system for QtWebEngine -- itself a fork of Chromium -- is becoming a pressing issue in Q4 and will no doubt chew up a lot of time in the coming months. __________________________________________________________________ Documentation Noteworthy changes in the documentation tree, in manpages, or in external books/documents. DOCNG on FreeBSD Links DOCNG Website Repo URL: https://gitlab.com/carlavilla/freebsd-hugo-website DOCNG Documentation Repo URL: https://gitlab.com/carlavilla/freebsd-hugo-documentation DOCNG Share Repo URL: https://gitlab.com/carlavilla/freebsd-hugo-data Contact: Sergio Carlavilla <carlavilla_at_FreeBSD.org> The Doc New Generation project aims to convert the website and all existing documentation to Hugo/AsciiDoctor. Right now almost everything is converted as you can see in the repositories. The objective of using Hugo and AsciiDoctor is to reduce the learning curve and let people to start quickly with our documentation system. Other benefits of using Hugo is that we can use other technologies aside from AsciiDoctor, like MarkDown, RST, Pandoc, etc. The remaining tasks include: * Finish the conversion of some books to AsciiDoctor. * Get some tweaks in the CSS to be responsive. * Add AsciiDoctor extensions to create an index of tables and figures. * Make a general review. The dates for making the migration have yet to be discussed. Patches, comments and objections are always welcome. __________________________________________________________________ Third-Party Projects Many projects build upon FreeBSD or incorporate components of FreeBSD into their project. As these projects may be of interest to the broader FreeBSD community, we sometimes include brief updates submitted by these projects in our quarterly report. The FreeBSD project makes no representation as to the accuracy or veracity of any claims in these submissions. Potluck - Flavour & Image Repository for pot Links Potluck Repository & Project URL: https://potluck.honeyguide.net/ Potluck on github URL: https://github.com/hny-gd/potluck pot project URL: https://pot.pizzamig.dev Contact: <> pot is a jail management tool that also supports orchestration through nomad. Potluck aims to be to FreeBSD and pot what Dockerhub is to Linux and Docker: A repository of pot flavours and complete images for usage with pot. In the last quarter, an initial set of Nomad, Consul and Traefik images has been created that are sufficient to run a simple virtual datacenter out of the box. A three-part article series explaining how to set this up is also available now. Furthermore, ready-made images suitable for scheduling via Nomad and Consul in such an environment have been created, e.g. a BackupPC or a Postfix Backup MX service. Future plans include additional images and exposing more configuration options in the existing images to allow a more flexible usage. Beside general feedback and tests, additional flavours and patches are very welcome! Sponsors: Honeyguide GmbH & Honeyguide Group (Pty) Ltd ## Puppet Puppet Puppet's FreeBSD slack channel Bolt Choria Puppet Team puppet_at_FreeBSD.org Since out last status report a few years ago, the puppet_at_ team regularly updated the various Puppet ports to follow upstream releases of Puppet 4, Puppet 5 and Puppet 6. Puppet 4 was removed when it reached EOL. More recently, an effort was made to enhance Facter 4 so that it can be used as a drop-in replacement of Facter 3 on FreeBSD. Facter 4 is a Ruby rewrite of Facter 3, the C++ rewrite of Facter 2 which was initially in Ruby. As a consequence we have two ports for Facter: sysutils/facter is the C++ implementation (Facter 3) and sysutils/rubygems-facter is the Ruby implementation (updated from Facter 2 to Facter 4 a few weeks ago). The Puppet 5 and Puppet 6 ports already allow to choose which version of Facter to use. Facter 4 will be the default version of Facter with Puppet 7 which is expected to be released soon. We are getting ready to add a port for Puppet 7 as sysutils/puppet7 when it is available, along with PuppetServer 7 (sysutils/puppetserver7), and PuppetDB 7 (databases/puppetdb7). Regarding orchestration, most Marionette Collective ports have been deprecated for a long time, and the last component sysutils/mcollective is expected to be deprecated soon: Marionette Collective was not shipped anymore with Puppet 6 and Bolt has been made available as a lightweight replacement. Bolt is already available in the ports tree as sysutils/rubygems-bolt), but if you are using Marionette Collective, you are invited to look into Choria which will reach the ports tree soon as sysutils/choria. Choria is a direct evolution of Marionette Collective allowing a smooth transition from MCollective. Once Choria is available in the ports tree, Marionette Collective will be deprecated. __________________________________________________________________ News Home | Status Home Site Map | Legal Notices | © 1995-2020 The FreeBSD Project. All rights reserved.
This archive was generated by hypermail 2.4.0 : Wed May 19 2021 - 11:41:25 UTC