Re: 64-bit inodes (ino64) Status Update and Call for Testing

On Thu, Apr 20, 2017 at 10:43:14PM +0300, Konstantin Belousov wrote:
> Inodes are data structures corresponding to objects in a file system,
> such as files and directories. FreeBSD has historically used 32-bit
> values to identify inodes, which limits file systems to somewhat under
> 2^32 objects. Many modern file systems internally use 64-bit identifiers
> and FreeBSD needs to follow suit to properly and fully support these
> file systems.
> 
> The 64-bit inode project, also known as ino64, started life many years
> ago as a project by Gleb Kurtsou (gleb_at_).  After that time several
> people have had a hand in updating it and addressing regressions, after
> mckusick_at_ picked up and updated the patch, and acted as a flag-waver.
> 
> Sponsored by the FreeBSD Foundation I have spent a significant effort
> on outstanding issues and integration -- fixing compat32 ABI, NFS and
> ZFS, addressing ABI compat issues and investigating and fixing ports
> failures.  rmacklem_at_ provided feedback on NFS changes, emaste_at_ and
> jhb_at_ provided feedback and review on the ABI transition support. pho_at_
> performed extensive testing and identified a number of issues that
> have now been fixed.  kris_at_ performed an initial ports investigation
> followed by an exp-run by antoine_at_. emaste_at_ helped with organization
> of the process.
> 
> This note explains how to perform useful testing of the ino64 branch,
> beyond typical smoke tests.
> 
> 1. Overview.
> 
> The ino64 branch extends the basic system types ino_t and dev_t from
> 32-bit to 64-bit, and nlink_t from 16-bit to 64-bit.  The struct dirent
> layout is modified due to the larger size of ino_t, and also gains a
> d_off (directory offset) member. As ino64 implies an ABI change anyway
> the struct statfs f_mntfromname[] and f_mntonname[] array length
> MNAMELEN is increased from 88 to 1024, to allow for longer mount path
> names.
> 
> ABI breakage is mitigated by providing compatibility using versioned
> symbols, ingenious use of the existing padding in structures, and by
> employing other tricks.  Unfortunately, not everything can be fixed,
> especially outside the base system.  For instance, third-party APIs
> which pass struct stat around are broken in backward and forward-
> incompatible way.
> 
> 2. Motivation.
> 
> The main risk of the ino64 change is the uncontrolled ABI breakage.
> Due to expansion of the basic types dev_t, ino_t and struct dirent,
> the impact is not limited to one part of the system, but affects:
> - kernel/userspace interface (syscalls ABI, mostly stat(2), kinfo
>   and more)
> - libc interface (mostly related to the readdir(3), FTS(3))
> - collateral damage in other libraries that happens to use changed types
>   in the interfaces.  See, for instance, libprocstat, for which compat
>   was provided using symbol versioning, and libutil, which shlib version
>   was bumped.
> 
> 3. Quirks.
> 
> We handled kinfo sysctl MIBs, but other MIBs which report structures
> depended on the changed type, are not handled in general.  It was
> considered that the breakage is either in the management interfaces,
> where we usually allow ABI slip, or is not important.
> 
> Struct xvnode changed layout, no compat shims are provided.
> 
> For struct xtty, dev_t tty device member was reduced to uint32_t.  It
> was decided that keeping ABI compat in this case is more useful than
> reporting 64bit dev_t, for the sake of pstat.
> 
> 4. Testing procedure.
> 
> The ino64 project can be tested by cloning the project branch from
> GitHub or by applying the patch <from the Phabricator review | located
> at URL | attached> to a working tree.  The authorative source is the
> GitHub, I do not promise to update the review for each update.
> 
> To clone from GitHub:
> % git clone -b ino64 https://github.com/FreeBSDFoundation/freebsd.git ino64
> 
> To fetch the patch from Phabricator:
> - Visit https://reviews.freebsd.org/D10439
> - Click "Download Raw Diff" at the upper right of the page
> 
> Or
> % arc patch D10439
> 
> After that, in the checkout directory do
> % (cd sys/kern && touch syscalls.master && make sysent)
> % (cd sys/compat/freebsd32 && touch syscalls.master && make sysent)
> If you use custom kernel configuration, ensure that
> 	options COMPAT_FREEBSD11
> is included into the config.  Then build world and kernel in the
> usual way, install kernel, reboot, install new world.  Do not make
> shortcuts in the update procedure.

Hey Kostik,

On my HardenedBSD system, world compiled fine with the patch, but the
kernel didn't compile. I've uploaded the log to:

https://hardenedbsd.org/~shawn/2017-05-15_ino64-r01.log

This was from importing the patch from Phabricator into
hardened/current/master in HardenedBSD.

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

GPG Key ID:          0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE