Re: HEADS UP: caution required with updates using custom kernels

> Am Fri, 24 Jun 2016 15:51:11 +0000
> Brooks Davis <brooks at freebsd.org> schrieb:
>
>> On Fri, Jun 24, 2016 at 06:00:19AM +0200, O. Hartmann wrote:
>> > Am Thu, 23 Jun 2016 21:07:51 +0000
>> > Brooks Davis <brooks at freebsd.org> schrieb:
>> >
>> > > Kernel config minimalists and those running aarch64 and riscv systems will
>> > > want to head this UPDATING message.
>> > >
>> > > In practice, if you're fairly up to date, doing installworld before
>> > > installkernel will also work (I've tested that case from ALPHA4), but is
>> > > always somewhat risky.
>> > >
>> > > -- Brooks
>> > >
>> > > ----- Forwarded message from Brooks Davis <brooks at FreeBSD.org> -----
>> > >
>> > > Date: Thu, 23 Jun 2016 21:02:05 +0000 (UTC)
>> > > From: Brooks Davis <brooks at FreeBSD.org>
>> > > To: src-committers at freebsd.org, svn-src-all at freebsd.org,
>> > > 	svn-src-head at freebsd.org
>> > > Subject: svn commit: r302152 - head
>> > >
>> > > Author: brooks
>> > > Date: Thu Jun 23 21:02:05 2016
>> > > New Revision: 302152
>> > > URL: https://svnweb.freebsd.org/changeset/base/302152
>> > >
>> > > Log:
>> > >   Add an UPDATING entry for the pipe() -> pipe2() transition.
>> > >
>> > >   Approved by:	re (gjb)
>> > >   Sponsored by:	DARPA, AFRL
>> > >
>> > > Modified:
>> > >   head/UPDATING
>> > >
>> > > Modified: head/UPDATING
>> > > ==============================================================================
>> > > --- head/UPDATING	Thu Jun 23 20:59:13 2016	(r302151)
>> > > +++ head/UPDATING	Thu Jun 23 21:02:05 2016	(r302152)
>> > > _at__at_ -31,6 +31,14 _at__at_ NOTE TO PEOPLE WHO THINK THAT FreeBSD 11
>> > >  	disable the most expensive debugging functionality run
>> > >  	"ln -s 'abort:false,junk:false' /etc/malloc.conf".)
>> > >
>> > > +20160622:
>> > > +	The the libc stub for the pipe(2) system call has been replaced with
>> > > +	a wrapper which calls the pipe2(2) system call and the pipe(2) is now
>> > > +	only implemented by the kernels which include "options
>> > > +	FREEBSD10_COMPAT" in their config file (this is the default).
>> > > +	Users should ensure that this option is enabled in their kernel
>> > > +	or upgrade userspace to r302092 before upgrading their kernel.
>> > > +
>> > >  20160527:
>> > >  	CAM will now strip leading spaces from SCSI disks' serial numbers.
>> > >  	This will effect users who create UFS filesystems on SCSI disks using
>> > >
>> > >
>> > > ----- End forwarded message -----
>> >
>> > Is this showing up, when one doesn't have the expected COMPAT_FREEBSD10 in kernel and
>> > updated kernel __before___ world?:
>>
>> You must include COMPAT_FREEBSD10 or have a new userspace.  Otherwise
>> things like your shell are unlikely to work.
>>
>> -- Brooks
>
>
> How can I fix this?
>
> On two boxes, I'm like a dead man in the water now.

Having just worked out how to recover from this on a mips64 host
(edgerouter lite), here's more or less what I did (minus all the
experimentation to get to a working solution).  What follows worked
for me - but no warranty is implied or given.

My machine with the new kernel, but old user binaries, failed to
reboot properly, and was left at a single user prompt:

start_init: trying /sbin/init
pid 23 (sh), uid 0: exited on signal 12
Jun 24 18:17:54 init: /bin/sh on /etc/rc terminated abnormally, going to 
single user mode
Enter full pathname of shell or RETURN for /bin/sh:

I hit return, and got a shell, albeit one that cannot call pipe().

If you cannot get to a single-user shell on the console, I don't
have any recommendations for recovering your system.

You will need to be able to get access to the /usr/src and /usr/obj
filesystems that you compiled from.  In my case, those filesystems
are NFS mounted, so I had to get some networking going, and then
mount those filesystems.  Commands I typed are after the $ prompts:

$ ifconfig octe0 192.168.16.138/24
$ ifconfig octe0

On the ERL, the Ethernet autoconfig is a bit unstable, until you
tickle the interface with the second 'ifconfig', the interface stays
down.  You probably don't need to do the second ifconfig

$ mount -u /
$ mount /boot
$ mount /tmp

Start some daemons needed for NFS.

$ rpcbind
$ rpc.lockd
$ rpc.statd

Mount the remote filesystems.

$ mount /usr/src
$ mount /usr/obj

At this point, I have access to the new user binaries, but 'make'
does not work (it calls pipe() a lot), so let's get it working:

First, fixup /sbin/init for the next reboot.  Note the rename into
/sbin/init.bak, which is one of the backup names for init that the
kernel knows about.

$ chflags noschg /sbin/init
$ cd /usr/obj/usr/src/sbin/init
$ mv /sbin/init /sbin/init.bak
$ cp -p init /sbin/init

Next, get the new /bin/sh installed.

$ cp -p /bin/sh /bin/sh.old
$ cd /usr/obj/usr/src/bin/sh
$ cp -p sh /bin/sh.new
$ mv /bin/sh.new /bin/sh

The tricky part: atomically replace the libc.so shared library.

$ chflags noschg /lib/libc.so.7
$ cp -p /lib/libc.so.7 /lib/libc.so.7.old
$ cd /usr/obj/usr/src/lib/libc
$ cp -p libc.so.7 /lib/libc.so.7.new
$ mv /lib/libc.so.7.new /lib/libc.so.7

If you system is still responding now, you're probably going to
make it!

Fixup make next:

$ cd /usr/obj/usr/src/usr.bin/bmake
$ mv /usr/bin/make /usr/bin/make.old
$ cp -p make /usr/bin/make

At this point, 'make' should function again, so you can just
do stuff like:

$ cd /usr/src/lib && make install
$ cd /usr/src/bin && make install
$ cd /usr/src/sbin && make install
$ cd /usr/src/usr.bin && make install
$ cd /usr/src/usr.sbin && make install

If you reboot after this, you should have a mostly working system.
Doing a full 'make installworld' would probably be a good idea.

-Kurt