Re: Jails that keep hanging around

On Sunday 15 February 2004 19:18, Robert Watson wrote:
> On Sun, 15 Feb 2004, Melvyn Sopacua wrote:
> > I have yet to figure out what triggers the bug, but I end up with
> > 'running' jails, without any processes. So I thought I'd create 'jld' to
> > remove a jail. However - prison_find isn't exported to userland.
> > Probably for good reason.
>

[ snip very informative analysis ]


> The only real exception to this is the tcpcb
> -- TCP connection remenants can last for quite a time after their socket
> exits, since they follow the TCP state machine (which has long waits).  So
> this could be it -- check netstat and see if there are any largely closed
> TCP sessions from the jails.  FYI, this is not a complete list of
> credential references, but it accounts for most of them.
>
> Are you using any services that pass references to sockets or other file
> descriptor objects in and out of the jail using UNIX domain sockets?  If
> so, that could also be it.

Mostly this happens when setting up the jails. The first one (10) with the 
jail-${IP}.idg.nl hostname, is created by a script 'makejail.sh', which is 
largely based upon the make steps descriped in jail(8). After the steps 
listed there, the jail is fired up to execute 'ports-jail.sh' inside the 
jail, which basically sets up some standard ports (portupgrade, shells, vim, 
postfix). I copy over distfiles, but it is possible, that fetch(1) came into 
play. After this script, all is done - the jail should exit.

The number 15 with the correct hostname, is me logging in and doing some 
additional work, mostly installing and configuring ports. The use of fetch(1) 
is very likely, as these ports are used inside jails only and typically don't 
exist in the host system's distfiles. I've logged out, without having stuff 
in the background.

I've tried to get the refcount, but get stuck on the proc structure - getting 
a bus error a few lines down. I couldn't find a faster road to this - if 
anybody can, I'd be happy to run some diagnostics. Probably the most 
interesting thing is to see whether the prison refcount is really 0.

My attempt below, was compiled with -D_KERNEL -lkvm.
-- 
Melvyn

=======================================================
FreeBSD sarevok.webteckies.org 5.2-CURRENT FreeBSD 5.2-CURRENT #1: Sat Feb 14 
02:48:30 CET 2004     
root_at_sarevok.webteckies.org:/usr/obj/usr/src/sys/SAREVOK_NOAPM_NODEBUG  i386
=======================================================

/* vim600: sw=4 ts=4 ai
 */
#include <sys/param.h>
#include <sys/types.h>
#include <fcntl.h>
#include <kvm.h>
#include <limits.h>
#include <sys/proc.h>
#include <sys/ucred.h>
#include <sys/jail.h>
#include <sys/sysctl.h>
#include <sys/user.h>
#include <sys/wait.h>

#include <err.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>


static struct prison *
prison_find(int prid)
{
	struct prison *pr;
	pid_t pid;
	char errbuf[_POSIX2_LINE_MAX];
	kvm_t *kern;
	struct kinfo_proc *p_procinfo;
	struct proc *p_proc;
	struct ucred *p_uc;
	int num_procs, i, status;

	pid = fork();
	pr = NULL;

	switch (pid)
	{
		case -1 :
			perror("fork()");
			break;
		case 0 :
			setsid();
			jail_attach(prid);
			sleep(60);
			exit(0);
		default:
			sleep(1); /* wait for jail_attach() */
			kern = kvm_openfiles(NULL, NULL, NULL, O_RDONLY, errbuf);
			if( kern == NULL )
			{
			   fprintf(stderr, "Cannot open kernel memory: %s", errbuf);
			   break;
			}
			p_procinfo = kvm_getprocs(kern, KERN_PROC_PID, pid, &num_procs);

			printf("Getting %d processes for pid %d and my pid is %d\n", num_procs, 
pid, getpid());
			for( i=num_procs; --i >= 0 ; ++p_procinfo )
			{
				p_proc = p_procinfo->ki_paddr;
				if( p_proc == NULL )
				{
					fprintf(stderr,
							"Cannot get proc struct from proinfo %d.\n", i);

					break;
				}
				p_uc = (struct ucred *)p_proc->p_ucred;
				if( p_uc == NULL )
				{
					fprintf(stderr,
							"Cannot get ucred struct from proc (pi %d).\n",
							i);
					break;
				}

				pr = (struct prison *)p_uc->cr_prison;
				if( pr != NULL )
					break;
			}
			kvm_close(kern);
			waitpid(pid, &status, WNOHANG);
			break;
	}

	return pr;
}

int main(int argc, char *argv[])
{
	struct prison *p;
	int jid;

	if( argc == 2 )
	{
		jid = (int)strtol(argv[1], NULL, 10);
		if( jid > 0 )
		{
			p = prison_find(jid);
			if( p == NULL )
			{
				fprintf(stderr, "Cannot find the prison!");
			}
			else
			{
				printf("Refcount is: %d\n", p->pr_ref);
			}
		}
		else
		{
			printf("Invalid jid: %d", jid);
		}
	}
	else
	{
		printf("No jid\n");
	}
	
	return 0;
}