Re: Possible bug in NFSv4 with krb5p security?

Elias Martenson wrote:
> Thanks for the information. I was looking a bit further into the
> tcpdump
> log, and this is what happens:
> 
> Here are some relevant packets:
> 
> ==== 115 ====
> NULL call establishing a mutual context(?)
> GSS-API:
> Kerberos AP-REQ:
> Ticket: Server Name (Principal): nfs/domainname
> ==== 117 ====
> NULL reply to packet 115
> ==== 169 ====
> NFSV4 Call. OPEN DH:.../foo.txt
> Credentials:
> Procedure: RPCSEC_GSS_DATA
> GSS Context: (reference to the context created in the request from
> frame 115)
> Verifier:
> GSS Token: (some long sequence of binary data)
> GSS-API: krb5_blob (more binary data)
> ==== 170 ====
> NFSV4 Reply
> OPEN Status: NFS4ERR_ACCESS
> =========
> 
> The weird thing here is that the OPEN request in packet 169 refers to
> a
> context that was created for the principal name "nfs/domainname" (i.e.
> the
> service principal).
Normally the client will get a service ticket for "nfs/server-host.domain" using
initiator credentials that are either "user" (via their TGT) or
"nfs/client-host.domain" via its keytab entry. The important part is what the
initiator principal is. (Watching what gets logged in your KDC's log as you
do the mount and access attempt might answer this if the wireshark trace doesn't.)

> Am I correct in my understanding that this context
> is
> different from the user principal that is accessing the data?
Ok, it sounds like the Linux client might be using a "host based initiator credential"
for this request, but I'm not sure. (This is the credential where the
initiator principal was "nfs/client-host.domain".)

Most NFSv4 clients use a "host based initiator credential" in a keytab file
for system operations. (The FreeBSD client can only do this if it is patched.)
(By system operations, I am referring to NFSv4 Operations that are related to
 maintenance of lock state: SetClientID, SetClientIDConfirm, Renew, ReleaseLockOwner
 and maybe a couple of others.)
This is done so that the mount doesn't break when a user's TGT expires and also
because few Kerberos setups have a user principal for root and root is usually
doing the mounts.

Now, what I think the client should be doing is a separate Null RPC sequence
for each user (like "elias") when that user first attempts to access files
within the mount. The user must have a valid TGT at this point for it to work.
Then it should use the credential handle acquired from that exchange for file
accesses (including Opens) for that user.

If wireshark understands the RPCSEC_GSS authenticator, try and look for the
credential handle field and also look for a separate Null RPC sequence for
user principal "elias" or whoever you are logged in as when trying to open
the file.

If you are correct and the Linux client is using the credential handle it
acquired for nfs/client-host.domain, then the FreeBSD gssd would map that to
"nobody" and it would explain what your problem is. I would consider that
a Linux client bug. Linux clients have been successfully tested against
the FreeBSD server (did not do the above) in the past, but things change/break.

> And if
> so,
> where is the details regarding that principal? Is it in the "verifier"
> part
> in packet 169?
> 
There is a credential handle (I might not have the correct name since I
haven't looked at the RFC in years) which identifies which principal it
is referred to. (The same credential handle is in the reply to the Null RPC
for the RPCSEC_GSS init.) So you need to find the Null RPC sequence that
creates the particular credential handle (just bits that are the same) used
in the RPCSEC_GSS data authenticator. The principal is in the ticket inside
the GSSAPI token in the Null RPC data.

> Secondly, what if the issue is gssd not correctly mapping the
> principals to
> Unix usernames? How can I determine if this is the case. There seems
> to be
> no logging options for gssd (-d does absolutely nothing other than
> prevent
> the process from detaching. It still doesn't log anything).
> 
Yep. I added a few cases that output debugging, but they're all on the
client side. (I wasn't the original author of this gssd.)

You could easily add some. It's the function with pname_to_uid in it
that does the translation. It basically does a gss_pname_to_uid()
followed by a getpwuid() to do the translation from principal name
to uid + gid list. If this fails, then it maps uid == 65534, which
is usually "nobody". (Why does the code has 65534 hardwired in it?
I have no idea.;-)

Just add fprintf()s and run it with "-d" to see what it is doing.

If the initiator principal is "nfs/client-host.domain" it will get
mapped to "nobody" as above.

Good luck with it, rick

> Regards,
> Elias
> 
> On 13 February 2013 06:47, Rick Macklem <rmacklem_at_uoguelph.ca> wrote:
> 
> > Elias Martenson wrote:
> > > On 12 February 2013 23:20, Rick Macklem < rmacklem_at_uoguelph.ca >
> > > wrote:
> > >
> > >
> > >
> > >
> > >
> > >
> > > There is (in case you missed it on google):
> > > http://code.google.com/p/macnfsv4/wiki/FreeBSD8KerberizedNFSSetup
> > > (Nothing much has changed since FreeBSD8, except the name of the
> > > client
> > > side patch for host based initiator credentials in the keytab
> > > file.)
> > > I was hoping others would add/update the wiki and it would
> > > eventually
> > > become FreeBSD doc, but that hasn't happened.
> > >
> > >
> > >
> > >
> > > Thank you for the link. I have indeed found that, and I have
> > > followed
> > > it to the letter.
> > >
> > >
> > > I have up the exact same thing from Ubuntu machines as well as
> > > from
> > > Solaris, and I do have a fairly good understanding of Kerberos.
> > > FreeBSD however, is pretty new to me.
> > >
> > >
> > > Other than that, the various discussions in the archive on this
> > > list
> > > may help. Unfortunately, I don't know of an easy way to figure out
> > > what is busted. I always suggest looking at the packets in
> > > wireshark,
> > > but for some reason, I get the impression that folk don't like
> > > doing this? It is what I do first when I run into NFS issues.
> > >
> > >
> > >
> > > I've been looking at the dumps using Wireshark. Well, I had to
> > > drop
> > > down the security since everything is encrypted when using keb5p.
> > > I do
> > > get the same errors using sec=krb5.
> > >
> > >
> > > When looking at this trace, I see a normal OPEN request followed
> > > by a
> > > NFS4ERR_ACCESS as a reply. The Kerberos credentials are of course
> > > encrypted, so I can't really say anything about that part.
> > >
> > Well, it sounds like you are doing all the right stuff, so I don't
> > know
> > why it is returning EACCES?
> >
> > I'm not a ZFS person, so I never test that. If you have a UFS file
> > system
> > you could export for testing, that might be worth a try. ZFS likes
> > to do
> > things differently;-)
> >
> > You can look at the authentication stuff in the RPC header:
> > Actually, the credentials in the RPC header aren't encrypted,
> > although
> > they are binary data. It's been a while since I looked at the RFC,
> > but
> > the authenticator is basically:
> > - an RPCSEC_GSS version (must be 1 for FreeBSD to be able to use it)
> > - a type that will be DATA in this case
> > - a credential handle (just a blob of bits the server uses as a
> > shorthand
> >     for the principal)
> > - a sequence# used to subvert replay attempts
> >
> > Then the authentication verifier is an encrypted checksum of the
> > above,
> > that the server uses to verify it.
> >
> > All the Kerberos stuff happens via NFS Null RPCs, where the GSSAPI
> > tokens
> > are passed as data (a Null RPC has no arguments) and the credential
> > handle
> > and a session key get established. (The Kerberos ticket is inside
> > the
> > GSSAPI
> > token for the first of these Null RPCs.)
> >
> > >
> > > Note that NFS4 with Kerberos security never uses the user ID
> > > numbers.
> > > They purely use the Kerberos principals for authorisation. This is
> > > different from the default "sys" security model that blindly uses
> > > user
> > > ID's.
> > >
> > Yep, of course. The Kerberos user principal "name_at_REALM" is
> > translated
> > to a uid + gid list by the gssd via a lookup of "name" as the
> > username.
> > The uid + gid list is then associated with that credential handle I
> > mentioned
> > above.
> >
> > >
> > >
> > >
> > > > nfscbd_enable="YES"
> > > Needed for client side only, and only if delegations are
> > > enabled at the server and you want the client to acquire
> > > delegations. (Delegations are not enabled by default on the
> > > FreeBSD NFSv4 server.)
> > >
> > >
> > >
> > > Noted. Thank you. I will change this.
> > >
> > >
> > >
> > > > rpc_lockd_enable="YES"
> > > > rpc_statd_enable="YES"
> > > You shouldn't need rpc_lockd or rpc_statd for NFSv4,
> > > since they are only used for NFSv3.
> > >
> > >
> > >
> > > Good point. I'll disable those too.
> > >
> > >
> > >
> > > I don't know why a Linux client would mix NFSv3 RPCs with NFSv4
> > > ones,
> > >
> > >
> > > I was suggested to set vfs.nfsd.server_min_nfsvers to 4 in order
> > > to
> > > completely disable NFS version below 4. I did this and got rid of
> > > the
> > > stray NFS3 requests. It didn't solve the original problem though.
> > >
> > >
> > >
> > > > If anyone is able to confirm whether or not this actually has
> > > > been
> > > > tested
> > > > in 9.1-CURRENT, I'd appreciate it. Also, if not, then I'd love
> > > > to
> > > > know
> > > > where I should start looking for a solution. I'm experienced in
> > > > system
> > > > level programming (having worked on Solaris at Sun in a previous
> > > > life), but
> > > > a pointer where to start would be helpful.
> > > >
> > > Usually, when everything is being done by "nobody", it indicates
> > > that
> > > the mapping between <uid, gids> <-> name_at_domain isn't working
> > > correctly.
> > > (Looking at the packets in wireshark, you need to look at the
> > > attributes
> > > called Owner and Owner_Group to see what they are being set to.)
> > >
> > >
> > >
> > > I actually doubt this. First of all, I have the correct idmapd
> > > setup
> > > working from FreeBSD to Ubuntu (I can see that since I can see the
> > > correct user names in "ls" even though the user ID's differ). On
> > > OSX I
> > > haven't got it to work yet.
> > >
> > >
> > > But, the behaviour is the same on both systems.
> > >
> > >
> > > This is actually expected, as the permission checks are orthogonal
> > > to
> > > the ID mapping.
> > >
> > >
> > > The most common problem (since you do have nfsuserd running on the
> > > server)
> > > is for the "domain" spec to be different between client and
> > > server.
> > > FreeBSD's nfsuserd defaults to the domain part of the machine's
> > > hostname.
> > > Linux's rpc.idmapd sets the domain from /etc/idmapd.conf (at least
> > > I
> > > think
> > > that's what it is called) and many distros ship with it set to
> > > "my.domain"
> > > or something like that.
> > >
> > >
> > >
> > > Correct. I have set this correctly. I know this, since once I did,
> > > "ls" started giving me the correct user names.
> > >
> > >
> > > As such, I'd start by checking the Linux client and seeing what it
> > > has
> > > for
> > > the domain spec. in /etc/idmapd.conf.
> > >
> > > If you want to override the default for FreeBSD, there is a
> > > command
> > > line
> > > option for nfsuserd to do this. I can't remember what it is, but
> > > "man
> > > nfsuserd"
> > > will give you the answer and it can be set in /etc/rc.conf using
> > > nfsuserd_flags.
> > >
> > >
> > >
> > > Thank you. I'm definitely willing to double-check this and I am
> > > not
> > > going to claim to know exactly what's going on in the realms of
> > > NFS
> > > security. :-)
> > >
> > >
> > > If this is configured correctly, then looking at the packets in
> > > wireshark is
> > > the best starting point w.r.t. figuring out what is broken. I do
> > > limited
> > > testing of this and it works for me. I don't know how many others
> > > use
> > > it,
> > > although some definitely have fun getting it working. (Usually it
> > > is
> > > the
> > > Kerberos part on the client side that causes the most grief.)
> > >
> > >
> > >
> > > It certainly is fun. But it gets frustrating when one fights a
> > > single
> > > problem for weeks on end.
> > >
> > >
> > > Far too few shops use Kerberos though.
> > >
> > Or maybe too many;-) A public key system might not be as secure, but
> > it
> > would be far easier to manage imho.
> >
> > >
> > >
> > > Regards,
> > > Elias
> >
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe_at_freebsd.org"