Re: Possible bug in NFSv4 with krb5p security?

Elias Martenson wrote:
> First of all, I used the "bug" word in the subject, and I'm not doing
> that
> lightly. I fully understand that the initial reaction to such claim is
> "he
> did something wrong", and frankly, that's what I'm hoping.
> 
> I've spent the last two weeks trying to get an NFS share working with
> krb5p
> security from a FreeBSD server to OSX and Ubuntu clients. I've
> followed all
> the documentation, read everything Google could find for me, asked on
> the
> IRC channel and even asked on Stackexchange, all to no avail.
> 
There is (in case you missed it on google):
http://code.google.com/p/macnfsv4/wiki/FreeBSD8KerberizedNFSSetup
(Nothing much has changed since FreeBSD8, except the name of the client
 side patch for host based initiator credentials in the keytab file.)
I was hoping others would add/update the wiki and it would eventually
become FreeBSD doc, but that hasn't happened.

Feel free to add to the wiki. All you need is a google login.

Other than that, the various discussions in the archive on this list
may help. Unfortunately, I don't know of an easy way to figure out
what is busted. I always suggest looking at the packets in wireshark,
but for some reason, I get the impression that folk don't like
doing this? It is what I do first when I run into NFS issues.

> In all my reading, something struck me as odd: Nowhere did I find any
> indication that anyone has actually set this up on 9.1-CURRENT. After
> receiving zero replies on Stackexchange I started to think that
> perhaps
> this is actually a bug.
> 
> Now, after all this talk, please let me explain what I've done. Most
> of
> this text is taken verbatim from my Stackexchange question here:
> 
> http://serverfault.com/questions/477118/permissions-are-not-taking-effect-with-kerberised-nfsv4-on-freebsd
> 
> Problem summary
> ===============
> 
> My goal is to achieve the following:
> 
> - Files served from the FreeBSD system
> - The only security model should be krb5p
> - Clients are Linux (Ubuntu) and OSX
> 
> The problem that I'm facing is that even though the Kerberos
> authentication
> works, all accesses are performed using the user "nobody".
> 
> I can see the permissions when I do "ls -l". Even the user mapping
> works
> correctly, but unless "nobody" has permission to do anything with the
> files, I get a permission denied.
> 
> Here's an example interaction from the client (Ubuntu in this case,
> but the
> same thing happens from OSX). In this example,
> /export/shared/testshare is
> the shared directory from the FreeBSD server:
> 
> (I have changed the actual domain name to `domain` and the Kerberos
> realm
> name to `REALM`)
> 
> $ kinit
> Password for elias_at_REALM:
> $ klist
> Ticket cache: FILE:/tmp/krb5cc_1000_GBjtDP
> Default principal: elias_at_REALM
> 
> Valid starting Expires Service principal
> 09/02/2013 09:40:47 10/02/2013 09:40:44 krbtgt/REALM_at_REALM
> $ sudo mount -t nfs4 -osec=krb5p,vers=4 lion:/export/shared/testshare
> /mnt
> $ ls -l /mnt
> total 4
> -rw-r--r-- 1 nobody nogroup 5 Feb 7 18:17 bar.txt
> -rw------- 1 elias nogroup 4 Feb 5 23:09 foo.txt
> $ cat /mnt/bar.txt
> blah
> $ echo foo >>/mnt/bar.txt
> bash: /mnt/bar.txt: Permission denied
> $ cat /mnt/foo.txt
> cat: /mnt/foo.txt: Permission denied
> $ klist
> Ticket cache: FILE:/tmp/krb5cc_1000_GBjtDP
> Default principal: elias_at_REALM
> 
> Valid starting Expires Service principal
> 09/02/2013 09:40:47 10/02/2013 09:40:44 krbtgt/REALM_at_REALM
> 09/02/2013 09:41:56 10/02/2013 09:40:44 nfs/lion.domain_at_REALM
> 
> Server configuration
> ====================
> 
> I have had quite some problems in finding a comprehensive guide to
> setting
> up NFSv4 on FreeBSD. This is somewhat surprising in itself as I have
> found
> that information on how to do things in FreeBSD to be very good.
> 
> Here are the relevant lines in /etc/rc.conf:
> 
> rpcbind_enable="YES"
> nfs_server_enable="YES"
> nfsv4_server_enable="YES"
> nfsuserd_enable="YES"
> nfscbd_enable="YES"
Needed for client side only, and only if delegations are
enabled at the server and you want the client to acquire
delegations. (Delegations are not enabled by default on the
FreeBSD NFSv4 server.)

> mountd_enable="YES"
> gssd_enable="YES"
> rpc_lockd_enable="YES"
> rpc_statd_enable="YES"
You shouldn't need rpc_lockd or rpc_statd for NFSv4,
since they are only used for NFSv3.

> zfs_enable="YES"
> 
> Here is the content of /etc/exports:
> 
> /export/shared/testshare -sec=krb5p
> V4: / -sec=krb5p
> 
> Another interesting aspect is that when I used `tcpdump` to record the
> NFS
> network traffic between the client and the server, I saw NFS3 packets
> together with the NFS4 packets. Both of these packet types contained
> encrypted data, so I still think Kerberos was used, but given the
> configuration above, I would have expected there to be nothing but
> NFS4
> traffic.
> 
I don't know why a Linux client would mix NFSv3 RPCs with NFSv4 ones,
but you can look at a raw packet capture done by tcpdump in wireshark.
(Unlike tcpdump, wireshark understands NFS and RPC packets, so you can
 look at what is there.) You might have been seeing rpc.lockd/rpc.statd
stuff. These are technically separate protocols from NFSv3, but might
be called NFSv3 to avoid confusion, since they are used to support NFSv3
file locking.

> If anyone is able to confirm whether or not this actually has been
> tested
> in 9.1-CURRENT, I'd appreciate it. Also, if not, then I'd love to know
> where I should start looking for a solution. I'm experienced in system
> level programming (having worked on Solaris at Sun in a previous
> life), but
> a pointer where to start would be helpful.
> 
Usually, when everything is being done by "nobody", it indicates that
the mapping between <uid, gids> <-> name_at_domain isn't working correctly.
(Looking at the packets in wireshark, you need to look at the attributes
 called Owner and Owner_Group to see what they are being set to.)

The most common problem (since you do have nfsuserd running on the server)
is for the "domain" spec to be different between client and server.
FreeBSD's nfsuserd defaults to the domain part of the machine's hostname.
Linux's rpc.idmapd sets the domain from /etc/idmapd.conf (at least I think
that's what it is called) and many distros ship with it set to "my.domain"
or something like that.

As such, I'd start by checking the Linux client and seeing what it has for
the domain spec. in /etc/idmapd.conf.

If you want to override the default for FreeBSD, there is a command line
option for nfsuserd to do this. I can't remember what it is, but "man nfsuserd"
will give you the answer and it can be set in /etc/rc.conf using nfsuserd_flags.

If this is configured correctly, then looking at the packets in wireshark is
the best starting point w.r.t. figuring out what is broken. I do limited
testing of this and it works for me. I don't know how many others use it,
although some definitely have fun getting it working. (Usually it is the
Kerberos part on the client side that causes the most grief.)

rick

> Thanks and regards,
> Elias
> _______________________________________________
> freebsd-current_at_freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to
> "freebsd-current-unsubscribe_at_freebsd.org"