Installworld fails with TMPDIR pointing to NFS mounted directory

If have this issue with 11-CURRENT on my raspberry 1 and 2, but I do not
think it is raspberry related or even 11-CURRENT related.

export TMPDIR=/media/usbdisk/tmp

make installword MAKEOBJDIRPREFIX=/media/swan/obj

Works as expected but fails cleaning up when TMPDIR points to an NFS
mounted directory:

export TMPDIR=/media/swan/tmp

The NFS server exports /media/swan which has a src/ obj/ and tmp/
subdirectory.
src/ has the sources, obj/ is filled correctly by makeworld.
The tmp dir has the correct permissions. The installworld runs till the
end, except for the last cleanup action which fails:

===> etc/sendmail (install)
cd /media/swan/src/etc/../share/man; make makedb
makewhatis /d/root11/usr/share/man
makewhatis /d/root11/usr/share/openssl/man
rm: /media/swan/tmp/install.xrgbPMy8/locale/en_US.UTF-8: Directory not empty
rm: /media/swan/tmp/install.xrgbPMy8/locale: Directory not empty
rm: /media/swan/tmp/install.xrgbPMy8: Directory not empty
*** Error code 1

Stop.
make[1]: stopped in /media/swan/src
*** Error code 1

Stop.
make: stopped in /media/swan/src

On some runs just a single error message that complains about:
/media/swan/tmp/install.xyz
not being empty, but an "ls" shows no files and an "rmdir /media/swan/tmp/
install.xyz" succeeds!
In the example above  "/media/swan/tmp/install.xrgbPMy8/locale/en_US.UTF-8"
IS empty!

It is as if a removed file remains visible for the client for a while.

The NFS server is running Ubuntu 15.10, NFSv3 is used, no other clients
access the NFS tmp directory,
no error messages on the client or server dmesg.

/etc/exports on the server:

/export/all/bsd
192.168.0.0/24(rw,no_root_squash,nohide,insecure,no_subtree_check,async)

The systems have completed many build/install world/kernel cycles using
this NFS mount and are rock solid.

Any hints would be appreciated.