Re: ACL issue (Was Re: HEADS UP: ZFSv28 is in!)

On Sun, Mar 06, 2011 at 09:43:34AM -0500, Steve Wills wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> On 03/06/11 08:35, Steve Wills wrote:
> > On 03/06/11 04:22, Edward Tomasz NapieraBa wrote:
> >> Wiadomo[ napisana przez Steve Wills w dniu 2011-03-06, o godz. 05:11:
> > 
> >> [..]
> > 
> >>> Thanks for your work on this, I'm very happy to have ZFS v28. I just
> >>> updated my -CURRENT system from a snapshot from about a month ago to
> >>> code from today. I have 3 pools and one of them is for ports tinderbox.
> >>> I only upgraded that pool. When I try to build something using
> >>> tinderbox, I get this error:
> >>>
> >>> cp: failed to set acl entries for
> >>> /usr/local/tinderbox/9-CURRENT-amd64-FreeBSD/buildscript: Operation not
> >>> supported
> > 
> >> What does "mount" show?
> > 
> > /dev/md4 12186190 332724 11853466 3%
> > /usr/local/tinderbox/9-CURRENT-amd64-FreeBSD
> > 
> > Sorry, I forgot about the mdmfs hacks I had in my local tinderd. Without
> > them, it works fine. So the problem seems to be in mfs rather than zfs.
> 
> I should have said mdmfs, but all that's doing is running mdconfig and
> newfs for me. I've reproduced the issue without mdmfs:
> 
> % mdconfig -a -t swap -s 12G -u 4
> % newfs -m 0 -o time /dev/md4
> [...]
> % mount /dev/md4 /tmp/foobar
> % cp -p /usr/local/tinderbox/scripts/lib/buildscript /tmp/foobar
> cp: failed to set acl entries for /tmp/foobar/buildscript: Operation not
> supported
> 
> Without -p it works fine. FWIW:
> 
> % getfacl /usr/local/tinderbox/scripts/lib/buildscript
> # file: /usr/local/tinderbox/scripts/lib/buildscript
> # owner: root
> # group: wheel
>             owner_at_:--------------:------:deny
>             owner_at_:rwxp---A-W-Co-:------:allow
>             group_at_:-w-p----------:------:deny
>             group_at_:r-x-----------:------:allow
>          everyone_at_:-w-p---A-W-Co-:------:deny
>          everyone_at_:r-x---a-R-c--s:------:allow
> 
> Any suggestions on where the problem could be?

At first glance it looks like acl_set_fd_np(3) isn't working on an
md-backed filesystem; specifically, it's returning EOPNOTSUPP.  You
should be able to reproduce the problem by doing a setfacl on something
in /tmp/foobar.

Looking through src/bin/cp/utils.c, this is the code:

420         if (acl_set_fd_np(dest_fd, acl, acl_type) < 0) {
421                 warn("failed to set acl entries for %s", to.p_path);
422                 acl_free(acl);
423                 return (1);
424         }

EOPNOTSUPP for acl_set_fd_np(3) is defined as:

     [EOPNOTSUPP]       The file system does not support ACL retrieval.

This would be referring to the destination filesystem.

Looking through the md(4) source for references to EOPNOTSUPP, we do
find some references:

$ egrep -n -r "EOPNOTSUPP|ENOTSUP" /usr/src/sys/dev/md
/usr/src/sys/dev/md/md.c:423:           return (EOPNOTSUPP);
/usr/src/sys/dev/md/md.c:475:                   error = EOPNOTSUPP;
/usr/src/sys/dev/md/md.c:523:           return (EOPNOTSUPP);
/usr/src/sys/dev/md/md.c:601:           return (EOPNOTSUPP);
/usr/src/sys/dev/md/md.c:731:                           error = EOPNOTSUPP;

Line 423 is within mdstart_malloc(), and it returns EOPNOTSUPP on any
BIO operation other than READ/WRITE/DELETE.  Line 475 is a continuation
of that.

Line 508 is within mdstart_vnode(), behaving effectively the same as
line 423.  Line 601 is within mdstart_swap(), behaving effectively the
same as line 423.

Line 731 is within md_kthread(), and indicates only BIO operation
BIO_GETATTR is supported.  This would not be an "ACL attribute" thing,
but rather getting attributes of the backing device itself.  The code
hints at that:

 722                 if (bp->bio_cmd == BIO_GETATTR) {
 723                         if ((sc->fwsectors && sc->fwheads &&
 724                             (g_handleattr_int(bp, "GEOM::fwsectors",
 725                             sc->fwsectors) ||
 726                             g_handleattr_int(bp, "GEOM::fwheads",
 727                             sc->fwheads))) ||
 728                             g_handleattr_int(bp, "GEOM::candelete", 1))
 729                                 error = -1;
 730                         else
 731                                 error = EOPNOTSUPP;
 732                 } else {

This leaves me with some ideas; just tossing them out here...

1. Maybe/somehow this is caused by swap being used as the backing
   type/store for md(4)?  Try using "mdconfig -t malloc -o reserve"
   instead, temporarily anyway.

2. Are you absolutely 100% sure the kernel you're using was built
   with "options UFS_ACL" defined in it?  Doing a "strings -a
   /boot/kernel/kernel | grep UFS_ACL" should suffice.

-- 
| Jeremy Chadwick                                   jdc_at_parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.               PGP 4BD6C0CB |