Finder’s Nasty Inherited ACL Bug (aka Error -41)

Finder’s Inherited ACL handling is broken

Support for inherited ACLs on folders is still in disarray in 10.6.7 (and has been since 10.6.5), there have been a few reports here and there, some mentioning Error -41, other saying it was AFP, but I’ve whittled it down and it’s a Finder flaw handling inherited ACLs that was introduced in 10.6.3!

Hands On Demo

Let’s create a folder in your home directory using Terminal to make a nice little nested folder set to play with :
mkdir -p ~/ACLShackles/1/2/3/4/5/6/7/8/9/10/11

Now let’s add an entry to the folder’s ACL, in this case  inherited folder and file Read & Write permissions, a common setup for workgroups that share a folder. OS X’s Server Admin sets inheritance by default when adding a user to an ACL, to do this with OS X Client you must use chmod:

chmod +a "$(whoami) allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit" ~/ACLShackles/

Make sure the above is all on one line and paste into Terminal.
Now, that the permissions are set double check with: ls -led ~/ACLShackles

Now duplicate folder structure we made called “1” with Command-D, now try and duplicate “1 copy”  — it fails. Finder Error -41.

Looking at the ACL in Terminal isn’t pretty:
ls -leR /Users/brunerj/ACLShackles/1\ copy\ 2

It goes on to 108…

If you don’t like Terminal, I’ve got an Applescript for you: Error41.zip

Just download and, open the resulting Error41.dmg, open and run Error41.scpt to get this result:

Step-by-step illustrated Demo

Let’s try another example and illustrate.

Now, let’s copy in Chess.app, take a look at the permissions with the Inspector (Command-Option-I), notice the Permissions (for this example I added a different user in the ACL to avoid confusion between the Unix permission and the ACL when looking at Finder’s Get Info, it’s the same silhouette):

Looks good, and how it should. Now duplicate Chess.app within this folder with a Command-D:

Uh-oh. Something’s not right. It’s duplicated the original ACE (Access Control Entry) and added one for me the duplicator that seems to mimic my Unix permissions.

Copy the copy.

Much worse. It’s now duplicating my erroneous ACE and the ACE of the test user as well.

Copy the copy.

Oh dear. Fail. Error -41. Checkmate.

Let’s look at that third failed and truncated copy in Finder

Care to see how long an ACL is inside the app?

Yes. 110 ACEs are on “Chess copy 3.app/Contents”!

Comparing other file manager’s behavior

cp doesn’t suffer the same fate as Finder when making copies of copies.

This simulates Finder’s copy of a copy, delete any existing duplicate folders and try this:

cp -Rp 1 1\ copy; cp -Rp 1\ copy 1\ copy\ 2; cp -Rp 1\ copy\ 2 1\ copy\ 3; cp -Rp 1\ copy\ 3 1\ copy\ 4

Take a look inside with ls -leR ~/ACLShackles, nice and clean, one ACE per ACL on each folder the way it should be.

Rixstep’s XFile – doesn’t exhibit this behavior when copying or duplicating either, Rixstep software is quite conscientious about doing the Right Thing™

Path Finder however duplicates ACLs just like Finder, but instead of Error -41 when an ACL gets too long and deep it just hangs instead. What’s interesting is what calls is PathFinder using? They have an SDK but have only taken a cursory look, regardless it’s the same result as Finder.

In conclusion…

For OS X Server environments, this affects crucial workflow behavior where multiple people act upon the same files and folders. ACLs quickly stack up and render Finder unusable. Currently the last known proper behavior for Finder was 10.6.2. Then 10.6.3 and 10.6.4, added the quirk of  adding an ACL entry that mimicked the UNIX permissions of the user doing the copy operation, but at least the ACEs weren’t duplicated ad nauseum. But taking the insanity to new heights was 10.6.5, 10.6.6, and now 10.6.7 with the duplication bug that makes working with inherited permissions unbearable. Luckily for OS X Client, this is minimized since Finder does not enable Inheritance for files or folders, but then you lose out on what inherited ACLs could do for you and your workgroup!

Since 10.6.7 is out now, 10.6.8 might be your last chance to have Snow Leopard’s Finder ACL behavior put back in working order. Let Apple know this is important to you, file a bug report at Apple’s Bug Reporter site, you can reference my bug number: 9160099 (you can view it at OpenRadar )

P.S. If you are wondering why I didn’t file this earlier since I am so detailed in the earlier 10.6 behavior? I actually became aware of something wrong in 10.6.5 but I also wasn’t sure what the cause was, Client, Server, AFP, Active Direcroty? Our UIDs are 10 digits long! During that time I hoped the soon to be released 10.6.6 would fix it, but when it didn’t I spent a lot of time just trying to fix it and clean the ACLs, it wasn’t until last week during the final seed of 10.6.7 that I found time to set up a new server and incrementally test 10.6, 10.6.2, 10.6.3, 10.6.4, 10.6.5, 10.6.6, and 10.6.7, recording QT Screen captures for each one and that took some time! My bug report only got in this Sunday. Oh well better late than never! :/

Remedies

It’s easy to nuke all the ACLs in a given folder, but then you’ll probably want them set back up again to be of any use! Anyway this will recursively wipe the ACLs from everything in the folder you’re in, be careful, you could wipe your own access! So it’s best for the server admin to to run from the server itself, then re-propagate the ACLs for the Share (actually you may not be able to delete ACLs from client anyway):

chmod -RN *

Update (3/23/11):

So I left my ACLShackles folder in my home foder and Time Machine failed to backup (backupd error -41, surprise.) so to avoid that, delete ACLShackles and/or turn off Time Machine while you play around with this…

Update (Sep 2011):
Just posting my comment down at the bottom so you don’t have to scroll all the way down there to read it…

10.6.8 has addressed it for the most part. You’ll still get weirdness when duplicating folders with inherited folders inside a folder with inherited permissions, but it’s a linear rather than geometric and that’s bearable.

27 thoughts to “Finder’s Nasty Inherited ACL Bug (aka Error -41)”

  1. I can reproduce your results to a tee, both using your Error41.scpt AppleScript, and also a similar Error41_prep.command shell script I prepared to set up test cases (that script appears below). I also agree that this is a fairly serious problem with ACLs that appears to have been introduced with OS X 10.6.0 which had what I’ll call “linear” growth of unwarranted ACLs during FInder copy / duplicate operations. With OS X 10.6.5, 10.6.6 and 10.6.7, I indeed see what I’d characterize as a “geometric” growth of unwarranted ACLs with identical Finder operations.

    First let me say that there’s a minor error in your instructions. You say that to set up folders with inherited ACLs you run these shell commands, but the order is incorrect:

    mkdir -p ~/ACLShackles/1/2/3/4/5/6/7/8/9/10/11
    chmod +a “$(whoami) allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit” ~/ACLShackles/

    But this is incorrect: you need to add the ACL to the folder *before* you create subfolders for the inherited ACLs to start appearing. More correctly, this would be:

    mkdir ~/ACLShackles
    chmod +a “$(whoami) allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit” ~/ACLShackles/
    mkdir -p ~/ACLShackles/1/2/3/4/5/6/7/8/9/10/11

    And in my full “FInder41_prep.command” script (below) I’m being a bit more paranoid about eliminating possible inherited ACLs in the test folder and eliminating any pre-existing ACLShackles folder.

    #!/bin/sh
    test=~/ACLShackles
    rm -fr $test ## Delete possible existing folder
    mkdir $test ## Create fresh test folder
    chmod -R -N $test ## Clear out any possible inherited ACLs
    chmod +a “$(whoami) allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit” $test

    My conclusions are these:

    OS X 10.5.8 has a number of ACL-related glitches, but doesn’t exhibit any ACL growth when you duplicate the “1” folder repeatedly in the Finder, as in “1” -> “1 copy” -> “1 copy 2” -> “1 copy 3” -> etc.

    With OS X 10.6.0 to 10.6.2, there is a linear growth rate of ACL “crud”. What seems to matter is how deep a folder was originally nested and not how many times it’s copied. So for example, a copied “11” folder will have eleven ACEs on it: the original one plus ten that are “crud” and extraneous. But it doesn’t matter how many times you copy the hierarchy, the amount of crud is determined by how deep the original folder hierarchy was, and not how many times it’s copied. Survivable, in other words.

    With OS X 10.6.3 (v1.1) and 10.6.4, things got a little worse in that an extra ACE or two was thrown in for “bad” measure, but overall the growth characteristic is linear, depending on the original depth of the hierarchy and not the number of times it is copied.

    With OS X 10.6.5, 10.6.6, and 10.6.7 I agree that’s when things literally blew up and became geometric. Here, the number of extra ACLs appears to depend on something like the product of the depth of a particular folder times the number of times it is copied. Thus if you copy even a relatively shallow hierarchy, as your Error41.scpt does quite well, you run into overflow problems. [Offhand I would guess there are 512 max ACEs allowed in an ACL, but this is conjecture.]

    I didn’t run OS X Leopard Server or Snow Leopard tests for lack of time, but would imagine that the growth pattern matches. HOWEVER, (and this is imporant), for CLIENTs of an OS X server running through an AFP mount, even if the server is running OS X 10.6.6 server as is our case, only a LINEAR growth rate is seen. I scoured our production server and found places where the four inherited ACLs present originally had been multiplied out to sixteen ACLs because of users copying the original folder hierarchy three times (as in ORIG -> copy A -> copy B -> copy C). However, I couldn’t find any instances of the geometric growth pattern across several million files and folders. Also, our IT admins are quite religious about running “cp -p -R” on any server-side copies, which works correctly, so they didn’t introduce ACL growth.

    So in our very specific case, the deepest the amount had crudded up in practical use was ACLs growing from four ACEs to sixteen, but I would say we’re pretty lucky because of the limited types of copying operations our users perform. Your milage may vary.

    Lastly, let me say that I also ran tests using the public beta of Lion and Lion Server, but cannot post comments about my results. I would recommend that if you have this software, you might want to run the Error41.scpt on it for yourself.

    Hope this helps.

  2. Thanks for sharing your findings Bert.

    Now — I would’t say my order of folder creation and setting permissions is “incorrect”, just different :) Actually, I _was_ wanting to work with a group of folders that had no permission on them at all, to clearly illustrate that nothing else was contributing to the rampant propagation of ACEs, just a simple copy on ‘clean’ folders with one inherited ACE in the parent ACL.

    I like that you were looking for the geometry of the propagation by applying the permissions before the creation of the nested folders and in turn finding that 10.6.3-10.6.4 with its extra ACE or two, the growth was linear for nested folders. Interesting that client with server is linear rather geometric? Hmmm. A few ways to skin this cat! Like you, I didn’t have the time to fully explore all aspects, so thanks again for taking some time and chiming in with your results. I hope you also wrote a bug report to Apple? :)

    One bug report does not a fix make. Surprisingly, some obvious things never get fixed because no one takes the time to bug Apple about it! (Google: “call support drivers”)

    And that goes for anyone else who wants this fixed: file a bug report and then either put it in open radar, or send me your radar number and I’ll put it in here… let’s make it a few pages long, gang! Yeah, that’s spirit! :D

  3. “What’s interesting is what calls is PathFinder using?”

    Path Finder uses standard calls from Mac OS X File Manager core service, more particularly FSCopyObjectSync() and FSCopyObjectAsync(). The same are apparently used by Apple’s Finder. So these calls seem to be root of the problem. I suppose XFile uses cp, but I can’t be sure about that.

    I will make sure to file a bug report at Apple’s Bug Reporter site and let you know. The more people do it, the more eager they will be to fix it (I guess).

  4. Great – the ball seems to be rolling now!
    So Bert, I may/may not have tested Lion for this behavior, but as NDA goes, can’t say anything. Regardless I think the point is for Apple to fix their currently shipping product before moving on to 10.7, even if they do fix it in 10.7, it’s not really aceptable for them to tell those who want this issue fixed to buy Lion. That’s not acceptable, especially if customers for one reason or another need Rosetta (not me, but there are many OS X users out there with different requirements)

    Dragan, thanks for filing a bug report! Actually I’ve heard from the XFile developer and they use NSWorkspace’s performFileOperation for file operations, so this is a good piece of info that should help narrow down the culprit. Looking at the File Manager API in XCode, it seems that half of it’s been deprecated, seems like Apple’s been slowly shuttering it, so I hope they either give FSCopyObjectSync some love or change Finder’s method to the properly behaving performFileOperation.

    We’ll see! Thanks again all.

  5. Bert Sierra:

    HOWEVER, (and this is imporant), for CLIENTs of an OS X server running through an AFP mount, even if the server is running OS X 10.6.6 server as is our case, only a LINEAR growth rate is seen.

    Actually, not in my environment. If you are bound to Active Directory and are a member of the group that is in the inherited ACL, the resulting copy will include the dupe inherited entries as well as the entries as it applies to you as a member of that group as well. As seen in the sencong copy of myPKG and the resulting Contents has the geometric growth as you describe not happening on AFP… regardless, there’s a good read over at openRadar for the bug. Nail on the head

    Here’s the example grom a 10.6.7 client (AD bound) making a Finder copy on a 10.6.7 Server (AD bound) domain and group names changed to protect the innocent:

    brunerd:staging brunerj$ ls -led /Volumes/Updates/
    drwx------@ 18 brunerj AD\domain users 568 Mar 18 00:06 /Volumes/Updates/
    0: group:AD\groupname allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit

    brunerd:staging brunerj$ ls -led /Volumes/Updates/staging/MyPKG.pkg/
    drwxr-xr-x+ 3 1024 staff 264 Mar 8 15:20 /Volumes/Updates/staging/MyPKG.pkg/
    0: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit

    brunerd:staging brunerj$ ls -led /Volumes/Updates/staging/MyPKG\ copy.pkg/
    drwxr-xr-x+ 3 brunerj staff 264 Mar 8 15:20 /Volumes/Updates/staging/MyPKG copy.pkg/
    0: user:brunerj inherited allow add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown,file_inherit,directory_inherit
    1: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit
    2: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit

    brunerd:staging brunerj$ ls -led /Volumes/Updates/staging/MyPKG\ copy.pkg/Contents/
    drwxr-xr-x+ 7 brunerj staff 264 Mar 8 15:20 /Volumes/Updates/staging/MyPKG copy.pkg/Contents/
    0: user:brunerj inherited allow add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown,file_inherit,directory_inherit
    1: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit
    2: user:brunerj inherited allow add_file,delete,add_subdirectory,delete_child,writeattr,writeextattr,chown,file_inherit,directory_inherit
    3: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit
    4: group:AD\groupname inherited allow list,add_file,search,add_subdirectory,delete_child,readattr,writeattr,readextattr,writeextattr,readsecurity,file_inherit,directory_inherit

  6. Been watching this and dealing with this issue for a while. There appears to be another variable that might be worth noting. All findings based on 10.6.7 client and 10.6.7 Server. Server is running OD Master and AFP. Accounts are defined as managed mobile accounts and when logged in at the workstation I am doing cached credentials.

    I am clearly seeing the issue in both OD/AD and Xsan environments. However, I noted that I was not getting these issues in environments in which users are standalone without ties to the domain.

    Here is an example. Log in as a local admin and connect to the file server using domain credentials. Files pulled to the local admin’s desktop do NOT keep the ACL from the server. They are removed during the copy process. Likewise, if the local admin copies items back to the server, they get proper inheritance.

    However, if I log into the workstation as the domain user, then the ACL inheritance mess begins.

    So I guess the question is, if this is a Finder issue, then why does it work one way for local admins and local accounts and another way for domain users?

    I am going to compile my findings and will post a bug report also.

    As you mention, I fear there will be no resolution in the Snow Leopard time frame. Clearly all efforts are being made to jam Lion down our throats.

  7. Yes I’ve seen this and I think it makes since. Because what good would an ACE that uses a UID from OD/AD do on an unbound Mac cannot resolve the UID to a user, so it ignores those entries when copying down to that machine.

  8. My users login to their machines using managed mobile accounts (from AD), on AD-bound machines…

  9. Can reproduce it easily …
    rdar://9395741

    Thanks for all your dissection!

  10. Thanks this really helped me. I have figured out how to edit the ACLs and now can get my backup to work (even if temporarily.)

  11. Thanks for the helpful write up. I have solved my copy issues for now. Working on script to weed this out in case Apple fails to patch in 10.6.8.

  12. Thanks for working so hard to get this one resolved. In my 5 minutes with 10.6.8 (OD bound systems only so far), I am seeing proper behavior. Wow, someone at Apple is still working on OS X. Who knew?

  13. I just tried the first two examples again on my test server, recently updated to 10.6.8 Server. No longer getting -41 errors, ACLs seem better behaved?

    Perhaps the issue is resolved?

  14. Yes! It’s true, the 10.6.8 seeds had taken care of this, but I didn’t want to break NDA or anything! (I felt like I was pushing the envelope enough by posting my bug report to Open Bugtracker!) Congrats to everyone for helping get this fixed with your bug reports, thanks! (Now get outside it’s a lovely weekend ;)

  15. It does look like they fixed the problem, though I am experiencing other problems with the update now. Users are reporting slow log in times on network log in accounts.

  16. 10.6.8 does not appear to fix this entirely, as I see it. While I’ve been unable to generate a Error -41 with some very quick cursory tests, the Finder continues to make superfluous explicit copies of inherited ACLs. While it’s a relief that my client’s Time Machine backups should no longer regularly fail because of the error, I’m not yet convinced yet that the problem is satisfactorily dealt with…

  17. Yes, indeed! I guess NDA is over for Lion eh? :) In fact when I filed this bug report, the 10.7 beta already had it fixed! Looking at Samples in Activity Monitor of Finder, you see a whole lot less calls in Lion on duplication, a lot more streamlined, they cleaned out some of that finder cruft… unfortunately the new Finder bug is its total lack of color!

  18. Hey there, it seems this problem is also apparent on SL Server. Copying files back and forth between shares on the server will create this huge ACL list of the same user, until eventually Finder will fail to copy the file that is screwed up.

    I created a little drag and drop script to fix this. Drag the offending file or folder on the the app and it will basically terminal chmod -R -N and remove ACL permissions. You can then transfer the file. It will eventually have to be done again and on others until its fixed, which I think has happened in 10.6.8.

    This addresses the error -50 that apple users will get.
    [Thanks, but no code links :]

  19. I am running 10.5.8 (hey, “if it ain’t broke, don’t fix it, right?”) – well, it’s broke. I have an external USB hard drive (ATA) with 3 partitions, and I have copied data to the drive(s) previously. Now, when I try to do a “Save as” or a “Create New Folder”, I get error code -50. I did a web search (a “werch?” “iSearch?”) and can only find refs to error code -41. WTF?

    I ran Disk Utility, and it was verified. I can’t get it to Repair (says “disk could not be unmounted” – WTF?) So I can’t repair permissions. Before I spend the next month re-learning UNIX from 20 years ago (it’s a Bash C Shell? Or Bourne?) and trying to reset the permissions, or just re-formatting the drive and copying again.. erg, this defeats the purpose of this backup drive. APPLE, WTF? Do I HAVE to now upgrade to Apple LionsTigers&Bear/Lynx/Cat/Pussy/yukyux, or is there some really urbane and anachronistic bit head that has this one sorted and can explain it without so much as a grep? Well, hell, do I have to log it.

    BTW – it’s AUG 6, 2011. Thanks, ASC!

  20. Well, I thought 10.6.8 had addressed it for the most part. You’ll still get weirdness when duplicating folders with inherited folders inside a folder with inherited permissions, but it’s a linear rather than geometric and that’s bearable.

  21. 10.6.8 has fixed this issue for us.
    Note: we had to update both clients and Metadatacontroller server to 10.6.8 for solving the problem.

    WOW: we can now copy files and folders present on the Apple Xsan volume from a 10.6.8 Xsan client and ACLs do not get duplicated.

  22. > fixed in Lion!

    Apparently not for those migrating files from older Macs/Systems. I got a mess when I tried, then looked for it and found many comments/questions at Apple’s support area about permission folders, authorizations needed to copy files or trash files, and on and on. Nobody seems to know for sure what’s up with this yet.

  23. Hi,

    I had a filepermission after a restore from my backup.
    Suddenly all my group-permissions where doubled and sometimes users from the same group where excluded for no reason.
    Thanks for your post I found. Now everything is working like before.

    One question though if you read this:
    Before I deleted all permissions and reenabled them for the group I noticed that Spotlight had some rights to the folders.
    Will Spotlight put the rights back or will they be gone?
    Do I have to put them back manually? If so – how?

    Sincerely,

    DB

    PS.
    Snow Leopard server on a MAC mini/ 5 users in 2 groups
    Backup thru Tri-Edre backup
    Virtualized 2 SLES 11 servers on VMware Fusion 3/ 1 mailserver/1 syncserver
    Functions perfectly

  24. If you use server admin, you can re-propagate the permissions, which will eliminate duplicates, also you can enable Spotlight indexing which will add the Spotlight ACL.

Comments are closed.