OS X : play with Extended Attributes

parse error

What are extended attributes ?

Extended Attributes (often shortened as xattrs) are a feature of (modern) filesystems that allows users to associate metadata with a file. These metadata are pairs of name-value that allows users to further describe a file. Extended Attributes is a very valuable tool since they allow users to add semantics to their files. For example, you can store the author(s) of a document in xattrs (authors: "John Doe, Jane Doe"). In macOS, xattrs are used to store tags (amongst other things).

The Extended Attributes are not stored in the file, they are stored in the filesystem itself and associated with the file they describe.

Guess what ? I LOVE extended attributes. macOS makes a pretty smart use of them. For example, when you download a file from the Internet, macOS automatically adds a bunch of xattrs to the downloaded file:

  • com.apple.quarantine that tells the system "Hey, this file might be dangerous, please manipulate with caution !" ;
  • com.apple.metadata:kMDItemDownloadedDate that stores the download date and time ;
  • com.apple.metadata:kMDItemWhereFroms that stores the URL from which the file was downloaded.

When used to store tags, it becomes a very powerful tool that allows you to get rid of the old tree metaphor to organise your files. Looking for photos of Uncle Ben ? Just search for them with the appropriate tag, no need to know where the files are.

How to work with xattrs (from the CLI) ?

Reading and listing

To list files with their extended attributes, you can simply use the @ flag of the ls command:

$ ls -l@
total 0
drwx------+ 12 francois  staff   408 19 sep 14:58 Desktop
drwx------+ 22 francois  staff   748  8 mai 22:23 Documents
drwx------+ 14 francois  staff   476 27 sep 14:37 Downloads
drwx------@ 63 francois  staff  2142 21 sep 14:10 Library
        com.apple.FinderInfo      32 
drwx------@  3 francois  staff   102  3 déc  2014 Movies
        com.apple.metadata:_kMDItemUserTags   50 
drwx------+  4 francois  staff   136  2 jui  2015 Music
drwx------+  5 francois  staff   170 25 jan  2016 Pictures
drwxr-xr-x+  5 francois  staff   170  3 déc  2014 Public

Notice that the permissions column also ends with an @ when the file has extended attributes. This is also the case when you list the directory with the simpler ls -l.

To list xattrs and their value, you can use xattr with the -l flag:

$ xattr -l ~/Downloads/KUBLER\ François\ -\ CV.pdf
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 BD 9A 9F 8C  |bplist00..3A....|
00000010  F4 E8 53 08 0A 00 00 00 00 00 00 01 01 00 00 00  |..S.............|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035
com.apple.metadata:kMDItemWhereFroms:
00000000  62 70 6C 69 73 74 30 30 A1 01 6F 10 3F 00 68 00  |bplist00..o.?.h.|
00000010  74 00 74 00 70 00 73 00 3A 00 2F 00 2F 00 66 00  |t.t.p.s.:././.f.|
00000020  72 00 61 00 6E 00 63 00 6F 00 69 00 73 00 2E 00  |r.a.n.c.o.i.s...|
00000030  6B 00 75 00 62 00 6C 00 65 00 72 00 2E 00 6F 00  |k.u.b.l.e.r...o.|
00000040  72 00 67 00 2F 00 6D 00 69 00 73 00 63 00 2F 00  |r.g./.m.i.s.c./.|
00000050  4B 00 55 00 42 00 4C 00 45 00 52 00 25 00 32 00  |K.U.B.L.E.R.%.2.|
00000060  30 00 46 00 72 00 61 00 6E 00 E7 00 6F 00 69 00  |0.F.r.a.n...o.i.|
00000070  73 00 25 00 32 00 30 00 2D 00 25 00 32 00 30 00  |s.%.2.0.-.%.2.0.|
00000080  43 00 56 00 2E 00 70 00 64 00 66 08 0A 00 00 00  |C.V...p.d.f.....|
00000090  00 00 00 01 01 00 00 00 00 00 00 00 02 00 00 00  |................|
000000A0  00 00 00 00 00 00 00 00 00 00 00 00 8B           |.............|
000000ad
com.apple.quarantine: 0083;57ea6802;Safari;D930F473-4B79-4AB2-8179-619CC9491A43

As you can see, macOS xattrs often have cryptic values. This is because they are meant to be used by macOS directly (and exposed through the Finder, for example). They don't have to be human readable.

If you want to focus on a specific xattr, the -p flag is your friend:

$ xattr -p com.apple.quarantine ~/Downloads/KUBLER\ François\ -\ CV.pdf
0083;57ea6802;Safari;D930F473-4B79-4AB2-8179-619CC9491A43

Writing

Creating an extended attribute is very easy:

$ touch ./test.pdf
$ xattr -w my_xattr "My Value" ./test.pdf

As you can see, you only have to provide the name of the extended attribute and its value.

Now let's read it:

$ xattr -p my_xattr ./test.pdf
My Value

Deleting

Deleting an extended attribute is also very easy. The -d flag allows you to delete a specific xattr whereas the -c flag will clear all extended attributes for the given file.

Finding files

macOS comes with a very useful command called mdfind that finds files matching a given query.

For example, if you are looking for files that have been downloaded from francois.kubler.org, you could run the following:

$ mdfind "kMDItemWhereFroms == '*francois.kubler.org*'"
/Users/francois/Downloads/KUBLER François - CV.pdf

To find files tagged with the keyword résumé:

$ mdfind "kMDItemUserTags == 'résumé'"
/Users/francois/Downloads/KUBLER François - CV.pdf

Let's try something else:

$ mdfind "my_xattr == '*Value*'"
$

As you can see, if we try to find the file for which we previously added a custom xattr, we don't get any result :(

Let's open a little parenthesis.

My guess is that macOS only indexes (= import in its metadata database) xattrs that are in the com.apple.metadata namespace. Using mdls seems to confort this assertion:

$ mdls test.pdf
_kMDItemOwnerUserID            = 501
kMDItemContentCreationDate     = 2016-09-27 12:37:20 +0000
kMDItemContentModificationDate  = 2016-09-27 12:37:21 +0000
kMDItemContentType             = "com.adobe.pdf"
kMDItemContentTypeTree         = (
    "com.adobe.pdf",
    "public.item",
    "com.adobe.pdf",
    "public.data",
    "public.composite-content",
    "public.content"
)
kMDItemDateAdded               = 2016-09-27 12:37:22 +0000
kMDItemDisplayName             = "test.pdf
kMDItemFSContentChangeDate     = 2016-09-27 13:10:03 +0000
kMDItemFSCreationDate          = 2016-09-27 13:10:03 +0000
kMDItemFSCreatorCode           = ""
kMDItemFSFinderFlags           = 0
kMDItemFSHasCustomIcon         = (null)
kMDItemFSInvisible             = 0
kMDItemFSIsExtensionHidden     = 0
kMDItemFSIsStationery          = (null)
kMDItemFSLabel                 = 0
kMDItemFSName                  = "test.pdf"
kMDItemFSNodeCount             = (null)
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 0
kMDItemFSTypeCode              = ""
kMDItemKind                    = "Portable Document Format (PDF)"
kMDItemLogicalSize             = 0
kMDItemPhysicalSize            = 0

See ? There is no trace of our previously created My Value value. I'm pretty sure mdfind only looks in this data to retrieve files.

Also notice how useful the output of this command is : it gives you a bunch of values to use with mdfind. For example, you now know how to find PDF files:

$ mdfind "kMDItemContentType == 'com.adobe.pdf'"

End of parenthesis.

And what about Linux ?

Nowadays, most distros come with a Kernel compiled with extended attributes support enabled. The most widely used filesystems on Linux (ext3, ext4, btrfs, ...) support xattrs. However, developers can't be sure that the system will actually support xattrs, which, of course, is a pain if your app depends on them. macOS doesn't have that kind of issue since Apple controls the whole software stack.

So, to my knownledge, a very few Linux applications make use of extended attributes. The biggest exceptions might be SELinux (that simply relies on them) and KDE's Baloo framework (for which I made a proposal to use xattrs to store tags in 2013).

It's also interesting to see that freedesktop.org has a page dedicated to extended attributes. So, maybe we will see improvements in the near future ? Let's hope so !

Anyway, if you are interested in using xattrs on your Linux machine, you should read the manpages for xattr, setfattr and getfattr.

Like me, you'll probably be disappointed by the lack of tools provided by Linux to deal with xattrs. For example, there is no ls -l@, cp or rsync don't preserve extended attributes unless you specify the good options (respectively --preserve=xattr and -X or --xattrs), and so on.

Going further here is out of the scope of this post, but feel free to get in touch with me if you want to know more about extended attributes in Linux.