Vous avez recherché tous les articles de la catégorie parse error, j'ai trouvé 13 résultats.

Hello World!

parse error

Comme il est généralement de coutume en informatique, voilà mon (nécessaire) Hello World! :)

Tiens, d'ailleurs, Google propose à partir d'aujourd'hui un nouveau joujou destiné aux développeurs : le SDK du très attendu Android. Petit tour du propriétaire...

Pour ceux qui dorment au fond, près du radiateur, Android est le nouvel outil de Google destiné aux téléphones mobiles. Il s'agit en fait d'une plateforme libre et gratuite permettant de faire tourner tout un tas de petites applications sur votre téléphone mobile. Le moins que l'on puisse dire est que Google facilite grandement la vie des développeurs, puisque ce SDK comprend une API, de la doc, un émulateur de terminal et même un plugin pour l'EDI Eclipse. Le tout est téléchargeable gratuitement, bien entendu.

A première vue, l'outil est très prometteur et fonctionne plutôt pas mal, à en juger par les premières vidéos :

Les habitués de la POO seront tout de suite dans le bain. En effet, le langage retenu est le Java. Google a même pensé à faciliter la création des IHM en permettant aux développeurs de décrire celles-ci via un fichier XML. Pratique et léger.

Enfin, pour les plus courageux, Google propose des exemples détaillés et commentés ainsi que des exercices. (en anglais, forcément).

Parmi ces exemples, le plus simple pour débuter reste forcément l'indispensable Hello World.

Quelque chose me dit que je ne vais pas tarder à mettre mon nez là-dedans, ne serait-ce que pour refaire un peu de Java. Reste à trouver une idée de programme utile et pas trop compliqué, histoire de bien commencer :)

Et voilà quelques liens pour les curieux :

J'ai hâte de voir comment s'en sortira OpenMoko et son Neo1973.

The WABAC Machine

parse error

Mise à jour du 14 novembre 2013 :

The WABAC Machine a beaucoup évolué depuis ! Suivez son développement via son dépôt GitHub.

The WABAC Machine dans les années 90

The WABAC Machine est un petit shellscript qui utilise RSync et dont je me sers quotidiennement pour faire une sauvegarde de mon /home.

Son fonctionnement est volontairement proche de celui de la Time Machine d'Apple. La WABAC Machine réalise chaque sauvegarde en 3 étapes :

  1. Elle calcule l'espace nécessaire pour réaliser la sauvegarde et vérifie que celui-ci est bien disponible sur le volume de sauvegarde. Si ce n'est pas le cas, la WABAC Machine supprime automatiquement les plus vieilles sauvegardes pour faire de la place.
  2. Elle réalise une sauvegarde incrémentale et datée, en hard-linkant les fichiers qui n'ont pas changé : l'espace occupé par la sauvegarde est donc le plus petit possible.
  3. Elle supprime les sauvegardes plus vieilles de x jours, x étant paramétrable, bien entendu.

Son efficacité est due à RSync, qui est un outil simplement formidable et aux hardlinks, hérités d'Unix.

Et voilà le script en question :

#!/bin/sh

#
#-------------------------------------------------------------------------------
#
# The WABAC Machine.
#
# This script tries to mimic Apple's TimeMachine, *thanks to rsync* :)
# Copyright François KUBLER, 2009.
#
# This program is free software; you can redistribute it and/or
# modify it under the terms of the GNU General Public License
# as published by the Free Software Foundation; either version 2
# of the License, or (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
#
#-------------------------------------------------------------------------------
#



# Edit these variables to fit your need ----------------------------------------

# Source :
SRC="/home/"

# Destination :
VOL="/media/HDD"
DST=$VOL"/BACKUP"

# Exclude file :
EXCLUDE_FILE="/home/user/wabac/exclude.txt"

# Remove saves older than (in days) :
REMOVE_OLDER_THAN=60

# Date format :
DATE_FORMAT="+%Y-%m-%d %H:%M:%S"




# Edit with care behind this line ! --------------------------------------------


check_mounted()
{
    cat /etc/mtab | grep $VOL >/dev/null
    if [ "$?" -ne "0" ]; then
        echo -n "    > $VOL is not mounted, trying to mount..."

        mount $VOL

        cat /etc/mtab | grep $VOL >/dev/null
        if [ "$?" -ne "0" ]; then
            echo "failed."
            echo "    > Unable to mount $VOL, aborting !"
            exit 2
        fi
        echo "OK."
    fi
}


needed_space()
{
    NEEDED_SPACE=$(rsync -a --stats --dry-run --exclude-from="$EXCLUDE_FILE" --link-dest="$DST/latest" "$SRC" "$DST/$NOW/" | grep "Total transferred file size" | tr -s " " | cut -d" " -f5)
    NEEDED_SPACE=$((NEEDED_SPACE/(1024*1024)))

    echo "    > Needed space : $NEEDED_SPACE Mo."
}


available_space()
{
    AVAIL_SPACE=$(df -m "$DST" | grep "/dev" | tr -s " " | cut -d" " -f4)

    echo "    > Available space : $AVAIL_SPACE Mo."
}


free_space()
{
    echo "+ PRE-BACKUP THINNING"

    needed_space
    available_space

    while [ $NEEDED_SPACE -gt $AVAIL_SPACE ]
    do
        # Damn, we need to delete the older directory, let's get its name :
        REM=$(find "$DST" -maxdepth 1 -mindepth 1 -type d | sort -f | head -1)
        echo "    > Deleting $REM to get enough free space."
        rm -Rf "$REM"

        available_space
    done
}


purge()
{
    NB_REM=$(find "$DST" -maxdepth 1 -mindepth 1 -type d -mtime +$REMOVE_OLDER_THAN | wc -l)

    if [ $NB_REM -gt 0 ]
        then
            echo "    > $NB_REM expired backup(s) will be removed."
            find "$DST" -maxdepth 1 -mindepth 1 -type d -mtime +$REMOVE_OLDER_THAN -print| while read obj
            do
            rm -Rf "$obj"
            echo "    > $obj has been removed."
            done
        else
            echo "    > OK."
    fi
}


make_link()
{
    rm "$DST/latest"
    N=$(find "$DST" -maxdepth 1 -mindepth 1 -type d | sort -rif | head -1)
    ln -s "$N" "$DST/latest"
}


rotate()
{
    echo "+ POST-BACKUP THINNING"

    purge
    make_link
}


backup()
{
    echo "+ BACKUP"
    echo "    > Backing up to: $DST/$NOW/"

    nice -n 19 rsync -ahS --numeric-ids --stats --exclude-from=$EXCLUDE_FILE --link-dest=$DST/latest $SRC "$DST/$NOW/"
    touch -a -m -c -d "$NOW" "$DST/$NOW"
}


wabac_machine()
{
    echo "+ WABAC MACHINE BACKUP STARTING (${NOW})"

    check_mounted
    free_space
    backup
    rotate

    NOW=$(date "$DATE_FORMAT")
    echo "+ WABAC MACHINE BACKUP DONE (${NOW})"
}



### Allright, let's start :

NOW=$(date "$DATE_FORMAT")

wabac_machine

exit 0



#EOF

Vous pouvez aussi télécharger le script. N'oubliez pas d'enlever le .txt, de lui donner les droits d'exécution (chmod u+x wabac_machine.sh) et de modifier les variables en fonction de vos besoins.

Time Machine : péripéties (2)

parse error

Parfois, on a quelques surprises après avoir appliqué les mises à jour de Mac OS X (10.6.4 Server dans ce cas). La dernière date d'hier, extrait du journal de backupd :

Starting standard backup
Attempting to mount network destination using URL: afp://admin@backup.local/backup
Mounted network destination using URL: afp://admin@backup.local/backup
Backup verification failed for image /Volumes/backup/server.sparsebundle!
Moved previous backup image to
        /Volumes/backup/server_2010-06-30-142551.sparsebundle
Recovery backup requested by user.
Deleted backup image: /Volumes/backup/server_2010-06-30-142551.sparsebundle.
        3724.5 GB available on host volume.
Creating disk image /Volumes/backup/server.sparsebundle</strong>
QUICKCHECK ONLY; FILESYSTEM CLEAN
Disk image /Volumes/backup/server.sparsebundle mounted at: /Volumes/Time Machine Backups
Backing up to: /Volumes/Time Machine Backups/Backups.backupdb
Ownership is disabled on the backup destination volume.  Enabling.
Backup content size: 12.7 GB excluded items size: 1.7 GB for volume disksystem
Backup content size: 337.3 GB excluded items size: 1.9 GB for volume DATA
No pre-backup thinning needed: 415.72 GB requested (including padding), 3.64 TB available

Sachez donc que Time Machine peut décider à tout moment de supprimer vos 6 derniers mois de sauvegardes et en recommencer une nouvelle, sans prévenir, ni vous demander votre avis, bien entendu. C'est ça aussi, travailler avec des produits Apple...

PyQt4 : QNetworkAccessManager, QEventLoop et synchronisation

parse error

Je me suis pas mal emm..amusé ces derniers jours avec PyQt4. L'énoncé du problème était pourtant assez simple : récupérer des infos d'un service web tout en gardant l'interface réactive. J'ai finalement trouvé la solution hier soir, et je me suis dit que ça pourrait peut-être en aider certains.

La première étape consiste à lancer une requête vers un service web et à récupérer le résultat. Qt4 facilite pas mal la tâche au développeur avec une classe appelée QNetworkAccessManager (du module QtNetwork) qui permet -entre autre- de faire ce genre de choses grâce à sa méthode get.

Petite difficulté à appréhender : la requête est asynchrone. Elle va donc se faire en arrière plan, et son résultat ne sera disponible (et exploitable) qu'au moment où notre QNetworkAccessManager émettra le signal finished. Cela signifie aussi que, pendant ce temps, les instructions suivantes du programme seront executées. Et, ô malheur, si ces instructions font appel au résultat de la requête... elles vont planter, tout simplement (éh oui, le résultat n'est pas forcément encore disponible) !

En rendant la requête totalement synchrone, on crée un autre problème : le programme reste bloqué jusqu'à l'obtention de la réponse à la requête, ce qui, dans certains cas, peut durer un certain moment.

Alors ? Alors la solution consiste bien à rendre notre requête synchrone (on a pas trop le choix) mais en l'executant dans une nouvelle boucle d'évènements (QEventLoop). Celle-ci aura pour effet de bloquer le programme jusqu'à la sortie de cette boucle, mais sans pour autant bloquer les évènements qui pourraient survenir dans la boucle d'évènements principale (celle dans laquelle s'execute notre interface graphique). Notre GUI reste donc réactive (elle ne freeze pas) et le programme attend sagement les données venant de la requête. Pari gagné !

Cette méthode est proche d'une méthode connue appelée busy waiting ou spinning, qui consiste à attendre, dans une boucle "infinie", qu'une condition soit remplie. Lorsque la condition est remplie, on sort de la boucle et on continue à exécuter les instructions suivantes. Cette méthode est très coûteuse en CPU, puisque le programme vérifie sans cesse si la condition est remplie ou non. Elle est donc à éviter.

Dans notre cas, Qt4 nous permet de réaliser une opération similaire mais beaucoup moins coûteuse, grâce à sa QEventLoop et aux signaux. La condition qui permet de poursuivre l'exécution du programme n'a pas besoin d'être vérifiée sans cesse : elle envoie un signal quand elle est remplie !

Bien entendu, j'ai mis tout ça dans une petite classe. Notez au passage que le QNetworkAccessManager ne dispose pas de système de timeout. Il faut l'implémenter soit même !

from PyQt4.QtCore import pyqtSlot, QObject, QEventLoop, QUrl, QTimer
from PyQt4.QtNetwork import QNetworkAccessManager, QNetworkRequest
from myown.Exceptions import NetworkException

class HttpRequest(QObject):
    """
    """
    def __init__(self):
        """
        """
        QObject.__init__(self)
        self.manager = QNetworkAccessManager()
        self.timer = QTimer()
        self.loop = QEventLoop()
        self.reply = None

        self.timer.setSingleShot(True)

        self.timer.timeout.connect(self.loop.quit)
        self.manager.finished.connect(self.loop.quit)

    def request(self, url):
        """
        Sends an HTTP request to the provided URL.
        """
        self.reply = self.manager.get(QNetworkRequest(QUrl(url)))
        self.reply.error.connect(self.handleError)

        self.timer.start(5000)
        self.loop.exec_()

        if self.timer.isActive():
            # Download completed before the end of the timer.
            self.timer.stop()
            r = self.handleReply()
        else:
            # Timeout.
            raise NetworkException("Timeout error.")

        return r

    @pyqtSlot()
    def handleError(self):
        """
        """
        raise NetworkException(self.reply.errorString())
        self.reply.deleteLater()

    def handleReply(self):
        """
        """
        raw_data = self.reply.readAll()
        self.reply.deleteLater()

        return raw_data

# Utilisation :

req = HttpRequest()

reply = req.request("http://francois.kubler.org/")

# Convert reply (a QByteArray) into unicode string and print :
print unicode(reply)

Et voilà ! Mon apprentissage de Python et de Qt4 se poursuit de jour en jour... Et je dois dire que je prends énormément de plaisir à travailler avec !

QtQuick : Helmo's rewrite - part 1

parse error

Qt logo

GUI is very important for a mediacenter. It has to be pleasant, easy to understand, easy to use and customizable. When I started Helmo a few monthes ago, QML was already promising but maybe not mature enough. So I decided to stick with QWidget, and to give customization capabilities through Qt StyleSheets (which is still better than nothing). The results are great, but it lacks transitions effects, capability to change the layout of items, etc.

So, I decided to take a look at QML again, and to actually give it a try. See where it could lead me. I started on monday, and I decided to write a few blog posts about it. So, here it is : the first one of a (hopefully) long season.

In this first chapter, I'll just explain some basic stuff. I need a starting point, right ?

Following the QtQuick doc, I started with a new Python file (HelloWorld.py) which is very simple :

# -*- coding: utf-8 -*-

import sys

from PyQt4.QtCore import QObject, QUrl
from PyQt4.QtGui import QApplication
from PyQt4.QtDeclarative import QDeclarativeView


class HelloWorld(QObject):
    def __init__(self):
        QObject.__init__(self)


app = QApplication(sys.argv)
h = HelloWorld()
v = QDeclarativeView()

v.setSource(QUrl(__file__.replace('.py', '.qml')))
v.setResizeMode(QDeclarativeView.SizeRootObjectToView)
v.show()

app.exec_()

As you can see, we first need to import some stuff : the sys module and the QObject, QUrl, QApplication and QDeclarative classes. This last one is a view that is able to display a Qt Declarative user interface. This is THE widget !

We then create a new HelloWorld class. It's empty for now, but it will get some stuff later.

The interesting stuff comes just after : we build a new QApplication, a new HelloWorld class instance and a QDeclarativeView instance. The view gets a new source, which is a QML file (HelloWorld.qml). This view will automatically resize its root item.

Finally, we display the view and run the app.

That's it. The Python file is ready to go. Let's head to the funny thing : the QML file.

A QML file has to start with this line :

import QtQuick 1.0

which is pretty easy to understand : we want to do some QtQuick stuff in here.

QtQuick provides a great collection of technologies to build user interface and much more. Among them is a very basic (but powerful) element : Rectangle. As you may have guess, the Rectangle element provides a simple rectangle that can be filled and that can have a border. We are going to draw one. Here is the HelloWorld.qml file :

import QtQuick 1.0

Rectangle {
    id: appWindow

    color: "#111111"
    height: 600
    width: 800        
}

Now, if you run your Python file (python HelloWorld.py), you should get a 800x600 darkgrey rectangle in a window. That's all. Try to resize the window : the Rectangle also resizes, thanks to the view's resizeMode property.

See ? Pretty easy, isn't it ? Well, the next parts will be much more interesting, I promise !

QtQuick : Helmo's rewrite - part 2

parse error

In the first chapter, we've seen how to make a very very basic application written in Python and QtQuick. In this second part, we'll go one step further and see how to share data between the application and the GUI.

We are going to add a _foo attribute to our HelloWorld class. To keep things simple, this attribute will be a simple unicode string.

The great thing with QML is that the internal engine can be automatically notified when the value of an attribute changes. To see the magic happen, we will need to create a specific Signal and make QML aware of it. To do so, we will declare foo as a Qt Property and thus, write a getter and a setter. The code for our HelloWorld class becomes :

from PyQt4.QtCore import pyqtSignal, pyqtProperty

class HelloWorld(QObject):

    fooModified = pyqtSignal()

    def __init__(self):
        QObject.__init__(self)
        self._foo = u'Hello'

    def getFoo(self):
        return self._foo

    def setFoo(self, newValue):
        if self._foo != newValue:
            self._foo = s
            self.fooModified.emit()

    foo = pyqtProperty(unicode, fget=getFoo, fset=setFoo, notify=fooModified)

See ? We tell QML that foo is a property with a getter (getFoo) and setter (setFoo). And also that the fooModified signal will be emitted whenever the value of _foo changes. The cool thing is that the view will be updated automatically with the new value ! Well, we still need to tell QML that the HelloWorld class exists. This is done via Contexts. Qt provides a QDeclarativeContext class to do so. Our Python file now looks like this :

# -*- coding: utf-8 -*-

import sys

from PyQt4.QtCore import QObject, QUrl, pyqtSignal, pyqtProperty
from PyQt4.QtGui import QApplication
from PyQt4.QtDeclarative import QDeclarativeView


class HelloWorld(QObject):

    fooModified = pyqtSignal()

    def __init__(self):
        QObject.__init__(self)
        self._foo = u"Hello"

    def getFoo(self):
        return self._foo

    def setFoo(self, newValue):
        if self._foo != newValue:
            self._foo = s
            self.fooModified.emit()

    foo = pyqtProperty(unicode, fget=getFoo, fset=setFoo, notify=fooModified)


app = QApplication(sys.argv)
h = HelloWorld()
v = QDeclarativeView()

context = v.rootContext()
context.setContextProperty('hello', h)

v.setSource(QUrl(__file__.replace('.py', '.qml')))
v.setResizeMode(QDeclarativeView.SizeRootObjectToView)
v.show()

app.exec_()

Okay, it's time to add some stuff to our QML file. We'll add a simple Text element to show the value of _foo. This value is available through the hello property (defined with context.setContextProperty('hello', h) :

import QtQuick 1.0

Rectangle {
    id: appWindow

    color: "#111"
    width: 1360
    height: 768
    Text {
        anchors.centerIn: parent
        font.pointSize: 32
        color: "white"
        text: hello.foo
    }
}

If everything's ok, you should see a darkgrey rectangle with a white centered label stating "Hello". That's it !

In the next part, we'll play a bit with a ListView, stay tuned !

QtQuick : Helmo's rewrite - part 3

parse error

Today we're going to play a bit with a ListView. A ListView is a simple but powerful widget that shows items in a list. In Helmo, the main menu I'm working on is a customized ListView.

Let's see how to start. We are going to start a new QML file. Let's call it MyList.qml :

import QtQuick 1.0

ListView {
    id: myList

    anchors.bottom: parent.bottom
    height: parent.height*0.25
    width: parent.width
    boundsBehavior: Flickable.DragOverBounds
    cacheBuffer: width
    focus: true
    highlightMoveDuration: 300
    orientation: ListView.Horizontal
    snapMode: ListView.SnapToItem
    spacing: 20
}

As you can see, we've built a simple horizontal list view (orientation property). The view will stick to its parent bottom (anchors.bottom property). We've also set a height and width. I strongly recommend you to RTFM if you want to know more about others properties :P

We still miss two important things : the data and how the data is displayed. Well, the list view is able to get the data from a model (you know MVC, right ?), thanks to the model property. Here, we're going to use a simple ListModel, but keep in mind that you can build your own model if you need to. Let's have a look at this model, stored in the MyListModel.qml file :

import QtQuick 1.0

ListModel {
    ListElement {
         name: "Plugin 1"
    }
    ListElement {
         name: "Plugin 2"
    }
    ListElement {
         name: "Plugin 3"
    }
    ListElement {
         name: "Plugin 4"
    }
    ListElement {
         name: "Plugin 5"
    }
    ListElement {
         name: "Plugin 6"
    }
}

Pretty easy to understand, isn't it ?

Now that the list view knows what to display, we still have to tell it how to display the data. Once again, the list view is very smart. It provides a delegate property to do that. A delegate is a simple template defining each item instanciated by the list view. And guess what, a delegate is a Component. So we're almost free to do whatever we want.

For this example, I chose a simple 256x75 text button as delegate. The source code is as follow :

Rectangle {
    color: "transparent"
    height: 75
    width: 256
    Text {
        anchors.bottom: parent.bottom
        anchors.horizontalCenter: parent.horizontalCenter
        color: "#FFFFFF"
        font.pointSize: 32
        horizontalAlignment: Text.AlignHCenter
        text: name
    }

    MouseArea {
        anchors.fill: parent
        onClicked: {
            console.log(name)
        }
    }
}

Now that everything is ready, let's stick all the pieces together. MyList.qml source code becomes :

import QtQuick 1.0

ListView {
    id: myList

    anchors.bottom: parent.bottom
    height: parent.height*0.25
    width: parent.width
    boundsBehavior: Flickable.DragOverBounds
    cacheBuffer: width
    focus: true
    highlightMoveDuration: 300
    orientation: ListView.Horizontal
    snapMode: ListView.SnapToItem
    spacing: 20

    model: MyListModel {}

    delegate : Component {
        Rectangle {
            color: "transparent"
            height: 75
            width: 256
            Text {
                anchors.bottom: parent.bottom
                anchors.horizontalCenter: parent.horizontalCenter
                color: "#FFFFFF"
                font.pointSize: 32
                horizontalAlignment: Text.AlignHCenter
                text: name
            }

            MouseArea {
                anchors.fill: parent
                onClicked: {
                    console.log(name)
                }
            }
        }
    }
}

Notice that I could have put the delegate in a seperate file (and I strongly encourage you to do so).

And do not forget to update your HelloWorld.qml file to add the list view :

import QtQuick 1.0

Rectangle {
    id: appWindow

    color: "#111"
    width: 1360
    height: 768
    Text {
        anchors.centerIn: parent
        font.pointSize: 32
        color: "white"
        text: hello.foo
    }

    MyList {}
}

Now just launch the Python file and see the magic happen. Isn't it great to build such a thing in minutes ?

rsync: unpack_smb_acl: sys_acl_get_info(): Unknown error: 0 (0)

parse error

When you run rsync on a HFS+ filesystem with ACL under Mac OS X, you may encounter that kind of error :

rsync: unpack_smb_acl: sys_acl_get_info(): Unknown error: 0 (0)

A little googling shows that rsync reports this error when it tries to copy a file that has an ACL for a user that doesn't exist anymore on the system.

Of course, rsync doesn't return a 0 exit status since it doesn't backup the problematic files/directories. The annoying thing (well, depends on which side you stand), is that rsync doesn't report which file is problematic.

When rsync reports this error hundreds of times, you can either spend hours and hours trying to figure out which files are problematic...

OR you can fire up a Terminal and run a little shellscript (can take some time) :

find /Your/Path -exec ls -lde {} \; | grep -E "[0-9]{1}: [0-9A-Z]{8}-[0-9A-Z]{4}-[0-9A-Z]{4}-[0-9A-Z]{4}-[0-9A-Z]{12}" -B 1

This will return all files that have ACE for a user that doesn't exist anymore. A quick chmod on the culprits should do the rest of the job and let rsync run peacefully.

Tip : adjust the value of the -B option if you have a lot of ACLs.

OS X : play with Extended Attributes

parse error

What are extended attributes ?

Extended Attributes (often shortened as xattrs) are a feature of (modern) filesystems that allows users to associate metadata with a file. These metadata are pairs of name-value that allows users to further describe a file. Extended Attributes is a very valuable tool since they allow users to add semantics to their files. For example, you can store the author(s) of a document in xattrs (authors: "John Doe, Jane Doe"). In macOS, xattrs are used to store tags (amongst other things).

The Extended Attributes are not stored in the file, they are stored in the filesystem itself and associated with the file they describe.

Guess what ? I LOVE extended attributes. macOS makes a pretty smart use of them. For example, when you download a file from the Internet, macOS automatically adds a bunch of xattrs to the downloaded file:

  • com.apple.quarantine that tells the system "Hey, this file might be dangerous, please manipulate with caution !" ;
  • com.apple.metadata:kMDItemDownloadedDate that stores the download date and time ;
  • com.apple.metadata:kMDItemWhereFroms that stores the URL from which the file was downloaded.

When used to store tags, it becomes a very powerful tool that allows you to get rid of the old tree metaphor to organise your files. Looking for photos of Uncle Ben ? Just search for them with the appropriate tag, no need to know where the files are.

How to work with xattrs (from the CLI) ?

Reading and listing

To list files with their extended attributes, you can simply use the @ flag of the ls command:

$ ls -l@
total 0
drwx------+ 12 francois  staff   408 19 sep 14:58 Desktop
drwx------+ 22 francois  staff   748  8 mai 22:23 Documents
drwx------+ 14 francois  staff   476 27 sep 14:37 Downloads
drwx------@ 63 francois  staff  2142 21 sep 14:10 Library
        com.apple.FinderInfo      32 
drwx------@  3 francois  staff   102  3 déc  2014 Movies
        com.apple.metadata:_kMDItemUserTags   50 
drwx------+  4 francois  staff   136  2 jui  2015 Music
drwx------+  5 francois  staff   170 25 jan  2016 Pictures
drwxr-xr-x+  5 francois  staff   170  3 déc  2014 Public

Notice that the permissions column also ends with an @ when the file has extended attributes. This is also the case when you list the directory with the simpler ls -l.

To list xattrs and their value, you can use xattr with the -l flag:

$ xattr -l ~/Downloads/KUBLER\ François\ -\ CV.pdf
com.apple.metadata:kMDItemDownloadedDate:
00000000  62 70 6C 69 73 74 30 30 A1 01 33 41 BD 9A 9F 8C  |bplist00..3A....|
00000010  F4 E8 53 08 0A 00 00 00 00 00 00 01 01 00 00 00  |..S.............|
00000020  00 00 00 00 02 00 00 00 00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 13                                   |.....|
00000035
com.apple.metadata:kMDItemWhereFroms:
00000000  62 70 6C 69 73 74 30 30 A1 01 6F 10 3F 00 68 00  |bplist00..o.?.h.|
00000010  74 00 74 00 70 00 73 00 3A 00 2F 00 2F 00 66 00  |t.t.p.s.:././.f.|
00000020  72 00 61 00 6E 00 63 00 6F 00 69 00 73 00 2E 00  |r.a.n.c.o.i.s...|
00000030  6B 00 75 00 62 00 6C 00 65 00 72 00 2E 00 6F 00  |k.u.b.l.e.r...o.|
00000040  72 00 67 00 2F 00 6D 00 69 00 73 00 63 00 2F 00  |r.g./.m.i.s.c./.|
00000050  4B 00 55 00 42 00 4C 00 45 00 52 00 25 00 32 00  |K.U.B.L.E.R.%.2.|
00000060  30 00 46 00 72 00 61 00 6E 00 E7 00 6F 00 69 00  |0.F.r.a.n...o.i.|
00000070  73 00 25 00 32 00 30 00 2D 00 25 00 32 00 30 00  |s.%.2.0.-.%.2.0.|
00000080  43 00 56 00 2E 00 70 00 64 00 66 08 0A 00 00 00  |C.V...p.d.f.....|
00000090  00 00 00 01 01 00 00 00 00 00 00 00 02 00 00 00  |................|
000000A0  00 00 00 00 00 00 00 00 00 00 00 00 8B           |.............|
000000ad
com.apple.quarantine: 0083;57ea6802;Safari;D930F473-4B79-4AB2-8179-619CC9491A43

As you can see, macOS xattrs often have cryptic values. This is because they are meant to be used by macOS directly (and exposed through the Finder, for example). They don't have to be human readable.

If you want to focus on a specific xattr, the -p flag is your friend:

$ xattr -p com.apple.quarantine ~/Downloads/KUBLER\ François\ -\ CV.pdf
0083;57ea6802;Safari;D930F473-4B79-4AB2-8179-619CC9491A43

Writing

Creating an extended attribute is very easy:

$ touch ./test.pdf
$ xattr -w my_xattr "My Value" ./test.pdf

As you can see, you only have to provide the name of the extended attribute and its value.

Now let's read it:

$ xattr -p my_xattr ./test.pdf
My Value

Deleting

Deleting an extended attribute is also very easy. The -d flag allows you to delete a specific xattr whereas the -c flag will clear all extended attributes for the given file.

Finding files

macOS comes with a very useful command called mdfind that finds files matching a given query.

For example, if you are looking for files that have been downloaded from francois.kubler.org, you could run the following:

$ mdfind "kMDItemWhereFroms == '*francois.kubler.org*'"
/Users/francois/Downloads/KUBLER François - CV.pdf

To find files tagged with the keyword résumé:

$ mdfind "kMDItemUserTags == 'résumé'"
/Users/francois/Downloads/KUBLER François - CV.pdf

Let's try something else:

$ mdfind "my_xattr == '*Value*'"
$

As you can see, if we try to find the file for which we previously added a custom xattr, we don't get any result :(

Let's open a little parenthesis.

My guess is that macOS only indexes (= import in its metadata database) xattrs that are in the com.apple.metadata namespace. Using mdls seems to confort this assertion:

$ mdls test.pdf
_kMDItemOwnerUserID            = 501
kMDItemContentCreationDate     = 2016-09-27 12:37:20 +0000
kMDItemContentModificationDate  = 2016-09-27 12:37:21 +0000
kMDItemContentType             = "com.adobe.pdf"
kMDItemContentTypeTree         = (
    "com.adobe.pdf",
    "public.item",
    "com.adobe.pdf",
    "public.data",
    "public.composite-content",
    "public.content"
)
kMDItemDateAdded               = 2016-09-27 12:37:22 +0000
kMDItemDisplayName             = "test.pdf
kMDItemFSContentChangeDate     = 2016-09-27 13:10:03 +0000
kMDItemFSCreationDate          = 2016-09-27 13:10:03 +0000
kMDItemFSCreatorCode           = ""
kMDItemFSFinderFlags           = 0
kMDItemFSHasCustomIcon         = (null)
kMDItemFSInvisible             = 0
kMDItemFSIsExtensionHidden     = 0
kMDItemFSIsStationery          = (null)
kMDItemFSLabel                 = 0
kMDItemFSName                  = "test.pdf"
kMDItemFSNodeCount             = (null)
kMDItemFSOwnerGroupID          = 20
kMDItemFSOwnerUserID           = 501
kMDItemFSSize                  = 0
kMDItemFSTypeCode              = ""
kMDItemKind                    = "Portable Document Format (PDF)"
kMDItemLogicalSize             = 0
kMDItemPhysicalSize            = 0

See ? There is no trace of our previously created My Value value. I'm pretty sure mdfind only looks in this data to retrieve files.

Also notice how useful the output of this command is : it gives you a bunch of values to use with mdfind. For example, you now know how to find PDF files:

$ mdfind "kMDItemContentType == 'com.adobe.pdf'"

End of parenthesis.

And what about Linux ?

Nowadays, most distros come with a Kernel compiled with extended attributes support enabled. The most widely used filesystems on Linux (ext3, ext4, btrfs, ...) support xattrs. However, developers can't be sure that the system will actually support xattrs, which, of course, is a pain if your app depends on them. macOS doesn't have that kind of issue since Apple controls the whole software stack.

So, to my knownledge, a very few Linux applications make use of extended attributes. The biggest exceptions might be SELinux (that simply relies on them) and KDE's Baloo framework (for which I made a proposal to use xattrs to store tags in 2013).

It's also interesting to see that freedesktop.org has a page dedicated to extended attributes. So, maybe we will see improvements in the near future ? Let's hope so !

Anyway, if you are interested in using xattrs on your Linux machine, you should read the manpages for xattr, setfattr and getfattr.

Like me, you'll probably be disappointed by the lack of tools provided by Linux to deal with xattrs. For example, there is no ls -l@, cp or rsync don't preserve extended attributes unless you specify the good options (respectively --preserve=xattr and -X or --xattrs), and so on.

Going further here is out of the scope of this post, but feel free to get in touch with me if you want to know more about extended attributes in Linux.

rsync : pack_smb_acl: sys_acl_init(): Cannot allocate memory (12) error

parse error

A few days ago, my (self-made) backup system (based on rsync) started to complain :

rsync: pack_smb_acl: sys_acl_init(): Cannot allocate memory (12)

Googling the error led me to this thread where the almighty Mike Bombich was asking some question about rsync source code. To demonstrate his issue, he wrote :

On Mac OS X, a file with the system limit of 128 ACEs will consistently run into the error noted in the subject.

D*mn ! What if some file somewhere on the server had more than 128 ACEs ? That could explain the error !

So, once more, I fired up Terminal and wrote a little shellscript to find files that have a suspicious high number of ACEs.

First, write a very simple program that will count ACEs :

#!/bin/sh

nb=$(ls -lde "$1" | wc -l | tr -d " ")
nbACE=$(($nb-1))

if [ $nbACE -gt 10 ]    # Change "10" with the value that best suits your need.
then
    echo "$1 ($nbACE)"
fi

exit 0

Save it as nbACE.sh and make it runnable.

And then, just use find like this (and as root) :

find "/Your/Path" -exec nbACE.sh {} \; > ./TooManyACEs.log

It can take a while to run, so be patient :) But when finished, you'll find the culprit(s) in TooManyACEs.log.

The WABAC Machine : Reloaded

parse error

Lorsque j'ai découvert la Time Machine d'Apple début 2009, je suis tout de suite tombé sous son charme : un système simple et intelligent de sauvegarde totalement intégré à l'OS et relativement transparent pour l'utilisateur. Voilà une avancée dans le domaine de l'informatique grand public !

Je dois avouer que j'ai été un peu jaloux aussi : pourquoi n'aurait-on pas droit à la même chose sous Linux ? Je me suis donc lancé dans l'écriture d'un programme similaire basé sur l'excellent rsync. Ça a donné the WABAC Machine, un shellscript de quelques dizaines de lignes.

Bien entendu, je n'en suis pas resté là et le programme a pas mal évolué en 4 ans. Hier, je me suis dit qu'il était temps de faire les choses correctement et j'ai décidé de créer un dépôt GitHub pour ce petit projet. Ceux que ça intéresse peuvent désormais participer à son développement :)

OS X : let's play with security

parse error

I have a few shellscripts that need username and passwords to accomplish some tasks. For example, I wrote a shellscript to backup some data in an encrypted sparsebundle file that is stored on an AFP share. So I have a username and a password to mount the share, and some others credentials to mount the sparsebundle.

When it came to automating the backup process, I had to face a simple issue : where do I store these usernames and passwords ?

Hard-code them in the script ? Naaah.

Create some kind of read-only file with proper permissions ? Sounds rather risky.

So, what would be the right way to do this with OSX ?

The answer is kind of obvious : OSX ships with a nice tool called Keychain Access.app. This service is intended to store and retrieve credentials, keys, passwords, certificates,... in a secure fashion. But how can I access it from the shell ?

This is where security comes to the rescue ! security is a Command Line Interface that allows you to administer and manipulate keychains, keys and certificates. In other words, security allows you to access Keychain (the app) from the shell (local shell ! SSH ! shellscript !). Wonderful, isn't it ?

Although it comes with a manpage, let's see some examples.

Adding items to a keychain

First, let's add a generic password to the default keychain. Here is what I'm using to store the credentials for my sparsebundle file :

security add-generic-password -a "username" -D "Image Disk Password" -s "MyFile.sparsebundle" -w "THis_IS_so_s3crEt_!"

The -a option allows you to specify the account name. -D is for the description. -s allows you to specify the service name (here I chose to set it to the sparsebundle filename but you can put whatever you want). And -w allows you to specify the password.

Adding an Internet password is a little bit different :

security add-internet-password -a "username" -D "AFP Share Password" -s "myserver.example.com" -p "share" -r "afp " -l "My AFP Share" -w "THis_IS_4lso_very_s3crEt_!" /Library/Keychains/System.keychain

This time we had to specify the server with -s, a path (here, the name of the shared point) with -p, a label with -l and the protocol (could have been http, ftp, or whatever) with -r.

WARNING : the argument for the -r option MUST be exactly 4 characters long. So you have to write "afp ", not "afp".

Please also notice that we decided to add this entry to the System keychain which is located in /Library/Keychains/System.keychain. We could have chosen to add it to another existing keychain by providing the path to this keychain, or we could have created a brand new one with :

security create-keychain /path/to/my/new/Personnal.keychain

Retrieving items

Retrieving an item in a keychain is pretty easy. Here I choose to retrieve the item by service (remember the -s option we used when we created the entry ?) :

security find-generic-password -g -s "MyFile.sparsebundle"

Which outputs the following :

keychain: "/Users/francoiskubler/Library/Keychains/login.keychain"
class: "genp"
attributes:
    0x00000007 <blob>="MyFile.sparsebundle"
    0x00000008 <blob>=<NULL>
    "acct"<blob>="username"
    "cdat"<timedate>=0x32303134313232393130323735365A00  "20141229102756Z\000"
    "crtr"<uint32>=<NULL>
    "cusi"<sint32>=<NULL>
    "desc"<blob>="Image Disk Password"
    "gena"<blob>=<NULL>
    "icmt"<blob>=<NULL>
    "invi"<sint32>=<NULL>
    "mdat"<timedate>=0x32303134313232393130323735365A00  "20141229102756Z\000"
    "nega"<sint32>=<NULL>
    "prot"<blob>=<NULL>
    "scrp"<sint32>=<NULL>
    "svce"<blob>="MyFile.sparsebundle"
    "type"<uint32>=<NULL>
password: "THis_IS_so_s3crEt_!"

But I could have chosen to retrieve it by account name (remember the -a option ?) :

security find-generic-password -g -a "username"

Or by account name AND service AND description in a specific keychain file :

security find-generic-password -g -a "username" -s "MyFile.sparsebundle" -D "Image Disk Password" /path/to/my/new/Personnal.keychain

If you have more than one entry that matches your criteria, security will only output the first one that matches. So, be careful when you add a new entry in a keychain, and make sure you'll be able to retrieve it easily later by specifying a different label (-l option) or a comment (-j option) or whatever you prefer.

Parsing the output

Obviously, the output of the command is kind of useless and needs to be parsed. On OSX prior to 10.9, when retrieving a password with the -g option (which asks security to print the password), the password was printed on stderr, so you had to play with shell redirections and write something like the following :

my_precious=$(security 2>&1 > /dev/null find-generic-password -g -s "MyFile.sparsebundle" | cut -d "\"" -f 2)

Starting with OSX 10.9, you can drop the -g flag and use -w instead :

my_precious=$(security find-generic-password -w -s "MyFile.sparsebundle")

The cool part here is that you can retrieve more than just the password. You can also retrieve the username (tested on OSX >= 10.6) :

output=$(security 2>&1 find-internet-password -s "myserver.example.com" -r "afp " -g)
username=$(head -n 7 <<< "$output" | tail -n 1 | cut -d "\"" -f 4)
my_precious=$(head -n 1 <<< "$output" | cut -d "\"" -f 2)

Of course, depending on your needs, you'll have to modify the commands, but you should now get the idea. The most important part here is to remember that when called with the -g option, security will print the password to stderr instead of stdout.

More : Certificates, Keys, ...

There are a bunch of option that allows you to manage certificates, keys and identities (certificate + private key). I've never had to deal with that stuff so I can't provide any decent example. But they do exist and a quick look at the manpages should be enough to help you.

OS X : play with RAID

parse error

Two Mac Mini

Since the release of El Capitan in september 2015, Disk Utility.app has lost all features related to RAID. No more GUI if you want to build or repair a RAID set. Pretty annoying, isn't it ?

Well, Apple's strategy is pretty clear : hide what's unecessary to common people, but still give possibilities through the Terminal. Which is pretty cool, because I love the CLI. That's not really a good excuse and they could probably find a better way to expose power-user functions through the GUI, but at least it's consistent.

So, for this example, we're going to build a RAID 1 (mirroring) set with two external USB disks.

First, let's get a list of all available disks:

host:~ francois$ diskutil list

/dev/disk0 (internal, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *500.1 GB   disk0
   1:                        EFI EFI                     209.7 MB   disk0s1
   2:                  Apple_HFS Macintosh HD            499.2 GB   disk0s2
   3:                 Apple_Boot Recovery HD             650.0 MB   disk0s3
/dev/disk1 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk1
   1:                        EFI EFI                     209.7 MB   disk1s1
   2:                  Apple_HFS G-DRIVE USB             4.0 TB     disk1s2
/dev/disk2 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:      GUID_partition_scheme                        *4.0 TB     disk2
   1:                        EFI EFI                     209.7 MB   disk2s1
   2:                  Apple_HFS G-DRIVE USB             4.0 TB     disk2s2

Here we can see that we have 3 disks:

  • The first one is an internal hard disk drive of 500 GB called /dev/disk0,
  • The second one is an external hard disk drive of 4 TB called /dev/disk1,
  • And the third one is also an external hard disk drive of 4 TB called /dev/disk2.

It is very important to make sure you can identify the disks without any doubt ! Ok ? Let's ask diskutil to build a RAID 1 set with disk1 and disk2. The resulting volume should be formatted with a journaled HFS+ filesystem and be named backup:

host:~ francois$ sudo diskutil appleRAID create mirror backup JHFS+ disk1 disk2

Started RAID operation
Unmounting proposed new member disk1
Unmounting proposed new member disk2
Repartitioning disk1 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk1s2 as a data slice
Repartitioning disk2 so it can be in a RAID set
Unmounting disk
Creating the partition map
Using disk2s2 as a data slice
Creating a RAID set
Bringing the RAID partitions online
Waiting for the new RAID to spin up "EE055FED-DDC8-4461-87E8-26CBBCBCC599"
Initialized /dev/rdisk3 as a 4 TB case-insensitive HFS Plus volume with a 311296k journal
Mounting disk
Finished RAID operation

Everything's allright. Right ? Let's check:

host:~ francois$ diskutil appleRAID list

AppleRAID sets (1 found)
===============================================================================
Name:                 backup
Unique ID:            EE055FED-DDC8-4461-87E8-26CBBCBCC599
Type:                 Mirror
Status:               Online
Size:                 4.0 TB (4000443039744 Bytes)
Rebuild:              manual
Device Node:          disk3
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk1s2   1A1937AF-8B61-4A64-A2A8-C890C2B5E9B7  Online     4000443039744
1  disk2s2   96713456-F4FD-401D-B6B5-F518F977DEEA  Online     4000443039744
===============================================================================

We can see that we have a RAID set named backup, of type Mirror, built with two disks that are both Online. So yes, everything seems fine ! That was easy, wasn't it ?

diskutil also gave our RAID set a unique identifier (UUID) that will allow us to manipulate it. In our case, this UUID is EE055FED-DDC8-4461-87E8-26CBBCBCC599.

Still, the Rebuild option is set to manual, which means we'll have to manually rebuild the RAID set if a disk fails. Let's ask OS X to try to rebuild it automatically in case of a failure:

host:~ francois$ sudo diskutil appleRAID update AutoRebuild 1 EE055FED-DDC8-4461-87E8-26CBBCBCC599

Password:
The RAID has been successfully updated

host:~ francois$ diskutil appleRAID list

AppleRAID sets (1 found)
===============================================================================
Name:                 backup
Unique ID:            EE055FED-DDC8-4461-87E8-26CBBCBCC599
Type:                 Mirror
Status:               Online
Size:                 4.0 TB (4000443039744 Bytes)
Rebuild:              automatic
Device Node:          disk3
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk1s2   1A1937AF-8B61-4A64-A2A8-C890C2B5E9B7  Online     4000443039744
1  disk2s2   96713456-F4FD-401D-B6B5-F518F977DEEA  Online     4000443039744
===============================================================================

Good !

So far, we only have 2 disks. In case of a failure, diskutil will try to re-use the failed drive to rebuild the RAID set. But if we had a third hard disk drive available, we could have asked OS X to use it as a spare drive:

host:~ francois$ sudo diskutil appleRAID add spare disk4 EE055FED-DDC8-4461-87E8-26CBBCBCC599

Started RAID operation on disk3 backup
Unmounting disk
Repartitioning disk4 for RAID
Adding disk4s2 to RAID Set
Finished RAID operation on disk3 backup

host:~ francois$ diskutil appleRAID list

AppleRAID sets (1 found)
===============================================================================
Name:                 backup
Unique ID:            EE055FED-DDC8-4461-87E8-26CBBCBCC599
Type:                 Mirror
Status:               Online
Size:                 4.0 TB (4000443039744 Bytes)
Rebuild:              automatic
Device Node:          disk3
-------------------------------------------------------------------------------
#  DevNode   UUID                                  Status     Size
-------------------------------------------------------------------------------
0  disk1s2   1A1937AF-8B61-4A64-A2A8-C890C2B5E9B7  Online     4000443039744
1  disk2s2   96713456-F4FD-401D-B6B5-F518F977DEEA  Online     4000443039744
-  disk4s2   B5069883-83D0-44F2-8D78-CCF3F1582595  Spare      4000443039744
===============================================================================

Note that adding a spare drive to a RAID set will also automatically set the Rebuild option to automatic.