I distribute binary packages of libgourou in both compiled version (tar.gzip) and AppImage (and maybe .deb in the future). AppImage is a quite old packaging system (2004), not deprecated, but that evolve slowly. It allows to run application without the need to install additional libraries (all is bundled into one binary package). There is not so much tools for building an Appimage and I had to rely mines on a Python bundle not really maintained on Github. Not perfect, but does the job and result worked fine for a while. Unfortunately, starting this summer, there was an issue in running packaged applications. It took me a lot of time to find it, but I finally did it !

First, let's try to run already packaged AppImage

> cd /tmp/libgourou_utils-0.8.7-x86_64.AppImage
> ./acsmdownloader -h
Unable to launch '/tmp/.mount_acsmdoKfAEabacsmdownloader'

If you look in details with strace

> strace ./acsmdownloader -h
execve("/tmp/.mount_acsmdojHgmJpacsmdownloader", ["/tmp/.mount_acsmdojHgmJpacsmdown"..., "-h"],
0x561f614f3790 /* 95 vars */) = -1 ENOENT (No such file or directory)

"ENOENT (No such file or directory)" is confusing and really non obvious to understand, because the file is there and has execution permission ! Most of the time, it means you try to run an application that has been compiled for another architecture (x86 32 bits, arm ...).

This error is raised by a core component of Linux systems : ld-linux-XXX.so. It's used to load ELF file into memory and resolve all dependencies (shared libraries). Useful information can be found with file util

> file /usr/bin/tail
/usr/bin/tail: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked,
interpreter /lib64/ld-linux-x86-64.so.2,
BuildID[sha1]=0d3a849e80ee804476c13ed25b3b1ba819847b35, for GNU/Linux 3.2.0, stripped

AppImage is an ELF binary statically linked

> file libgourou_utils-0.8.7-x86_64.AppImage
libgourou_utils-0.8.7-x86_64.AppImage: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), static-pie linked,
BuildID[sha1]=b15b018a4f56042ad61c2b9e0504773f7da1ac38, stripped

It contains compiled binaries and libraries in a squashfs filesystem mounted by libfuse (which is its only dependency). We can access to them with

> ./acsmdownloader --appimage-extract
> cd squashfs-root

> strace ./acsmdownloader
execve("./acsmdownloader", ["./acsmdownloader"], 0x7ffdeb43fa30 /* 49 vars */) = -1 ENOENT (No such file or directory)
strace: exec: No such file or directory

Our core error, let's look deeper

> file acsmdownloader 
acsmdownloader: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked,
interpreter usr/lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0,
BuildID[sha1]=aaefe8dfc25186c9c3bb2c0e254f8ba09cdd7f7b, not stripped

We can see that the path of ld-linux-x86-64.so.2 is usr/lib64/ld-linux-x86-64.so.2 which is invalid !!!. Ahead "/" is missing (it has been removed by builder Python script). Moreover, since version 13 (or a bit earlier), Debian seems to have moved ld-linuxXXX.so from /lib64 to /usr/lib64 which break compatibility.

My solution is to patch binaries with patchelf in AppImageBuilder.yml, after they are copied into AppDir/usr/bin/

find AppDir/usr/bin/ -maxdepth 1 -type f -executable -exec  patchelf --set-interpreter /usr/lib64/ld-linux-x86-64.so.2 \{\} \;

Après plusieurs mois de gel, permettant des tests approfondis, la dernière version stable de Debian (nom de code Trixie) est sortie le 9 août !

Qu'est-ce que Debian ? On pourrait dire basiquement qu'il s'agit d'un système d'exploitation. Pour être exact, Debian est une "distribution", c'est à dire un assemblage de tous les composants nécessaires à la réalisation d'un système d'exploitation. La nuance tient au fait que chaque composant (ou presque) est développé par des équipes indépendantes. Les composants majeurs sont le noyau (ici Linux), l'environnement de bureau (GNOME, KDE...), les gestionnaire de logiciels/paquets (apt, fait par Debian), l'installeur (fait par Debian) et tous les logiciels (dont les plus connus : LibreOffice, Chrome, Firefox, Gimp ...).

Organisé sous forme de fondation à but non lucratif, le travail des développeurs Debian consiste donc à s'assurer que tout ces composants fonctionnent ensemble. Côté grand public, on a plutôt tendance à résumer les choses en parlant de Linux, sans faire référence à la distribution. Pourtant, il y a des milliers de personnes de part le monde qui développent (la plupart du temps bénévolement) tous ces logiciels.

Debian est l'une des plus anciennes distributions Linux (première version en 1993), sortie seulement 2 ans après la première version du noyau. Il y a donc beaucoup de distributions dérivées de cette dernière. Un peu comme Android qui est développé par Google, mais qui est personnalisé par les constructeurs de téléphones. Mes serveurs fonctionnent tous grâce à Debian. C'est également le cas pour mon PC personnel depuis plus de 15 ans ! Mieux encore, je l'ai aussi installé sur un ordinateur plus ancien lors du passage forcé à Windows 10 (l'histoire se répète avec Windows 11...), alors que l'ordinateur en question n'était pas compatible. C'est donc un système utilisable par le grand public, pour peu que l'on soit un peu curieux en cas d'erreur...

Ma découverte de ce système s'est faite lors de mes études à l'IUT où nous avions accès à des serveurs sous Linux (avec une interface graphique nativement accessibles à distance grâce au serveur X \o/). Mais à l'époque, l'environnement manquait de maturité et n'était pas facilement exploitable par tout le monde. L'interface graphique primaire de GNOME était assez moche comparé à celle de Windows XP. Je me souviens également avoir partitionné un disque dur pour y installer la distribution Mandrake (il y avait plusieurs CD) quand j'étais au lycée. Ça marchotait, mais sans soutien extérieur (Internet était plus basique, avec seulement une connexion ADSL à 128kb/s et la plupart des ressources an Anglais), j'ai rapidement abandonné. Il faut dire qu'en ces temps, bien que curieux, j'étais un Windowsien convaincu.

Vint ensuite l'école d'ingénieur. Le Conseil Général nous a gracieusement prêté des ordinateurs portables pour notre cycle supérieur. Ils étaient configurés avec la distribution Fedora (et une interface KDE proche de celle de Windows). Et puis la révolution Ubuntu est arrivée, avec son bureau à effet whaou Compiz (comprendre : avec des animations 3D et de la transparence dans les fenêtres). À l'époque, Canonical (la société derrière Ubuntu), offrait gratuitement les CD d'installation ! C'est également à ce moment où j'ai commencé à m'intéresser à la couche système et aux systèmes embarqués, en gros : mon métier ! L'outillage disponible dans le monde Linux (et qui est directement hérité de l'univers UNIX, système historique de l'informatique moderne) est sans commune mesure par rapport à Windows, qui a toujours eu une approche utilisateur standard : applications métiers, bureautique, jeux. Après ce (petit) passage avec Ubuntu, j'ai migré sur Debian, pour ne plus jamais le lâcher !

Quelle plaie de voir aujourd'hui des entreprises m'imposer un Windows sur ma station de travail (et tout particulièrement la version 11 qui est une bouse sans nom). Mon ordinateur d'il y a 15 ans (un Core i5 M450 quad core) avec Linux est plus rapide que celui du travail avec son processeur Intel Core i7 vPro, ses 14 cœurs et ses 32GO de RAM. Tout ça pour simplifier le travail des administrateurs systèmes qui ne veulent pas gérer plusieurs systèmes. Pour respirer un peu, il y a bien les machines virtuelles (parfois limitées) et/ou le sous-système Linux (WSL), mais c'est loin d'être parfait (surtout quand il faut accéder aux périphériques).

La mise à jour de mon serveur avec cette nouvelle version s'est bien déroulée, mis à part la configuration mail (toujours aussi cryptique) qu'il a fallu retravailler un peu, quelques soucis PHP et Django, ainsi que le serveur de nom de domaine (bind9). J'en ai profité pour recompiler mon noyau Linux afin d'être aligné avec celui de Debian.

NB: Le serveur X a longtemps été le serveur graphique utilisé par Linux. En cas de soucis, une recherche sur internet avec ces termes ne donnait pas forcément les réponses attendues...

(Ooops) I did it again ! After doing the CRC32 optimization, I tried the same for CRC16-CCITT. This one is harder (but not so hard) to optimize for a C compiler because in modern CPU we mainly have 32 bits/64 bits registers, but for CRC16, we have to play with 16 bits values, split into upper and lower parts which need shift + mask operations.

Whatever, context is the same : target is ARM Cortex M4, no data/instruction cache, SRAM memory and GCC 9. One interesting point is that this time, I didn't target armv6, but armv7 (we'll see why later). Figures are still impressive, with a gain between 50% and 60% !

• -Os compilation : 21.7 milliseconds
• -O2 compilation : 17.2 milliseconds
• -Os + optimizations : 8.8 milliseconds

I used the same optimization tricks than CRC32 (see article) plus this ones :

Use specific instruction if you can

ARMv7 instruction set provides thumb2 instructions which contains bitfield extraction. This is really really (yes 2 times) nice ! instruction ubfx (and variants) allows to extract a specified range of bits from a register and thus avoid to do shift + mask (2 instructions).

Be careful on instruction size

Thumb2 is really nice because you can mix 16 bits and 32 bits instructions. But, in order to save space (thus speed), you have to carefully choose your instructions. The case here is :

lsrs r5, r5, #24 C equivalent r5 = r5 >> 24

and

ubfx r5, r5, #24, #8 C equivalent r5 = (r5 & 0xff000000) >> 24

They both do have the same result but the first one is an "old" instruction and can be encoded on 16 bits while the second is new and is encoded into 32 bits.

Don't take care on unused register part

At some point, I do a 32 bits xor operation which generate random values on bits 31..15. But we don't care because we have to focus on 16 bits lower part.

Here is the optimized function. Whole C file can be found here. Optimization is effective for 16 bytes blocks (aligned).

uint16_t crc16_ccitt_opt16(
        const unsigned char*     block,
        unsigned int            blockLength,
        uint16_t          crc)
{
    /* unsigned int i; */

    /* for(i=0U; i<blockLength; i++){ */
    /*     uint16_t tmp = (crc >> 8) ^ (uint16_t) block[i]; */
    /*     crc = ((uint16_t)(crc << 8U)) ^ crc16_ccitt_table[tmp]; */
    /* } */

/*
      r0 -> s
      r1 -> len
      r2 -> crc16val
      r3 -> crc16tab
      r4 -> curval[0]
      r5 -> (crc >> 8) ^ (uint16_t) block[i]
      r6 -> crc16_ccitt_table[(crc >> 8) ^ (uint16_t) block[i])
      r7 -> curval[1]
      r8 -> curval[2]
      r9 -> curval[3]
     */
    __asm__ volatile (
        "mov r0, %1\n"
        "mov r1, %2\n"
        "mov r2, %3\n"
        "mov r3, %4\n"

        "push {r7, r8, r9}\n"

        "crc16_opt16_loop:\n"
        "ldm r0!, {r4, r7, r8, r9}\n"

        // curval[0]
        "eor r5, r4, r2, lsr #8\n"
        "uxtb r5, r5\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r4, r2\n"
        "ubfx r5, r5, #8, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r4, r2, lsl #8\n"
        "ubfx r5, r5, #16, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r4, r2, lsl #16\n"
        "lsrs r5, r5, #24\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        // curval[1]        
        "eor r5, r7, r2, lsr #8\n"
        "uxtb r5, r5\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r7, r2\n"
        "ubfx r5, r5, #8, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r7, r2, lsl #8\n"
        "ubfx r5, r5, #16, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r7, r2, lsl #16\n" 
        "lsrs r5, r5, #24\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        // curval[2]        
        "eor r5, r8, r2, lsr #8\n"
        "uxtb r5, r5\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r8, r2\n"
        "ubfx r5, r5, #8, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r8, r2, lsl #8\n"
        "ubfx r5, r5, #16, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r8, r2, lsl #16\n"
        "lsrs r5, r5, #24\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        // curval[3]        
        "eor r5, r9, r2, lsr #8\n"
        "uxtb r5, r5\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r9, r2\n"
        "ubfx r5, r5, #8, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r9, r2, lsl #8\n"
        "ubfx r5, r5, #16, #8\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"
        "eor r2, r6, r2, lsl #8\n"

        "eor r5, r9, r2, lsl #16\n"
        "lsrs r5, r5, #24\n\n"
        "ldrh r6, [r3, r5, lsl #1]\n"

        // Last two lines inverted
        "subs r1, r1, #16\n"
        "eor r2, r6, r2, lsl #8\n"

        "bne crc16_opt16_loop\n"

        "pop {r7, r8, r9}\n"
        "strh r2, %0\n"
        : "=m" (crc)
        : "r" (block), "r" (blockLength), "r" (crc), "r" (crc16_ccitt_table)
          // Missing r7-r9, manually save it
        : "r0", "r1", "r2", "r3", "r4", "r5", "r6"
        );

    return crc;
}

We can see that computation is not the same for all parts of the 32 bits register while it was really symmetric in CRC32.

Code has to be compiled with minimum -O1 or -Os option

For comparison, the (quite good) code generated by GCC 12 with -Os, working on a single byte :

 594:   428b            cmp     r3, r1
 596:   d100            bne.n   59a <crc16_ccitt+0x12>
 598:   bd30            pop     {r4, r5, pc}
 59a:   f813 2b01       ldrb.w  r2, [r3], #1
 59e:   0204            lsls    r4, r0, #8
 5a0:   b2a4            uxth    r4, r4
 5a2:   ea82 2210       eor.w   r2, r2, r0, lsr #8
 5a6:   f835 2012       ldrh.w  r2, [r5, r2, lsl #1]
 5aa:   ea84 0002       eor.w   r0, r4, r2
 5ae:   e7f1            b.n     594 <crc16_ccitt+0xc>

It's clearly focus on Armv6 compatibility as it use masking operation + shift at lines 59e and 5a0.

At work I played with an ultra low power SoC powered by a single core ARM Cortex M4. To check some data integrity, we have to use CRC32 but there is no hardware peripheral to speed up computation and ARMv6 doesn't have special instruction for this (it starts from Armv8.1). After some researches, I was surprised not to find any optimized implementation on Internet. So, I wrote it by myself and the result is quite impressive : my version is ~50% faster ! Some figures (on ~30KB of data) :

• -Os compilation : 19.4 milliseconds
• -02 compilation : 15.2 milliseconds
• -0s + optimizations : 8.2 milliseconds

Here, we have to consider that Cortex M4 doesn't have Data nor Instruction cache and memory accesses are done on a SRAM. Compilation is done in thumb mode with GCC 9.

Original version is the one from Wang Yaofu licensed under Apache2. It's quite simple and very academic. C primitives doesn't allows to optimize this algorithm so much because CRC has to be processed byte by byte, so we have to do some assembly !

I used multiple optimization tricks :

Use all registers available

The idea is to play with registers from r0 to 10 and not be limited to r0-r5 as commonly used.

Unroll loop

Avoid to break CPU pipeline by doing some checks + jump. Code is bigger and repetitive, but faster. We may write macro to reduce source code, not my choice here.

Do memory burst instead of unitary access

Especially when there is no cache, memory burst accesses are really faster. Here we load 4*32 bits at a time and keep all data into registers. Bytes outside burst window are computed using non optimized version.

Use shifts and rotates within load and eor instructions

ARM instructions allows to shift registers values within load and eor (and some other instructions) without having to do it in a separate line.

Avoid pipeline register lock

When it's possible, we can invert assembly lines to avoid working on the same registers on consecutive instructions (and thus avoid to lock them).

Update condition flags in sub instruction

Use subs variant to update EQ flag and avoid to check it for 0 in a separate instruction.

Do aligned accesses

In the calling function there is some code to inject "s" as a 32 bits aligned pointer (extra bytes processed by standard code).

Here is the optimized function. Whole C file can be found here

/**
 * Optimized version of _update_crc32 for 16 bytes blocks
 */
static void _update_crc32_opt16(const unsigned char *s, unsigned int len)
{
    /* unsigned int i; */

    /* for (i = 0;  i < len;  i++) { */
    /*     crc32val = crc32_tab[(crc32val ^ s[i]) & 0xFF] ^ ((crc32val >> 8) & 0x00FFFFFF); */
    /* } */

    /*
      r0 -> s
      r1 -> len
      r2 -> crc32val
      r3 -> crc32tab
      r4 -> curval[0]
      r5 -> (crc32val ^ s[i]) & 0xFF
      r6 -> crc32_tab[(crc32val ^ s[i]) & 0xFF]
      r7 -> curval[1]
      r8 -> curval[2]
      r9 -> curval[3]
     */
    __asm__ volatile (
        "mov r0, %1\n"
        "mov r1, %2\n"
        "mov r2, %3\n"
        "mov r3, %4\n"

        "push {r7, r8, r9}\n"

        "crc32_opt16_loop:\n"
        "ldm r0!, {r4, r7, r8, r9}\n"

        // curval[0]
        "eor r5, r2, r4\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r4, ror #8\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r4, ror #16\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r4, ror #24\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        // curval[1]        
        "eor r5, r2, r7\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r7, ror #8\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r7, ror #16\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r7, ror #24\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        // curval[2]        
        "eor r5, r2, r8\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r8, ror #8\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r8, ror #16\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r8, ror #24\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        // curval[3]        
        "eor r5, r2, r9\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r9, ror #8\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r9, ror #16\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"
        "eor r2, r6, r2, lsr #8\n"

        "eor r5, r2, r9, ror #24\n"
        "uxtb r5, r5\n"
        "ldr r6, [r3, r5, lsl #2]\n"

        // Last two lines inverted
        "subs r1, r1, #16\n"
        "eor r2, r6, r2, lsr #8\n"

        "bne crc32_opt16_loop\n"

        "pop {r7, r8, r9}\n"
        "str r2, %0\n"
        : "=m" (crc32val)
        : "r" (s), "r" (len), "r" (crc32val), "r" (crc32_tab)
          // Missing r7-r9, manually save it
        : "r0", "r1", "r2", "r3", "r4", "r5", "r6"
        );
}

Code has to be compiled with minimum -O1 or -Os option

For comparison, the (quite good) code generated by GCC 12 with -Os, working on a single byte :

 570:   4288            cmp     r0, r1
 572:   d100            bne.n   576 <_update_crc32+0x12>
 574:   bd30            pop     {r4, r5, pc}
 576:   6814            ldr     r4, [r2, #0]
 578:   f810 3b01       ldrb.w  r3, [r0], #1
 57c:   4063            eors    r3, r4
 57e:   b2db            uxtb    r3, r3
 580:   f855 3023       ldr.w   r3, [r5, r3, lsl #2]
 584:   ea83 2314       eor.w   r3, r3, r4, lsr #8
 588:   6013            str     r3, [r2, #0]
 58a:   e7f1            b.n     570 <_update_crc32+0xc>

Not so far from mine, but main weakness is result write at each loop, which is really time consuming.

Reminder : Libgourou is an open source ADEPT protocol implementation (ePub DRM management from Adobe) that helps download ACSM files on Linux system (and remove DRM).

Libgourou v0.8.7 is out. I just realized that I missed to announce v0.8.6 release and the mysterious v0.8.5 has just been dropped... There is only few updates since v0.8.4, mainly small bugfixes :

• Use of $HOME environment variable if available instead of static /home/XXX
• Fix a use after free for sendHTTPRequest()
• Remove EBX object (that contains DRM information) when removing DRM from PDF
• Handle empty names in adept_load_mgt

I initialy did not planed to release v0.8.7 so early, but libzip has been updated into Debian (v4 -> v5). So I felt it was a good opportunity to provide updated binary release. The nice thing with this version is that I received 3 external contributions, which is great !

You can find source code and binaries in my forge