InEnglish

Debian : Failed to find logical volume at boot

Thursday, 17 January 2019
|
Écrit par
Grégory Soutadé

After further investigations, I found a correct fix to this. In facts, its my configuration that is wrong. I have LUKS on LVM schema like this :

system_group (LVM)
    --> system_crypt (LUKS)
swap_group (LVM)
    --> swap_crypt (LUKS)
home_group (LVM)
    --> home_crypt (LUKS)

To be right handled, you need to declare :

In /etc/crypttab :

home--group-home_crypt UUID=349ca075-2922-4c9c-a52b-8dce587767ea /root/home.key luks
swap--group-swap_crypt UUID=4490ce3c-8700-4e90-81df-250cd3573b7c /root/swap.key luks
system--group-system_crypt UUID=95e39100-25c2-41be-829a-bd84fcb21d0a none luks

use blkid command to get right UUID values

In /etc/fstab :

/dev/mapper/system--group-system_crypt /           ext4    errors=remount-ro 0       1
UUID=6866a661-0424-472c-853e-6daa20d15d74 /boot    ext4    defaults          0       2
/dev/mapper/home--group-home_crypt /home           ext4    defaults          0       2
/dev/mapper/swap--group-swap_crypt none            swap    sw                0       0

In /etc/initramfs-tools/conf.d/resume :

UUID=none

Then, do sudo update_initramfs -u and restart.


It's the second time it happens ! After an update my Debian refuse to boot. This time, I wasn't asked for anything !!

I have an LUKS on LVM configuration. I don't use UUID in my cryptroot and Debian scripts only activate devices with it ! I need to manually add "vgchange" command to mount all devices. Unfortunately, the patched script was overwrote by update.

If it happens, follow this procedure :

  • At boot, wait for the rescue shell (~5 minutes)
  • Enter "vgchange -ay"
  • Enter "exit"

Now, you may be able to boot into your system. Then :

  • Edit /usr/share/initramfs-tools/scripts/local-top/cryptroot
  • Add "vgchange -ay" in wait_for_source() function
  • Update initramfs with "sudo update-initrmafs -u"

Safely reboot !

Tip: keyctl in a bash script

Monday, 06 August 2018
|
Écrit par
Grégory Soutadé

Here is a simple tip to use keyctl in a bash script. keyctl is a wrapper for Linux kernel key management interface. It allows to securely save data in kernel memory. The man documentation is very bigcomplete but I didn't find any example on internet. What I initially wanted to do is to safely store a password entered by user inside a bash shell script and keep private to it (don't share with other processes).

Basically the script looks like :

#!/bin/bash

password=SecretPassword

keyctl new_session > /dev/null
keyid=`keyctl add user mail $password @s`
keyctl show
# echo "KEYID $keyid"
keyctl print $keyid

The first thing to do is to create a new session (to detach the current shared one).

Then we will add the password in the new item "mail". We don't have other choice to set type to "user". The item will be placed into the session keyring (@s). We could create new keyrings to store it with keyctl newring command. The command return item id as a big integer. We can use this integer or its name "%user:mail" for further references.

There is also a command keyctl padd which read data from stdin, but I don't recommend to use it as data is displayed in clear on the terminal.

Finally we show keyring information and print our password. We use print command to have an human friendly output, keyctl read command display it in hex format...

Live Stock Monitor

Friday, 27 July 2018
|
Écrit par
Grégory Soutadé

Today, a small Python script to track live stock exchanges. It fetch data from boursorama website and format it for "Generic Monitor" XFCE applet which allows to display result of a command line script. Just setup the path of this script in genmon properties and set the delay to 60s (to avoid flooding website).

#!/usr/bin/python

#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <https://www.gnu.org/licenses/>.
#

import requests
import json

params_gettickseod = {"symbol":"%s","length":"1","period":"0","guid":""}
params_updatecharts = {"symbol":"%s","period":"-1"}

base_headers = {
    'Host': 'www.boursorama.com',
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'fr,en-US;q=0.7,en;q=0.3',
    'DNT': '1',
    'Upgrade-Insecure-Requests': '1',
    'Pragma': 'no-cache',
    'Cache-Control': 'no-cache',
}
base_address = 'https://www.boursorama.com/cours/'

headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
    'Accept': 'application/json, text/javascript, */*; q=0.01',
    'Accept-Language': 'fr,en-US;q=0.7,en;q=0.3',
    'Accept-Encoding': 'gzip, deflate, br',
    'Referer': 'https://www.boursorama.com/cours/%s/',
    'Content-Type': 'application/json; charset=utf-8',
    'X-Requested-With': 'XMLHttpRequest',
    'DNT': '1',
    'Connection': 'keep-alive',
}

xhr_address = 'https://www.boursorama.com/bourse/action/graph/ws/'
address_gettickseod = xhr_address + 'GetTicksEOD'
address_updatecharts = xhr_address + 'UpdateCharts'

cookies = None

def _do_request(address, params, headers):
    if cookies is None:
        req = requests.get(address, params=params, headers=headers)
    else:
        req = requests.get(address, params=params, headers=headers, cookies=cookies)

    if req.status_code == requests.codes.ok:
        j = req.json()
        if len(j) == 0:
            raise Exception('Not available')
        return j
    else:
        raise Exception("Request error!")

def getStock(stock, display_name=None):
    my_headers = headers.copy()
    my_headers['Referer'] = headers['Referer'] % (stock)

    close_value = 0
    res = ''

    my_params  = params_updatecharts.copy()
    my_params["symbol"] = stock
    try:
        j = _do_request(address_updatecharts, my_params, my_headers)
    except:
        req = requests.get(base_address + stock, headers=base_headers)
        # cookies = req.cookies
        j = _do_request(address_updatecharts, my_params, my_headers)

    current = float(j['d'][0]['c'])
    my_params  = params_gettickseod.copy()
    my_params["symbol"] = stock
    try:
        j = _do_request(address_gettickseod, my_params, my_headers)
        close_value = float(j['d']['qv']['c'])
    except Exception, e:
        if not len(j):
            raise e
        close_value = float(j['d'][0]['o']) # Open value

    if close_value != 0:
        var = ((current/close_value) - 1)*100
    else:
        var = 0
    if current < close_value:
        color = 'red'
        var = -var
    else:
        color = 'green'
    if not display_name is None:
        res += '%s ' % (display_name)
    res += '%.3f <span fgcolor="%s">%.2f</span>' % (current, color, var)

    return res

def getMail():
    res = ''
    nb_messages = ''
    pipew = open("/tmp/gmail-pipe-w", "wb+")
    piper = open("/tmp/gmail-pipe-r", "rb+")
    pipew.write("a\n")
    pipew.flush()
    while not len(nb_messages):
        nb_messages = piper.readline()
    if len(nb_messages):
        nb_messages = int(nb_messages)
        if nb_messages == 1:
            res = ', 1 msg'
        elif nb_messages > 1:
            res = ', %d msgs' % (nb_messages)
    pipew.close()
    piper.close()

    return res

def getStocks(stocks):
    res = ''
    for stock in stocks:
        if res != '': res += ', '
        try:
            res += getStock(*stock)
        except Exception, e:
            if len(stock) > 1:
                res += "%s %s" % (stock[1], str(e))
            else:
                res += str(e)
    res += getMail()
    print('<txt>%s</txt>' % (res))

getStocks([('1rPENX', 'Euronext'), ('1rPAIR',)])

Get stock code id from website URL (last part). A file version is available here.

I added another part to get email count from gmail. It relies on a bash script that fetches RSS feeds when data is wrote in the FIFO.

Body of the script :

#!/bin/bash

USER='soutade'

while [ 1 ] ; do
    echo -n "Please enter gmail account password : "
    read -s password
    echo ""
    echo -n "Confirm password : "
    read -s password2
    echo ""
    if [ "$password" != "$password2" ] ; then
        echo -e "Passwords doesn't match !!\n"
        continue
    fi
    break
done

pipew="/tmp/gmail-pipe-w"
piper="/tmp/gmail-pipe-r"

rm -f $pipew $piper
mkfifo $pipew $piper

while [ 1 ] ; do
    read line < $pipew
    feeds=`curl -u "$USER:$password" --silent "https://mail.google.com/mail/feed/atom"`
    echo $feeds | sed  s/.*\<fullcount\>//g | sed  s/\<\\/fullcount\>.*//g > $piper
done

You can hardcode password in the script, but I don't like having my password in clear on the harddrive. A file version is available here.

Tips : Linux kernel coding rules

Friday, 01 December 2017
|
Écrit par
Grégory Soutadé

Tux, the Linux mascot

For some projects I have to work inside Linux kernel code and, as a developer, I have my own preferences for coding rules (not stable for variable/function naming in facts...) . Especially, I prefer 4 spaces for indentation, curly brackets at a newline. But, the kernel is full of narcissists dictatorscoders and a strict set of coding rules has been determined some years ago. It can be found in Documentation/CodingStyle. Linux is a big project with thousands of people working on it, so I agree that it requires some code normalization for all contribution. Even if you don't plan to verse your patches into upstream, it's good to follow these rules. Here is some tips to comply with it.

First, as an emacs user I have my own rules in my ~/.emacs configuration file. But, when I work on Linux kernel, I want "linux" rules to be applied. A tip from emacswiki allows to automatically switch when a file with "linux" in its path name is found :

(defun maybe-linux-style ()
  (when (and buffer-file-name
         (string-match "linux" buffer-file-name))
    (c-set-style "Linux")
    (c-set-indentation-style "linux")
    (indent-tabs-mode t)
    ))
(add-hook 'c-mode-hook 'maybe-linux-style)
(add-hook 'before-save-hook 'delete-trailing-whitespace)

Another tip I use is a modified pre-commit hook that will checks my modifications before validate the commit. Edit you .git/hooks/pre-commit with the following lines :

#!/bin/sh
#
# An example hook script to verify what is about to be committed.
# Called by "git commit" with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# To enable this hook, rename this file to "pre-commit".

temp_file=`mktemp`
files=`git diff --name-only`
to_diff=""
for file in ${files} ; do
    # Filter .c and .h files
    echo ${file} | grep ".h$" > /dev/null 2>&1
    if [ $? -ne 0 ] ; then
    echo ${file} | grep ".c$" > /dev/null 2>&1
    if [ $? -ne 0 ] ; then
        continue
    fi
    fi
    # Add not deleted file
    if [ -f ${file} ] ; then
    to_diff="${to_diff} ${file}"
    fi
done
git diff --no-color -u --summary ${to_diff} > ${temp_file}
./scripts/checkpatch.pl --no-signoff --min-conf-desc-length=0 --no-summary --mailback ${temp_file}
ret=$?
rm -f ${temp_file}
exit $ret

Don't forget chmod +x .git/hooks/pre-commit

Unfortunately, this hook can't be stored into the central repository and have to be copied each time you clone it (or install a server hook). If you want to bypass it (for some reasons), just commit with "--no-verify" option.

Finally, when you already have a custom codebase ready and commited, you can use some scripts provided by the kernel team to check for coding rules. The first is scripts/Lindent that will indent your file following the kernel coding rules via indent util (needs to be installed). The second is scripts/cleanfile which remove unnecessary whitespaces. The third is scripts/checkpatch.pl -f that will checks the whole file (and not just a patch).

Division by 10

Wednesday, 27 September 2017
|
Écrit par
Grégory Soutadé

For a human, it's pretty simple to divide a number by ten because we used to calculate in ten base everyday. But ... a computer handle numbers in base 2. It doesn't means that it can't compute a division with a number that is not a power of 2, but operations are really faster in this base. Especially if you don't have floating point unit.

At work, I had to re implement the function "printf". To display decimal integers, I use an algorithm like :

while (value)
{
    *cur_ptr = '0' + (value%10);
    value = value/10;
    ...
}

This works fine, but two GCC builtin functions udivdi3 and umoddi3 are called which represent an amount of 3.5kB of code. So, I was looking for a code size optimized implementation on the Internet and didn't found my way.

Finally, I wrote my own. It's a basic one inspired from child learning method :

01. void div10(unsigned value, unsigned* _res, unsigned* _mod)
02. {
03.    unsigned res = value / 8;
04.    unsigned mod = value - (res*10);
05.
06.    while (mod > 10)
07.    {
08.        res -= 1;
09.        mod += 10;
10.    }
11.
12.    *_res = res;
13.    *_mod = mod;
14. }

This algorithm is a basic approach to division. It tries all numbers until it find the good one.

First thing : why I use variables instead of directly write values to pointers ? It's to indicate to GCC that they are temporary values which can be kept into registers and not written every loop into the memory (save instructions and memory accesses).

Line 3 is the begining. We will start at value / 8 which can be easily done by the computer because it is equivalent to a right shift of 3 bits (only one instruction). Note that 8 is the closest power of two to 10 and x/8 is greater than x/10 .

Line 4 is the computation of difference (distance) between my result multiplied by 10 and the current value. For the final result, this difference must be less than 10 (which correspond to the modulo).

Line 6 : while its not the case, we decrement result and increment modulo. Why incrementing modulo ? It's an optimization of the re computation of :

mod = value - (res*10);

If res is decremented, modulo is incremented as value is fixed. So, a simple addition is sufficient here.

There is another big trick in this code : the substract line 4 is done with UNSIGNED values and the result of line 4 is most of the time negative ! Which corresponds to big unsigned value (> 2147483648) that also implies > 10. We have to wait an integer overflow for mod to become positive and when it's done, we get the current modulo value (at least MAXLONGINT+10 = 9) !

If we does opposite operation ((res*10) - value), we have to decrement mod until it becomes less or equals to 0. But, in this case, all operations must be done in signed mode and the final modulo must be inverted at the end (more instructions) :

void div10(unsigned long value, unsigned long* _res, unsigned long* _mod)
{
    unsigned long res = value / 8;
    unsigned long mod = (res*10) - value;

    while (((signed long)mod) > 0)
    {
        res -= 1;
        mod -= 10;
    }

    *_res = res;
    *_mod = (unsigned long)-(signed long)mod;
}

Facts : the unsigned version of my algorithm is 15 instructions while udivdi3 + umoddi3 is 881 instructions. Wonderful !

Beware : this algorithm is slow. For small numbers it's not import because x/8 =~ x/10, but when x becomes bigger, the difference can be huge and requires one decrement, one increment and one test multiplied (x/8 - x/10)/10 times. For 32 bits numbers, it's 10 737 418 loops...

This algorithm can be extended to any divisor by replacing hardcoded divisor with a parameter and a function that finds the nearest and inferior power of 2.

void div(unsigned long value, unsigned long divisor, unsigned long* _res, unsigned long* _mod)
{
    unsigned long res, mod, tmp = divisor, power2 = 0;

    /* Nearest inferior power of 2 */
    while (tmp > 1)
    {
        tmp >>=1;
        power2++;
    }

    res = value >> power2;
    mod = value - (res*divisor);

    while (mod > divisor)
    {
        res -= 1;
        mod += divisor;
    }

    *_res = res;
    *_mod = mod;
}