SheevaPlug

IWLA 0.7

Sunday, 17 March 2024
|
Écrit par
Grégory Soutadé

Capture d'écran IWLA

Here is the work done for version 0.7 of IWLA (Intelligent Web Log Analyzer witten in Python) since one year and a half :

Core

  • Awstats data updated (7.9)
  • Remove detection from awstats dataset for browser
  • Don't analyze referer for non viewed hits/pages
  • Remove all trailing slashs of URL before starting analyze
  • Improve page/hit detection
  • Main key for visits is now "remote_ip" and not "remote_addr"
  • Add IP type plugin to support IPv4 and IPv6
  • --display-only switch now takes an argument (month/year), analyze is not yet necessary
  • Add --disable-display option

Plugins

  • Geo IP plugin updated (use of ip-api.com)
  • Update robot detection
  • Display visitor IP is now a filter
  • Add subdomains plugin

HTML

  • Generate HTML part in dry run mode (but don't write it to disk)
  • Set lang value in generated HTML page
  • Bugfix: flags management for feeds display
  • New way to display global statistics : with links in months names instead of "Details" button

Config

  • Add no_referrer_domains list to defaut_conf for website that defines this policy
  • Add excluded domain option
  • Set count_hit_only_visitors to False by default

A demo instance (for forge.soutade.fr) is available here

Debian stock on Cubox-i, USB issue

Sunday, 25 February 2024
|
Écrit par
Grégory Soutadé

My backup server was off for a while now. Something broken in my Sheevaplug power supply. And, if I do regular backup of my data on laptop, this is not the case for my server, so it was urgent to find a solution. I first though to buy a Cubox-M which is more powerful, but it's 250$ ! Quite expensive for this kind of stuff ! So I bought a second hand Cubox-i for for only 30€. This is a clone of my current server. I had to install everything from scratch, so I follow Solid Run instructions to prepare my SDCard with UBoot (with a new 2022 release and the ability to read ext4 partitions !).

> sudo dd if=SPL of=/dev/sdX bs=1k seek=1 conv=sync
> sudo dd if=u-boot.img of=/dev/sdX bs=1k seek=69 conv=sync

Then, I wanted to try to install a Debian distribution in stock version (with "nothing" from SolidRun) for an easier maintenance.

My SDCard partitioning looks like (partitioned with gparted) :

Cubox SDCard partitioning

On the main partition, I downloaded generic netboot console install from Debian server (netboot.tar.gz) and uncompress all files in the main partition. But I think it's better to put all in a directory named boot. Then put it in my Cubox and start to install using serial console.

> screen /dev/ttyUSB0 115200

Everything went fine, except that the image is configured for old stable release. So, I have to migrate to stable one (and don't forgot to remove installation files !).

When all was installed, I tried to put a USB key, but nothing happened... USB seems to work :

> lsusb
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

After looking for kernel logs for a while (and a bit despairs), I found the solution : Debian compile its kernel with only few drivers linked. Most of them are in modules (located in /lib/modules/kernelname). And the one missing is USB PHY driver phy-mxs-usb ! In my main server, I had it compiled because I use target imx6/7.

So, then simplest way to test is :

> sudo modprobe phy-mxs-usb

And for automatic loading at boot time :

> sudo echo phy-mxs-usb >> /etc/modules

I also added tun, dm-mod and dm-crypt.

Goodbye inDefero

Sunday, 10 December 2023
|
Écrit par
Grégory Soutadé

And thanks for all ! This week, I decided to replace my old software forge inDefero by a Gitea instance. inDefero project was started in 2008 by Loïc d'Anterroches after Google decided to stop GoogleCode. Unfortunately, it was discontinued some years after because Loïc was almost alone to develop it and don't earn enough money with it (even with a SaaS/pay option). Plus, when you do the same things for a long time, and not necessary related to your core studies, you want to do something else. Written in PHP5, it has a lot of features :

  • Multi CVS support : SVN, mercurial, git, monotone
  • Tickets
  • Code review
  • Wiki
  • Full Open Source
  • Nice design

It's fast and extensible, can be run on a low power server. This is why I really, really thanks Loïc for his hard work. More than ten years after, I still can run it (with some patches) !

Nevertheless, web technologies evolve too fast and I don't want to run PHP5 anymore (nor create a container for it). I wanted something light & fast, something I can hack, something nice for the eyes. After looking on all available forges, I decided to test Gitea. I don't know Go programming language, so I can't patch it, but I was really impressed to see how fast it is (on a low power server). Concerning UI, it's a clone of Github, but it's almost the case for all competitors... Not really nice or innovative, but it's okay. The funny fact is that I can't compile it by myself because it take too much memory... Fortunately, the precompiled binary works like a charm !

Why not Github (or other online forge) ? More than code hosting, Github offers a community. It has become the reference for all developpers (you can even run your own company instance !). I'm pretty sure I would get more contributions to my projets if I put them on it. From a technical point of view, it works very well and offer all you need (except project management). So, the reason to not choose an external forge is more a philosophical decision : it's better to keep your projects/data in your own home more than elsewhere, to have a full control on it (ok, git is by design a distributed software). It's the same for the blog in comparison of Facebook/Instagram/Whatever... Sure, it's less fast, can have some downtime and I had to take care on backups and administration by myself. It's the price of freedom.

The new address for my projects is more generic : forge.soutade.fr. I tried to map old inDefero URLs to redirect in the new one, but didn't migrate tickets & user accounts (sorry everybody).

Tip: Fight against SPAM in comments

Sunday, 29 October 2023
|
Écrit par
Grégory Soutadé

What a surprise when you manage a server and see in the morning your mailbox full of mails telling there is an issue ! It starts like a bad day... First thing : connect to the server and blacklist the attacker IP. This one was not from China or Russia, but from Germany ! It tries to do code and SQL injection on all web pages, with a delay between each bunch of requests to remain undercover. Fortunately, all dynamic web pages in my server are behind a login form. Public part use only statically generated HTML pages. This is done by my static blog generator Dynastie. It's a 10 years old project written in Python/Django. If I have to write it again, I would use Python templates and not XML, but this one has been especially written for my needs and perfectly fit it. So, even if the IHM is very basic, the rendering is aux petits oignons.

One of great feature (not available in other static generators), except dynamic post management, is dynamic comment support. Unfortunately, a website offering public comments without registration is a target for spammers. My automatic comment filtering works well since 2014, but has been bypassed this week. Here is how I fixed it.

As robot doesn't load CSS and JavaScript resources, we can play with hidden fields in the comment's form and do checks on webserver side. So, I added an hidden field which is filled by Javascript when user press on "Comment" button. Value set is a timestamp + a magic number that is then checked by server. So, if the spammer doesn't run Javascript, it'll be blocked ! For sure, this trick is very easy to break and a spammer can easily bypass it with a smart/targeted robot or by doing manual SPAM. In this case, the only solutions is a complex captcha/registration and/or manual comment validation. But it requires more complex modules and work from both parts (user and webmaster), which is overkill for small a website.

Gandi (no) bullshit

Sunday, 02 July 2023
|
Écrit par
Grégory Soutadé

Gandi est un acteur bien connu en France pour son activité de "Gestionnaire de nom de domaine" (registrar en Anglais). Enfin, tout du moins par les profils techniques qui cherchent à gérer leurs noms de domaine ! La réputation de la société (fondée en 2000) s'est bâtie sur sa devise "no bullshit" : l'offre commerciale n'est pas la plus avantageuse, mais derrière l'on retrouve des équipes solides techniquement avec un sens éthique développé (de nombreux organismes à but non lucratif sont sponsorisés). Pour se diversifier, ils ont étendus au fil des années leur offre avec des certificats SSL, ainsi que des hébergement virtuels et physiques.

Malheureusement en 2019, Gandi se fait racheter par un fond de capital-investissement (Montefiore Investissements). Il y a une première vague de départs de la part des clients, alors qu'aucune annonce concrète n'est faite (mais parce-que l'on sait tous comment ça va se finir). En 2023, après quatre années stables, nouveau bouleversement avec la fusion entre Gandi et le groupe Néerlandais Total Webhosting Solutions (TWS) pour former Your.Online. Suite à cette fusion, l'ensemble des clients a eu la surprise de découvrir une augmentation générale des tarifs, ainsi que la partie mail va devenir payante (4€ HT/mois/boîte pour l'offre de base). On peut comprendre une augmentation des tarifs vu de l'inflation actuelle (minime pour un .fr (< 0.5€HT/an)). Mais, ne plus avoir ne serait-ce qu'une adresse mail associée à son nom de domaine (qui est une pratique courante dans le milieu) est rédhibitoire pour beaucoup de personnes. Il y a clairement une recherche maximale de rentabilité au détriment des clients et de l'éthique. Tout du moins de la part de la direction, les équipes techniques devant se contenter de suivre.

Pour ma part, je vais encore rester chez Gandi, car leur solution technique tient la route et que je n'utilisais le mail que comme "relai" pour émettre mon courrier (et ne pas tomber dans les filtres anti-spams), gérant moi-même mes serveurs mails. Néanmoins, je dois configurer une nouvelle entrée "PTR" dans le DNS (qui n'est autre qu'un DNS inversé), notamment requis par Gmail. J'en profite donc pour mettre mon script à jour. Si ça ne tient pas sur le long terme, j'utiliserai de nouveau un relai, notamment via Proton Mail qui a l'air fort sympathique.

Le script est disponible ici

#!/usr/bin/env python3
#-*- encoding: utf-8 -*-

# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

import requests
import json
import re
import socket
import ipaddress

# Config
domain="soutade.fr"
API_KEY = "MY_API_KEY"
livedns_api = "https://dns.api.gandi.net/api/v5/"
dyndns_url = 'http://checkip.dyndns.com/'
headers = {
    'X-Api-Key': API_KEY,
}
A_RECORD_NAME="@" # Target record to update

# https://www.programcreek.com/python/?CodeExample=get+local+address
def get_ipv6():
    s = socket.socket(socket.AF_INET6, socket.SOCK_DGRAM)
    s.connect(('2001:4860:4860::8888', 1))
    return s.getsockname()[0]

def get_ipv4():
    response = requests.get(dyndns_url)
    if response.ok:
        pattern = re.compile('[^:]*(\d+\.\d+\.\d+\.\d+)')
        result = pattern.search(response.text, 0)
        if result == None:
            print("No IP found")
            exit(1) 
        else:
            return result.group(0).strip()
    # Bad gateway
    elif response.status_code in (502,504):
        exit(0)
    else:
        print("Connexion error")
        response.raise_for_status()
        exit(1)

def update_gandi_record(domain_records_href, target, value):
    # Get recorded IP
    response = requests.get(f'{domain_records_href}/{A_RECORD_NAME}/{target}', headers=headers)

    if (response.ok):
        record = response.json()
    else:
        print("Failed to look for recorded IP")
        if response.status_code != 502: # Bad gateway
            response.raise_for_status()
        exit(1)

    if value != record['rrset_values'][0]:
        record['rrset_values'][0] = value

        print(f'URL {domain_records_href}/{A_RECORD_NAME}/{target}')

        # PUT new IP
        response = requests.put(f'{domain_records_href}/{A_RECORD_NAME}/{target}',
                                headers=headers, json=record)

        if (response.ok):
            print("IP updated")
        else:
            print("something went wrong")
            if response.status_code != 502: # Bad gateway
                response.raise_for_status()
            exit(1)

        return 34 # IP updated return !!

    return 0

def create_gandi_record(domain_records_href, name, _type, value):
    request = {
        'rrset_name':name,
        'rrset_type': _type,
        'rrset_values': [value],
        'rrset_ttl': 300
        }

    response = requests.post(f'{domain_records_href}', headers=headers, json=request)

    if response.status_code == 201:
        return 0
    else:
        print(response)
        return 1

def delete_gandi_records(domain_records_href, _type):
    response = requests.get(f'{domain_records_href}?rrset_type={_type}', headers=headers)

    if (response.ok):
        json_resp = response.json()
        for record in json_resp:
            requests.delete(record['rrset_href'], headers=headers)
        return 0
    else:
        print(response)
        return 1

# Get current IP
current_ip_v4 = get_ipv4()
print(f'Your Current IP is {current_ip_v4}')

# Retrieve domains address
response = requests.get(livedns_api + "domains", headers=headers)
if (response.ok):
    domains = response.json()
else:
    if response.status_code != 502: # Bad gateway
        response.raise_for_status()
    exit(1)

domain_index = next((index for (index, d) in enumerate(domains) if d["fqdn"] == domain), None)

if domain_index == None:
    # domain not found
    print("The requested domain {domain} was not found in this gandi account")
    exit(1)

domain_records_href = domains[domain_index]["domain_records_href"]
ret = update_gandi_record(domain_records_href, 'A', current_ip_v4)

current_ip_v6 = get_ipv6()
print(f'Your Current IP is {current_ip_v6}')

domain_records_href = domains[domain_index]["domain_records_href"]
ret |= update_gandi_record(domain_records_href, 'AAAA', current_ip_v6)

if ret == 34:
    # Delete all PTR records
    delete_gandi_records(domain_records_href, 'PTR')

    # Update PTR v4
    reverse_ip = '.'.join(current_ip_v4.split('.')[::-1])
    ptr_ip = f'{reverse_ip}.in-addr.arpa'
    create_gandi_record(domain_records_href, ptr_ip, 'PTR', f'{domain}.')

    # Update PTR v6
    full_ip = ipaddress.ip_address(current_ip_v6).exploded
    reverse_ip = '.'.join(full_ip.replace(':', '')[::-1])
    ptr_ip = f'{reverse_ip}.in-addr.arpa'
    create_gandi_record(domain_records_href, ptr_ip, 'PTR', f'{domain}.')

exit(ret)