About Archive Tags RSS Feed

 

Brexit has come

5 January 2021 13:00

Nothing too much has happened recently, largely as a result of the pandemic killing a lot of daily interests and habits.

However as a result of Brexit I'm having to do some paperwork, apparently I now need to register for permanent residency under the terms of the withdrawal agreement, and that will supersede the permanent residency I previously obtained.

Of course as a UK citizen I've now lost the previously-available freedom of movement. I can continue to reside here in Helsinki, Finland, indefinitely, but I cannot now move to any other random EU country.

It has crossed my mind, more than a few times, that I should attempt to achieve Finnish citizenship. As a legal resident of Finland the process is pretty simple, I just need two things:

  • Prove I've lived here for the requisite number of years.
  • Pass a language test.

Of course the latter requirement is hard, I can understand a lot of spoken and written Finnish, but writing myself, and speaking a lot is currently beyond me. I need to sit down and make the required effort to increase my fluency. There is the alternative option of learning Swedish, which is a hack a lot of immigrants use:

  • Learning Swedish is significantly easier for a native English-speaker.
  • But the downside is that it would be learning a language solely to "cheat" the test, it wouldn't actually be useful in my daily life.

Finland has two official languages, and so the banks, the medical world, the tax-office, etc, are obliged to provide service in both. However daily life, ordering food at restaurants, talking to parents in the local neighborhood? Finnish, or English are the only real options. So if I went this route I'd end up in a weird situation where I had to learn a language to pass a test, but then would continue to need to learn more Finnish to live my life. That seems crazy, unless I were desperate for a second citizenship which I don't think I am.

Learning Finnish has not yet been a priority, largely because I work in English in the IT-world, and of course when I first moved here I was working (remotely) for a UK company, and didn't have the time to attend lessons (because they were scheduled during daytime, on the basis that many immigrants are unemployed). Later we had a child, which meant that early-evening classes weren't a realistic option either.

(Of course I learned a lot of the obvious things immediately upon moving, things like numbers, names for food, days of the week were essential. Without those I couldn't have bought stuff in shops and would have starved!)

On the topic of languages a lot of people talk about how easy it is for children to pick up new languages, and while that is broadly true it is also worth remembering just how many years of correction and repetition they have to endure as part of the process.

For example we have a child, as noted already, he is spoken to by everybody in Finnish. I speak to him in English, and he hears his mother and myself speaking English. But basically he's 100% Finnish with the exception of:

  • Me, speaking English to him.
  • His mother and I speaking English in his hearing.
  • Watching Paw Patrol.

If he speaks Finnish to me I pretend to not understand him, even when I do, just for consistency. As a result of that I've heard him tell strangers "Daddy doesn't speak Finnish" (in Finnish) when we've been stopped and asked for directions. He also translates what some other children have said into English for my benefit which is adorable

Anyway he's four, and he's pretty amazing at speaking to everybody in the correct language - he's outgrown the phase where he'd mix different languages in the same sentence ("more leipä", "saisinko milk") - when I took him to the UK he surprised and impressed me by being able to understand a lot of the heavy/thick accents he'd never heard before. (I'll still need to train him on Rab C. Nesbitt when he's a wee bit older, but so far no worries.)

So children learn languages, easily and happily? Yes and no. I've spent nearly two years correcting his English and he still makes the same mistake with gender. It's not a big deal, at all, but it's a reminder that while children learn this stuff, they still don't do it as easily as people imagine. I'm trying to learn and if I'd been corrected for two years over the same basic point you'd rightly think I was "slow", but actually that's just how it works. Learning languages requires a hell of a lot of practice, a lot of effort, and a lot of feedback/corrections.

Specifically Finnish doesn't have gendered pronouns, the same word is used for "he" and "she". This leads to a lot of Finnish people, adults and children, getting the pronouns wrong in English. In the case of our child he'll say "Mommy is sleeping, when he wake up?" In the case of adults I've heard people say "My girlfriend is a doctor, he works in a hospital", or "My dad is an accountant, she works for a big firm". As I say I've spent around two years making this correction to the child, and he's still nowhere near getting it right. Kinda adorable actually:

  • "Mommy is a woman we say "when she wakes up"..."
  • "Adriana is a girl we say "her bike".."

| 3 comments

 

Archiving Debian-Administration.org, for real

1 November 2020 13:00

Back in 2017 I announced that the https://Debian-Administration.org website was being made read-only, and archived.

At the time I wrote a quick update to save each requested page as a flat-file, hashed beneath /tmp, with the expectation that after a few months I'd have a complete HTML-only archive of the site which I could serve as a static-website, instead of keeping the database and pile of CGI scripts running.

Unfortunately I never got round to archiving the pages in a git-repository, or some other store, and I usually only remembered this local tree of content was available a few minutes after I'd rebooted the server and lost the stuff - as the reboot would reap the contents of /tmp!

Thinking about it today I figured I probably didn't even need to do that, instead I just need to redirect to the wayback machine. Working on the assumption that the site has been around for "a while" it should have all the pages mirrored by now I've made a "final update" to Apache:

 RewriteEngine on
 RewriteRule   ^/(.*)  "http://web.archive.org/web/https://debian-administration.org/$1"  [R,L]

Assuming nobody reports a problem in the next month I'll retire the server and make a simple docker container to handle the appropriate TLS certificate renewal, and hardwire the redirection(s) for the sites involved.

| 1 comment

 

Offsite-monitoring, from my desktop.

20 October 2020 13:00

For the past few years I've had a bunch of virtual machines hosting websites, services, and servers. Of course I want them to be available - especially since I charge people money to access at some of them (for example my dns-hosting service) - and that means I want to know when they're not.

The way I've gone about this is to have a bunch of machines running stuff, and then dedicate an entirely separate machine solely for monitoring and alerting. Sure you can run local monitoring, testing that services are available, the root-disk isn't full, and that kind of thing. But only by testing externally can you see if the machine is actually available to end-users, customers, or friends.

A local-agent might decide "I'm fine", but if the hosting-company goes dark due to a fibre cut you're screwed.

I've been hosting my services with Hetzner (cloud) recently, and their service is generally pretty good. Unfortunately I've started to see an increasing number of false-alarms. I'd have a server in Germany, with the monitoring machine in Helsinki (coincidentally where I live!). For the past month I've started to get pinged with a failure every three/four days on average, "service down - dns failed", or "service down - timeout". When the notice would wake me up I'd go check and it would be fine, it was a very transient failure.

To be honest the reason for this is my monitoring is just too damn aggressive, I like to be alerted immediately in case something is wrong. That means if a single test fails I get an alert, as rather than only if a test failed for something more reasonable like three+ consecutive failures.

I'm experimenting with monitoring in a less aggressive fashion, from my home desktop. Since my monitoring tool is a single self-contained golang binary, and it is already packaged as a docker-based container deployment was trivial. I did a little work writing an agent to receive failure-notices, and ping me via telegram - instead of the previous approach where I had an online status-page which I could view via my mobile, and alerts via pushover.

So far it looks good. I've tweaked the monitoring to setup a timeout of 15 seconds, instead of 5, and I've configured it to only alert me if there is an outage which lasts for >= 2 consecutive failures. I guess the TLDR is I now do offsite monitoring .. from my house, rather than from a different region.

The only real reason to write this post was mostly to say that the process of writing a trivial "notify me" gateway to interface with telegram was nice and straightforward, and to remind myself that transient failures are way more common than we expect.

I'll leave things alone for a moment, but it was a fun experiment. I'll keep the two systems in parallel for a while, but I guess I can already predict the outcome:

  • The desktop monitoring will report transient outages now and again, because home broadband isn't 100% available.
  • The heztner-based monitoring, in a different region, will report transient problems, because even hosting companies are not 100% available.
    • Especially at the cheap prices I'm paying.
  • The way to avoid being woken up by transient outages/errors is to be less agressive.
    • I think my paying users will be OK if I find out a services is offline after 5 minutes, rather than after 30 seconds.
    • If they're not we'll have to talk about budgets ..

| No comments

 

Writing an assembler.

3 October 2020 13:00

Recently I've been writing a couple of simple compilers, which take input in a particular format and generate assembly language output. This output can then be piped through gcc to generate a native executable.

Public examples include this trivial math compiler and my brainfuck compiler.

Of course there's always the nagging thought that relying upon gcc (or nasm) is a bit of a cheat. So I wondered how hard is it to write an assembler? Something that would take assembly-language program and generate a native (ELF) binary?

And the answer is "It isn't hard, it is just tedious".

I found some code to generate an ELF binary, and after that assembling simple instructions was pretty simple. I remember from my assembly-language days that the encoding of instructions can be pretty much handled by tables, but I've not yet gone into that.

(Specifically there are instructions like "add rax, rcx", and the encoding specifies the source/destination registers - with different forms for various sized immediates.)

Anyway I hacked up a simple assembler, it can compile a.out from this input:

.hello   DB "Hello, world\n"
.goodbye DB "Goodbye, world\n"

        mov rdx, 13        ;; write this many characters
        mov rcx, hello     ;; starting at the string
        mov rbx, 1         ;; output is STDOUT
        mov rax, 4         ;; sys_write
        int 0x80           ;; syscall

        mov rdx, 15        ;; write this many characters
        mov rcx, goodbye   ;; starting at the string
        mov rax, 4         ;; sys_write
        mov rbx, 1         ;; output is STDOUT
        int 0x80           ;; syscall


        xor rbx, rbx       ;; exit-code is 0
        xor rax, rax       ;; syscall will be 1 - so set to xero, then increase
        inc rax            ;;
        int 0x80           ;; syscall

The obvious omission is support for "JMP", "JMP_NZ", etc. That's painful because jumps are encoded with relative offsets. For the moment if you want to jump:

        push foo     ; "jmp foo" - indirectly.
        ret

:bar
        nop          ; Nothing happens
        mov rbx,33   ; first syscall argument: exit code
        mov rax,1    ; system call number (sys_exit)
        int 0x80     ; call kernel

:foo
        push bar     ; "jmp bar" - indirectly.
        ret

I'll update to add some more instructions, and see if I can use it to handle the output I generate from a couple of other tools. If so that's a win, if not then it was a fun learning experience:

| 2 comments

 

Using a FORTH-like language for something useful

22 September 2020 13:00

So my previous post was all about implementing a simple FORTH-like language. Of course the obvious question is then "What do you do with it"?

So I present one possible use - turtle-graphics:

\ Draw a square of the given length/width
: square
  dup dup dup dup
  4 0 do
    forward
    90 turn
  loop
;

\ pen down
1 pen

\ move to the given pixel
100 100 move

\ draw a square of width 50 pixels
50 square

\ save the result (png + gif)
save

Exciting times!

| No comments

 

Implementing a FORTH-like language ..

16 September 2020 21:00

Four years ago somebody posted a comment-thread describing how you could start writing a little reverse-polish calculator, in C, and slowly improve it until you had written a minimal FORTH-like system:

At the time I read that comment I'd just hacked up a simple FORTH REPL of my own, in Perl, and I said "thanks for posting". I was recently reminded of this discussion, and decided to work through the process.

Using only minimal outside resources the recipe worked as expected!

The end-result is I have a working FORTH-lite, or FORTH-like, interpreter written in around 2000 lines of golang! Features include:

  • Reverse-Polish mathematical operations.
  • Comments between ( and ) are ignored, as expected.
    • Single-line comments \ to the end of the line are also supported.
  • Support for floating-point numbers (anything that will fit inside a float64).
  • Support for printing the top-most stack element (., or print).
  • Support for outputting ASCII characters (emit).
  • Support for outputting strings (." Hello, World ").
  • Support for basic stack operations (drop, dup, over, swap)
  • Support for loops, via do/loop.
  • Support for conditional-execution, via if, else, and then.
  • Load any files specified on the command-line
    • If no arguments are included run the REPL
  • A standard library is loaded, from the present directory, if it is present.

To give a flavour here we define a word called star which just outputs a single start-character:

: star 42 emit ;

Now we can call that (NOTE: We didn't add a newline here, so the REPL prompt follows it, that's expected):

> star
*>

To make it more useful we define the word "stars" which shows N stars:

> : stars dup 0 > if 0 do star loop else drop then ;
> 0 stars
> 1 stars
*> 2 stars
**> 10 stars
**********>

This example uses both if to test that the parameter on the stack was greater than zero, as well as do/loop to handle the repetition.

Finally we use that to draw a box:

> : squares 0 do over stars cr loop ;
> 4 squares
****
****
****
****

> 10 squares
**********
**********
**********
**********
**********
**********
**********
**********
**********
**********

For fun we allow decompiling the words too:

> #words 0 do dup dump loop
..
Word 'square'
 0: dup
 1: *
Word 'cube'
 0: dup
 1: square
 2: *
Word '1+'
 0: store 1.000000
 2: +
Word 'test_hot'
  0: store 0.000000
  2: >
  3: if
  4: [cond-jmp 7.000000]
  6: hot
  7: then
..

Anyway if that is at all interesting feel free to take a peak. There's a bit of hackery there to avoid the use of return-stacks, etc. Compared to gforth this is actually more featureful in some areas:

  • I allow you to use conditionals in the REPL - outside a word-definition.
  • I allow you to use loops in the REPL - outside a word-definition.

Find the code here:

| No comments

 

I'm a bit of a git (hacker?)

28 July 2020 21:00

Sometimes I enjoy reading the source code to projects I like, use, or am about to install for the first time. This was something I used to do on a very regular basis, looking for security issues to report. Nowadays I don't have so much free time, but I still like to inspect the source code to new applications I install, and every now and again I'll find the time to look at the source to random projects.

Reading code is good. Reading code is educational.

One application I've looked at multiple times is redis, which is a great example of clean and well-written code. That said when reading the redis codebase I couldn't help noticing that there were a reasonably large number of typos/spelling mistakes in the comments, so I submitted a pull-request:

Sadly that particular pull-request didn't receive too much attention, although a previous one updating the configuration file was accepted. I was recently reminded of these pull-requests when I was when I was doing some other work. So I figured I'd have a quick scan of a couple of other utilities.

In the past I'd just note spelling mistakes when I came across them, usually I'd be opening each file in a project one by one and reading them from top to bottom. (Sometimes I'd just open files in emacs and run "M-x ispell-comments-and-strings", but more often I'd just notice them with my eyes). It did strike me that if I were to do this in a more serious fashion it would be good to automate it.

So this time round I hacked up a simple "dump comments" utility, which would scan named files and output the contents of any comments (be they single-line, or multi-line). Once I'd done that I could spell-check easily:

 $ go run dump-comments.go *.c > comments
 $ aspell -c comments

Anyway the upshot of that was a pull-request against git:

We'll see if that makes its way live sometime. In case I get interested in doing this again I've updated my sysbox-utility collection to have a comments sub-command. That's a little more robust and reliable than my previous hack:

$ sysbox comments -pretty=true $(find . -name '*.c')
..
..

The comments sub-command has support for:

  • Single-line comments, for C, as prefixed with //.
  • Multi-line comments, for C++, as between /* and */.
  • Single-line comments, for shell, as prefixed with #.
  • Lua comments, both single-line (prefixed with --) and multiline between --[[ and --]].

Adding new support would be trivial, I just need a start and end pattern to search against. Pull-requests welcome:

| 1 comment

 

Growing food is fun.

27 July 2020 12:00

"I grew up on a farm" is something I sometimes what I tell people. It isn't true, but it is a useful shorthand. What is true is that my parents both come from a farming background, my father's family up in Scotland, my mother's down in Yorkshire.

Every summer my sisters and myself would have a traditional holiday at the seaside, which is what people do in the UK (Blackpool, Scarborough, Great Yarmouth, etc). Before, or after, that we'd spend the rest of the summer living on my grandmother's farm.

I loved spending time on the farm when I was a kid, and some of my earliest memories date from that time. For example I remember hand-feeding carrots to working dogs (alsatians) that were taller than I was. I remember trying to ride on the backs of those dogs, and how that didn't end well. In fact the one and only time I can recall my grandmother shouting at me, or raising her voice at all, was when my sisters and I spent an afternoon playing in the coal-shed. We were filthy and covered in coal-dust from head to toe. Awesome!

Anyway the only reason I bring this up is because I have a little bit of a farming background, largely irrelevant in my daily life, but also a source of pleasant memories. Despite it being an animal farm (pigs, sheep, cows) there was also a lot of home-grown food, which my uncle Albert would deliver/sell to people nearby out of the back of a van. That same van that would be used to ferry us to see the fireworks every November. Those evenings were very memorable too - they would almost always involve flasks of home-made vegetable soup.

Nowadays I live in Finland, and earlier in the year we received access to an allotment - a small piece of land (10m x 10m) for €50/year - upon which we can grow our own plants, etc.

My wife decided to plant flowers and make it look pretty. She did good.

I decided to plant "food". I might not have done this stuff from scratch before, but I was pretty familiar with the process from my youth, and also having the internet to hand to make the obvious searches such as "How do you know when you can harvest your garlic?"

Before I started I figured it couldn't be too hard, after all if you leave onions/potatoes in the refrigerator for long enough they start to grow! It isn't like you have to do too much to help them. In short it has been pretty easy and I'm definitely going to be doing more of it next year.

I've surprised myself by enjoying the process as much as I have. Every few days I go and rip up the weeds, and water the things we've planted. So far I've planted, and harvested, Radish, Garlic, Onions, and in a few more weeks I'll be digging up potatoes.

I have no particular point to this post, except to say that if you have a few hours spare a week, and a slab of land to hand upon which you can dig and plant I'd recommend it. Sure there were annoyances, and not a single one of the carrot-seeds I planted showed any sign of life, but the other stuff? The stuff that grew? Very tasty, om nom nom ..

(It has to be said that when we received the plot there was a jungle growing upon it. Once we tidied it all up we found raspberries, roses, and other things. The garlic I reaped was already growing so I felt like a cheat to harvest it. That said I did plant a couple of bulbs on my balcony so I could say "I grew this from scratch". Took a while, but I did indeed harvest my own garlic.)

| No comments

 

Writing a brainfuck compiler.

14 June 2020 19:00

So last night I had the idea that it might be fun to write a Brainfuck compiler, to convert BF programs into assembly language, where they'd run nice and quickly.

I figured I could allocate a day to do the work, and it would be a pleasant distraction on a Sunday afternoon. As it happened it only took me three hours from start to finish.

There are only a few instructions involved in brainfuck:

  • >
    • increment the data pointer (to point to the next cell to the right).
  • <
    • decrement the data pointer (to point to the next cell to the left).
  • +
    • increment (increase by one) the byte at the data pointer.
  • -
    • decrement (decrease by one) the byte at the data pointer.
  • .
    • output the byte at the data pointer.
  • ,
    • accept one byte of input, storing its value in the byte at the data pointer.
  • [
    • if the byte at the data pointer is zero, then instead of moving the instruction pointer forward to the next command, jump it forward to the command after the matching ] command.
  • ]
    • if the byte at the data pointer is nonzero, then instead of moving the instruction pointer forward to the next command, jump it back to the command after the matching [ command.

The Wikipedia link early shows how you can convert cleanly to a C implementation, so my first version just did that:

  • Read a BF program.
  • Convert to a temporary C-source file.
  • Compile with gcc.
  • End result you have an executable which you can run.

The second step was just as simple:

  • Read a BF program.
  • Convert to a temporary assembly language file.
  • Compile with nasm, link with ld.
  • End result you have an executable which you can run.

The following program produces the string "Hello World!" to the console:

++++++++[>++++[>++>+++>+++>+<<<<-]>+>+>->>+[<]<-]>>.>---.+++++++..+++.>>.<-.<.+++.------.--------.>>+.>++.

My C-compiler converted that to the program:

extern int putchar(int);
extern char getchar();

char array[30000];
int idx = 0;

int main (int arc, char *argv[]) {
  array[idx]++;
  array[idx]++;
  array[idx]++;
  array[idx]++;
  array[idx]++;
  array[idx]++;
  array[idx]++;
  array[idx]++;

  while (array[idx]) {
    idx++;
    array[idx]++;
    array[idx]++;
    array[idx]++;
    array[idx]++;

    while (array[idx]) {
      idx++;
      array[idx]++;
      array[idx]++;
      idx++;
      array[idx]++;
      array[idx]++;
      array[idx]++;
    ..
    ..

The assembly language version is even longer:


global _start
section .text

_start:
  mov r8, stack
  add byte [r8], 8
label_loop_8:
  cmp byte [r8], 0
  je close_loop_8
  add r8, 1
  add byte [r8], 4
label_loop_14:
  cmp byte [r8], 0
  je close_loop_14
  add r8, 1
  add byte [r8], 2
  add r8, 1
  add byte [r8], 3
  add r8, 1
  add byte [r8], 3
  add r8, 1
  add byte [r8], 1
  sub r8, 4
  sub byte [r8], 1
  jmp label_loop_14
close_loop_14:
  add r8, 1
  add byte [r8], 1
  ..

  mov rax, 60
  mov rdi, 0
  syscall
section .bss
stack: resb 300000

Annoyingly the assembly language version ran slower than the C-version, which I was sneakily compiling with "gcc -O3 .." to ensure it was optimized.

The first thing that I did was to convert it to fold adjacent instructions. Instead of generating separate increment instructions for ">>>" I instead started to generate "add xx, 3". That didn't help as much as I'd hoped, but it was still a big win.

After that I made a minor tweak to the way that loops are handled to compare at the end of the loop as well as the start, and that shaved off a fraction of a second.

As things stand I think I'm "done". It might be nice to convert the generated assembly language to something gcc can handle, to drop the dependency on nasm, but I don't feel a pressing need for that yet.

Was a fun learning experience, and I think I'll come back to optimization later.

| No comments

 

Updated my linux-security-modules for the Linux kernel

22 May 2020 09:00

Almost three years ago I wrote my first linux-security-module, inspired by a comment I read on LWN

I did a little more learning/experimentation and actually produced a somewhat useful LSM, which allows you to restrict command-execution via the use of a user-space helper:

  • Whenever a user tries to run a command the LSM-hook receives the request.
  • Then it executes a userspace binary to decide whether to allow that or not (!)

Because the detection is done in userspace writing your own custom rules is both safe and easy. No need to touch the kernel any further!

Yesterday I rebased all the modules so that they work against the latest stable kernel 5.4.22 in #7.

The last time I'd touched them they were built against 5.1, which was itself a big jump forwards from the 4.16.7 version I'd initially used.

Finally I updated the can-exec module to make it gated, which means you can turn it on, but not turn it off without a reboot. That was an obvious omission from the initial implementation #11.

Anyway updated code is available here:

I'd kinda like to write more similar things, but I lack inspiration.

| No comments