Serverless deployment via docker

Monday, 19 March 2018

I've been thinking about serverless-stuff recently, because I've been re-deploying a bunch of services and some of them could are almost microservices. One thing that a lot of my things have in common is that they're all simple HTTP-servers, presenting an API or end-point over HTTP. There is no state, no database, and no complex dependencies.

These should be prime candidates for serverless deployment, but at the same time I don't want to have to recode them for AWS Lamda, or any similar locked-down service. So docker is the obvious answer.

Let us pretend I have ten HTTP-based services, each of which each binds to port 8000. To make these available I could just setup a simple HTTP front-end:


We'd need to route the request to the appropriate back-end, so we'd start to present URLs like:


Here any request which had the prefix steve/foo would be routed to a running instance of the docker container steve/foo. In short the name of the (first) path component performs the mapping to the back-end.

I wrote a quick hack, in golang, which would bind to port 80 and dynamically launch the appropriate containers, then proxy back and forth. I soon realized that this is a terrible idea though! The problem is a malicious client could start making requests for things like:


That would trigger my API-proxy to download the containers and spin them up. Allowing running arbitrary (albeit "sandboxed") code. So taking a step back, we want to use the path-component of an URL to decide where to route the traffic? Each container will bind to :8000 on its private (docker) IP? There's an obvious solution here: HAProxy.

So I started again, I wrote a trivial golang deamon which will react to docker events - containers starting and stopping - and generate a suitable haproxy configuration file, which can then be used to reload haproxy.

The end result is that if I launch a container named "foo" then requests to will reach it. Success! The only downside to this approach is that you must manually launch your back-end docker containers - but if you do so they'll become immediately available.

I guess there is another advantage. Since you're launching the containers (manually) you can setup links, volumes, and what-not. Much more so than if your API layer span them up with zero per-container knowledge.

| 1 comment.


A change of direction ..

Friday, 9 March 2018

In my previous post I talked about how our child-care works here in wintery Finland, and suggested there might be a change in the near future.

So here is the predictable update; I've resigned from my job and I'm going to be taking over childcare/daycare. Ideally this will last indefinitely, but it is definitely going to continue until November. (Which is the earliest any child could be moved into public day-care if there problems.)

I've loved my job, twice, but even though it makes me happy (in a way that several other positions didn't) there is no comparison. Child-care makes me happier-still. Sure there are days when your child just wants to scream, refuse to eat, and nothing works. But on average everything is awesome.

It's a hard decision, a "brave" decision too apparently (which I read negatively!), but also an easy one to make.

It'll be hard. I'll have no free time from 7AM-5PM, except during nap-time (11AM-1PM, give or take). But it will be worth it.

And who knows, maybe I'll even get to rant at people who ask "Where's his mother?" I live for those moments. Truly.



How we care for our child

Monday, 19 February 2018

This post is a departure from the regular content, which is supposed to be "Debian and Free Software", but has accidentally turned into a hardware blog recently!

Anyway, we have a child who is now about 14 months old. The way that my wife and I care for him seems logical to us, but often amuses local people. So in the spirit of sharing this is what we do:

  • We divide the day into chunks of time.
  • At any given time one of us is solely responsible for him.
    • The other parent might be nearby, and might help a little.
    • But there is always a designated person who will be changing nappies, feeding, and playing at any given point in the day.
  • The end.

So our weekend routine, covering Saturday and Sunday, looks like this:

  • 07:00-08:00: Husband
  • 08:01-13:00: Wife
  • 13:01-17:00: Husband
  • 17:01-18:00: Wife
  • 18:01-19:30: Husband

Our child, Oiva, seems happy enough with this and he sometimes starts walking from one parent to the other at the appropriate time. But the real benefit is that each of us gets some time off - in my case I get "the morning" off, and my wife gets the afternoon off. We can hide in our bedroom, go shopping, eat cake, or do anything we like.

Week-days are similar, but with the caveat that we both have jobs. I take the morning, and the evenings, and in exchange if he wakes up overnight my wife helps him sleep and settle between 8PM-5AM, and if he wakes up later than 5AM I deal with him.

Most of the time our child sleeps through the night, but if he does wake up it tends to be in the 4:30AM/5AM timeframe. I'm "happy" to wake up at 5AM and stay up until I go to work because I'm a morning person and I tend to go to bed early these days.

Day-care is currently a complex process. There are three families with small children, and ourselves. Each day of the week one family hosts all the children, and the baby-sitter arrives there too (all the families live within a few blocks of each other).

All of the parents go to work, leaving one carer in charge of 4 babies for the day, from 08:15-16:15. On the days when we're hosting the children I greet the carer then go to work - on the days the children are at a different families house I take him there in the morning, on my way to work, and then my wife collects him in the evening.

At the moment things are a bit terrible because most of the children have been a bit sick, and the carer too. When a single child is sick it's mostly OK, unless that is the child which is supposed to be host-venue. If that child is sick we have to panic and pick another house for that day.

Unfortunately if the child-carer is sick then everybody is screwed, and one parent has to stay home from each family. I guess this is the downside compared to sending the children to public-daycare.

This is private day-care, Finnish-style. The social-services (kela) will reimburse each family €700/month if you're in such a scheme, and carers are limited to a maximum of 4 children. The net result is that prices are stable, averaging €900-€1000 per-child, per month.

(The €700 is refunded after a month or two, so in real-terms people like us pay €200-€300/month for Monday-Friday day-care. Plus a bit of beaurocracy over deciding which family is hosting, and which parents are providing food. With the size being capped, and the fees being pretty standard the carers earn €3600-€4000/month, which is a good amount. To be a school-teacher you need to be very qualified, but to do this caring is much simpler. It turns out that being an English-speaker can be a bonus too, for some families ;)

Currently our carer has a sick-note for three days, so I'm staying home today, and will likely stay tomorrow too. Then my wife will skip work on Wednesday. (We usually take it in turns but sometimes that can't happen easily.)

But all of this is due to change in the near future, because we've had too many sick days, and both of us have missed too much work.

More news on that in the future, unless I forget.

| 1 comment.


Updated my package-repository

Saturday, 17 February 2018

Yesterday I overhauled my Debian package-hosting repository, in response to user-complaints.

I started down the rabit hole due to:

  W: No Hash entry in Release file /.._._Release which is considered strong enough for security purposes

I fixed that by changing my hashes from SHA1 to SHA256 + SHA512, but I was only making a little progress, due to the more serious problem, my repository-signing key was DSA-based and "small". I replaced it with a modern key, then changed how I generate my packages and all is well.

In the past I was generating the Release files manually, via a silly shell-script. Anyway here is my trivial Makefile for making the per-project and per-distribution archive, no doubt it could be improved:

   all: repo

       @rm -f InRelease Packages Sources Packages.gz Sources.gz Release Release.gpg

   Packages: $(wildcard *.deb)
       @apt-ftparchive packages . > Packages 2>/dev/null
       @gzip -c Packages > Packages.gz

   Sources: $(wildcard *.tar.gz)
       @apt-ftparchive sources . > Sources 2>/dev/null
       @gzip -c Sources > Sources.gz

   repo: Packages Sources
       @apt-ftparchive release . > Release
       @gpg --yes --clearsign -o InRelease Release
       @gpg --yes -abs -o Release.gpg Release

In conclusion, in the unlikely event you're using my packages please see GPG-instructions. I've also hidden any packages which were solely for Squeeze and Wheezy, but they continue to exist to avoid breaking links.

| No comments


Decoding 433Mhz-transmissions with software-defined radio

Sunday, 11 February 2018

This blog-post is a bit of a diversion, and builds upon my previous entry of using 433Mhz radio-transmitters and receivers with Arduino and/or ESP8266 devices.

As mentioned in my post I've recently been overhauling my in-house IoT buttons, and I decided to go down the route of using commercially-available buttons which broadcast signals via radio, rather than using IR, or WiFi. The advantage is that I don't need to build any devices, or worry about 3D-printing a case - the commercially available buttons are cheap, water-proof, portable, and reliable, so why not use them? Ultimately I bought around ten buttons, along with a radio-receiver and radio-transmitter modules for my ESP8266 device. I wrote code to run on my device to receive the transmissions, decode the device-ID, and take different actions based upon the specific button pressed.

In the gap between buying the buttons (read: radio transmitters) and waiting for the transmitter/receiver modules I intended to connect to my ESP8266/arduino device(s) I remembered that I'd previously bought a software-defined-radio receiver, and figured I could use it to receive and react to the transmissions directly upon my PC.

The dongle I'd bought in the past was a simple USB-device which identifies itself as follows when inserted into my desktop:

  [17844333.387774] usb 3-9: New USB device found, idVendor=0bda, idProduct=2838
  [17844333.387777] usb 3-9: New USB device strings: Mfr=1, Product=2, SerialNumber=3
  [17844333.387778] usb 3-9: Product: RTL2838UHIDIR
  [17844333.387778] usb 3-9: Manufacturer: Realtek
  [17844333.387779] usb 3-9: SerialNumber: 00000001

At the time I bought it I wrote a brief blog post, which described tracking aircraft, and I said "I know almost nothing about SDR, except that it can be used to let your computer do stuff with radio."

So my first step was finding some suitable software to listen to the right-frequency and ideally decode the transmissions. A brief search lead me to the following repository:

The RTL_433 project is pretty neat as it allows receiving transmissions and decoding them. Of course it can't decode everything, but it has the ability to recognize a bunch of commonly-used hardware, and when it does it outputs the payload in a useful way, rather than just dumping a bitstream/bytestream.

Once you've got your USB-dongle plugged in, and you've built the project you can start receiving and decoding all discovered broadcasts like so:

  skx@deagol ~$ ./build/src/rtl_433 -U -G
  trying device  0:  Realtek, RTL2838UHIDIR, SN: 00000001
  Found Rafael Micro R820T tuner
  Using device 0: Generic RTL2832U OEM
  Exact sample rate is: 250000.000414 Hz
  Sample rate set to 250000.
  Bit detection level set to 0 (Auto).
  Tuner gain set to Auto.
  Reading samples in async mode...
  Tuned to 433920000 Hz.

Here we've added flags:

  • -G
    • Enable all decoders. So we're not just listening for traffic at 433Mhz, but we're actively trying to decode the payload of the transmissions.
  • -U
    • Timestamps are in UTC

Leaving this running for a few hours I noted that there are several nearby cars which are transmitting data about their tyre-pressure:

  2018-02-10 11:53:33 :      Schrader       :      TPMS       :      25
  ID:          1B747B0
  Pressure:    2.500 bar
  Temperature: 6 C
  Integrity:   CRC

The second log is from running with "-F json" to cause output to be generated in JSON format:

  {"time" : "2018-02-10 09:51:02",
   "model" : "Toyota",
   "type" : "TPMS",
   "id" : "5e7e0637",
   "code" : "63e6026d",
   "mic" : "CRC"}

In both cases we see "TPMS", and according to wikipedia that is Tyre Pressure Monitoring System. I'm a little shocked to receive this data, unencrypted!

Other events also become visible, when I left the scanner running, which is presumably some kind of temperature-sensor a neighbour has running:

  2018-02-10 13:19:08 : RF-tech
     Id:              0
     Battery:         LOW
     Button:          0
     Temperature:     0.0 C

Anyway I have a bunch of remote-controlled sockets, branded "NEXA", which look like this:

Radio-Controlled Sockets

When I press the remote I can see the transmissions and program my PC to react to them:

  2018-02-11 07:31:20 : Nexa
    House Code:  39920705
    Group:  1
    Channel: 3
    State:   ON
    Unit:    2

In conclusion:

  • SDR can be used to easily sniff & decode cheap and commonly available 433Mhz-based devices.
  • "Modern" cars transmit their tyre-pressure, apparently!
  • My neighbours can probably overhear my button presses.



Creating an IoT button, the smart way

Friday, 9 February 2018

There are several projects out there designed to create an IoT button:

  • You press a button.
  • Magic happens, and stuff runs on your computer, or is otherwise triggered remotely.

I made my own internet-button, an esp8266-based alarm-button, and recently I've wanted to have a few more dotted around our flat. To recap, the basic way these things work is that you have a device with a button on it.

Once deployed you would press the button, your device wakes up, connects to your WiFi and sends a "message". That message then goes on to trigger some kind of defined action. In my case my button would mute any existing audio-playback, then trigger the launch of an "alarm.mp3" file. In short - if somebody presses the button I would notice.

I wanted a few more doing slightly more complex things in the flat, such as triggering lights and various programs. Unfortunately these buttons are actually relatively heavy-weight, largely because connecting to WiFi demands a reasonable amount of power-draw. Even with deep-sleeping between invocations, driving such a device from battery-power means the life-span is not great. (In my case I cheat, my button is powered by a small phone-charger, which means power isn't a concern, but my "button" is hobbled.)

Ideally what everybody wants is security, simplicity, and availability. Running from batteries, avoiding the need to program WiFi credentials and having a decent form-factor makes an IoT button a lot simpler to sell - you don't need to do complicated setup, and things are nice and neat.

So I wondered is such an impossible dream actually possible, and it turns out that yes, such a device is trivial.

Instead of building WiFi into a bunch of buttons you could you build the smarts into one device, a receiver, connected to your PC via a USB cable - the buttons are very very simple, don't use WiFi, don't need to be programmed, and don't draw a lot of current. How? Radio.

There exist pre-packaged and simple radio-based buttons, such as this one:


You press a button and it broadcasts a simple message on 433Mhz. There exist very cheap and reliable 433Mhz recievers which you can connect to an arduino, or ESP8266-based device. Which means you have a different approach:

  • You build a device based upon an Arduino/ESP8266/similar.
  • It listens for 433Mhz transmissions, decodes the device ID.
  • Once it finds something it recognizes it can write to STDOUT (more or less)
  • The host system opens /dev/ttyUSB0 and reads the output
    • Which it can then use to trigger actions.

The net result is you can buy a bunch of buttons, for €5 each and add them to your system. The transmissions are easy to decode, and each button has a unique ID. You don't need to program them with your WiFi credentials, or set them up - except on the host - and because these devices don't do much, they just sleep waiting for a press, make a brief radio-transmission, then go back to sleep, their batteries can last for months.

So that's what I've done. I've written a simple program which decodes the trasmissions and posts to an MQ instance "button-press-a", "button-press-b", etc, and I can react to them uniquely. (In some cases the actions taken depend upon the time of day.)

No code to show here, because it depends upon the precise flavour of button(s) that you buy. But I had fun because some of the remote-controls around my house use the same frequency - and a lot of the cheap "remote-controlled power-switches" use this fequency too. If you transmit as well as receive you can have a decent amount of fun. :)

Of course radio is "broadcast", so somebody nearby could tell I've pressed a button, but as far as security goes there are no obvious IoT-snafus that I think I care about.

In my next post I might even talk about something more interesting - SMS-based things. In Europe (data)-roaming fees have recently been abolished, and anti-monopoly regulations make it "simple" to get your own SIMs made. This means you can buy a bunch of SIMS, stick them in your devices and get cheap data-transfer Europe-wide. There are obvious commercial aspects available if you go down this route, if you can accept the caveat that you need to add a SIM-card to each transmitter and each receiver. If you can a lot of things that become possible, especially when coupled with GPS. Not only do you gain the ability to send messages/events/data, but you can see where it came from, physically/geographically, and that is something that I think has a lot of interesting use-cases.



More ESP8266 projects, radio and epaper

Saturday, 6 January 2018

I finally got the radio-project I've been talking about for the past while working. To recap:

  • I started with an RDA5807M module, but that was too small, and too badly-performing.
  • I moved on to using an Si4703-based integrated "evaluation" board. That was fine for headphones, but little else.
  • I finally got a TEA5767-based integrated "evaluatioN" board, which works just fine.
    • Although it is missing RDS (the system that lets you pull the name of the station off the transmission).
    • It also has no (digital) volume-control, so you have to adjust the volume physically, like a savage.

The project works well, despite the limitations, so I have a small set of speakers and the radio wired up. I can control the station via my web-browser and have an alarm to make it turn on/off at different times of day - cheating at that by using the software-MUTE facility.

All in all I can say that when it comes to IoT the "S stands for Simplicity" given that I had to buy three different boards to get the damn thing working the way I wanted. That said total cost is in the region of €5, probably about the same price I could pay for a "normal" hand-held radio. Oops.

The writeup is here:

The second project I've been working on recently was controlling a piece of ePaper via an ESP8266 device. This started largely by accident as I discovered you can buy a piece of ePaper (400x300 pixels) for €25 which is just cheap enough that it's worth experimenting with.

I had the intention that I'd display the day's calendar upon it, weather forecast, etc. My initial vision was a dashboard-like view with borders, images, and text. I figured rather than messing around with some fancy code-based grid-layout I should instead just generate a single JPG/PNG on a remote host, then program the board to download and display it.

Unfortunately the ESP8266 device I'm using has so little RAM that decoding and displaying a JPG/PNG from a remote URL is hard. Too hard. In the end I had to drop the use of SSL, and simplify the problem to get a working solution.

I wrote a perl script (what else?) to take an arbitrary JPG/PNG/image of the correct dimensions and process it row-by-row. It would keep track of the number of contiguous white/black pixels and output a series of "draw Lines" statements.

The ESP8266 downloads this simple data-file, and draws each line one at a time, ultimately displaying the image whilst keeping some memory free.

I documented the hell out of my setup here:

And here is a sample image being displayed:



Translating my website to Finnish

Thursday, 28 December 2017

I've now been living in Finland for two years, and I'm pondering a small project to translate my main website into Finnish.

Obviously if my content is solely Finnish it will become of little interest to the world - if my vanity lets me even pretend it is useful at the moment!

The traditional way to do this, with Apache, is to render pages in multiple languages and let the client(s) request their preferred version with Accept-Language:. Though it seems that many clients are terrible at this, and the whole approach is a mess. Pretending it works though we render pages such as:


Then "magic happens", such that the right content is served. I can then do extra-things, like add links to "English" or "Finnish" in the header/footers to let users choose.

Unfortunately I have an immediate problem! I host a bunch of websites on a single machine and I don't want to allow a single site compromise to affect other sites. To do that I run each website under its own Unix user. For example I have the website "" running as the "s-fi" user, and my blog runs as "s-blog", or "s-blogfi":

root@www ~ # psx -ef | egrep '(s-blog|s-fi)'
s-blogfi /usr/sbin/lighttpd -f /srv/ -D
s-blog   /usr/sbin/lighttpd -f /srv/ -D
s-fi     /usr/sbin/lighttpd -f /srv/ -D

There you can see the Unix user, and the per-user instance of lighttpd which hosts the website. Each instance binds to a high-port on localhost, and I have a reverse proxy listening on the public IP address to route incoming connections to the appropriate back-end instance.

I used to use thttpd but switched to lighttpd to allow CGI scripts to be used - some of my sites are slightly/mostly dynamic.

Unfortunately lighttpd doesn't support multiviews without some Lua hacks which will require rewriting - as the supplied example only handles Accept rather than the language-header I want.

It seems my simplest solution is to switch from having lighttpd on the back-end to running apache2 instead, but I've not yet decided which way to jump.

Food for thought, anyway.

hyvää joulua!



IoT radio: Still in-progress ..

Sunday, 17 December 2017

So back in September I was talking about building a IoT Radio, and after that I switched to talking about tracking aircraft via software-defined radio. Perhaps time for a followup.

So my initial attempt at a IoT radio was designed with RDA5807M module. Frustratingly the damn thing was too small to solder easily! Once I did get it working though I found that either the specs lied to me, or I'd misunderstood them: It wouldn't drive headphones, and performance was poor. (Though amusingly the first time I got it working I managed to tune to Helsinki's rock-station, and the first thing I heard was Rammstein's Amerika.)

I made another attempt with an Si4703-based "evaluation board". This was a board which had most of the stuff wired in, so all you had to do was connect an MCU to it, and do the necessary software dancing. There was a headphone-socket for output, and no need to fiddle with the chip itself, it was all pretty neat.

Unfortunately the evaluation board was perfect for basic use, but not at all suitable for real use. The board did successfully output audio to a pair of headphones, but unfortunately it required the use of headphones, as the cable would be treated as an antenna. As soon as I fed the output of the headphone-jack to an op-amp to drive some speakers I was beset with the kind of noise that makes old people reminisce about how music was better back in their day.

So I'm now up to round 3. I have a TEA5767-based project in the works, which should hopefully resolve my problems:

  • There are explicit output and aerial connections.
  • I know I'll need an amplifier.
  • The hardware is easy to control via arduino/esp8266 MCUs.
    • Numerous well-documented projects exist using this chip.

The only downside I can see is that I have to use the op-amp for volume control too - the TEA5767-chip allows you to mute/unmute via software but doesn't allow you to set the volume. Probably for the best.

In unrelated news I've got some e-paper which is ESP8266/arduino controlled. I have no killer-app for it, but it's pretty great. I should write that up sometime.

| No comments repository cleanup, and email-changes.

Sunday, 3 December 2017

I've shuffled around all the repositories which are associated with the blogspam service, such that they're all in the same place and refer to each other correctly:

Otherwise I've done a bit of tidying up on virtual machines, and I'm just about to drop the use of qpsmtpd for handling my email. I've used the (perl-based) qpsmtpd project for many years, and documented how my system works in a "book":

I'll be switching to pure exim4-based setup later today, and we'll see what that does. So far today I've received over five thousand spam emails:

  steve@ssh /spam/today $ find . -type f | wc -l

Looking more closely though over half of these rejections are "dictionary attacks", so they're not SPAM I'd see if I dropped the qpsmtpd-layer. Here's a sample log entry (for a mail that was both rejected at SMTP-time by qpsmtpd and archived to disc in case of error):

    "reason":"Mail for juha not accepted at",
    "subject":"Viagra Professional. Beyond compare. Buy at our shop.",

I suspect that with procmail piping to crm114, and a beefed up spam-checking configuration for exim4 I'll not see a significant difference and I'll have removed something non-standard. For what it is worth over 75% of the remaining junk which was rejected at SMTP-time has been rejected via DNS-blacklists. So again exim4 will take care of that for me.

If it turns out that I'm getting inundated with junk-mail I'll revert this, but I suspect that it'll all be fine.

| 1 comment.


Recent Posts

Recent Tags