More ESP8266 projects, radio and epaper

Saturday, 6 January 2018

I finally got the radio-project I've been talking about for the past while working. To recap:

  • I started with an RDA5807M module, but that was too small, and too badly-performing.
  • I moved on to using an Si4703-based integrated "evaluation" board. That was fine for headphones, but little else.
  • I finally got a TEA5767-based integrated "evaluatioN" board, which works just fine.
    • Although it is missing RDS (the system that lets you pull the name of the station off the transmission).
    • It also has no (digital) volume-control, so you have to adjust the volume physically, like a savage.

The project works well, despite the limitations, so I have a small set of speakers and the radio wired up. I can control the station via my web-browser and have an alarm to make it turn on/off at different times of day - cheating at that by using the software-MUTE facility.

All in all I can say that when it comes to IoT the "S stands for Simplicity" given that I had to buy three different boards to get the damn thing working the way I wanted. That said total cost is in the region of €5, probably about the same price I could pay for a "normal" hand-held radio. Oops.

The writeup is here:

The second project I've been working on recently was controlling a piece of ePaper via an ESP8266 device. This started largely by accident as I discovered you can buy a piece of ePaper (400x300 pixels) for €25 which is just cheap enough that it's worth experimenting with.

I had the intention that I'd display the day's calendar upon it, weather forecast, etc. My initial vision was a dashboard-like view with borders, images, and text. I figured rather than messing around with some fancy code-based grid-layout I should instead just generate a single JPG/PNG on a remote host, then program the board to download and display it.

Unfortunately the ESP8266 device I'm using has so little RAM that decoding and displaying a JPG/PNG from a remote URL is hard. Too hard. In the end I had to drop the use of SSL, and simplify the problem to get a working solution.

I wrote a perl script (what else?) to take an arbitrary JPG/PNG/image of the correct dimensions and process it row-by-row. It would keep track of the number of contiguous white/black pixels and output a series of "draw Lines" statements.

The ESP8266 downloads this simple data-file, and draws each line one at a time, ultimately displaying the image whilst keeping some memory free.

I documented the hell out of my setup here:

And here is a sample image being displayed:

| 2 comments.

 

Translating my website to Finnish

Thursday, 28 December 2017

I've now been living in Finland for two years, and I'm pondering a small project to translate my main website into Finnish.

Obviously if my content is solely Finnish it will become of little interest to the world - if my vanity lets me even pretend it is useful at the moment!

The traditional way to do this, with Apache, is to render pages in multiple languages and let the client(s) request their preferred version with Accept-Language:. Though it seems that many clients are terrible at this, and the whole approach is a mess. Pretending it works though we render pages such as:

index.html
index.en.html
index.fi.html

Then "magic happens", such that the right content is served. I can then do extra-things, like add links to "English" or "Finnish" in the header/footers to let users choose.

Unfortunately I have an immediate problem! I host a bunch of websites on a single machine and I don't want to allow a single site compromise to affect other sites. To do that I run each website under its own Unix user. For example I have the website "steve.fi" running as the "s-fi" user, and my blog runs as "s-blog", or "s-blogfi":

root@www ~ # psx -ef | egrep '(s-blog|s-fi)'
s-blogfi /usr/sbin/lighttpd -f /srv/blog.steve.fi/lighttpd.conf -D
s-blog   /usr/sbin/lighttpd -f /srv/blog.steve.org.uk/lighttpd.conf -D
s-fi     /usr/sbin/lighttpd -f /srv/steve.fi/lighttpd.conf -D

There you can see the Unix user, and the per-user instance of lighttpd which hosts the website. Each instance binds to a high-port on localhost, and I have a reverse proxy listening on the public IP address to route incoming connections to the appropriate back-end instance.

I used to use thttpd but switched to lighttpd to allow CGI scripts to be used - some of my sites are slightly/mostly dynamic.

Unfortunately lighttpd doesn't support multiviews without some Lua hacks which will require rewriting - as the supplied example only handles Accept rather than the language-header I want.

It seems my simplest solution is to switch from having lighttpd on the back-end to running apache2 instead, but I've not yet decided which way to jump.

Food for thought, anyway.

hyvää joulua!

| 2 comments.

 

IoT radio: Still in-progress ..

Sunday, 17 December 2017

So back in September I was talking about building a IoT Radio, and after that I switched to talking about tracking aircraft via software-defined radio. Perhaps time for a followup.

So my initial attempt at a IoT radio was designed with RDA5807M module. Frustratingly the damn thing was too small to solder easily! Once I did get it working though I found that either the specs lied to me, or I'd misunderstood them: It wouldn't drive headphones, and performance was poor. (Though amusingly the first time I got it working I managed to tune to Helsinki's rock-station, and the first thing I heard was Rammstein's Amerika.)

I made another attempt with an Si4703-based "evaluation board". This was a board which had most of the stuff wired in, so all you had to do was connect an MCU to it, and do the necessary software dancing. There was a headphone-socket for output, and no need to fiddle with the chip itself, it was all pretty neat.

Unfortunately the evaluation board was perfect for basic use, but not at all suitable for real use. The board did successfully output audio to a pair of headphones, but unfortunately it required the use of headphones, as the cable would be treated as an antenna. As soon as I fed the output of the headphone-jack to an op-amp to drive some speakers I was beset with the kind of noise that makes old people reminisce about how music was better back in their day.

So I'm now up to round 3. I have a TEA5767-based project in the works, which should hopefully resolve my problems:

  • There are explicit output and aerial connections.
  • I know I'll need an amplifier.
  • The hardware is easy to control via arduino/esp8266 MCUs.
    • Numerous well-documented projects exist using this chip.

The only downside I can see is that I have to use the op-amp for volume control too - the TEA5767-chip allows you to mute/unmute via software but doesn't allow you to set the volume. Probably for the best.

In unrelated news I've got some e-paper which is ESP8266/arduino controlled. I have no killer-app for it, but it's pretty great. I should write that up sometime.

| No comments

 

BlogSpam.net repository cleanup, and email-changes.

Sunday, 3 December 2017

I've shuffled around all the repositories which are associated with the blogspam service, such that they're all in the same place and refer to each other correctly:

Otherwise I've done a bit of tidying up on virtual machines, and I'm just about to drop the use of qpsmtpd for handling my email. I've used the (perl-based) qpsmtpd project for many years, and documented how my system works in a "book":

I'll be switching to pure exim4-based setup later today, and we'll see what that does. So far today I've received over five thousand spam emails:

  steve@ssh /spam/today $ find . -type f | wc -l
  5731

Looking more closely though over half of these rejections are "dictionary attacks", so they're not SPAM I'd see if I dropped the qpsmtpd-layer. Here's a sample log entry (for a mail that was both rejected at SMTP-time by qpsmtpd and archived to disc in case of error):

   {"from":"<clzzgiadqb@ics.uci.edu>",
    "helo":"adrian-monk-v3.ics.uci.edu",
    "reason":"Mail for juha not accepted at steve.fi",
    "filename":"1512284907.P26574M119173Q0.ssh.steve.org.uk.steve.fi",
    "subject":"Viagra Professional. Beyond compare. Buy at our shop.",
    "ip":"2a00:6d40:60:814e::1",
    "message-id":"<p65NxDXNOo1b.cdD3s73osVDDQ@ics.uci.edu>",
    "recipient":"juha@steve.fi",
    "host":"Unknown"}

I suspect that with procmail piping to crm114, and a beefed up spam-checking configuration for exim4 I'll not see a significant difference and I'll have removed something non-standard. For what it is worth over 75% of the remaining junk which was rejected at SMTP-time has been rejected via DNS-blacklists. So again exim4 will take care of that for me.

If it turns out that I'm getting inundated with junk-mail I'll revert this, but I suspect that it'll all be fine.

| 1 comment.

 

Paternity-leave is half-over

Tuesday, 14 November 2017

I'm taking the month of November off work, so that I can exclusively take care of our child. Despite it being a difficult time, with him teething, it has been a great half-month so far.

During the course of the month I've found my interest in a lot of technological things waning, so I've killed my account(s) on a few platforms, and scaled back others - if I could exclusively do child-care for the next 20 years I'd be very happy, but sadly I don't think that is terribly realistic.

My interest in things hasn't entirely vanished though, to the extent that I found the time to replace my use of etcd with consul yesterday, and I'm trying to work out how to simplify my hosting setup. Right now I have a bunch of servers doing two kinds of web-hosting:

Hosting static-sites is trivial, whether with a virtual machine, via Amazons' S3-service, or some other static-host such as netlify.

Hosting for "dynamic stuff" is harder. These days a trend for "serverless" deployments allows you to react to events and be dynamic, but not everything can be a short-lived piece of ruby/javascript/lambda. It feels like I could setup a generic platform for launching containers, or otherwise modernising FastCGI, etc, but I'm not sure what the point would be. (I'd still be the person maintaining it, and it'd still be a hassle. I've zero interest in selling things to people, as that only means more support.)

In short I have a bunch of servers, they mostly tick over unattended, but I'm not really sure I want to keep them running for the next 10+ years. Over time our child will deserve, demand, and require more attention which means time for personal stuff is only going to diminish.

Simplify things now wouldn't be a bad thing to do, before it is too late.

| 3 comments.

 

Possibly retiring blogspam.net

Thursday, 2 November 2017

For the past few years I've hosted a service for spam-testing blog/forum comments, and I think it is on the verge of being retired.

The blogspam.net service presented a simple API for deciding whether an incoming blog/forum comment was SPAM, in real-time. I used it myself for two real reasons:

  • For the Debian Administration website.
    • Which is now retired.
  • For my blog
    • Which still sees a lot of spam comments, but which are easy to deal with because I can execute Lua scripts in my mail-client

As a result of the Debian-Administration server cleanup I'm still in the process of tidying up virtual machines, and servers. It crossed my mind that retiring this spam-service would allow me to free up another host.

Initially the service was coded in Perl using XML/RPC. The current version of the software, version 2, is written as a node.js service, and despite the async-nature of the service it is still too heavy-weight to live on the host which runs most of my other websites.

It was suggested to me that rewriting it in golang might allow it to process more requests, with fewer resources, so I started reimplementing the service in golang at 4AM this morning:

The service does the minimum:

  • Receives incoming HTTP POSTS
  • Decodes the body to a struct
  • Loops over that struct and calls each "plugin" to process it.
    • If any plugin decides this is spam, it returns that result.
  • Otherwise if all plugins have terminated then it decides the result is "OK".

I've ported several plugins, I've got 100% test-coverage of those plugins, and the service seems to be faster than the node.js version - so there is hope.

Of course the real test will be when it is deployed for real. If it holds up for a few days I'll leave it running. Otherwise the retirement notice I placed on the website, which chances are nobody will see, will be true.

The missing feature at the moment is keeping track of the count of spam-comments rejected/accepted on a per-site basis. Losing that information might be a shame, but I think I'm willing to live with it, if the alternative is closing down..

| 2 comments.

 

A busy week or two

Thursday, 12 October 2017

It feels like the past week or two has been very busy, and so I'm looking forward to my "holiday" next month.

I'm not really having a holiday of course, my wife is slowly returning to work, so I'll be taking a month of paternity leave, taking sole care of Oiva for the month of November. He's still a little angel, and now that he's reached 10 months old he's starting to get much more mobile - he's on the verge of walking, but not quite there yet. Mostly that means he wants you to hold his hands so that he can stand up, swaying back and forth before the inevitable collapse.

Beyond spending most of my evenings taking care of him, from the moment I return from work to his bedtime (around 7:30PM), I've made the Debian Administration website both read-only and much simpler. In the past that site was powered by a lot of servers, I think around 11. Now it has only a small number of machines, which should slowly decrease.

I've ripped out the database host, the redis host, the events-server, the planet-machine, the email-box, etc. Now we have a much simpler setup:

  • Front-end machine
    • Directly serves the code site
    • Directly serves the SSL site which exists solely for Let's Encrypt
    • Runs HAProxy to route the rest of the requests to the cluster.
  • 4 x Apache servers
    • Each one has a (read-only) MySQL database on it for the content.
      • In case of future-compromise I removed all user passwords, and scrambled the email-addresses.
      • I don't think there's a huge risk, but better safe than sorry.
    • Each one runs the web-application.
      • Which now caches each generated page to /tmp/x/x/x/x/$hash if it doesn't exist.
      • If the request is cached it is served from that cache rather than dynamically.

Finally although I'm slowly making progress with "radio stuff" I've knocked up a simple hack which uses an ultrasonic sensor to determine whether I'm sat in front of my (home) PC. If I am everything is good. If I'm absent the music is stopped and the screen locked. Kinda neat.

(Simple ESP8266 device wired to the sensor. When the state changes a message is posted to Mosquitto, where a listener reacts to the change(s).)

Oh, not final. I've also transfered my mobile phone from DNA.fi to MoiMobile. Which should complete soon, right now my phone is in limbo, active on niether service. Oops.

| No comments

 

Tracking aircraft in real-time, via software-defined-radio

Thursday, 5 October 2017

So my last blog-post was about creating a digital-radio, powered by an ESP8266 device, there's a joke there about wireless-control of a wireless. I'm not going to make it.

Sticking with a theme this post is also about radio, software-defined radio. I know almost nothing about SDR, except that it can be used to let your computer "do stuff" with radio. The only application I've ever read about that seemed interesting was tracking aircraft.

This post is about setting up a Debian GNU/Linux system to do exactly that, show aircraft in real-time above your head! This was almost painless to setup.

  • Buy the hardware.
  • Plug in the hardware.
  • Confirm it is detected.
  • Install the appropriate sdr development-package(s).
  • Install the magic software.
    • Written by @antirez, no less, you know it is gonna be good!

So I bought this USB device from AliExpress for the grand total of €8.46. I have no idea if that URL is stable, but I suspect it is probably not. Good luck finding something similar if you're living in the future!

Once I connected the Antenna to the USB stick, and inserted it into a spare slot it showed up in the output of lsusb:

  $ lsusb
  ..
  Bus 003 Device 043: ID 0bda:2838 Realtek Semiconductor Corp. RTL2838 DVB-T
  ..

In more detail I see the major/minor numbers:

  idVendor           0x0bda Realtek Semiconductor Corp.
  idProduct          0x2838 RTL2838 DVB-T

So far, so good. I installed the development headers/library I needed:

  # apt-get install librtlsdr-dev libusb-1.0-0-dev

Once that was done I could clone antirez's repository, and build it:

  $ git clone https://github.com/antirez/dump1090.git
  $ cd dump1090
  $ make

And run it:

  $ sudo ./dump1090 --interactive --net

This failed initially as a kernel-module had claimed the device, but removing that was trivial:

  $ sudo rmmod dvb_usb_rtl28xxu
  $ sudo ./dump1090 --interactive --net

Once it was running I'd see live updates on the console, every second:

  Hex    Flight   Altitude  Speed   Lat       Lon       Track  Messages Seen       .
  --------------------------------------------------------------------------------
  4601fc          14200     0       0.000     0.000     0     11        1 sec
  4601f2          9550      0       0.000     0.000     0     58        0 sec
  45ac52 SAS1716  2650      177     60.252    24.770    47    26        1 sec

And opening a browser pointing at http://localhost:8080/ would show that graphically, like so:

NOTE: In this view I'm in Helsinki, and the airport is at Vantaa, just outside the city.

Of course there are tweaks to be made:

  • With the right udev-rules in place it is possible to run the tool as non-root, and blacklist the default kernel module.
  • There are other forks of the dump1090 software that are more up-to-date to explore.
  • SDR can do more than track planes.

| 2 comments.

 

Started work on an internet-of-things Radio

Tuesday, 26 September 2017

So recently I was in York at the Bytemark office, and I read a piece about building a radio in a Raspberry Pi magazine. It got me curious, so when I got home to sunny Helsinki I figured I'd have a stab at it.

I don't have fixed goal in mind, but what I do have is:

  • A WeMos Mini D1
    • Cost €3.00
    • ESP8266-powered board, which can be programmed easily in C++ and contains on-board WiFi as well as a bunch of I/O pins.
  • A RDA5807M FM Radio chip.
    • Cost 37 cents.
    • With a crystal for support.

The initial goal is simple wire the receiver/decoder to the board, and listen to the radio.

After that there are obvious extenstions, such as adding an LCD display to show the frequency (What's the frequency Kenneth), and later to show the station details, via RDS.

Finally I could add some buttons/switches/tweaks for selecting next/previous stations, and adjusting the volume. Initially that'll be handled by pointing a browser at the IP-address of the device.

The first attempt at using the RDA5807M chip was a failure, as the thing was too damn small and non-standardly sized. Adding header-pins to the chips was almost impossible, and when I did get them soldered on the thing just gave me static-hisses.

However I later read the details of the chip more carefully and realized that it isn't powerfull enough to drive (even) headphones. It requires an amp of some kind. With that extra knowledge I was able to send the output to the powered-speakers I have sat beside my PC.

My code is basic, it sets up the FM-receiver/decoder, and scans the spectrum. When it finds a station it outputs the name over the serial console, via RDS, and then just plays it.

I've got an PAM8403-based amplifier board on-order, when that arrives I'll get back to the project, and hookup WiFi and a simple web-page to store stations, tuning, etc.

My "token goal" at the moment is a radio that switches on at 7AM and switches off at 8AM. In addition to that it'll serve a web-page allowing interactive control, regardless of any buttons that are wired in.

I also have another project in the wings. I've ordered a software-defined radio (USB-toy) which I'm planning to use to plot aircraft in real-time, as they arrive/depart/fly over Helsinki. No doubt I'll write that up too.

| 2 comments.

 

Retiring the Debian-Administration.org site

Thursday, 21 September 2017

So previously I've documented the setup of the Debian-Administration website, and now I'm going to retire it I'm planning how that will work.

There are currently 12 servers powering the site:

  • web1
  • web2
  • web3
  • web4
    • These perform the obvious role, serving content over HTTPS.
  • public
    • This is a HAProxy host which routes traffic to one of the four back-ends.
  • database
    • This stores the site-content.
  • events
    • There was a simple UDP-based protocol which sent notices here, from various parts of the code.
    • e.g. "Failed login for bob from 1.2.3.4".
  • mailer
    • Sends out emails. ("You have a new reply", "You forgot your password..", etc)
  • redis
    • This stored session-data, and short-term cached content.
  • backup
    • This contains backups of each host, via Obnam.
  • beta
    • A test-install of the codebase
  • planet
    • The blog-aggregation site

I've made a bunch of commits recently to drop the event-sending, since no more dynamic actions will be possible. So events can be retired immediately. redis will go when I turn off logins, as there will be no need for sessions/cookies. beta is only used for development, so I'll kill that too. Once logins are gone, and anonymous content is disabled there will be no need to send out emails, so mailer can be shutdown.

That leaves a bunch of hosts left:

  • database
    • I'll export the database and kill this host.
    • I will install mariadb on each web-node, and each host will be configured to talk to localhost only
    • I don't need to worry about four database receiving diverging content as updates will be disabled.
  • backup
  • planet
    • This will become orphaned, so I think I'll just move the content to the web-nodes.

All in all I think we'll just have five hosts left:

  • public to do the routing
  • web1-web4 to do the serving.

I think that's sane for the moment. I'm still pondering whether to export the code to static HTML, there's a lot of appeal as the load would drop a log, but equally I have a hell of a lot of mod_rewrite redirections in place, and reworking all of them would be a pain. Suspect this is something that will be done in the future, maybe next year.

| 2 comments.

 

Recent Posts

Recent Tags