Building a computer - part 1

Thursday, 11 July 2019

I've been tinkering with hardware for a couple of years now, most of this is trivial stuff if I'm honest, for example:

  • Wiring a display to a WiFi-enabled ESP8266 device.
    • Making it fetch data over the internet and display it.
  • Hooking up a temperature/humidity sensor to a device.
    • Submit readings to an MQ bus.

Off-hand I think the most complex projects I've built have been complex in terms of software. For example I recently hooked up a 933Mhz radio-receiver to an ESP8266 device, then had to reverse engineer the protocol of the device I wanted to listen for. I recorded a radio-burst using an SDR dongle on my laptop, broke the transmission into 1 and 0 manually, worked out the payload and then ported that code to the ESP8266 device.

Anyway I've decided I should do something more complex, I should build "a computer". Going old-school I'm going to stick to what I know best the Z80 microprocessor. I started programming as a child with a ZX Spectrum which is built around a Z80.

Initially I started with BASIC, later I moved on to assembly language mostly because I wanted to hack games for infinite lives. I suspect the reason I don't play video-games so often these days is because I'm just not very good without cheating ;)

Anyway the Z80 is a reasonably simple processor, available in a 40PIN DIP format. There are the obvious connectors for power, ground, and a clock-source to make the thing tick. After that there are pins for the address-bus, and pins for the data-bus. Wiring up a standalone Z80 seems to be pretty trivial.

Of course making the processor "go" doesn't really give you much. You can wire it up, turn on the power, and barring explosions what do you have? A processor executing NOP instructions with no way to prove it is alive.

So to make a computer I need to interface with it. There are two obvious things that are required:

  • The ability to get your code on the thing.
    • i.e. It needs to read from memory.
  • The ability to read/write externally.
    • i.e. Light an LED, or scan for keyboard input.

I'm going to keep things basic at the moment, no pun intended. Because I have no RAM, because I have no ROM, because I have no keyboard I'm going to .. fake it.

The Z80 has 40 pins, of which I reckon we need to cable up over half. Only the arduino mega has enough pins for that, but I think if I use a Mega I can wire it to the Z80 then use the Arduino to drive it:

  • That means the Arduino will generate a clock-signal to make the Z80 tick.
  • The arduino will monitor the address-bus
    • When the Z80 makes a request to read the RAM at address 0x0000 it will return something from its memory.
    • When the Z80 makes a request to write to the RAM at address 0xffff it will store it away in its memory.
  • Similarly I can monitor for requests for I/O and fake that.

In short the Arduino will run a sketch with a 1024 byte array, which the Z80 will believe is its memory. Via the serial console I can read/write to that RAM, or have the contents hardcoded.

I thought I was being creative with this approach, but it seems like it has been done before, numerous times. For example:

  • http://baltazarstudios.com/arduino-zilog-z80/
  • https://forum.arduino.cc/index.php?topic=60739.0
  • https://retrocomputing.stackexchange.com/questions/2070/wiring-a-zilog-z80

Anyway I've ordered a bunch of Z80 chips, and an Arduino mega (since I own only one Arduino, I moved on to ESP8266 devices pretty quickly), so once it arrives I'll document the process further.

Once it works I'll need to slowly remove the arduino stuff - I guess I'll start by trying to build an external RAM/ROM interface, or an external I/O circuit. But basically:

  • Hook the Z80 up to the Arduino such that I can run my own code.
  • Then replace the arduino over time with standalone stuff.

The end result? I guess I have no illusions I can connect a full-sized keyboard to the chip, and drive a TV. But I bet I could wire up four buttons and an LCD panel. That should be enough to program a game of Tetris in Z80 assembly, and declare success. Something like that anyway :)

Expect part two to appear after my order of parts arrives from China.

| No comments

 

Upgraded my first host to buster

Tuesday, 9 July 2019

I upgrade the first of my personal machines to Debian's new stable release, buster, yesterday. So far two minor niggles, but nothing major.

My hosts are controlled, sometimes, by puppet. The puppet-master is running stretch and has puppet 4.8.2 installed. After upgrading my test-host to the new stable I discovered it has puppet 5.5 installed:

root@git ~ # puppet --version 5.5.10

I was not sure if there would be compatibility problems, but after reading the release notes nothing jumped out. Things seemed to work, once I fixed this immediate problem:

     # puppet agent --test
     Warning: Unable to fetch my node definition, but the agent run will continue:
     Warning: SSL_connect returned=1 errno=0 state=error: dh key too small
     Info: Retrieving pluginfacts
     ..

This error-message was repeated multiple times:

SSL_connect returned=1 errno=0 state=error: dh key too small

To fix this comment out the line in /etc/ssl/openssl.cnf which reads:

CipherString = DEFAULT@SECLEVEL=2

The second problem was that I use borg to run backups, once per day on most systems, and twice per day on others. I have an invocation which looks like this:

borg create ${flags} --compression=zlib  --stats ${dest}${self}::$(date +%Y-%m-%d-%H:%M:%S) \
   --exclude=/proc \
   --exclude=/swap.file \
   --exclude=/sys  \
   --exclude=/run  \
   --exclude=/dev  \
   --exclude=/var/log \
   /

That started to fail :

borg: error: unrecognized arguments: /

I fixed this by re-ordering the arguments such that it ended "destination path", and changing --exclude=x to --exclude x:

borg create ${flags} --compression=zlib  --stats \
   --exclude /proc \
   --exclude /swap.file \
   --exclude /sys  \
   --exclude /run  \
   --exclude /dev  \
   --exclude /var/log \
   ${dest}${self}::$(date +%Y-%m-%d-%H:%M:%S)  /

That approach works on my old and new hosts.

I'll leave this single system updated for a few more days to see what else is broken, if anything. Then I'll upgrade them in turn.

Good job!

| No comments

 

Radio gaga, the odessy of automation

Thursday, 6 June 2019

Recently I wanted to monitor the temperature and humidity of a sauna. I figured the safest way to go would be to place a battery-powered temperature/humidity sensor on a shelf, the kind of sensor that is commonly available on AliExpress for €1-5 each.

Most of the cheap "remote sensors" transmit their data over a short 433Mhz radio-transmission. So I just assumed it'd be possible to work something out.

The first step was to plug an SDR-dongle into my laptop, that worked just fine when testing, I could hear "stuff". But of course a Sauna is wood-lined, and beyond a tiled-shower area. In practice I just couldn't recieve the signal if my laptop lived in its usual location.

So I came up with a fall-back plan:

  • Wire a 433Mhz receiver to an ESP8266 device.
  • Sniff the radio-transmission.
    • Decode it
    • Inject into an MQ-host, via WiFi

Since the receiver could be within 10m of the transmitter I figured that would work fine - and it did. The real problem came when I tried to do this. There are a few projects you can find for acting as a 433Mhz -> WiFi bridge and none of them understood the transmission(s) my sensor was submitting.

In the end I had to listen for packets, work out the bit-spacing, and then later work out the actual contents of the packets. All by hand.

Anyway the end result is that I have something which will sniff the packets from the radio-transmitter, correctly calculate the temperature/humidity values and post them to MQ. From MQ a service polls the values and logs them to SQLite for later display. As a bonus I post to Slack the first time the temperature exceeds 50 &degree; a day:

  • "Hot? It's like a sauna in here."
  • "Main steam on, somebody set us up the beer."
  • etc.

Next week I'll talk about how I had a similar (read: identical) problem reacting to the 433Mhz transmission triggered by a doorbell. None of the gateways I looked at logged a thing when the button was pressed. So I'll have to save the transmission via rtl_433, analyze it with audacity, and do the necessary.

For reference these are the three existing firmwares/solutions/projects I tried; on a Wemos Mini D1:

  • https://github.com/xoseperez/espurna
  • https://github.com/arendst/Sonoff-Tasmota
    • Liked this one best for the UI.
  • https://github.com/1technophile/OpenMQTTGateway
    • This was sparse, but recommended too.
    • Tried two different modules: RF, RFPilight, wasn't clear what the difference was, no change anyway.

| No comments

 

Language/Communication development is fascinating

Sunday, 26 May 2019

We have a child who is now reaching 2.5 years old, and watching the language develop is fascinating. No doubt every parent experiences a similar level of amazement, but it's still new to me.

Part of the fun of watching our child grow his communication skills is obviously his bilingual nature; his mother speaks Finnish to him, and I speak English. Of course the pair of us communicate in English almost exclusively, but by contrast basically every other conversation he hears will be in Finnish.

Today I was struck by a new milestone, as he said the word "dog" for the first time.

I continue to read books to him, and of course they're simple books with lots of pictures. For over a year now he's been able to follow simple instructions:

  • Can you point to the dog?
  • Can you point to the cat?

I remember being really impressed when he was coming up to two years old, and he was consistently able to play that game - despite the various dogs/cats being drawn in different styles, and from different perspectives. (Cartoon dogs vary a lot; but he always was able to recognize them. He's obviously internalized the nature of the dog...)

Anyway he speaks pretty well, getting into two-four word sentences now. His favourite words are predictably enough "Äiti" (mother) and "en" (no). But he's never said dog until today, instead he's said:

  • Woof-woof
    • Probably as a result of months of me saying "dogs go woof", "cats go miow", & etc.
  • Hauva
    • Finnish for "doggy".
  • Dana
    • We have a neighbour with a dog. This dog moved in with us for a few days, and he fell in love.
    • That dog is called Dana, so suddenly all dogs became Dana.

Anyway today we were walking to the park and he said "iso-dog", "iso" being Finnish for "big". Indeed there was a big dog in front of him.

Good dog. Good boy.

Some of our conversations are quite intricate, some of the instructions we give him he can clearly understand/follow along - in two languages - but when I hear him use a new word, especially an English word, I'm suddenly reminded how awesome everything is.

| 3 comments.

 

Parsing PHP for fun and profit

Sunday, 7 April 2019

Recently I've been dealing with a lot of PHP code, and coders. I'm still not a huge fan of the language, but at the same time modern PHP is a world apart from legacy PHP which I dismissed 10ish years ago.

I've noticed a lot of the coders have had a good habit of documenting their code, but also consistently failing to keep class-names up to date. For example this code:

 <?php

 /**
  * Class Bar
  *
  * Comments go here ..
  */
 class Foo
 {
    ..

The rest of the file? Almost certainly correct, but that initial header contained a reference to the class Bar even though the implementation presented Foo.

I found a bunch of PHP linters which handle formatting, and coding-style checks, but nothing to address this specific problem. So I wrote a quick hack:

  • Parse PHP files.
    • Look for "/*".
    • Look for "*/".
    • Look for "class".
    • Treat everything else as a token, except for whitespace which we just silently discard.

Once you have a stream of such tokens you can detect this:

  • Found the start of a comment?
    • Clear the contents of any previously saved comment, in lastComment.
    • Append each subsequent token to "lastComment" until you hit EOF, or the end of a comment token.
  • Found a class token?
    • Look at the contents of the lastComment variable and see if it contains "class", after all the class might not have documentation that refers to any class.
    • If there is "class xxx" mentioned check it matches the current class name.

There were some initial false-positives when I had to handle cases like this:

throw new \Exception("class not found");

(Here my naive handling would decide we'd found a class called not.)

Anyway the end result was stable and detected about 150 offenses in the 2000 file codebase I'm looking at.

Good result. Next step was integrating that into the CI system.

And that concludes my recent PHP adventures, using go to help ;)

(Code isn't public; I suspect you could rewrite it in an hour. I also suspect I was over-engineering and a perl script using regexp would do the job just as well..)

| No comments

 

Raising a bilingual child

Tuesday, 5 March 2019

The last time I talked about parenting it was in the context of a childcare timetable, where my wife and I divide the day explicitly hour by hour so that one of us is "in charge" at all times.

For example, I might take care of Oiva from 7AM-12pm on Saturdays, then she takes over until 5pm, and I take 5-7PM (bed-time). We alternate who gives him a bath and sits/reads with him until he's asleep.

Even if all three of us are together there is always one person who is in-charge, and will handle nappies, food, and complaints. The system works well, and has done since he was a few weeks old. The big benefit is that both of us can take time off, avoiding burnout and frustration.

Anyway that's all stable, although my wifes overnight shifts sometimes play havoc with the equality, and I think we're all happy with it. The child himself seems to recognize who is in charge, and usually screams for the appropriate parent as required.

Today's post is more interesting, because it covers bilingual children, which our child is:

  • His mother is Finnish.
    • She speaks Finnish to him, exclusively.
  • I'm from the UK.
    • I speak English to him, exclusively.

Between ourselves we speak English 95% of the time and Finnish 2% of the time. The rest of our communication involves grunting, pointing, and eye-contact.

He's of an age now where he's getting really good at learning new words, and you can usually see who he learned them from. For example he's obsessed with (toy) cars. One of his earlier words was "auto", but these days he sometimes says "car" to me. He's been saying "ei" for months now, which is Finnish for "no". But now he's also started to say "no" in English.

We took care of a neighbours dog over the weekend, and when the dog tried to sniff one of his cars he pointed a finger at it, and said "No!". That was adorable.

Anyway his communication is almost exclusively single-words so far. If he's hungry he might say:

  • leipä! leipä! leipä!
    • Bread! Bread! Bread!
  • muesli! muesli! muesli!
    • muesli! muesli! muesli!

He understands complex ideas, commands, instructions, and sentences in both English and Finnish ("We're going to the shop", "Would you like to play in the park?", and many many more). But he's only really starting to understand that he can say the same thing in multiple languages - as per the example above of "ei" vs "no", or "car" vs "auto".

Usually he uses the word in the language he heard it in first. For example he'll say goodbye to people by saying "moi moi", but greet them with "hello". There are fun words though. For example 99% of the time a dog is a "woof woof", but sometimes recently he's been describing them as "hauva". A train is a "choo choo", as is a tram, and a rabbit is a "pupu".

He's started saying "kissa" for cat, but when watching cartoons or reading books he's more likely to identify them as dogs.

No real conclusion here, but it's adorable when he says isä/isi for Daddy, and äiti for Mummy. Or when he's finished at the dining table and sometimes he says "pois" and other times says "away".

Sometimes you can see confusion when we both refer to something with different words, but he seems pretty adept at understanding. I'm looking forward to seeing him flip words between languages more often - using each one within a couple of minutes. He has done that sometimes, but it's a rare thing. He'll sometimes say "daddy car" and "äiti auto", but more often than not the association seems random. He's just as likely to say "more kala" as "more fish".

| No comments

 

Experimenting with github actions

Tuesday, 26 February 2019

Recently I heared that travis-CI had been bought out, and later that they'd started to fire their staff.

I've used Travis-CI for a few years now, via github, to automatically build binaries for releases, and to run tests.

Since I was recently invited to try the Github Actions beta I figured it was time to experiment.

Github actions allow you to trigger "stuff" on "actions". Actions are things like commits being pushed to your repository, new releases appearing, and so on. "Stuff" is basically "launch a specific docker container".

The specified docker container has a copy of your project repository cloned into it, and you can operate upon it pretty freely.

I created two actions (which basically means I authored two Dockerfiles), and setup the meta-information, so that now I can do what I used to do with travis easily:

  • github-action-tester
    • Allows tests to be run whenever a new commit is pushed to your repository.
    • Or whenever a pull-request is submitted, or updated.
  • github-actions-publish-binaries
    • If you create a new release in the github UI your project is built, and the specified binaries are attached to the release.

Configuring these in the repository is very simple, you have to define a workflow at .github/main.workflow, and my projects tend to look very similar:

  # pushes trigger the testsuite
  workflow "Push Event" {
    on = "push"
    resolves = ["Test"]
  }

  # pull-requests trigger the testsuite
  workflow "Pull Request" {
    on = "pull_request"
    resolves = ["Test"]
  }

  # releases trigger new binary artifacts
  workflow "Handle Release" {
    on = "release"
    resolves = ["Upload"]
  }

  ##
  ## The actions
  ##

  ##
  ## Run the test-cases, via .github/run-tests.sh
  ##
  action "Test" {
     uses = "skx/github-action-tester@master"
  }

  ##
  ## Build the binaries, via .github/build, then upload them.
  ##
  action "Upload" {
    uses = "skx/github-action-publish-binaries@master"
    args = "math-compiler-*"
    secrets = ["GITHUB_TOKEN"]
  }

In order to make the actions generic they both execute a shell-script inside your repository. For example the action to run the tests just executes

  • .github/run-tests.sh

That way you can write the tests that make sense. For example a golang application would probably run go test ..., but a C-based system might run make test.

Similarly the release-making action runs .github/build, and assumes that will produce your binaries, which are then uploaded.

The upload-action requires the use of a secret, but it seems to be handled by magic - I didn't create one. I suspect GITHUB_TOKEN is a magic-secret which is generated on-demand.

Anyway I updated a few projects, and you can see their configuration by looking at .github within the repository:

All in all it was worth the few hours I spent on it, and now I no longer use Travis-CI. The cost? I guess now I'm tied to github some more...

| No comments

 

Updated myy compiler, and bought a watch.

Saturday, 16 February 2019

The simple math-compiler I introduced in my previous post has had a bit of an overhaul, so that now it is fully RPN-based.

Originally the input was RPN-like, now it is RPN for real. It handles error-detection at run-time, and generates a cleaner assembly-language output:

In other news I bought a new watch, which was a fun way to spend some time.

I love mechanical watches, clocks, and devices such as steam-engines. While watches are full of tiny and intricate parts I like the pretence that you can see how they work, and understand them. Steam engines are seductive because their operation is similar; you can almost look at them and understand how they work.

I've got a small collection of watches at the moment, ranging from €100-€2000 in price, these are universally skeleton-watches, or open-heart watches.

My recent purchase is something different. I was looking at used Rolexs, and found some from 1970s. That made me suddenly wonder what had been made the same year as I was born. So I started searching for vintage watches, which had been manufactured in 1976. In the end I found a nice Soviet Union piece, made by Raketa. I can't prove that this specific model was actually manufactured that year, but I'll keep up the pretence. If it is +/- 10 years that's probably close enough.

My personal dream-watch is the Rolex Oyster (I like to avoid complications). The Oyster is beautiful, and I can afford it. But even with insurance I'd feel too paranoid leaving the house with that much money on my wrist. No doubt I'll find a used one, for half that price, sometime. I'm not in a hurry.

(In a horological-sense a "complication" is something above/beyond the regular display of time. So showing the day, the date, or phase of the moon would each be complications.)

| No comments

 

I decided it was time to write a compiler

Thursday, 31 January 2019

I've spent some time in the recent past working with interpreters, and writing a BASIC interpreter, but one thing I'd not done is write a compiler.

Once upon a time I worked for a compiler-company, but I wasn't involved with the actual coding at that time. Instead I worked on other projects, and did a minor amount of system-administration.

There are enough toy-languages that it didn't seem worthwhile to write a compiler for yet another one. At the same time writing a compiler for a full-language would get bogged down in a lot of noise.

So I decided to simplify things: I would write a compiler for "maths". Something that would take an expression and output assembly-language, which could then be compiled.

The end result is this simple compiler:

Initially I wrote something that would parse expressions such as 3 + 4 * 5 and output an abstract-syntax-tree. I walked the tree and started writing logic to pick registers, and similar. It all seemed like more of a grind than a useful exercise - and considering how ludicrous compiling simple expressions to assembly language already was it seemed particularly silly.

So once again I simplified, deciding to accept only a simple "reverse-polish-like" expression, and outputing the assembly for that almost directly.

Assume you want to calculate "((3 * 5) +2)" you'd feed my compiler:

  3 5 * 2 +

To compile that we first load the initial state 3, then we walk the rest of the program always applying an operation with an operand:

  • Store 3
  • 5 * -> multiply by 5.
  • 2 + -> add 2.
  • ..

This approach is trivial to parse, and trivial to output the assembly-language for: Pick a register and load your starting value, then just make sure all your operations apply to that particular register. (In the case of Intel assembly it made sense to store the starting value in EAX, and work with that).

A simple program would then produce a correspondingly simple output. Given 1 1 + we'd expect this output:

  .intel_syntax noprefix
  .global main

  .data
    result: .asciz "Result %d\n"

  main:
    mov rax, 1
    add rax, 1

    lea rdi,result
    mov rsi, rax
    xor rax, rax
    call printf
    xor rax, rax
    ret

With that output you can assemble the program, and run it:

 $ gcc -static -o program program.s
 $ ./program
 Result 2

I wrote some logic to allow calculating powers too, so you can output 2 ^ 8, etc. That's just implemented the naive-way, where you have a small loop and multiply the contents of EAX by itself the appropriate number of times. Modulus is similarly simple to calculate.

Adding support for named variables, and other things, wouldn't be too hard. But it would involve register-allocation and similar complexity. Perhaps that's something I need to flirt with, to make the learning process complete, but to be honest I can't be bothered.

Anyway check it out, if you like super-fast maths. My benchmark?

$  time perl -e 'print 2 ** 8 . "\n"'
256
real    0m0.006s
user    0m0.005s
sys     0m0.000s

vs.

$ math-compiler -compile '2 8 ^'
$ time ./a.out
Result 256

real   0m0.001s
user   0m0.001s
sys    0m0.000s

Wow. Many wow. Much speed. All your base-two are belong to us.

| No comments

 

This is mostly how I make bread

Sunday, 28 October 2018

I've talked about making bread in the past on this blog, here's my typical recipe - this recipe requires only two bowls, a spatula, a dutch-oven, oven-gloves, and some scales.

Ingredients:

  • 1kg flour
  • 950g water.
    • If you want to keep it simple you could use 1kg of water instead. It simplifies the measuring if you're using a balance-scale
  • 1.5 teaspoon salt.
  • 1 teaspoon instant/dried yeast.

Here is the flour, water, & yeast - not pictured salt!

Mix all ingredients together. It will be sticky, and your hands will become messy. Embrace it. (ProTip: Take off your watch and wedding-ring(s) if applicable.) Expect it to take 2-5 minutes to do a decent job. Ensure you scoop your hand right into the bottom of the mixture, to make sure there is no flour clumped together at the bottom of the bowl which is not fully mixed in.

The end result is a sticky mess which will look something like this, perhaps your bowl will be cleaner and you'll have done a better job at mixing all the flour!

Cover the bowl with cling-film, and stick in the fridge overnight. (I tend to mix stuff at 6PM in the evening, then come back to it around 9AM the following morning, which means the bowl sits in the fridge for 14 hours or so.)

Take the bowl out of the fridge and you should see it has "grown", and it will have a lot of bubbles on the top, as growth of the yeast emitted CO2.

You'll also see that it is significantly more gloopy, as chains of gluten have formed

Anyway now your bowl is on the counter, out of the fridge, you want turn on your oven and set it to 250°C, with the dutch oven inside it. While you're waiting for the oven to heat up transfer the sticky mess to a new bowl, lined with baking-paper. This will make it easier to add to the pot when we're ready to actually cook it.

As per the previous video the mixture will be very sticky, but you should be able to manage it. Don't worry too much about the shape, it'll become a "loaf-shape" when it cooks, the only reason we're moving it is because it is much easier to lift the mixture into the pot by holding the paper, than trying to scrape it from your cold bowl to your very hot dutch-oven. Anyway once you've moved it to a new bowl you'll have something like this:

When your oven has reached the right temperature carefully transfer the mixture, in its paper, to the dutch oven which you'll then return to the oven.

  • Cook for 40 minutes at 250°C
  • Cook for an additional 20 minutes at 200°C
    • Just turn down the temperature-dial.
  • Finally open the oven, remove the lid from the pot, and cook for a further 15 minutes (still at 200°C)

The end result will be something similar to this:

Enjoy!

Notes:

  • You can see vestiages of the paper-wrapper in the final result.
  • I like my bread dark.
  • Let it cool down before you eat it, something like 45-60 minutes once you've removed from the oven.

| 7 comments.

 

Recent Posts

Recent Tags