Entries tagged bytemark

Lars Wirzenius recently released, and packaged for Debian, a simple script to make release tarballs. He calls it Unperish.

It makes me wonder how many other people use that kind of system?

Of the top of my head the only similar thing I can recall using is Brad Fitzpatrick's ShipIt - another moduler/plugin-based system (Perl rather than Python this time.)

For my needs I tend to just write a Makefile which has a "dist" target, and then I have a simple script called "release". This runs:

1. make dist / make release.
2. creates a gpg signature of the release.
3. scp's the resulting files to a remote source.

All this is configurable via a per-project .release file.

The configuration files are very simple, the script itself is almost trivial but being able to sit in a random project directory and have a new tarball on my webserver just by typing "release" is enormously useful.

There are times when I think I should make it a mini-project of its own, with the ability to auto-build Debian packages, etc. Other times I just think .. well its a hell of a lot better than my previous ad-hoc solution.

At the very least I think I will make the cosmetic change of updating the script to run "make test" if there is a test/ or t/ directory inside the generated tarball.

In real news - tomorrow I leave for a two week holiday with my partner's parents. Yesterday I got back from a night spent with her in York. The Bytemark staff night out. Lots of fun. Over too soon, but lots of fun.

Tags: brad, bytemark, christmas, lars, randomness, release, software, travel | 3 comments

So I've had a hectic few days, and I'm getting close to having caught up with the things that I've been sitting on whilst I've been away.

ObRandom: Several people, independantly, have told me within the past few days that "whilst" is not a real word. it is. End of ..

Some interesting things I've been working upon recently include a fun little firewall tool. Once upon a time I wrote a firewall script which worked like this:

firewall/
`-- incoming.d
    |-- smtp
    |-- ssh
    `-- www
`-- outgoing.d
    |-- ssh
    |-- smtp
    |-- dns
    `-- icmp

When you executed the magic firewall script it would scan the incoming.d directory, and for each file it found lookup the relevant port in /etc/services. These port numbers would then be opened. And at the end you'd just have a "-j DROP".

After a long phone conversation to a colleague on Thursday/Friday of last week I've now reworked this idea anew. There is still the notion of filenames referring to what is allowed for a pair of directories (incoming.d/ + outgoing.d/) but even more flexability and no hardwired use of /etc/servvices.

I guess some ideas are just too simple to give up ..?

Anyway there are a plethora of different firewall applications of varying sophistication and complexity in the world. I don't really want to go out of my way to promote this one - but at the same time it might be a useful idea for somebody?

The next (work) job I have is determining how to make a "kernel" + "kernel-dev" RPM package based on Debian sources. Joy. Actually the more I look around the more fiddly, annoying, and troublesome I suspect this is going to be. Sigh.

ObQuote: The Grudgy

Tags: bytemark, centos, firewalls, kernels, rpm | 3 comments

Over the past couple of months the machine which hosts the Debian Administration website has been struggling with two distinct problems:

The dreaded scheduler bug/issue

The machine would frequently hang with the messages of the form:

Task xxx blocked for more than 120 seconds

This would usually require the application of raised elephants to recover from.

OOM-issues

The system would exhaust the generous 2Gb of memory it possessed, and start killing random tasks until the memory usage fell - at which point the server itself stopped functioning in a useful manner.

Hopefully these problems are now over:

• Bytemark have generously increased the memory installed upon this (donated) machine from 2Gb to 4Gb.
• I've cleaned up some of the site code and switched to using a combination of nginx & apache 2 to serve the site. (I've documented this upon the site itself : Speeding up dynamic websites via an nginx proxy.)

The combination of these two changes should resolve the memory issues, and I've installed a home-made 2.6.31.4 kernel which appears to have corrected the task-blocking scheduler issue.

ObTitle: Bridget Jones: The Edge of Reason

Tags: apache2, bytemark, debian-administration, nginx, performance | 2 comments

Symbiosis

Symbiosis is the collective name given to a group of Debian GNU/Linux packages which implement simple virtual hosting. It is developed by my employers Bytemark.

Symbiosis is basically a collection of configuration snippets, code, and libraries which works to offer virtual hosting in a reliable consistent and easy to understand fashion.

You implement hosting for a new domain by merely creating a directory tree. So for example you might configure the hosting for the domain example.com by running:

mkdir -p /srv/example.com/public/htdocs
echo "hello, world" >> /srv/example.com/public/htdocs/index.html

mkdir -p /srv/example.com/mailboxes/webmaster
echo "super-secret" > /srv/example.com/mailboxes/webmaster/password

mkdir -p /srv/example.com/config
echo "3l33t" > /srv/example.com/mailboxes/config/ftp-password

There you are, now http://www.example.com/ and http://example.com/ will work, and you may login to check mail with the email address webmaster@example.com via POP3, IMAP, IMAPS, or POP3S. Finally you can FTP with username example.com and be dropped into the public directory.

The mail handling is very flexible, and the webhosting supports wonderful things.

I don't generally talk about work-stuff explicitly, but we've just made a major new release of the Symbiosis system such that it works upon Squeeze and has lots of IPv6 support out of the box. (Email, DNS, HTTP, Firewalling, FTP etc.)

All in all it is simple, well-documented, and open-source with a reasonably large user-base. More external testers, users, and developers would be a wonderful thing..

Mutt Mailboxes & Idle Hooks?

Mutt is wonderful but I'm starting to get annoyed by its lack of auto-mailbox discovery.

Assuming you use procmail you might deliver mail to ~/Maildir/.foo/ and mutt won't notice that if the directory is created once it starts.

(This is because generally mailboxes are defined via "mailboxes =one =two ..", even if you use a shell snippet it won't get updated unless you re-read configuration, or re-exec mutt).

I wish it were possible to use inotify/dnotify/something magic such that everything beneath ~/Maildir would just work.

(Re-reading mailboxes manually is one solution but it is .. nasty?)

I'm thinking that of all the possible solutions one of the most potentially interesting would be to define a new hook: "idle-hook command .."

That way "command" would be executed every time the client is idle. (This is a distinct state unrelated to IMAP IDLE times.)

Nopte: There is already "mail_check" & "timeout" options. Even running a defined command immediately following the code for mail_check would be reasonable.

Reverse Proxy

I continue to use, love, and enjoy my node.js-based reverse HTTP proxy, and pub discussions seemed to suggest it is a great idea (due to flexibility) but it will never take on because people don't trust node.

I'm almost tempted to re-code it in LUA & C. But I can't help but think that would be a waste of time which would not increase adoption - after all most people use "simple" reverse proxies, and they are well suited by Apache, nginx, or even varnish.

Still no rush I suppose.

In more personal news after living in this flat for 7 years, or so, I'm getting a new bathroom designed and deployed. Good times.

In the meantime I've been steadily watching Stargate SG-1 having recently purchased a box-set of series 1-10. I've just started series six this evening, and I'm enjoying it a lot.

ObQuote: "You have been recruited by the Star League to defend the frontier against Xur and the Ko-Dan armada. " - The Last Starfighter (1984). First film I ever saw at a cinema as a child.

Tags: bytemark, mutt, node.js, random, symbiosis | 9 comments

The following companies are amongst those showing Christmas Adverts on television before the start of December:

• Tesco
• Homebase.
• M&S
• Waitrose.
• John Lewis.

I will boycott these companies until next year.

In happier news I've spent the past week or two replacing the monitoring system that we use at work.

Our previous monitoring system had been struggling to keep up with the sheer number of tests it was being asked to process. This was partly because we carry out many ping-tests, ssh-tests, http-tests, dns-tests, etc. The other reason was that our monitoring system was a behemoth of threaded-ruby, which all ran upon a single host. This made adding another monitoring host a complex undertaking.

The new solution uses a work-queue:

• Tests to apply are parsed and inserted into a single, global, beanstalkd queue.
• Workers continuously poll the queue for tests to execute. They then execute them, and alert on failures as appropriate.

The code is open-source, written in Ruby, and available here:

• https://projects.bytemark.co.uk/projects/custodian/wiki

I've completed the process of tidying up the code to the extent I'm happy with it, and I believe I've also abstracted away the work-specific pieces of the code.

That said I'd not be surprised if it needs a few minor tweaks before it it useful for other people.

Tags: bytemark, christmas, custodian | 2 comments

After seven years working from home I've resigned from my position at Bytemark.

Why? A combination of wanting to do something different coupled with the desire to reclaim my second bedroom, which is currently tied up as an office.

Working in an office in the future will be weird ("You mean I have to get dressed every day?!") but hopefully not unduly burdonsome.

My two-year plan still remains in effect: Pay off this flat as soon as possible, then purchase another and rent this one out. Giving me some income of my own, which I will need.

The "five" year plan involves me quitting work, so that I can stay home and raise children. That makes sense because sometime next year I'll become the partner who earns the least amount of monies, and I'll also be the partner with the lowest upper-bound on salary potential (short of moving to London/similar which I've always ruled out).

Having rental income for myself means I'm not utterly dependant on other money, and all being well this place will be 100% paid off within 18 months.

(After that lots of saving will take place for a deposit for the second place. We did bid on a couple of places locally, which were outstanding, but it is perhaps for the best we didn't win them. No more looking at ESPC!)

Bytemark now becomes a company I recommend 100% for hosting in the UK. In the past I've always said nice things, but I've not strongly recommended them/us, because I'm too biased.

All my personal hosting, except for one virtual machine, will remain at Bytemark indefinitely. Lovely, flexible, and great.

(I have one outside guest for the purposes of diversification. That currently lives at Mythic Beasts.)

Tags: bytemark, flat, life, work | 7 comments

Today was my last day working at Bytemark, and I found it a lot harder than expected.

For better or worse I finished earlier than expected; having been gradually removing my accounts and privileges over the past few weeks I'd revoked my OpenVPN key this morning.

Mid-afternoon my openvpn connection tried to renegotiate session keys, or similar, and failed. So I stopped work a few hours early. That meant I managed to avoid sending my "goodbye world" email, which is probably for the best - after all a lovely company, lovely people, and a good environment, what can you say besides things that are lovely?

I think I largely wrapped things up neatly, and I'm pleased that one of my photos is hanging on the office wall. (I look forward to seeing that actually, I've only rarely made canvas prints.)

The only other thing of note this week has been the sharp rise in blogspam I've detected. Black Friday alive and well, on the internets ..

Tags: blogspam, bytemark | No comments

Today I should have been heading down to York, to attend the Bytemark Christmas party. Instead I'm here in Edinburgh, because wind/storms basically shutdown the rail network in Scotland for the morning.

Technically I could have probably made it, but only belatedly and only at a huge cost to my sanity. The train-station was insane with stranded people, and there seemed no guarantee the recently-revived service would continue.

So instead I'm sulking at home.

I had a lot of other things scheduled to do in York/London today/tomorrow, for reasons that will become apparent next week, so to say I'm annoyed is an understatement.

In happier news I'm not dead.

Walking to work this morning was horrific, there was so much wind 70-100mph, that I counldn't actually cross a bridge, on Ocean Drive, because I just kept getting blown into the road. (Yeah, that's a road that is very close to the coast. Driving wind. Horrible rain. Storming sea. Fun.)

I ended up retracing my steps, and taking a detour. (PS. My boots leaked.)

Not a good day. Enjoy some software instead - a trivial HTTP / XMPP bridge.

Tags: bytemark, cron, jabber, weather, wind, xmpp | No comments

Once upon a time I worked from home for seven years, for a company called Bytemark. Then, due to a variety of reasons, I left. I struck out for adventures and pastures new, enjoyed the novelty of wearing clothes every day, and left the house a bit more often.

Things happened. A year passed.

Now I'm working for Bytemark again, although there are changes and the most obvious one is that I'm working in a shared-space/co-working setup, renting a room in a building near my house instead of being in the house.

Shame I have to get dressed, but otherwise it seems to be OK.

Tags: bytemark | 3 comments

Over the past week I've mailed many of the people who had signed my previous GPG key and who had checked my ID as part of that process. My intention was to ask "Hey you trusted me before, would you sign my new key?".

So far no replies. I may have to be more dedicated and do the local-thing with people.

In other news Bytemark, who have previously donated a blade server, sponsored Debconf, and done other similar things, have now started offering free hosting to Debian-developers.

There is a list of such offers here:

• https://wiki.debian.org/ServicesHosting.

I think that concludes this months blog-posting quota. Although who knows? I turn 39 in a couple of days, and that might allow me to make a new one.

Tags: bytemark, gpg, hosting | 2 comments

In my previous post I talked about how our child-care works here in wintery Finland, and suggested there might be a change in the near future.

So here is the predictable update; I've resigned from my job and I'm going to be taking over childcare/daycare. Ideally this will last indefinitely, but it is definitely going to continue until November. (Which is the earliest any child could be moved into public day-care if there problems.)

I've loved my job, twice, but even though it makes me happy (in a way that several other positions didn't) there is no comparison. Child-care makes me happier-still. Sure there are days when your child just wants to scream, refuse to eat, and nothing works. But on average everything is awesome.

It's a hard decision, a "brave" decision too apparently (which I read negatively!), but also an easy one to make.

It'll be hard. I'll have no free time from 7AM-5PM, except during nap-time (11AM-1PM, give or take). But it will be worth it.

And who knows, maybe I'll even get to rant at people who ask "Where's his mother?" I live for those moments. Truly.

Tags: bytemark, childcare, finland | 4 comments