Entries tagged git

Today mostly consisted of a new release of the chronicle blog compiler. Interestingly this received several random mails today. I wonder what caused that all of a sudden?

The release of the compiler is timely, as it reminds me I've still not managed to find a decent gallery compiler. Although the thought of writing my own, rightly, fills me with depression.

I've been interested in reading more about both Git and SELinux upon Planet Coker Debian recently. I've switched several small projects over to GIT but I've not yet listed them publically. First of all I would like to see if there is a version of trac that I can install which supports git repositories. I guess that's a job to research tomorrow.

I wonder if I would confuse people by hosting GIT projects upon cvsrepository.org? ;)

Tags: chronicle, cvs, git | No comments

I'm still in the middle of a quandry with regards to revision control.

90% of my open code is hosted via CVS at a central site.

I wish to migrate away from CVS in the very near future, and having ummed and ahhed for a while I've picked murcurial as my system of choice. There is extensive documentation, and it does everything I believe I need.

The close-runner was git, but on balance I've decided to choose mercurial as it wins in a few respects.

Now the plan. I have two options:

• Leave each project in one central site.
• Migrate project $foo to its own location.

e.g. My xen-tools could be hosted at mercurial.xen-tools.org, my blog compiler could live at mercurial.steve.org.uk.

Alternatively I could just leave the one site in place, ignoring the fact that the domain name is now inappropriate.

The problem? I can't decide which approach to go for. Both have plusses and minuses.

Suggestions or rationales welcome - but no holy wars on why any particular revision control system is best...

I guess ultimately it matters little, and short of mass-editing links its 50/50.

Tags: chronicle, cvs, cvsrepository, git, lazyweb, mercurial, todo | No comments

Once upon a time a company started using distributed version control, and setup several project repositories using darcs.

Over time people became more sane and new projects were created in mercurial.

Later still Git became available, and was used by a few of the brave.

Sadly each of these projects is hosted on the same host, and in the home directory of the same user. This means these two commands work:

hg clone ssh://[email protected]/foo

git clone ssh://[email protected]/bar

I'm now wanting to setup per-repository ACLs and have hit a problem...

There are several git-wrappers such as gitolite and gitosis. There is also the excellent hg-gateway and mercurial-server for dealing with mercurial.

However I've yet to find a wrapper which will handle both git & mercurial repositories, under the same UID. (+ Darcs too, of course).

So my question - is there such a beast out there, or do we need to write it? I expect such a thing would be useful for many people, so I'm surprised I've not yet found it.

Tags: git, lazyweb, mercurial | 12 comments

To round up the discussion of the Debian Administration site yesterday I flipped the switch on the load-balancing. Rather than this:

  https -> pound \
                  \
  http  -------------> varnish  --> apache

We now have the simpler route for all requests:

http  -> haproxy -> apache
https -> haproxy -> apache

This means we have one less HTTP-request for all incoming secure connections, and these days secure connections are preferred since a Strict-Transport-Security header is set.

In other news I've been juggling git repositories; I've setup an installation of GitBucket on my git-host. My personal git repository used to contain some private repositories and some mirrors.

Now it contains mirrors of most things on github, as well as many more private repositories.

The main reason for the switch was to get a prettier interface and bug-tracker support.

A side-benefit is that I can use "groups" to organize repositories, so for example:

• http://git.steve.org.uk/websites - Contains website source code.
• http://git.steve.org.uk/blogspam - Contains grouped repositories on a single theme.

Most of those are mirrors of the github repositories, but some are new. When signed in I see more sources, for example the source to http://steve.org.uk.

I've been pleased with the setup and performance, though I had to add some caching and some other magic at the nginx level to provide /robots.txt, etc, which are not otherwise present.

I'm not abandoning github, but I will no longer be using it for private repositories (I was gifted a free subscription a year or three ago), and nor will I post things there exclusively.

If a single canonical source location is required for a repository it will be one that I control, maintain, and host.

I don't expect I'll give people commit access on this mirror, but it is certainly possible. In the past I've certainly given people access to private repositories for collaboration, etc.

Tags: git, github, yawns | No comments

Yesterday I carried out the upgrade of a Debian host from Squeeze to Wheezy for a friend. I like doing odd-jobs like this as they're generally painless, and when there are problems it is a fun learning experience.

I accidentally forgot to check on the status of the MySQL server on that particular host, which was a little embarassing, but later put together a reasonably thorough serverspec recipe to describe how the machine should be setup, which will avoid that problem in the future - Introduction/tutorial here.

The more I use serverspec the more I like it. My own personal servers have good rules now:

shelob ~/Repos/git.steve.org.uk/server/testing $ make
..
Finished in 1 minute 6.53 seconds
362 examples, 0 failures

Slow, but comprehensive.

In other news I've now migrated every single one of my personal mercurial repositories over to git. I didn't have a particular reason for doing that, but I've started using git more and more for collaboration with others and using two systems felt like an annoyance.

That means I no longer have to host two different kinds of repositories, and I can use the excellent gitbucket software on my git repository host.

Needless to say I wrote a policy for this host too:

#
#  The host should be wheezy.
#
describe command("lsb_release -d") do
  its(:stdout) { should match /wheezy/ }
end


#
# Our gitbucket instance should be running, under runit.
#
describe supervise('gitbucket') do
  its(:status) { should eq 'run' }
end

#
# nginx will proxy to our back-end
#
describe service('nginx') do
  it { should be_enabled   }
  it { should be_running   }
end
describe port(80) do
  it { should be_listening }
end

#
#  Host should resolve
#
describe host("git.steve.org.uk" ) do
  it { should be_resolvable.by('dns') }
end

Simple stuff, but being able to trigger all these kind of tests, on all my hosts, with one command, is very reassuring.

Tags: git, github, serverspec | 1 comment

It looks like I'll be spending a lot of time working with puppet over the coming weeks.

I've setup some toy deployments on virtual machines, and have converted several of my own hosts to using it, rather than my own slaughter system.

When it comes to puppet some things are good, and some things are bad, as exected, and as any similar tool (even my own). At the moment I'm just aiming for consistency and making sure I can control all the systems - BSD, Debian GNU/Linux, Ubuntu, Microsoft Windows, etc.

Little changes are making me happy though - rather than using a local git pre-commit hook to validate puppet manifests I'm now doing that checking on the server-side via a git pre-receive hook.

Doing it on the server-side means that I can never forget to add the local hook and future-colleagues can similarly never make this mistake, and commit malformed puppetry.

It is almost a shame there isn't a decent collection of example git-hooks, for doing things like this puppet-validation. Maybe there is and I've missed it.

It only crossed my mind because I've had to write several of these recently - a hook to rebuild a static website when the repository has a new markdown file pushed to it, a hook to validate syntax when pushes are attempted, and another hook to deny updates if the C-code fails to compile.

Tags: git, puppet | 3 comments

I've been allowing users to upload SSH public-keys, and displaying them online in a form. Displaying an SSH public key is a pain, because they're typically long. That means you need to wrap them, or truncate them, or you introduce a horizontal scroll-bar.

So rather than displaying them I figured I'd generate a fingerprint when the key was uploaded and show that instead - This is exactly how github shows your ssh-keys.

Annoyingly there is only one reasonable way to get a fingerprint from a key:

• Write it to a temporary file.
• Run "ssh-keygen -lf temporary/file/name".

You can sometimes calculate the key via more direct, but less obvious methods:

awk '{print $2}' ~/.ssh/id_rsa.pub | base64 -d | md5sum

But that won't work for all key-types.

It is interesting to look at the various key-types which are available these days:

mkdir ~/ssh/
cd ~/ssh/
for i in dsa ecdsa ed25519 rsa rsa1 ; do
  ssh-keygen -P "" -t $i -f ${i}-key
done

I've never seen an ed25519 key in the wild. It looks like this:

$ cat ~/ssh/ed25519-key.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMcT04t6UpewqQHWI4gfyBpP/ueSjbcGEze22vdlq0mW skx@shelob

Similarly curve-based keys are short too, but not as short:

ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLTJ5+  \
 rWoq5cNcjXdhzRiEK3Yq6tFSYr4DBsqkRI0ZqJdb+7RxbhJYUOq5jsBlHUzktYhOahEDlc9Lezz3ZUqXg= skx@shelob

Remember what I said about wrapping? Ha!

Anyway for the moment I've hacked up a simple perl module SSH::Key::Fingerprint which will accept a public key and return the fingerprint, as well as validating the key is well-formed and of a known-type. I might make it public in the future, but I think the name is all wrong.

The only code I could easily find to do a similar job is this node.js package, but it doesn't work on all key-types. Shame.

And that concludes this weeks super-happy fun-time TODO-list item.

Tags: git, ssh | 4 comments

Like many people I use a password-manage to record logins to websites. I previously used a tool called pwsafe, but these days I switched to using pass.

Although I don't like the fact the meta-data is exposed the tool is very useful, and its integration with git is both simple and reliable.

Reading about the security issue that recently affected cloudflare made me consider rotating some passwords. Using git I figured I could look at the last update-time of my passwords. Indeed that was pretty simple:

git ls-tree -r --name-only HEAD | while read filename; do
  echo "$(git log -1 --format="%ad" -- $filename) $filename"
done

Of course that's not quite enough because we want it sorted, and to do that using the seconds-since-epoch is neater. All together I wrote this:

#!/bin/sh
#
# Show password age - should be useful for rotation - we first of all
# format the timestamp of every *.gpg file, as both unix+relative time,
# then we sort, and finally we output that sorted data - but we skip
# the first field which is the unix-epoch time.
#
( git ls-tree -r --name-only HEAD | grep '\.gpg$' | while read filename; do \
      echo "$(git log -1 --format="%at %ar" -- $filename) $filename" ; done ) \
        | sort | awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}'

Not the cleanest script I've ever hacked together, but the output is nice:

 steve@ssh ~ $ cd ~/Repos/personal/pass/
 steve@ssh ~/Repos/personal/pass $ ./password-age | head -n 5
 1 year, 10 months ago GPG/[email protected]
 1 year, 10 months ago GPG/[email protected]
 1 year, 10 months ago GPG/[email protected]
 1 year, 10 months ago Git/git.steve.org.uk/root.gpg
 1 year, 10 months ago Git/git.steve.org.uk/skx.gpg

Now I need to pick the sites that are more than a year old and rotate credentials. Or delete accounts, as appropriate.

Tags: git, passwords | 4 comments

Sometimes I enjoy reading the source code to projects I like, use, or am about to install for the first time. This was something I used to do on a very regular basis, looking for security issues to report. Nowadays I don't have so much free time, but I still like to inspect the source code to new applications I install, and every now and again I'll find the time to look at the source to random projects.

Reading code is good. Reading code is educational.

One application I've looked at multiple times is redis, which is a great example of clean and well-written code. That said when reading the redis codebase I couldn't help noticing that there were a reasonably large number of typos/spelling mistakes in the comments, so I submitted a pull-request:

• https://github.com/redis/redis/pull/6058

Sadly that particular pull-request didn't receive too much attention, although a previous one updating the configuration file was accepted. I was recently reminded of these pull-requests when I was when I was doing some other work. So I figured I'd have a quick scan of a couple of other utilities.

In the past I'd just note spelling mistakes when I came across them, usually I'd be opening each file in a project one by one and reading them from top to bottom. (Sometimes I'd just open files in emacs and run "M-x ispell-comments-and-strings", but more often I'd just notice them with my eyes). It did strike me that if I were to do this in a more serious fashion it would be good to automate it.

So this time round I hacked up a simple "dump comments" utility, which would scan named files and output the contents of any comments (be they single-line, or multi-line). Once I'd done that I could spell-check easily:

 $ go run dump-comments.go *.c > comments
 $ aspell -c comments

Anyway the upshot of that was a pull-request against git:

• https://github.com/gitgitgadget/git/pull/685

We'll see if that makes its way live sometime. In case I get interested in doing this again I've updated my sysbox-utility collection to have a comments sub-command. That's a little more robust and reliable than my previous hack:

$ sysbox comments -pretty=true $(find . -name '*.c')
..
..

The comments sub-command has support for:

• Single-line comments, for C, as prefixed with //.
• Multi-line comments, for C++, as between /* and */.
• Single-line comments, for shell, as prefixed with #.
• Lua comments, both single-line (prefixed with --) and multiline between --[[ and --]].

Adding new support would be trivial, I just need a start and end pattern to search against. Pull-requests welcome:

• https://github.com/skx/sysbox

Tags: git, github, redis, sysbox | 1 comment

Like many I use pass for storing usernames and passwords. This gives me easy access to credentials in a secure manner.

I don't like the way that the metadata (i.e. filenames) are public, but that aside it is a robust tool I've been using for several years.

The last time I talked about pass was when I talked about showing the age of my credentials, via the integrated git support.

That then became a pass-plugin:

  frodo ~ $ pass age
  6 years ago GPG/[email protected]
  6 years ago GPG/[email protected]
  ..
  4 years, 8 months ago Domains/Domain.fi.gpg
  4 years, 7 months ago Mobile/dna.fi.gpg
  ..
  1 year, 3 months ago Websites/netlify.com.gpg
  1 year ago Financial/ukko.fi.gpg
  1 year ago Mobile/KiK.gpg
  4 days ago Enfuce/sre.tst.gpg
  ..

Anyway today's work involved writing another plugin, named env. I store my data in pass in a consistent form, each entry looks like this:

   username: steve
   password: secrit
   site: http://example.com/login/blah/
   # Extra data

The keys vary, sometimes I use "login", sometimes "username", other times "email", but I always label the fields in some way.

Recently I was working with some CLI tooling that wants to have a username/password specified and I patched it to read from the environment instead. Now I can run this:

     $ pass env internal/cli/tool-name
     export username="steve"
     export password="secrit"

That's ideal, because now I can source that from within a shell:

   $ source <(pass env internal/cli/tool-name)
   $ echo username
   steve

Or I could directly execute the tool I want:

   $ pass env --exec=$HOME/ldap/ldap.py internal/cli/tool-name
   you are steve
   ..

TLDR: If you store your password entries in "key: value" form you can process them to export $KEY=$value, and that allows them to be used without copying and pasting into command-line arguments (e.g. "~/ldap/ldap.py --username=steve --password=secrit")

Tags: git, pass, passwordstore, shell | 7 comments