Continuous integration that uses chroots?

Sunday, 12 June 2011

I'd like to setup some auto-builders for some projects - and theese projects must be built upon Lenny, Squeeze, Lucid, and multiple other distros. (i386 and amd64 obviously.)

Looking around I figure it should be simple. There are a lot of continuous integration tools out there - but when looking at them in depth it seems like they all work in temporary directories and are a little different to how I'd expect them to be.

Ultimately I want to point a tool at a repository (mercurial), and receive a status report and a bunch of .deb packages for a number of distributions.

The alternative seems to be to write a simple queue submission system, then for each job popped from the queue run:

  • Creates a new debootstrap-based chroot.
  • Installs build-essential, mercurial, etc.
  • Fetches the shource.
  • Runs make.
  • Copies the files produced in ./binary-out/ to a safe location.
  • Cleans up.

Surely this wheel must already exist? I guess its a given that we have to find build-dependencies, and that we cannot just run "pbuilder *.dsc" - as the dsc doesn't exist in advance. We really need to run "make dependencies test build", or similar.

Hudson looked promising, but it builds things into /var/lib/hudson, and doesn't seem to support the use of either chroots or schroots.

ObQuote: "I feel like I should get you another sweater." - "Friends"

| 8 comments.

 

Comments On This Entry

[gravitar] Asheesh Laroia (paulproteus)

Submitted at 19:08:53 on 12 june 2011

With Hudson, you can just add a few build steps as shell commands:

1. Create the chroot and install dependencies

2. Bind-mount /var/lib/hudson into there

3. Write "chroot /chroot/X" before your other build steps, so they run in the chroot

4. Un-bind-mount /var/lib/hudson

Then your resulting files will still be in /var/lib/hudson where Hudson expects them, so its built-in way to expose successful build result files (e.g. *.deb) should work fine.

This is kind of like how pbuilder works. I think this would be reasonably easy.

Let me know what exactly you've tried, and I can try to be more helpful. I run a few Hudson/Jenkins instances, and I've found the web interface to be a pleasure, by and large.

[gravitar] mirabilos

Submitted at 15:30:13 on 13 june 2011

Hm,

that should be possible with cowbuilder – bind mounting is one half of the trick (see what Asheesh wrote), the other half is hook scripts. Something like this as hookdir/B50jenkins should work (then ignore /var/cache/pbuilder/result/*):

#!/bin/sh
for a in /tmp/buildd/*; do
test -d "$a" || cp $a /var/lib/hudson/.../
done

Of course, it can get even fancier – copy only the files you actually need out:

#!/bin/sh
cat >/tmp/cf <<EOF
[cowkins]
method = local
allow_unsigned_uploads = 1
incoming = /var/lib/hudson/jobs/.../
EOF
dput -c /tmp/cf cowkins /tmp/buildd/*.changes

Have a look at https://evolvis.org/plugins/scmgit/cgi-bin/gitweb.cgi?p=shellsnippets/shellsnippets.git;a=blob;f=mksh/sysadmin/mvndput.sh;h=34a9b04b61f144073741053ec77120e339e3b4e1;hb=HEAD for a more full-featured version that also creates an APT repository per job automatically (you need both files), although that (of course) can’t run in the chroot.

So the general idea is to have a job run things like this:

wd=$(pwd)
mkdir hooks outdir
cat >hooks/B50jenkins < [cowkins]
method = local
allow_unsigned_uploads = 1
incoming = $wd/outdir
EOF
dput -c /tmp/cf cowkins /tmp/buildd/*.changes
EOS
chmod +x hooks/B50jenkins
cd svnworkingcopy/subdir/packagename
dpkg-buildpackage -rfakeroot -S
cd ..
sudo cowbuilder --hookdir "$wd/hooks" --bindmount "$wd" --build foo*.dsc
cd "$wd/outdir"
# do something with them, for example
mksh /opt/mvn-debs/mvndput.sh jobname squeeze main *.changes

[gravitar] Jon Topper

Submitted at 13:52:39 on 14 june 2011

If you're on a debian based system, you'll want to look at the "schroot" tool which gets you most of the way there. You build a basic chroot into an LVM volume, and 'schroot' lets you snapshot this, mount it up, use it, and then throw it away. "sbuild" extends that and lets you use an schroot environment for debuild-related activities.

[gravitar] wqeeqwqweqe

Submitted at 12:54:39 on 22 june 2011

"These", not "theese".

[gravitar] matthieu

Submitted at 03:06:38 on 7 march 2012

We use buildbot for this, it's as simple as
1)creating all the chroot you need (could be vm as well)
2)install buildbot inside chroot
3)run buildbot master from your host
4)commit anychange
5)wait and profit

[gravitar] Brandon Holtsclaw

Submitted at 06:36:04 on 7 march 2012

You might look into travis-ci too, its what your looking for only for webapps, i'm sure it could be easily adapted for debs

[gravitar] Anonymous

Submitted at 08:43:44 on 7 march 2012

A note regarding previous comments: you want to look at Jenkins, not Hudson. Hudson refers to the old project that Oracle took over and kicked out all the people working on it. Jenkins refers to the new version maintained and developed by all the original Hudson developers. Unless you want to get a support contract from Oracle (and if so, why), you want Jenkins. (See also LibreOffice, for very similar reasons.)

[gravitar] Martijn

Submitted at 09:07:54 on 7 march 2012

We use an Ubuntu-tool for automated backporting called 'prevu' (which is unfortunately abandonned in favour of 'backportpackage', but we're stuck with it for the time being).

Prevu uses pbuilder (or cowbuilder in our case), which takes care of the chroot stuff.

Finally, we have a relatively simple eternally running shell script, that basically does a 'git pull && prevu' on the source trees. It monitors a stamp file, that gets touched from a git hook when the commit contains a 'debian/changelog' file and fires off a build in that case.

 

Comments are closed on posts which are more than ten days old.

Recent Posts

Recent Tags