Introduction

Currently you can only know if a package cross-builds successfully if you set up an environment and try it.

Being able to look this up online and provide automatic feedback to package maintainers is very useful, and allows collaboration and for people to fix the stuff they care about.

The ultimate aim is to have nice infrastructure with online summaries and a backend-database, doing cross-rebuilds on newly-uploaded packages, as Debian and Ubuntu do for native builds.

However that will take a while to fill in missing infrastrucutre pieces, so in the meantime we have a rather simpler setup, which at least provides the basic info.

Current buildd

The current output is found at http://people.linaro.org/~wookey/buildd/

Analysis of build issues is here: Multiarch Crossbuild status

This buildd is set up using reprepro, rebuildd, sbuild, schroot and a few glue scripts and cron jobs.

logfiles are htmlised, indexed and synced https://wiki.linaro.org/Platform/DevPlatform/CrossCompile/MultiarchCrossBuildStatus to the above location

Currently supported build flavour is: precise multiarch builds using sbuild (called 'sbuild')

An older scheme using xbuilder to process a list of packages 'single-shot' was used for Natty and Oneiric dpkg-cross style builds.

All are building the linaro-nano package set.

Implementation notes

A full buildd setup needs these parts:

1. A repository to store built packages and to provide packages and sources lists to compare for what is out of date.
2. A database to track what is/isn't up-to-date
3. Something to query the out-of-date list and schedule builds on suitable machines
4. Chroots to do builds in
5. Builder to get sources, gets deps and run builds
6. Nice web UI to examine results

Real distros have complicated and slightly mysterious tools for some of this making setups that are hard to reproduce, but most of it is modularised and here is the easiest way to make your own:

1. reprepro (packaged)
2. rebuildd (packaged)
3. schroot (packaged)
4. sbuild (packaged)

None of this really changes for cross-building. Just different-arch chroots are used and sbuild is passed an arch to build for.

Reprepro can produce a need-sbuild list on its own to feed almost directly to sbuild, so for a simple case we can cut this down to just use:

• 1 & 2 Reprepro

• 3 xbuilder scripts

• 4 & 5 sbuild

• 6 rsync

Setup

Get current versions of tools

You need these recently-modified versions: sbuild v0.63 (in Debian testing) rebuildd 0.3.12cross1 (from linaro cross-tool PPA) (to do arch-all builds you need reprepro 4.9.0 or later)

Add this source to your build machine deb http://ppa.launchpad.net/linaro-foundations/cross-build-tools/ubuntu precise main

apt-get install rebuildd reprepro sbuild schroot

Create build chroot

Follow Preparing multiarch chroot

Set up rebuildd

Edit these config files: /etc/default/rebuildd

START_REBUILDD=1
START_REBUILDD_HTTPD=1
BUILDER_BIN=/usr/sbin/sbuild

None of the rest of this file actually matters unless you want to use the maintenance scripts

We need a rebuildd config file to tell it which builder command and apt-get source command to use /etc/rebuildd/rebuilddrc

[build]
check_every = 300
max_threads = 2
max_jobs = 5
kill_timeout = 90
source_cmd = apt-get -q --download-only -t precise source ${p}=${v}
#build_cmd = pbuilder build --basetgz /var/cache/pbuilder/${d}-${a}.tgz ${p}_${v}.dsc
build_cmd = sbuild --host=armel --verbose --nolog -d ${d} -c precise-amd64-cross-sbuild ${p}_${v}.dsc
post_build_cmd =
dists = precise-ma-cross
work_dir = /srv/buildd/flavours/precise
database_uri = sqlite:///var/lib/rebuildd/rebuildd.db
build_more_recent = 1
more_archs = armel arhmf

[mail]
from = rebuildd@localhost
mailto = wookey@arm.com
subject_prefix = [rebuildd]
smtp_host = localhost
smtp_port = 25

[telnet]
port = 9999
ip = 127.0.0.1
prompt = rebuildd@localhost->
motd = Connected to rebuildd on linaro cross-buildd

[http]
port = 9998
ip = 127.0.0.1
log_lines_nb = 30
templates_dir = /usr/share/rebuildd/templates
cache = 1
logfile = /var/log/rebuildd/httpd.log

[log]
file = /srv/buildd/rebuildd.log
time_format = %Y-%m-%d %H:%M:%S
logs_dir = /srv/buildd/flavours/precise/build_logs
mail_failed = 1
mail_successful = 0

Note that this config will only build the arch-dependent binary packages. If you want to cross-build arch-all packages too (rather than just do a native build on any handy arch), then you need to change build_cmd to add --arch-all:

build_cmd = sbuild --host=armel --arch-all --verbose --nolog -d ${d} -c precise-amd64-cross-sbuild ${p}_${v}.dsc

Rebuildd downloads the sources before changing into the chroot, so you need to ensure that they are available to apt: add this to the local config: /etc/apt/sources.list.d/cross-repo

deb-src copy:/srv/buildd/repo/ precise-ma-cross main universe
#deb-src http://localhost/builddrepo/ precise-ma-cross main universe

If you prefer to use http over copy then you need to add a link in /var/www:

sudo ln -s /srv/buildd/repo/ /var/www/builddrepo

Before rebuildd can do anything you need to initialise its database:

rebuildd init

And you also need to make sure that if the [log] mail* settings are '1' then you need to have a setup that will at least accept mail. (ssmtp or nullmailer are very simple local MTA packages which just pass mail stright on to your normal smarthost)

Set up reprepro

This is the biggest piece of setup. Reprepro needs config files to define the available distros, where to pull in updates from, how to process incoming built packages, filters to restrict the archive coverage, and scripts to help with all this.

The design is that the repo sits between the upstream distro and the buildd. We run reprepro update to pull in new sources. reprepro build-needing to get the list of unbuilt binaries, which is passed via format-conversion to rebuildd-job to queue the jobs. reprepro processincoming is run on completed jobs to uploaded them into the repo.

In more detail that needs these config files:

This file defines the distributions ('codenames') in the repo. The log feature at the end is a handy way to run a script after each upload. That will be used to trigger log-syncing to the public server (but isn't yet).

/srv/buildd/repo/conf/distributions

Codename: precise-ma-cross
Label: Multiarch auto-cross-built packages
Version: 12.04
Architectures: armel armhf source
Components: main universe
UdebComponents: main universe
Description: Local Precise multiarch-style cross-builds
#SignWith: wookey@wookware.org
DebIndices: Packages Release . .gz .bz2
DscIndices: Sources Release .gz .bz2
Tracking: all includechanges
Update: precise-upstream
Log: logfile
#  --changes bin/upload_log

/srv/buildd/repo/conf/updates

Name: precise-upstream
Method: http://archive.ubuntu.com/ubuntu
Suite: precise
Components: main
#VerifyRelease: 40976EAF437D05B5
VerifyRelease: blindtrust
Architectures: source
FilterSrcList: deinstall filters/nano filters/alip

/srv/buildd/repo/conf/incoming

Name: precise-ma-cross
IncomingDir: /srv/buildd/flavours/precise
Allow: precise->precise-ma-cross
Default: precise-ma-cross
Tempdir: /tmp

This is not necessary just a debug convenience:

/srv/buildd/repo/conf/options

verbose

You can add more 'verbose' lines to make reprepro more verbose.

This is a simple script to convert reprepro build-needing output format into rebuildd-job input format. It is triggered by cron each time the repo is updated with new sources.

Eventually rebuildd will be taught to understand the format directly, so the awk bit will go.

/srv/buildd/repo/bin/queue-builds

#!/bin/bash 

repopath="/srv/buildd/repo"

reprepro -b $repopath build-needing precise-ma-cross armel | awk '{print $1,$2,"1 precise-ma-cross armel"}' | sudo rebuildd-job add

To do arch-all builds as well (see above for corresponding change to /etc/rebuildd/rebuilddrc) add this line:

reprepro -b $repopath build-needing precise-ma-cross all | awk '{print $1,$2,"1 precise-ma-cross"}' | sudo rebuildd-job add

for test purposes it's useful to have a corresponding script that just lists the jobs: /srv/buildd/repo/bin/list-builds

#!/bin/bash 

repopath="/srv/buildd/repo"

reprepro -b $repopath build-needing precise-ma-cross armel | awk '{print $1,$2,"1 precise-ma-cross armel"}'
#reprepro -b $repopath build-needing precise-ma-cross all | awk '{print $1,$2,"1 precise-ma-cross"}'

package-list filters

In order not to pull the whole of debian/ubuntu into a local repo just to rebuild it we use a filter list to restrict the set of packages of interest (see the 'update' file). These filters are in dpkg --get-selections format, one line for each package of interest. So they are very easy to generate from an image on a machine you are interested in. I just list part of the file here to show the format.

We currently have 'nano' and 'alip' filters used in the update rule.

/srv/buildd/repo/conf/filters/nano

adduser install
apt install
base-passwd install
bash install
busybox install
ca-certificates install
consolekit install
coreutils install
cpio install
dash install
debconf install
debianutils install
...
ucf install
udev install
upstart install
util-linux install
xfonts-terminus install
xkeyboard-config install
xz-utils install
zlib install

set up cron jobs

This job updates the local repo from upstream with latest sources, then sends a job-list of everything that is now/still out of date to the buildd

Create a file /etc/cron.daily/buildd (don't forget to chmod +x it

#!/bin/sh -e

builddpath="/srv/buildd"

echo "*** update local repositories with current source ***"
reprepro -b ${buildpath}/repo/ update precise-ma-cross

echo "*** feed rebuildd with packages in need of building ***"
${buildpath}/repo/bin/queue-builds

This job processes any packages that have been built and uploads them into the local repository, so they will hencforth be deemed 'up to date' and further builds will not be attempted until new sources appear.

/etc/cron.hourly/buildd-outputs

#!/bin/sh -e

builddpath="/srv/buildd"

reprepro -b ${builddpath}/repo/ processincoming precise-ma-cross

set up syncing

This is current work-in-progress. It can be made a lot slicker. You do not need this part for a purley local buildd. This is just for exposing the logs/status out on the real internet.

Logs are synced with a cron job:

rsync -a ${base}/flavours/${flavour}/logs/ wookey@people.linaro.org:public_html/buildd/${flavour}/

For this to work smoothly from cron, without stalling on ssh password handshake, your specific IP needs to be given permission by the admins.

before syncing each log dir has build-status run over the log files to determine build status and linkify-build-status to make it into html table.

This is controlled with make and an iterator script, which is a bit baroque, and designed for batch uploading. Once we have per logfile uploading it should probably be driven differently. Anyway here are the scripts for the record:

Makefile

#!/usr/bin/make

files = $(wildcard *.log)

status.txt: *.log
        -build-status $^ > status.txt
        
status.html: status.txt
        linkify-filelist $^ < status.txt > status.html

/srv/buildd/buildd-synclogs

#!/bin/sh -x

base="/srv/buildd"
uploaduser="wookey"
server="people.linaro.org"
#base path on server to copy to
htmlpath="public_html/buildd"
flavourlist="precise"

for flavour in ${flavourlist}; do
  for dir in ${base}/flavours/${flavour}/logs/sbuild/*; do
    make -f ${base}/Makefile -C $dir status.html
  done
  rsync -a ${base}/flavours/${flavour}/logs/ ${uploaduser}@${server}:${htmlpath}/${flavour}/
done

Testing

Check that your chroot is accessible with:

schroot -c <chroot_name>

Check that sbuild can cross-build something in it with:

apt-get source --download-only sourcepackage
sbuild  --host=armel -d precise <sourcepackage_ver>.dsc

outside the chroot, check that reprepro source updates work reprepro -b /srv/buildd/repo update precise-ma-cross

check that reprepro now lists a set of packages to build: /srv/buildd/repo/bin/list-builds

queue the builds up with try queuing a single build with: echo "zlib 1:1.2.3.4.dfsg-3ubuntu4 1 precise-ma-cross" | sudo rebuildd-job add

telnet into rebuildd (or use the http access) and see what's up: telnet localhost 9999 > job status <packagename>

look for logfiles. if it looks like it's working set it going: /srv/buildd/repo/bin/queue-builds

Generating summaries

A pile of log files tells you all you need to know, but not in a very digestible form.

So we generate a summary file by grepping those logs for tell-tale info. xbuilder contains a couple of scripts for this purpose build-status and build-status-single. (The former can take a set of files, the latter just one).

build-status is run over each set of build-logs to generate a status.html file which provides a table summary of 'Package: Built OK, maybe-built, failed (reason)'. Those are uploaded with the logs.

more to come in here....

Using PPAs in buildd chroots

If you want to include updates from the cross-PPA do this too:

Add this source: deb [arch=amd64,armel] http://ppa.launchpad.net/linaro-foundations/cross-alip/ubuntu oneiric main deb-src http://ppa.launchpad.net/linaro-foundations/cross-alip/ubuntu oneiric main

Add this file:

/etc/apt/preferences.d/alipprefs

Package: *
Pin: release o=LP-PPA-linaro-foundations-cross-alip
Pin-Priority: 600

Package: *
Pin: release o=LP-PPA-linaro-foundations-cross-build-tools
Pin-Priority: 600

Faster building using apt-cacher

THe slowest part of building things over and over is the downloading of toolchains and dependencies. THis wastes everyones bandwidth and takes time, as generally each build downloads the same stuff over and over.

Installing apt-cacher means packages are cached on the local machine and downloads are _much_ faster.

apt-get installl apt-cacher

Edit /etc/default/apt-cacher to enable the demon

You can either edit deb source lines you want cached to include localhost:3142 or set an apt-proxy heading which applies everywhere. The latter is easier. Create /etc/apt/apt.conf.d/10proxy

Acquire::http::proxy "http://e102475-lin:3142";

Now packages should download much faster after the 1st time. (You can check it's working by tail -f /var/log/apt-cacher/access.log) whilst apt-getting some packages.

xdeb/xbuilder based build-runs

An older scheme using xbuilder to process a list of packages 'single-shot' was used for natty and onieric dpkg-cross style builds.

xbuilder is a set of scripts and config for running a cross-buildd. It is available from here: http://ppa.launchpad.net/linaro-foundations/cross-build-tools/ubuntu

The 0.3 version in natty is intended for use with xdeb and a package-build list to do a run-through building all the packages in the list. It also supports pdebuild-cross builds.

These flavours were built that way. natty + ALIP PPA using xdeb (called 'natty-ppa') oneiric + ALIP PPA using xdeb (called oneiric-ppa')

xbuilder v0.5 supports sbuild-based builds using multiarch in precise and wheezy. It is still under development and is only for the enthusiast right now (Jan 2012).

setup procedure

Make a plain debootstrap chroot build_arch chroot, then:

Install dpkg-dev build-essential fakeroot binutils-multiarch dpkg-cross ubuntu-keyring gnupg xapt realpath

Install g++-arm-linux-gnueabi libc-dev-armel-cross linux-libc-dev-armel-cross from universe

natty

Install xdeb dpkg-cross from http://ppa.launchpad.net/linaro-foundations/cross-build-tools/ubuntu

oneiric, precise

Install dpkg-cross from http://ppa.launchpad.net/linaro-foundations/cross-build-tools/ubuntu

Install g++-arm-linux-gnueabi libc-dev-armel-cross linux-libc-dev-armel-cross from universe

Ensure that sudo doesn't just sit there waiting for a prompt all the time: replace

%sudo ALL=(ALL:ALL) ALL

with

%sudo ALL=NOPASSWD: ALL

in /etc/sudoers (need to use chmod +w etc/sudoers; <edit>; chmod -w etc/sudoers)

Set APT::Install-Recommends "0"; in /etc/apt/apt.conf.d/norecommends

run apt-get update once inside the chroot to give a based package-file set.

CategoryCrossCompile